While browsing the web, Professor Mark Chang came across the work of Jeff Han at NYU: http://cs.nyu.edu/~jhan/ftirtouch/. He approached several students, Matt Donahoe, Anthony Roldan, Chris Stone, Jeff DeCew, Olek Lorenc, and Jon Tse, and asked if we wanted to develop a similar touchscreen based on Frustrated Total Internal Reflection (FTIR) technology.

At the time, the idea of applying FTIR to touchscreen interfaces was relatively new, so we investigated its feasibility. We developed some prototypes over the course of the first semester, which were functional, but did not give us the dynamic response we were hoping for. The following semester, Matt and Jeff left the team to develop the K’Nex Computer, and Chris, Anthony, Olek, and Jon continued work on the touchscreen.

In the last phase, Anthony and were are the only two students working on the project. We’ve moved from FTIR to diffuse illumination (DI), as it is easier to implement and is more forgiving.

Touch screen interfaces have been around for quite some time, but haven’t really been in general use until recently. Smartphones and tablet PCs are becoming more and more prevalent, and the touch screen interface is growing both in complexity, ease of use, and popularity.

One of the main issues with touch screens is scalability. It’s difficult or prohibitively expensive to build a reliable table-sized touchscreen with traditional sensor technology. Additionally, traditional sensors are grid-based; they determine the cartesian coordinates of each touch. Unfortunately, these systems usually can only read one set of coordinates.

Multitouch interfaces, like Jeff Han’s work and the Apple iPhone, are capable of handling multiple touches. FTIR and DI-based multitouch interfaces are unique in that they are cheaply scaleable.

Frustrated Total Internal Reflection (FTIR) is one way of detecting multiple touches. The mechanism is similar to fiber optics. We begin by placing a strong infrared (IR) source along the edges of the plexiglass. This is typically done by arraying IR LEDs.

The IR light is not columated, but some of the light that enters the edge of the plexiglass reflects back and forth between the inside planes of the plexiglass, much like light in a fiber optic cable. This continuous reflection is known as Total Internal Reflection (TIR). Fortunately, the light that does not undergo TIR leaves the plexiglass near the edge.

The plexiglass is now flooded with IR light. By touching the surface with something that conforms to the shape of the glass, such as a human finger, the reflection is disturbed, and the finger appears as an IR "blob," hence the nomenclature Frustrated Total Internal Reflection. As described earlier, the blobs are picked up by the camera behind the plexiglass and processed into Cartesian coordinates.

Diffuse Illumination (DI) is another way of detecting multiple touches. The setup is largely the same as FTIR; theoretically, both technologies could be combined to greater effect. The central idea behind DI is to flood the surface of the plexiglass from the projector side with diffuse IR light, illuminating the fingertips of the user. The reason to use diffuse IR light is to ensure a uniform light intensity at all points on the plexiglass.

Objects closer to the surface of the plexiglass will reflect more light, so by very carefully filtering and thresholding the camera image, the fingers touching the surface of the plexiglass can be isolated, again as blobs. The downside to this process is that any object close to the surface will show up, including the user’s palms, shirtsleeves, or bracelets. With careful tweaking of the filtering and thresholding algorithms, these effects can be minimized.

Our earliest prototype used both visible and IR light. We were curious to see if the FTIR effect was feasible, so we used visible light for quick testing and debugging. The best results were achieved using a piece of plexiglass, a laser pointer, and a mirror. Putting the edge of the plexiglass up to the mirror and shining the laser pointer in from the opposite edge, we were able to set up a nice TIR. Touching the plexiglass with our fingers resulted in a very clear, sharp FTIR emission.

Having validiated that FTIR works, we moved to IR light. We used a modifed webcam with IR notch filter removed, and the exposed, developed 35mm film as an IR bandpass filter. We saw minimal results with the FTIR unless in near pitch-black darkness, but we weren’t using many IR LEDs.

Revision 1 was short-lived. We had decided to move to a larger piece of plexiglass, both to make projection feasible and to increase the available edge area for more IR LEDs. We built a wooden frame to hold the plexiglass, and clamped it to a table.

The biggest contribution of Revision 1 was the new IR LED arrays. Using techniques learned in Olin’s freshman course Design Nature, I designed some LED holders out of Delrin that clip onto the edge of the plexiglass and maintain good spacing between LEDs for regular lighting.

The downsides of Revision 1 were primarily the lack of adequate support for the plexiglass. We were taking the plexiglass out of the frame frequently to work with it, so a permanent clamp on it was unacceptable. We used a block of wood on the back side to brace the plexiglass against our pressing on it, but there was nothing holding it in from front other than some thumbtacks.

Olek, having seen the shortcomings of Revision 1, took it upon himself to build a new frame for us, using aluminum L-brackets.

Olek welded L-brackets together and used screws and washers to clamp down on the plexiglass. The screws were easily removeable, and they were more than adequate for holding the plexiglass in the frame.

We used the Delrin LED array brackets on Revision 2, sanding down the tips of the LEDs and epoxying them into place. The epoxy, we hoped, would ensure a good optical channel for the IR light to travel through. However, even though we were flooding the plexiglass with IR, the FTIR wasn’t performing adequately.

We used Revision 2 for our first demonstration to the rest of the college, but we had to do it in a dark room due to the sensitivity to ambient IR. Also, to ensure good contact between human skin and the surface of the plexiglass, we dipped our fingers in water before using the touchscreen. It also became very tiring to use the touchscreen, since it was angled vertically, which is a very awkward position for the user’s hands, especially when the user has to push very hard to get a response.

We decided we would address several issues with the Revision 3 touchscreen.

The Revision 3 touchscreen is about three feet tall, contains the camera and projector mount, and has panels to cover the sides.

After all of these FTIR related issues, we decided to move to Diffuse Illumination (DI). We used the Revision 3 design, and simply covered the inside surfaces with aluminum foil to reflect IR light.

We initially tried using an incandescent lamp as an IR source, which worked exceptionally well as a prototype. It wasn’t diffuse enough, even with diffusing surfaces in front of it, and the camera picked up the 60Hz flicker from the mains.

We bought some IR LED spotlights, which are typically used for as illumination for IR security cams. The LED spotlights proved to be easier to control and diffuse, and we were seeing excellent results very early in the process.

Finally free of the problems from FTIR, we started building a new prototype designed around DI. It is still under construction, but essentially it is a cube made of Medium Density Fibreboard (MDF) with a cutout on top for the plexiglass. The projector and camera will be contained inside the box, as in Revision 3.

We decided that a photo tagging application was a good fit for the interface. We can quickly sort photos by flicking them to various places on the screen with our fingers, much like we would an actual physical pile of photos.

Tagging photos is as simple as touching the tag you want, and the photos you want tagged. It’s a very simple and intuitive interface and we hope to develop it further once the hardware is up to snuff.

As far as Touchlib is concerned, it’s a really nice tool for getting actual data out of your setup very fast. We never bothered trying to get it to run under Linux, but we were really only interested in getting the blob coordinates.

Our very first prototype used the Java library Processing, which I believe will run under Linux. We gave that a shot at one point, but I couldn’t get the webcam drivers to integrate with it right so we gave up and settled for the Windows version. Here’s some example Processing FTIR Stuff: http://lowres.ch/ftir/

Our next software prototype used VVVV. It’s a really handy graphical language, and we were able to make more progress faster than we could with Processing. It just made changing parameters really easy because it was graphical. It also has some really nifty implementations of random media-related things, which if nothing else are amusing to look at.

As far as Linux vs Windows, we saw pretty heavy processor usage running any of the software packages just for one camera, so we actually broke our system into two computers. One computer ran the image processing, and the other ran the actual Multitouch app.

I’m reasonably certain that all three packages output OSC (OpenSound Control) packets if you want them to that contain blob position information, so you can actually develop your app on any platform that supports OSC that you want (I know python does, so your options are pretty broad).

I’d suggest using a Windows box for now to do the image processing. If you’re really really gung ho about it you can always use an FPGA to run the image processing and output an OSC (or whatever protocol you choose) stream. We saw ourselves going down that line in the long term, but never got a chance to finish.

We wrote a pretty simple game in Python to demo the hardware, and I can provide the source code to you if you’d like. The benefit to using something like OSC is that we kept the game unchanged, basically, and kept switching both hardware and software implementations and the game just kept on working, which was excellent for debugging.

You also could have multiple computers with cameras doing image processing and sending OSC streams to another computer that is actually running the multitouch app.

We got stuck at the end sorting out DI hardware issues—i.e. we were swamped with the whole getting out of college process, so we never got past testing the hardware with touchlib alone. I’d honestly say that touchlib was an excellent prototyping platform, and made everything easy. It does all of the filtering and image processing for you, and it even does the matrix transforms you need to calibrate the screen properly. I know it runs on Windows, but I’d honestly say that it’s worth it. I’m a huge Linux fanboy myself, so I don’t say that lightly.

Well, it’s been awhile, but if I remember correctly we used a background subtract filter, a high-pass filter, and a thresholding filter. There might have been some smoothing filters in there as well, but those are the big three important ones.

That seemed to work very well for us. The biggest problem for us was achieving uniform illumination across the whole screen. We were using IR spotlights (typically used for home security camera night-vision purposes) and aluminum foil as reflectors to illuminate our screen. This really affects the thresholding algorithm, because the filter assumes that the light intensity is uniform.