I was recently reading an article on the web going on about what a gimmick multi-touch technology is, and how they don’t see any good use for it. Now, I could not agree less. Sure, the ways it has been used so far are pretty gimmicky, but that doesn’t mean it has to be. I haven’t worked as an interface designer in years, but even I can see the exciting possibilities presented by the technology. In fact, I can think of tons of way to use it, that I could probably make a fortune off of patenting, but patents aren’t cheap, and I don’t have the kinds of financial resources to patent every good idea I come up with. So, if you are the wise ass who will read this article, and then go out and file a patent from what you read, then at least have the decency to offer me a job, ok?
Anyway, the ways I see it being really useful are mainly for creative applications, as that is where I spend most of my time. I’m sure it has exciting applications elsewhere, but I will focus on the creative ones.
Video Editing Systems
This is a big one. Just imagine, if you wanted to take a little off the front and back of a clip in your timeline. Instead of doing what you do right now, which is click on the track to select it, then move the cursor to one edge, then move that edge, then go to the other edge, and move that one, you could instead just reach up to the screen, put one finger on one edge, on finger on the other, and then move your fingers in towards each other to change the in point and out point simultaneously! That would save a ton of time. Or, imagine grabbing four pieces of media at once from your bin, each with a different finger. Imagine grabbing two clips, and then swapping their places just by dragging them around on the screen. Oh, it would be so nice. Right now one of the main reasons NLE systems feel so clunky, is because you are constantly having to click to select, then work on one clip at a time. Something as simple as changing the order of clips can be a needless pain, because you constantly have to move things one at a time. Being able to grab as many items as you want at once, and move them independently, would just be a dream come true. Realistically, I think you would probably end up most often just working with two clips at a time, because anything else requires some pretty dexterous contortion of your fingers, however even just two clips at a time is still twice as fast as one.
3D Systems
This one has so many possible applications it boggles the mind. I get woozy when I think about deformers where I could move the deformer tool handle, at the same time I am also moving the deformer control, to get a really complex deformation on a part of a model. Right now you have to do an annoyingly clunky process of positioning the handle, then manipulating the control handle, then going back to the tool handle to change the position, then readjust the control handle again to fine tune the deformation. It is a real pain, and about as removed from acual modeling or sculpting as you can get. For that matter, how great would it be if every handle recognized two fingers, and you could change the position, scale, and rotation just by sliding those two fingers around? If you were using one of the multi-touch systems that could also recognize pressure, then it would just give you an insanely natural amount of control! My dream system would allow you to just put a finger on an object, move your finger around to translate it in an XY plane set to the view, and press harder to push it back in the Z relative to the view. Then, if you kept your finger on the object, and put down a second finger within the object, that would give you rotational control, in XZ on the same viewplane. The first finger would still be controlling position, so you could drag it, while it tried to keep the rotation aiming at wherever you put the second figure. That way you could transform and rotate simultaneously. YOu could also have pressure on the finger rotate the object in the Y axis relative to the viewplane, so you had full positional and rotational control of the object, just with two fingers on the screen. Throw in a little modifier, like touching a third finger to the model to toggle + or – on the pressure manipulations, and you have full six-axis control, right on the screen, with nothing more complicated than your fingers. Of course, imilar control schemes could be used for scale or other transformations. Imagine being able to pose an IK system, where your first finger controlled the position of the root joint of the IK chain, and your second finger controlled the position of the end effector of the chain. It would be like moving around a little on-screen stop motion armature, instead of the current clumsy control schemes. The same goes for your animation curves, where you could hold one finger to anchor a keyframe, and then move another finger to adjust the curve.
And that is all with just two fingers! What if every animatable channel could throw up a little on-screen slider? Say goodbye to those MIDI devices and dial boxes crowding your desk! You could manipulate 10 channels at once, just like working with an audio mixer. It would be every animators dream come true. Direct and immediate control of several aspects of your animation at the same time, to see how they all effect each other, instead of having to do pass after pass. Then there are particle tools! Being able to seat the spread, velocity, and decay of an emitter, just by swiping your fingers across the scene like you want the particles to fly, would eliminate hours of scripting and trial and error. I can’t even begin to fathom all the implications of a system like this, but in just a couple of minutes of thought, I see that it would totally change the nature of 3D software.
Audio Systems
Of course, this is one of the few systems that is already enjoying these benefits, thanks to products like the ones made by JazzMutant. It would, however, still be fantastic to be able to manipulate all your tracks, samples, and controls directly on the screen, instead of having to map a select few to something like the Lemur. You could have all the benefits already discussed in the Video System section,plus the more traditional feel that audio engineers are already used to from a mixing board. It would be great, and is probably the first place you will see this sort of integration.
Compositing and Paint Systems
This is another one where it would completely change the nature of the software. I don’t really see it replacing the precision and natural feel of a good pressure sensitive tablet like a Wacom, but it would definitely augment it in some great ways. Being able to quickly resize, move or rotate a layer without having to select a tool, would alone be a big enough productivity gain to be worth it, but I can think of so much more. Being able to smear you artwork by actually smearing your hand, or a finger across it, and having the tool change its size, shape, and opacity based on the pressure and size of the area smeared would be just one example. Another would be the ability to edit multiple vector curve handles simultaneously, instead of having to do them one at a time. I can even see finger painting broad strokes, or using your fingers to define the bounds of an arc, spiral, or geometric tool. For node-based systems, it would let you connect complex node structure very quickly, as you just drew the connections between various nodes with your fingers, while at the same time quite naturally dragging the nodes around like pieces of a puzzle on a table. Hell, it would just be great to be able to move the canvas around while you were in the middle of drawing something! I can’t be the only one who has ever been working on something zoomed in, only to realize that I was going to have to exist the tool and start over, because I was running off the part of the image that was in frame. Working with a lot of particle emtter tools would also be a dream for some of the reasons I discussed in the 3D section. Of course, it could also give rise to entirely new paint tools, like splatter tools, that flung blobs of paint on the canvas based on the relative velocity of your fingers as you flung them across the canvas. You would also benefit from the same multiple finger sliders. Imagine the joy of being able to tweak all your color, contrast, brightness, contrast and exposure controls on an image at the same time, interactively, seing how they all effect each other, instead of having to go back and forth from one control to another, then undo, try again, and hope for the best. It would be like the fun you had (or at least I did) playing with all the hew and saturation knobs on old TVs, or professional monitors.
All in all, I would say there are almost unlimited possibilities for creative multi-touch applications, and I hope that someone is working on these as we speak, or will soon, perhaps after reading this. The metaphors being used in current creative apps are pretty much stretched to their limits, and are getting very tired and worn around the edges as they are asked to do more and more with one mouse, and one cursor. I hope that multi-touch will usher in a complete rethink of how we approach creative interaction with the computer.