Operating Environmentalism

In a grand new tradition of using Universe as lodging for really interesting "supplemental material," I present to you the history (and mystery) of g-speak, an incredible new spatial operating environment, as told to me by John Underkoffler, chief scientist at Oblong Industries. Underkoffler designed the fantasy computer systems in Minority Report, then made g-speak, an almost frighteningly futuristic interface that will throw the proverbial brick through the computer screen. Check out the video above to get a sense of it in its full, dizzying glory.

My full article about g-speak is over at GOOD Magazine.

"We've built g-speak from the ground up to be a completely general computing environment -- the idea is that anything you might want to do with a computer can be done as a dialog between you and g-speak. The really interesting thing is that what it looks like on screen, what it feels like to your hands and your mind, is radically different from the GUI [Graphical User Interface] that you're used to."

"Every bit of the on-screen experience that we've all come to regard as basic or elemental over the last twenty-five years is predicated on one thing: the mouse. The whole semi-overlapping-windows scheme, and all the little gewgaws that come along with it (pulldown menus, little nubs you click on to close or bloat windows, sliders, scrollbars, etc.) were designed to accommodate the mouse. Once you replace the mouse with something vastly more capable -- i.e. unfettered human hands -- the stuff that's usually on screen is immediately inappropriate. One of the exciting breakthroughs for us has been to show that many of those artifacts are necessary because you can't see enough at one time: consider what a scrollbar does and why that's necessary. But if you can imbue the operating environment with a more fundamental way of navigating around, a way that's implicit in how you already interact with the world, then it's not like you replace the scrollbar with a gestural equivalent. You fundamentally don't need the scrollbar any longer."

"I'm afraid that I'm the Minority Report culprit. I'd been building human-machine interface stuff like this for years as part of my work at MIT (in the Media Laboratory), and when a kind of advance team (principally Alex McDowell, the brilliant production designer) showed up at the lab to "scout" technology ideas for the movie, the HMI [Human Machine Interface] work seemed to resonate. So I became the science advisor for the film and slightly adapted what I'd been building at MIT -- and that's what you see in the various scenes in which the characters are doing police forensics work on giant screens. The screens were blank for shooting (we didn't have time to actually build the system), but the actors really knew the gestural language, so when we shot the gestural scenes they weren't making anything up. In a way, they were genuinely operating a g-speak system. There's no question for me that that shows vividly when you watch the movie."

"Once the movie came out, we'd built g-speak twice: once in an academic lab, which has certain constraints and lacks others, and once in an extremely visible piece of popular media, which works a completely different way. Audiences really responded to those scenes -- you could tell, talking to people about it, that they felt like they'd seen something that either was real or should be. And since we're most of us engineers and couldn't stop building things if we wanted to, it was inevitable that we'd return to the lab and the workbench and build this stuff a third time. This time, though, it was clear it had to be in the context of a company making a commercial product. That's the only way to get the stuff out there into the world as broadly as we intend. We sincerely believe that the entire world will use their computers this way at some point down the line. Could be six years; could be ten; but it has to come. The interface we've been using for a quarter of a century just isn't keeping up, mainly because of the giant gap that's opened between what the computer (with its incredible processors, giant memory, profound graphics, and networked view of the world) can express and what the mouse and windows GUI allows us humans to express."

"For some information problems, there's no real alternative to g-speak. To comprehend and then be able (in real time) to act on such volumes of data takes more than visualization alone; eyes aren't enough. You have to enlist another giant chunk of the human brain, the part that deals with muscles and muscle memory and proprioception and all that. That chunk of brain knows as much about space as the human visual system does, and they're actually evolved to work together. That's why were all such experts at getting around and manipulating the real world. So it seems clear to us that computers should work the same way -- and that's what g-speak is. It engages both parts of your brain to let you get at digital information the same way you get at the real world. That means reaching into data; stretching it; pointing at it and poking it; spinning it around."