Imagine controlling Apple iTunes from inside Microsoft Word without having to switch applications. That could be possible, according to researchers at the University of Washington who are working on a project that could essentially make any proprietary software open source.
“Microsoft and Apple aren’t going to open up all their stuff. But they all create programs that put pixels on the screen. And if we can modify those pixels, then we can change the programs’ apparent behavior,” said James Fogarty, a University of Washington assistant professor of computer science and engineering.
Almost everything seen on a display is made of prefabricated blocks of code, and the tool, called Prefab, looks for those blocks as often as 20 times per second and alters their behavior.
Fogarty likened the idea to HTML. “Everybody can see the HTML. But that hasn’t been possible on the desktop,” he said. A standard Web page can display a map from Google, an embedded YouTube video and a constantly updating RSS feed all at the same time, but achieving something similar with native PC programs is hard to do.
Fogarty has been working on Prefab for about two years, along with a University of Washington doctoral student in computer science and engineering, Morgan Dixon.
Dixon described Prefab’s capabilities thus: “I’m using some program that I love, and I’m going to stick in some features from some other program that I love, so I have a more unified interface.”
While the main part of the program is done, Dixon and Fogarty are still developing Prefab and exploring options for commercializing it.
Prefab doesn’t actually reveal or manipulate the source code for programs, since it can’t see it in proprietary software. It can only manipulate and combine what’s visible on the computer screen. “Even if it’s in a menu six layers down, if your eyes can see it, so can Prefab,” Fogarty said.
He showed a few examples of how Prefab could work in a video using target-aware pointing techniques like a bubble cursor, which dynamically sizes the cursor so that it always captures the nearest target. Rather than needing to click directly on a button, the bubble cursor allows users to be in the vicinity of a button and still be able to click it. It can be particularly helpful for people with disabilities.
The bubble cursor could be added to any application by Prefab, which would start by copying the pixels from a source window. Prefab then interprets the pixels and adds enhancements — in this case the bubble cursor. It then outputs the final product to a target window, including both the pixels from the source window and the enhancements.
Making changes to software from Microsoft, Apple and other companies could potentially lead to legal problems, but Fogarty argued that “there’s a lot of value we can provide these companies.”
He said he plans to use “crowd-sourcing” and create an online community for people to share ideas for Prefab. “We’ve seen the mashup culture flourish online, and this is not fundamentally contrary.”
He plans to show the software on April 14 in Atlanta at the Computer Human Interface conference. One of the demonstrations will use Adobe Photoshop and involves creating multiple previews of a single image file. In the background, Prefab moves the controls to adjust color and contrast and so on, captures the resulting images and displays them all on a single screen. That could save time when editing images by displaying a range of possible results to users.