Microsoft is using machine learning to make its Windows apps a lot smarter. At a Technology Sneak Peek on Monday at Microsoft’s Build developer conference in Seattle, executives used Windows Ink to demonstrate the potential of Windows ML, a machine learning API that debuted in the April 2018 Update to Windows 10.
Because Windows ML is an API, regular users will never encounter it. But the API provides the framework for developers to tap the power of your CPU or GPU to anticipate and act upon your needs, in Windows and Windows apps.
The executives at the demo made it clear that the Ink features they were showing off could drastically change, or might never be released at all. Still, the future they showed moves a few steps forward from the way Windows works today.
Case in point: Today, OneNote’s app for Windows 10 is smart enough to translate an inked math problem into a solution that can be solved. But each time you want to perform that action, you have to “lasso” that equation with your pen, signalling to Windows what you want it to do.
The Windows Ink teams showed off how Windows ML could address that and other issues with the inking experience. For example, ink could be used to:
- Ink the values you want on a slider bar, rather than making you move the slider back and forth until you find the right value.
- Allow Windows to recognize when users are inking math, a flowchart, or a table.
- Change behavior when ink is used in OneNote, as opposed to Windows’ Sticky Notes or WordPad apps.
- Alter the texture of a paintbrush application by speed.
- Use the recognizer to give Office Lens OCR capabilities.
Mark Hachman / IDG
Windows ML could adjust the depth and texture of an ink brush on the fly.
The Office Lens OCR example shows the potential of the recognizer platform created with Windows ML. Office Lens is a mobile app that’s superb at, say, aligning a photo of a whiteboard for easy reading. But it can’t yet “read” the text that’s been written on it. The ability to “see” the real world has been a Microsoft priority, and machine learning could move that priority forward for Office Lens and more.
As for whether Office Lens would actually gain this capability, Kyle Beck, a program manager at Microsoft said that his team just designs the inking platform—where it ends up isn’t his decision. But, he added, Office Lens would be the “natural fit” for it.
Mark Hachman / IDG
This is how Ink will (or. at least, could) recognize inked equations.