One of the most significant features of Windows 10’s Anniversary Update was the addition of pen computing, known as Windows Ink, which we criticized as falling short of the average consumer’s needs. We don’t know whether any new inking features will be announced at Microsoft’s Windows event on Wednesday (or the event that follows on November 2). Recently, however, we took a deeper dive into the capabilities, when we tried the new Math and Replay features within Windows 10’s OneNote UWP app.
Math translates and solves inked equations, while Replay records your series of ink strokes and can play them back. On paper, both sound like terrific ideas. But the devil’s in the details, and the challenges of both features show how Windows Ink is struggling with the realities of handwriting recognition and data wrangling.
OneNote Math: It doesn’t always add up
As part of Microsoft’s August updates for Office 365 subscribers, OneNote gained a new superpower: equation calculation. Unfortunately, poor inking recognition is its Kryptonite, rendering it virtually unusable with a pen.
When it works, OneNote’s new Math feature translates your electronic ink into an equation, and then solves it for you—and even shows its work. It resembles mathematics powerhouse Wolfram Alpha, which, like Microsoft, charges money for this capability. Unfortunately, the Math feature hasn’t yet appeared on OneNote 2016, where other, useful features like real-time audio annotations reside.
To use OneNote Math, you must ink the equation, lasso it, then click the Math button.
You’ll find the new Math feature in the Draw tab’s toolbar. You ink an equation, select it using the lasso tool on the far left of the toolbar, and click the Math button. OneNote then interprets your ink and provides you with a list of options to solve the equation.
Unfortunately, the feature didn’t quite live up to my expectations. Sure, it was able to calculate my equations.You can see in the screenshot that it does a nifty job of displaying solutions and showing the work.
MARK HACHMANWhen OneNote’s math function works, it’s pretty neat.
But despite its aptitude with a single equation, as shown above, OneNote appeared to be incapable of solving a series of linear equations (say, two equations with two variables). The app also doesn’t plot any graphs in this version of the software, and you won’t find many of the higher mathematical functions a site like Wolfram Alpha supports. Or so it seems.
The other thing is, OneNote is extremely finicky about how it interprets your digital ink. Each time you enter an equation, you need to ink it, circle it, and pray. Write the number ‘4’ in the open-top style, as I do, and Math may not interpret it correctly. Ditto if you place a ‘+’ sign out of alignment, or if your cotangent (“cot”) abbreviation isn’t inked perfectly.
I had a tough time getting Math to understand the notation for an integral, at least the way I inked it. I also ran into an occasional bug where the box that displays the interpreted ink was partially obscured. That left me guessing whether my equation had actually been interpreted correctly.
Mark Hachman Poor ink recognition is an issue Microsoft must solve to make OneNote Math a winner.
Also, inking with the stylus produced worse results on my Surface Book than using my finger. Maybe it was because I inked larger symbols using my finger, but the difference was baffling. Another limitation: The lasso tool won’t automatically extend beyond the limits of the screen to the left and right, so inked equations have to fit within those parameters.
If you ink an equation incorrectly (or if OneNote misinterprets your ink) you have a chance to go back and select Fix It. That allows you to lasso a specific character or characters and do a rewrite. You then have to re-lasso the entire equation and resubmit it to be solved.
OneNote won’t always show its work.
If the formula is interpreted correctly, OneNote works its magic, providing a drop-down menu with different solution options, such as to differentiate for x, or integrate it. In most cases OneNote will show the steps it took to arrive at the answer, but it seemed to me that the more complex the problem, the less likely it was to do so.
The Ink to Math feature takes your inked equation and replaces it with text characters within the main window. Unfortunately, those characters aren’t editable. I suppose the eventual purpose of Ink to Math is to translate your ink into something that can be used in a more professional document. Right now, however, it’s just frustrating.
OneNote Math’s other problem: Competition
OneNote’s Math features also look underwhelming when you consider the existing competition. Here are other alternatives to consider:
1. Nebo, could do a better job of interpreting your ink into mathematical equations, even though its mathematical range may be more limited.
2. Wolfram, of Mathematica fame (that professional-grade app begins at $250), offers the WolframAlpha Windows 10 app for $2.99 per month. It includes step-by-step solutions, but without some of the richer features offered by the formal Pro subscription ($5.49 per month).
3. The FluidMath Windows 10 app ($14.99 for one year) is what OneNote Math should be: FluidMath supports digital inking and seems to do a better job than OneNote at interpreting your jottings (though it’s still finicky). Like Nebo, FluidMath shows you what it’s “seeing” as you’re inking it.
FluidMath’s ink interpretation isn’t great, either, but the app is simple and relatively easy to use.
A few things I particularly like about FluidMath: It considers all of the equations on the page, so you can ink a series of linear equations with, say, three variables. At the end, you can ink a solution set and it will fill in the answers. Alternatively, you can also ink an equation—then with a gesture, graph it. Finally, I like FluidMath’s liberal trialware system: While you only get a few minutes to try it (and it erases your work when the trial expires), you can open it up again and ink another equation.
4. Finally, there’s an oldie-but-goodie: Microsoft Mathematics. Dating from the Windows 7 days and boasting a nostalgic Windows CE-style interface, Microsoft Mathematics seems perfectly functional in Windows 10, albeit more like a graphing calculator than a direct rival to OneNote Math.
Microsoft Mathematics looks a little clunky, but seems surprisingly useful. Note the “business calculator” UI to the right, which stacks results on top of one another.
On the other hand, it’s free, extremely functional, and has a wealth of information under the hood, including common equations. About the only thing I couldn’t do with Microsoft Mathematics was ink: For some reason, the built-in inking function wouldn’t recognize my Surface Book stylus. Still, I was much more productive here than when I was using the early version of OneNote Math. Microsoft may be onto something with OneNote Math, but a reliance on ink is currently holding it back.
OneNote Replay: The promise, the reality
Not everyone can remember the details of a meeting the next day, let alone the next month—which is why there are note-taking apps like Microsoft OneNote. A new Replay feature lets you track the origin of whiteboarded visual ideas, too, though the concept needs some work.
As you might guess, Replay records the digital ink applied to a OneNote notebook as one or more people draw upon it. The end result is a stop-motion progression of still images, which you can scrub through like a movie, as an idea moves from concept to reality. Digital whiteboards like the KappIQ have talked about replay features, but Microsoft, its Surface Hub, and OneNote appear to have beaten everyone to the punch.
Unfortunately, OneNote’s Replay suffers from the artificial feature split between Microsoft’s productivity apps. Replay—as well as the new Math feature we evaluated—appear in the OneNote UWP app for Windows 10. That app is a subset of the more full-featured app, OneNote 2016, which has neither the Replay or the Math features and may never get them. You’ll also need to subscribe to Office 365 to receive Replay, via a Store app update.
Mark Hachman This Surface Hub is shown using the Sketchpad app that’s now in Windows Ink, but it can run the OneNote UWP app, and it probably should.
How Replay works
Replay hides within the View tab in OneNote. Keep in mind that the OneNote UWP app requires a touchscreen, if not a stylus—you can’t just begin drawing lines with a mouse. In practice, the app works well for individuals on a tablet like the Surface Book or Surface Pro 4. But while I didn’t have a Surface Hub to try with it, I can see how it would really shine on a large collaborative whiteboard.
Once you begin drawing within OneNote’s Replay, the app caches each stroke of your pen and each line of text. Changes, erasures, or adjustments aren’t recorded, so if you freehand a circle and ask OneNote to convert it to a shape, only the finished circle is recorded. Ditto for asking OneNote to interpret ink as mathematical equations. Clicking the Replay button plays back the ink (and text) at about a stroke (or line) per half-second or so, like a stop-motion movie. And that’s where OneNote’s inking team needs to begin making additional improvements.
First, there’s the issue of time constraints: Replay means watching your inkstrokes through from the beginning until the end, though you can use the scrubbing feature to accelerate the process. It took several attempts for the scrubbing feature to display properly, however. During the replay process, you can pause and add ink, and that ink will remain when the replay completes.
Mark Hachman As of right now, Replay hasn’t been integrated with the more powerful OneNote 2016—whose existing audio recording and syncing feature might match up well with the new Replay feature.
Second, there’s the issue of irrevocable loss. Replay only stores ink that’s added, while notes that you erase disappear permanently. That process of elimination runs counter to the way we often generate ideas—by adding, subtracting, and trying out new things. Users who try to keep everything by erasing nothing risk cluttering up the notebook.
Maybe there’s room for a compromise: greater control over what ink is saved, and better still, the ability to go back and resurface ideas that were tossed out. This happens in real life: Imagine you planned out a strategic direction for next year. Then your budget was cut. Then your division was merged with a more profitable one. All of this would require re-adjustments to the original plan, but not necessarily a fresh start—like “forking” software, where you copy an existing code base, then head in a separate, parallel direction.
Right now, all you can do is highlight a portion of the final ink and copy it to another region on the page or to a new OneNote page. You can’t do that while the Replay feature is running, though, and that, too, would be handy.
In any event, OneNote’s new Replay capabilities provide the jumping-off point to something better. I’m a frequent user of OneNote, and its synced audio note-taking feature is a powerful tool for almost any scenario. Replay, if developed further, might have similar potential.
Inking might have a future as well. For now, however, the first implementations of inking within Windows need further development before they’re truly useful.