Cortana can see you. Maybe. A mysterious, cone-shaped device, shown in a demo Monday at Microsoft’s Build event in Seattle, hinted at a big leap in capability for the company’s digital assistant: the ability to see and identify people.
In the ruthless war among Amazon’s Alexa, Apple’s Siri, and Google’s Assistant, this is a feature we haven’t yet seen from the competition. What we don’t know is what this prototype is, and whether it will ever come out.
Microsoft’s official, overshadowed demonstration showed us the conference room of the future. Microsoft incorporated Cortana, the Surface Hub, and Skype’s translation and transcription features, together with PowerBI and Microsoft Teams, into a cohesive working environment.
In a presentation led by Raanah Amjadi, a marketing manager for Microsoft Teams, Microsoft showed off how simply scheduling the meeting room could be facilitated by Cortana, who was asked to set up a meeting with the “smart building team” to discuss issues with a hypothetical product.
The prototype device resembled the Harman/Kardon Invoke that was shown off in previous demos. According to Amjadi, however, this device could both see and hear. “It picks up both audio and video that you can pair with AI services and Microsoft 365 to help with key tasks,” Amjadi said, “like identification, voice transcription, and even real-time translation.”
Mark Hachman / IDG
Microsoft’s cognitive services recorded and transcribed the conversation, taking notes as appropriate.
In the demo, as each attendee walked in, the device identified each person and greeted them by first name. When the leader, Amjadi, entered, the device also asked her whether she wanted to start the meeting. As Amjadi did so, a nearby Surface Hub began transcribing the conversation, separately taking meeting notes of key tasks. (Apparently, one of the trigger words was “follow up,” acting on Cortana’s existing capability to track emails and identify commitments or promises that you make to others.)
Above, a panoramic window showed all of the participants, apparently from the perspective of the prototype device. Microsoft’s AI services also provided a translation of the meeting for a colleague in China.
Mark Hachman / IDG
Microsoft even provided more than one translation at a time.
Software, services connect employees
Microsoft’s Teams app stood in as a sort of back-channel communication between the participants, providing links to shared services such as GitHub or Trello, but also facilitating follow-ups that the team members wanted to set. In one example, Cortana stepped in to volunteer her services to find a convenient meeting time among the participants. Likewise, Teams was smart enough to know which “Q2 sales forecast” the participants wanted, tapping into a feature within Microsoft Delve for surfacing relevant documents.
Mark Hachman / IDG
Microsoft apparently sees Teams as a supplementary services that participants will run in the background of a meeting.
Microsoft showed off features within Word that allowed users to send “@” messages to other members of the team as a short message, without the need to open Outlook or a chat application. Those messages even included apps, such as a message to the Microsoft To-Do app. In the latter case, Cortana stepped in, again, and suggested information that the employee had used before.
Mark Hachman / IDG
Teams allows users to send messages to other Teams members inside of apps like Word, and Cortana can step in and help out.
Microsoft also showed off a brief demo of Your Phone, a prototype app in which users can manage their phones directly from their PC. (Again, this feature is already built into Windows, but somewhat hidden within the Action Menu notifications. Your Phone will be launched to Insiders soon, Microsoft said.) Finally, the team “solved” the problem by visualizing the data in various ways using PowerBI, and exploring the data using the HoloLens.
A second Cortana device?
At press time, Microsoft officials had no further comment beyond what Amjadi said on stage. But it’s also true that Microsoft has had high hopes to push Cortana past the Invoke, into a world of cars and other connected devices. That, at least, was the original vision Microsoft promoted at the end of 2016.
Harman Kardon
The Harman / Kardon Invoke was the first and really only consumer device to be powered by Cortana.
Since then, though, Cortana has appeared in the Invoke and the GLAS smart thermostat from Johnson Controls—and that’s pretty much it. Amazon, meanwhile, is encouraging the use of its Alexa assistant in just about anything—including Windows PCs, which Microsoft has agreed to in a partnership.
So far, however, Alexa, Cortana, and Apple’s Siri are voice-powered devices. Amazon and Apple haven’t shown off machine-vision capabilities, though Amazon has demonstrated an Alexa-powered device with a screen. At the Microsoft Build keynote on Monday, Microsoft announced Project Kinect for Azure, the fourth generation of the Kinect sensor that other hardware makers can use to build their own hardware that can “see.”
That hardware, of course, would presumably use Microsoft’s services. Would a smart device that could see you be appealing to consumers, or simply creepy? In the home, consumers may worry that Microsoft could see them in their underwear, or monitor them around their home.
But Microsoft has turned away from the consumer market, favoring businesses and their larger, fixed budgets. A series of business devices powered by a sighted Cortana assistant makes sense—even if it’s still a little creepy.