Creating the user experience for the digital car of the future — today

Multisensory interaction combining voice, gaze and head tracking

At the heart of Nuance’s CES presence was a 220° immersive theater experience coupled with the in-car demo in the Mercedes V-Class that brought the city of New York to Las Vegas.

Gaze interaction with POI featuring augmented reality

This combination ensures a very intuitive interaction with the outside world, that is becoming more like interacting with a human.

  • Ask questions and follow-up questions about POI to get general information, opening hours, ratings and more
  • Results are highlighted and supported by augmentation on a smart windshield
Button-free car: super-responsive in-car controls

This level of assistance allows OEMs to offer more & more car functions without having to clutter the car with as many buttons and dials. It also reduces driver distraction and increases overall passenger level of comfort by eliminating the need to remember the location and role of every button.

  • Just Talk interaction, with no need for Push-To-Talk button or WuW allowing to control many car functions, e.g. climate control, windows, doors, music, and map
  • The embedded voice interaction is super responsive and supports follow up commands in a natural and human-like style
  • Interactive car manual
Button-free car: Natural language understanding meets head tracking

Operating car controls without a need for buttons is not only very convenient, it will also help users control in-car functions once they are no longer tied to a driver's seat or when using robo-taxis.

  • Dragon Drive fuses head movement results with voice recognition to enable intuitive in-car controls, for example by looking at the passenger-side window and saying: “Open that window”.
Button-free car: In-sight widgets

Using the windshield as an additional interface to provide users with information or access to services will become a key UX feature in the context of the increasingly digital and autonomous car.

  • Using voice commands in combination with eye-tracking, users can intuitively interact with “widgets” displayed on the windshield to refine and select services and information – phone, contacts, weather, music – that would traditionally be shown on the console display.
Advanced NLU including multi-intent and conversational context
  • The assistant understands multiple-intent requests (e.g. triple intent phrases like “Take me to the Empire State Building, send the location to Frank Baker, and then play Jimmie Hendrix”) and uses advanced dialog to walk the user through the completion of every task, one-by-one
  • It remembers the conversation history, therefore being able to understand sentences such as “What’s the weather like there” or “How about Albany”, which refer to earlier parts of the conversation, and provide relevant answers

Key components

  • Advanced Natural Language Understanding with enhanced context capabilities & Just Talk​
  • Sensor fusion
  • Automotive-grade eye tracking camera
  • Smart windshield technology
  • Deep integration into the car
  • Nuance Dragon Drive Innovation Showcase

Leveraging in-car surveillance systems to react to the driver's cognitive & emotional state

Detecting and counteracting drowsiness

This technology could enhance safety on the road by preventing distracted, drowsy and impaired driving

  • The assistant detects signs of drowsiness and offers countermeasures in the form of human-like, proactive dialogs and recommendations

Key components

  • Affectiva Emotion AI
  • Nuance's adaptive-style text-to-speech
  • Natural Language Generation
Emotionally intelligent response

An assistant that can adjust its tone and style to users and specific situations may improve trust and rapport with users, thus leading to more use and providing an overall more engaging and valued experience.

  • The assistant adapts its behavior to the driver's emotional state by changing both its response style and tone of voice to match the situation.

Voice Biometrics and Plug & Play apps

Voice Biometrics and Plug & Play apps

Personalization, security and obsolescence prevention are important topics for the digital car. Dragon Drive offers features to address these OEM and consumer needs.

  • Voice Biometrics for secure transactions, such as payments and fund transfers
  • Plug & Play apps, which can be added to cars leveraging the cloud, even after SOP
  • Multi-Zone intelligence
Key components

  • Voice biometrics
  • Over-the-air updates
  • Interoperability with third-party systems

What CES visitors are saying about our demo

Schedule your individual demo and experience yourself.