Mitsubishi Electric today announced in a company press release that it has developed what it believes to be the world’s first Scene-Aware Interaction technology capable of highly natural and intuitive interaction with humans based on a scene-aware capability that can translate multimodal sensing information into natural language.
The technology recognises contextual objects and events based on multimodal sensing information that allows for natural and intuitive interaction with humans through the context-dependent generation of natural language.
The company claims that Scene-Aware Interaction technology will provide drivers with intuitive and accurate route guidance for their current location car journeys by providing scene relative guidance.
The Scene-Aware Interaction uses a number of tools to recognise contextual objects and events that include cameras to capture imagery and video, microphones to record audio information, and LiDAR to measure localisation information.
Mitsubishi are expecting Scene-Aware Interaction technology to have wider applications across different industries and be able to display the information in human-machine interfaces (HMIs).
Only recently Advantech announced it was introducing a new AI-based infection prevention solution and Scene-Aware Interaction technology could be a potential add-on to these types of solutions that are being increasingly rolled out due to COVID-19.
By having Scene-Aware Interaction technology in place it has the potential to enable help and efficiency in a number of industries from healthcare through to manufacturing by encouraging social distancing through to more detailed interaction with robots in an automated production facility.
An example of the technology in use for Mitsubishi’s electric cars is that rather than instructing the driver to “turn right in 50m,” the system would provide scene-aware guidance, such as “turn right before the postbox” or “follow that gray car turning right” giving an instruction to a driver they might more easily understand due to visual based direction.
Mitsubishi explains in the press release that the technology is capable of highlighting health and safety risks such as pedestrians crossing the road with the scene-aware guidance system generating voice warnings, such as “a pedestrian is crossing the street,” when nearby objects and people are predicted to intersect with the path of the car.
The scene-aware interaction technology incorporates Mitsubishi Electric’s proprietary Maisart®* compact AI technology which is applied across the different functions to enhance the features and allow it to translate the data into natural language.
To prioritise the different categories of information, Mitsubishi Electric developed the Attentional Multimodal Fusion technology, which is capable of automatically weighting salient unimodal information to support appropriate word selections for describing scenes with accuracy.
In benchmark testing using a common test set, the Attentional Multimodal Fusion technology used audio and visual information to achieve a Consensus-Based Image Description Evaluation (CIDEr) score that was 29 percentage points higher than in the case of using visual information only.
Mitsubishi Electric’s combination of Attentional Multimodal Fusion, scene understanding technology, and context-based natural language generation offers a potentially powerful and effective end-to-end Scene-Aware Interaction system for intuitive interaction with users in diverse and current situations.
To achieve this functionality, the system analyses scenes to identify distinguishable, visual landmarks, and dynamic elements of the scene, and then uses those recognized objects and events to generate intuitive sentences for guidance as mentioned above.
Mitsubishi explains that the recent advances in object recognition, video description, natural language generation, and spoken dialog technologies using deep neural networks are enabling machines to better understand their surroundings and interact with humans more naturally and intuitively.
[…] Mitsubishi Electric develops scene-aware interaction technology […]