Mudra
Left Border
Prof. Dr. Beat Signer
Vrije Universiteit Brussel
Department of Computer Science
Pleinlaan 2, 1050 Brussels
(Belgium)
+32 2 629 1239, bsigner@vub.be
Office: PL9.3.60 (Pleinlaan 9)
VUB
View Beat Signer's profile on LinkedIn twitter View Beat Signer's profile on Facebook View Beat Signer's profile on YouTube Instagram View Beat Signer's profile on academia.edu View Beat Signer's profile on Google Scholar View Beat Signer's profile on ResearchGate View Beat Signer's profile in the ACM Digital Library View Beat Signer's ORCID profile Slideshare View Beat Signer's profile on Speaker Deck View Beat Signer's profile on 500px View Beat Signer's profile on SmugMug

Mudra: A Unified Multimodal Interaction Framework

Multimodal interfaces have become an important solution in the domain of post-WIMP interfaces. However, significant challenges still have to be overcome before multimodal interfaces can reveal their true potential. We addressed the challenge of managing multimodal input data coming from different levels of abstraction. Our investigation of related work shows that existing multimodal fusion approaches can be classified in two main categories: data stream-oriented solutions and semantic inference-based solutions. We further highlighted that there is a gap between these two categories and most approaches trying to bridge this gap introduce some ad-hoc solutions to overcome the limitations imposed by initial implementation choices. The fact that most multimodal interaction tools have to introduce these ad-hoc solutions at one point confirms that there is a need for a unified software architecture with fundamental support for fusion across low-level data streams and high-level semantic inferences.
Mudra architecture
Fig. 1: Mudra architecture

We presented Mudra, a unified multimodal interaction framework for the processing of low-level data streams as well as high-level semantic inferences. Our approach is centred around a fact base that is populated with multimodal input from various devices and recognisers. Different recognition and multimodal fusion algorithms can access the fact base and enrich it with their own interpretations. A declarative rule-based language is used to derive low-level as well as high-level interpretations of information stored in the fact base. By presenting a number of low-level and high-level input processing examples, we have demonstrated that Mudra bridges the gap between data stream-oriented and semantic inference-based approaches and represents a promising direction for future unified multimodal interaction processing frameworks.


Related Publications

  • 2015

  • 2014

  • thumb Engineering Gestures for Multimodal User Interfaces, , , , and , Proceedings of EICS 2014, 6th International Conference on Engineering Interactive Computing Systems, Rome, Italy, June, 2014
    Available:  document ACM digital library
  • 2013

  • 2011