MiNT: Multimodal Interaction for Modeling and Model Refactoring

The development of software brings together participants from different backgrounds, such as domain experts, analysts, designers, programmers, managers, technical writers, graphic designers, and users. No single participant can understand or control all aspects of the system under development, and thus, all participants depend on others to accomplish their work. Moreover, any change in the system or the application domain requires all participants to update their understanding of the system. The importance of continuous involvement of domain experts in the modeling process is well known. But domain experts are usually not proficient with the modeling tools used by the software developers and as a result are often limited to the initial requirements elicitation. Researchers have provided substantial evidence that multimodal interfaces can greatly expand the accessibility of interfaces to diverse and nonspecialist users. To address these limitations in the collaboration between application domain experts and modelers, we developed MiNT, an extensible platform to add new modalities and to configure multimodal fusion in CASE tools. MiNT is based on the M3 framework that allows capturing multimodal interaction during the design process of new multimodal interfaces. The M3 framework has been developed in a bootstrapping process during the development of MiNT. The viability of MiNT was demonstrated in two reference implementations; Mint Eclipse and Mint Mobile. MiNT Eclipse used the MiNT framework to add multimodality to Eclipsebased modeling. MiNT Mobile provides multimodal modeling and model transformations on mobile devices. We conducted two controlled experiments to study the feasibility and applicability of multimodal interfaces for modeling and model refactoring. The results of the first experiment show that multimodal interfaces employing speech as an input modality improve the efficiency of modelers. Speech additionally allows modelers to verbalize their thoughts and is suitable for collaborative modeling sessions. The results of the second experiment show that a multimodal interface which provides a combination of touch, speech, and touch gestures is more useful than a multimodal interface employing only touch and speech.