Robots and Wizards: An Investigation Into Natural Human–Robot Interaction

Wizard-of-Oz Study on Natural Robot Control using Touch and Speech

This study presents the first prototype of RoSA (Robot System Assistant), a multimodal robot interaction system designed for intuitive control in industrial environments. Using a Wizard-of-Oz (WoZ) approach, 36 participants interacted with what they believed to be an intelligent robot system—when in fact it was secretly controlled by a human operator.

The setup involved a UR5e robot, speech and gesture input, and a task-based scenario where users had to request blocks, spell words, and build a pyramid using voice and pointing gestures. A deliberate error was introduced during each task to observe how users adapt to misunderstandings and failures.

Key findings:

• 97% of participants used speech input, with 72% combining it with gestures, confirming the natural tendency toward multimodal communication.
• Despite system flaws and simulated failures, users rated RoSA highly, with an average UX score of 74% across four usability questionnaires (SUS, UMUX, PSSUQ, ASQ).
• The experiment showed strong correlations between gesture and descriptive speech, and between micro/macro gestures and task efficiency.
• Users often assumed RoSA to be similar to AI assistants like Alexa or Google, indicating preconceptions shaped interaction behavior.
• A gesture catalog was derived from the participants’ suggestions, identifying common intuitive motions for actions like move, stop, and gripper control.

This study not only validates the intuitive potential of the RoSA concept but also provides a catalog of interaction strategies and gestures that laid the groundwork for the development of RoSA 2.0 and 3.0.

IEEE Access Best Multimedia Award

This video is a promotional prize winner of the 2020 IEEE Access Best Multimedia Award

Fulltext Access

https://doi.org/10.1109/ACCESS.2020.3037724

Citing

@ARTICLE{RoSA1,
  author={Strazdas, Dominykas and Hintz, Jan and Felßberg, Anna-Maria and Al-Hamadi, Ayoub},
  journal={IEEE Access}, 
  title={Robots and Wizards: An Investigation Into Natural Human–Robot Interaction}, 
  year={2020},
  volume={8},
  pages={207635--207642},
  keywords={Service robots; Task analysis; Correlation; Robot kinematics; Cameras; Education; Robot vision systems; Activity recognition; Cooperative systems; Gesture recognition; Human-robot interaction; Intelligent robots; Interactive systems; Robot control; Robot learning; Telerobotics},
  doi={10.1109/ACCESS.2020.3037724}
}