An Immersive Game-like Teaching Environment with Simulated Teacher and Hybrid World

DOI: 10.5176/978-981-08-8227-3_cgat08-30

Authors: Richard Leibbrandt, Martin Luerssen, Takeshi Matsumoto, Kenneth Treharne, Trent Lewis, Martin Li Santi, David Powers

Abstract:

In this research we describe the development of a modular Teaching Head platform for research into the linguistic and social aspects of effective communication with and by an Artificial Intelligence. A number of embedding scenarios will be discussed, focusing in particular on the embedding combination of the Thinking Head, a simulated Robot World, and a monitored Real World stage, in the context of game-like and puzzle-like teaching scenarios.

The Thinking Head platform consists very simply of a perceptual/sensory input stage, a mid-brain or dialogue manager, and an effectual/motor output stage. The input may be text or speech, and the output
is audio-visual synchronized speech. This simple chain is designed to allow straightforward substitution of modules, or expansion to provide additional multimodality.

The Microjaea and Magrathea Robot Worlds were designed as platforms for language and ontology learning. The environment allows simple real world scenarios to be simulated with unlimited variations. Originally it was designed and used to teach computers language using simulated grounding of semantics, but now we are adapting it to allow the computer to demonstrate it’s understanding of a concept or in fact to teach a concept or language, to a human participant.

The composite Teaching Head platform includes a number of cameras and microphones – these not only handle speech recognition, but allow lip-reading, expression-reading, speaker identification and tracking, and object identification and tracking. Conversely, the AV output not only shows a correctly articulating face synchronized with the speech, but allows expressions appropriate to the content, or on
demand, to be interspersed or overlaid.

Keywords: Robot World, Robot Vision, Serious Games, Educational Games, Second Language Learning, Foreign Language Learning, Thinking Head, Teaching Head