Presentation of the dissertation project and the research results to date:

Motivation

There is a remarkable success of robots in automation industries. In structured environments the robots’ speed, accuracy and reliability by far exceed human capabilities. But in natural human environments robots are often unable to perform simple tasks.

An advanced memory and knowledge processing system is essential for mobile service robots with sophisticated cognitive capabilities [Tenorth and Beetz, 2008].

Goals

The initial questions of the dissertation project are:

• How can different modalities be represented in such a way as to enable a mobile service robot to integrate the available sensing and acting modalities spontaneously into reasonable cross-modal skills?

• Can a reasonably executable model that mimics the cross-modal integration, representation and reasoning techniques of natural cognitive systems be constructed and applied to a service robot platform?

• How can a memory system that is based on the advanced reasoning techniques of an existing knowledge representation system be constructed in a way that enables a mobile service robot to improve its cognitive capabilities?

• How can such a memory system be integrated with an AI-based general purpose planner?

The dissertation project with the working title “Cross-Modal Enhanced Memory for Mobile Service Robots” tries to develop a biologically inspired cross-modal memory system for mobile service robots. The memory system will be based on the current neuroscientific understanding of cross-modal integration and knowledge representation. From general ontological knowledge down to low-level information, knowledge will be represented at different abstraction levels. Abstract concepts will be grounded to physical sensing and acting modalities.

The memory system will further be based on a knowledge representation system with advanced reasoning capabilities. It will therefore be able to deduce new knowledge. The final goal is to enable the robot to understand high-level tasks, reason about high-level concepts and decompose abstract tasks into executable robot skills.

The memory system will be integrated with the state-of-the-art service robot platform TASER [Weser, 2009]. TASER works in terms of plan-based robot control and is equipped with a general purpose deliberative planner. The planner requires knowledge about it its environment in order to find a reasonable plan for a task. This knowledge about the environment should be provided by the memory system. If necessary information is not represented explicitly, the system will try to acquire the information via perception, deduction or human-robot interaction. Recently the planner received its own rather rudimental knowledge representation technique that states information as “raw” facts. Relationships between abstract concepts cannot be modelled and general inference rules are not supported. Therefore, the planner should leave the knowledge representation to the memory system. It is further impossible to integrate external knowledge sources (e.g. perception, human-robot interaction or deduction) into the deliberative planning approaches. An interface between the planner and the memory system will be defined. Furthermore, the planner will be adapted in way that enables it to use the memory system, take advantage of the advanced reasoning capabilities and integrate external knowledge sources into the planning process.

The different sensing and acting modalities will be represented explicitly. This approach makes it easier for the planner to compose different modalities dynamically into complex robots skills. Inspired by natural multi-modal cognitive systems and the neuroscientific dissertation projects of CINACS, the planner will in so doing be qualified to find plans even in the presence of malfunctions. Unavailable modalities will be compensated in several situations via an appropriate composition of the remaining modalities.

In contrast to other existing service robots, TASER is endowed with a general purpose planner from the AI community that is not explicitly developed for a specific domain. This approach has the following advantages:

• Maintenance and further development are provided by a large open source community.

• State-of-the-art methods from theoretical AI are implemented; provably good performance.

• The well-defined interface to reactive parts of the architecture permits future replacement of the planning component.

But applicable general purpose planners presume the closed world assumption and are therefore unable to deal with incomplete information. Since mobile service robots act in a real dynamic environment and construct or adapt their world model autonomously based on sensory data, they are inevitably confronted with uncertain and incomplete information about the world. The memory system will therefore be able to represent uncertain and incomplete information about the world and additionally allow for reasoning about this kind of knowledge.

In a nutshell, the dissertation project is expected to lead to concepts and implementations that can improve the cognitive capabilities of a service robot in the following ways:

1. Available modalities can be composed into new cross-modal robot skills. In this way, the malfunction of modalities can in several situations be compensated by the remaining robot skills.

2. The exploitation of a multi-modal memory system will improve the planning process in the following ways:

1.  

   1. The efficient generation of reasonable plans will be enabled in more complex scenarios.

   2. Unnecessary, often time-intensive perception processes are avoided when the memory system can provide the needed information.

   3. Necessary unknown information can be acquired from external knowledge sources.

1. The presence of grounded high-level concepts permits instructors to formulate tasks in terms of abstract high-level concepts.

2. High-level knowledge and reasoning can be exploited to deduce new explicit information.

Current State of Work

Currently, an important part of the memory system has already been developed and implemented. Robot skills are modelled in a way that enables the planner to autonomously compensate the malfunction of modalities by composing the remaining modalities dynamically into reasonable robot skills that are able to fulfil a given task. The memory system is able to represent factual, conceptual and axiomatic knowledge. From robot platform specific knowledge to common sense knowledge all kinds of concepts and information can be expressed with this approach. The applied knowledge representation techniques have been significantly inspired and influenced by the knowledge representation techniques of natural cognitive systems. An existing state-of-the-art planning approach from the artificial intelligence community has been enhanced. The new planning approach is able to take advantage of natural cognitive systems’ methods like problem solving strategies, representing and reasoning with an incomplete world model and autonomously acquiring necessary information from external knowledge sources. The memory and planning systems are integrated into a coherent artificial cognitive system.

How I plan to proceed

In the next weeks the developed artificial cognitive system will be applied to the service robot platform TASER. Further experiments with the robot platform will be carried out in order to demonstrate the scientific innovations of the current approach.

The further course of action is:

• Extend the memory system

•  

  • Extend the cross-modal reasoning component in order to support more scenarios

  • Ground high-level concepts to the physical acting and sensing modalities

• Extend the planning system

•  

  • Integrate knowledge from general purpose concept databases

  • Improve –- inspired by natural cognitive systems — the selection of relevant information in a given context

The work will conclude with the evaluation phase of the developed and implemented architecture. Experiments will demonstrate the cognitive capabilities of the improved robot platform TASER. Finally, the results will be interpreted and the robot’s cognitive capabilities will be compared with other state-of-the-art service robots.

Links to other projects of the IRTG

The dissertation project is strongly connected to the work of Martin Weser who contributed important innovations by composing different modalities into coherent cross-modal robot skills. His dissertation project further demonstrates how the integration of cross-modal robot skills with a forefront deliberative planning system can improve the cognitive capabilities of a service robot platform. My dissertation project goes into a similar direction and will benefit substantially from Martin Weser’s results.

Furthermore there is a strong connection to Jianhua Zhang's project. We are developing a coherent artificial cognitive system together. Our cooperation includes the grounding of high-level concepts to acting and sensing modalities and the composition of these modalities to executable robot skills. Moreover we are using the same experimental robot platform (TASER) and the same knowledge representation system. Several joint experiments and evaluations are scheduled.

It is further linked to Christopher Baumgärtner’s and Nils Beuck’s project, because we are going to integrate the multi-modal natural language understanding systems of their dissertation projects into the artificial cognitive system of my project. Several profitable discussions about adequate knowledge representation techniques and interfaces helped me to make my own approach more flexible and adaptable. The integrated cognitive system will finally be implemented on the service robot platform TASER.

Moreover my project is linked to the projects of Dong Wang, Xiaobing Liu and Cailiang Liu. They are all working on visual concept representation and image/video analysis. We planned to share data and software systems. My work benefits a lot from their broad knowledge in this research area. I will try to integrate their results into my artificial cognitive system. All our dissertation projects will additionally profit from joint experiments on our service robot platform (TASER).

There is also an important connection to Mario Maiworm’s dissertation project. His project focuses on the integrative processing of different sources of information in humans. Maximum likelihood estimation models are used to describe cross-modal integration processes. In several discussions we gathered ideas of how to integrate these models into a coherent artificial cognitive system.

I further had several inspiring discussion about conceptual knowledge representation with Kris Lohmann, because that issue also plays an important role in his project.

Integration of visits abroad into the dissertation project

During my stay at the Tsinghua University in September 2009 I had several profitable discussions that influenced my dissertation projects. I had the opportunity to present my dissertation project to scientists from the robotic laboratory. They gave me several helpful hints about connected projects and literature. In this way they had an important impact on the further direction of my dissertation project. I further discussed several ideas with neuroscientists. These discussions helped me to get a better understanding of cognitive systems in general and influenced some parts of my artificial cognitive system.

References

M. Weser. Hierarchical Memory Organization of Multimodal Robot Skills for Plan-based Robot Control. Doctoral Thesis, Hamburg, Germany, 2009.

M. Tenorth and M. Beetz. Towards practical and grounded knowledge representation systems for autonomous household robots. Proc. IWCTS 2008 - the 1st International Workshop of Cognition for Technical Systems, München, Germany, Oct 06-07, 2008