Sensori-Motor Augmented Reality for Tele-robotics:
Intelligent Mediation System

M. Rokonuzzaman, R.G. Gosine

CóCORE (CDN)

Résumé

Le téléguidage des véhicules dans le domaine de l'exploitation minière exige une attention incessante de la part de l'opérateur humain, ce qui nuit à l'efficacité. D'autre part, le temps mis par l'information pour arriver au poste de conduite et en repartir influe négativement sur la qualité de la perception, d'où risque de fausse manoeuvre. Enfin, l'interaction directe entre l'homme et la machine expose à des incidents de parcours. Ce sont là autant de restrictions majeures dont souffrent les techniques de manoeuvre à distance classiques lorsqu'on opère en terrain non aménagé présentant des conditions difficiles. Les engins ont tendance à évoluer au ralenti du fait de la nature du terrain et de leur grosseur relative, et la complexité des interactions avec le milieu environnant leur interdit une autonomie complète. Pour échapper à ces contraintes, on propose une démarche nouvelle faisant appel aux techniques de télé-robotique avec complément sensori-moteur (projet SMART). La dissociation entre spécification et exécution des tâches, grâce à l'insertion d'une réplique virtuelle de l'équipement à l'endroit où le travail doit s'effectuer, promet de rationaliser l'exploitation et de réduire les risques d'incident de parcours.

Introduction

The objective of this program is to develop expertise to design, implement, and test an innovative approach to tele-robotics for interventions in harsh environment, based upon a concept of sensori-motor augmented reality, with the goal of producing more efficient, safer, and realistic alternatives to present-day operations. The utility of this innovative tele-robotics concept will be verified in the mining sector because of mining's significant position in Canadian economy and Canada's leading role in mining technology development. However, other application sectors, such as hazardous waste disposal, forestry, space exploration, remote excavation, and oil and gas infrastructure maintenance could benefit from this concept. This innovative tele-robotic concept consists of four facets: interactive three-dimensional sensing, enhanced perception, intelligent mediation, and intelligent control (as shown in Figure 1) in order to treat each element of a task distinctly - specification, execution, and supervision.

block diagram of tele-robotic system

Figure 1. The block diagram representation of a tele-robotic system using the SMART concept.

Facet 1. Interactive three-dimensional sensing

Interactive three-dimensional sensing promises to reduce the effect of transmission delays to and from the operator's site with real-time synthesised upon-request view from a small representative subset of all necessary information.

Facet 2. Enhanced perception

The development of enhanced perception will encompass issues of image-based three-dimensional reasoning, to properly handle quantitative spatial interactions between the virtual model and the three-dimensional stereoscopic real scene.

Facet 3. Intelligent mediation

The unstructured nature of the targeted application environments precludes complete autonomy, at this stage. It is anticipated that any time human intervention is essential to perform complex tasks in unstructured environments with the help of tele-operated robotics systems. The objective of intelligent mediation system is to provide a dual-mode control approach in order to accomplish a task in co-operation with autonomous and tele-operated control. The objective of the development of intelligent mediation system (IMS) is to allow a failure of a task to be traced to a single lower level task which could be corrected by human intervention through direct tele-operation and then to resume the autonomous execution of the remaining tasks after the correction.

Despite the development of supervisory control (i.e., dual mode control) solutions for well defined and simple tasks, the state-of-the-art of supervisory control techniques suggest that solutions involve considerable art. The lack of a mathematical formalism makes it difficult to trace the potential failure of a task to a single lower level task which could be corrected by human intervention and to resume the autonomous execution of the remaining tasks after the correction.

In order to accomplish the objective of IMS, every high level task will be decomposed into lower level tasks as a hierarchical discrete event system (DES). This hierarchical discrete event (DEV) model of task decomposition will be used as reference to generate control signals and monitor the state of task execution. A sensing system will monitor the quality of completion of each elementary task; in case of failure to complete a task, the human operator will intervene in order to correct it. The autonomous execution will resume after the completion of the incomplete elementary task by human operator through direct tele-operation. The development of a discrete event tool to analyse and design robotic tasks as a hierarchical finite state machine is underway. A discrete event model of typical mining tasks with the provision of human intervention is shown in Figure 2. This same tool will be used to generate control signals to the robot and monitor the state of execution in order to trace a failure of a task to a single lower level task which could be corrected by human intervention.

model of interctions

Figure 2. The discrete model of interactions among typical asynchronous and concurrent mining tasks with the provision of human intervention.

Facet 4. Intelligent control

This facet is concerned with the development of commands that will execute various actions, with sensorial feedback from sensors, while monitoring the status of task execution and surrounding environment, to suspend the execution of sub-task and to refer to the human operator. Intelligent control deals with the issues: (1) Plan generation, (2) Autonomous execution of sub-tasks, (3) Automated reasoning about the task execution.

Project status

SMART is in the first year of its four years life span. The literature survey, the development of co-operation with potential partners and preliminary development of all four facets are underway.

Future outlook

This innovative tele-robotics concept, based on sensori-motor augmented reality, will be matured over the next four years by demonstrating its utility in mining applications. The concept of different commercial products based on this novel tele-robotics technology will be identified. Present partnership among research institutes and industries will be strengthened and extended to include more potential partners in order to mature SMART. These partnerships will work to develop and commercialise products based on this innovative concept for efficient and safe intervention in harsh environments.

Table 1. The Participants
  Research Institutes
C-CORE, Memorial University of Newfoundland (CDN)
Ecole Polytechnique (CDN)
University of Toronto (CDN)
University of Prince Edward Island (CDN)
University of Ottawa (CDN) 

Government and other agencies
Institute for Robotics and Intelligent Systems (IRIS) (CDN)
European Space Agency

  Industrial
Atlantic Nuclear Services Ltd. (ANSL) (CDN)
AQUILA Mining Systems Limited (CDN)
Aquatic Sciences Inc. (ASI) (CDN)
Canpolar East Inc. (CDN)
Defence and Civil Institute of Environmental Medicine (CDN)
Dynacon Enterprises Limited (CDN)
Environment Test Facility Inc. (CDN)
MPB Technologies Inc. (CDN)
La Compagnie Miniére Québec Cartier (CDN)
Spar Space Systems (CDN)
Tektrend International Inc. (CDN)


About | Search | Feedback

RightLeftUpHomeTTP homepagePreparing for the Future Vol. 8 No. 4
Published December 1998.