Sensori-Motor Augmented Reality for Tele-robotics:
Intelligent Mediation System
M. Rokonuzzaman, R.G. Gosine
Le téléguidage des véhicules dans le domaine de
l'exploitation minière exige une attention incessante de la part
de l'opérateur humain, ce qui nuit à l'efficacité.
D'autre part, le temps mis par l'information pour arriver au poste de conduite
et en repartir influe négativement sur la qualité de la perception,
d'où risque de fausse manoeuvre. Enfin, l'interaction directe entre
l'homme et la machine expose à des incidents de parcours. Ce sont
là autant de restrictions majeures dont souffrent les techniques
de manoeuvre à distance classiques lorsqu'on opère en terrain
non aménagé présentant des conditions difficiles.
Les engins ont tendance à évoluer au ralenti du fait de la
nature du terrain et de leur grosseur relative, et la complexité
des interactions avec le milieu environnant leur interdit une autonomie
complète. Pour échapper à ces contraintes, on propose
une démarche nouvelle faisant appel aux techniques de télé-robotique
avec complément sensori-moteur (projet SMART). La dissociation entre
spécification et exécution des tâches, grâce
à l'insertion d'une réplique virtuelle de l'équipement
à l'endroit où le travail doit s'effectuer, promet de rationaliser
l'exploitation et de réduire les risques d'incident de parcours.
The objective of this program is to develop expertise to design, implement,
and test an innovative approach to tele-robotics for interventions in harsh
environment, based upon a concept of sensori-motor augmented reality, with
the goal of producing more efficient, safer, and realistic alternatives
to present-day operations. The utility of this innovative tele-robotics
concept will be verified in the mining sector because of mining's significant
position in Canadian economy and Canada's leading role in mining technology
development. However, other application sectors, such as hazardous waste
disposal, forestry, space exploration, remote excavation, and oil and gas
infrastructure maintenance could benefit from this concept. This innovative
tele-robotic concept consists of four facets: interactive three-dimensional
sensing, enhanced perception, intelligent mediation, and intelligent control
(as shown in Figure 1) in order to treat each element of a task distinctly
- specification, execution, and supervision.
Figure 1. The block diagram representation of a tele-robotic system
using the SMART concept.
Facet 1. Interactive three-dimensional sensing
Interactive three-dimensional sensing promises to reduce the effect of
transmission delays to and from the operator's site with real-time synthesised
upon-request view from a small representative subset of all necessary information.
Facet 2. Enhanced perception
The development of enhanced perception will encompass issues of image-based
three-dimensional reasoning, to properly handle quantitative spatial interactions
between the virtual model and the three-dimensional stereoscopic real scene.
Facet 3. Intelligent mediation
The unstructured nature of the targeted application environments precludes
complete autonomy, at this stage. It is anticipated that any time human
intervention is essential to perform complex tasks in unstructured environments
with the help of tele-operated robotics systems. The objective of intelligent
mediation system is to provide a dual-mode control approach in order to
accomplish a task in co-operation with autonomous and tele-operated control.
The objective of the development of intelligent mediation system (IMS)
is to allow a failure of a task to be traced to a single lower level task
which could be corrected by human intervention through direct tele-operation
and then to resume the autonomous execution of the remaining tasks after
Despite the development of supervisory control (i.e., dual mode control)
solutions for well defined and simple tasks, the state-of-the-art of supervisory
control techniques suggest that solutions involve considerable art. The
lack of a mathematical formalism makes it difficult to trace the potential
failure of a task to a single lower level task which could be corrected
by human intervention and to resume the autonomous execution of the remaining
tasks after the correction.
In order to accomplish the objective of IMS, every high level task will
be decomposed into lower level tasks as a hierarchical discrete event system
(DES). This hierarchical discrete event (DEV) model of task decomposition
will be used as reference to generate control signals and monitor the state
of task execution. A sensing system will monitor the quality of completion
of each elementary task; in case of failure to complete a task, the human
operator will intervene in order to correct it. The autonomous execution
will resume after the completion of the incomplete elementary task by human
operator through direct tele-operation. The development of a discrete event
tool to analyse and design robotic tasks as a hierarchical finite state
machine is underway. A discrete event model of typical mining tasks with
the provision of human intervention is shown in Figure 2. This same tool
will be used to generate control signals to the robot and monitor the state
of execution in order to trace a failure of a task to a single lower level
task which could be corrected by human intervention.
Figure 2. The discrete model of interactions among typical asynchronous
and concurrent mining tasks with the provision of human intervention.
Facet 4. Intelligent control
This facet is concerned with the development of commands that will execute
various actions, with sensorial feedback from sensors, while monitoring
the status of task execution and surrounding environment, to suspend the
execution of sub-task and to refer to the human operator. Intelligent control
deals with the issues: (1) Plan generation, (2) Autonomous execution of
sub-tasks, (3) Automated reasoning about the task execution.
SMART is in the first year of its four years life span. The literature
survey, the development of co-operation with potential partners and preliminary
development of all four facets are underway.
This innovative tele-robotics concept, based on sensori-motor augmented
reality, will be matured over the next four years by demonstrating its
utility in mining applications. The concept of different commercial products
based on this novel tele-robotics technology will be identified. Present
partnership among research institutes and industries will be strengthened
and extended to include more potential partners in order to mature SMART.
These partnerships will work to develop and commercialise products based
on this innovative concept for efficient and safe intervention in harsh
Table 1. The Participants
C-CORE, Memorial University of Newfoundland (CDN)
Ecole Polytechnique (CDN)
University of Toronto (CDN)
University of Prince Edward Island (CDN)
University of Ottawa (CDN)
Government and other agencies
Institute for Robotics and Intelligent Systems (IRIS) (CDN)
European Space Agency
Atlantic Nuclear Services Ltd. (ANSL) (CDN)
AQUILA Mining Systems Limited (CDN)
Aquatic Sciences Inc. (ASI) (CDN)
Canpolar East Inc. (CDN)
Defence and Civil Institute of Environmental Medicine (CDN)
Dynacon Enterprises Limited (CDN)
Environment Test Facility Inc. (CDN)
MPB Technologies Inc. (CDN)
La Compagnie Miniére Québec Cartier (CDN)
Spar Space Systems (CDN)
Tektrend International Inc. (CDN)
| Search |
for the Future Vol. 8 No. 4
Published December 1998.