In recent years, computer scientists and roboticists have developed a variety of technological tools to aid human agents during critical missions, such as military operations or search and rescue efforts. Unmanned aerial vehicles (UAVs) have proved to be particularly valuable in these cases, as they can often enter remote or dangerous areas that are inaccessible to humans.
Researchers at Polytechnique Montréal recently developed a new system to control UAVs deployed during critical missions. This system allows users to control the UAVs via a head-mounted display (HMD).
The overall objective of the research was to explore the potential of new technologies, particularly AR, for assisting humans in critical situations. The researchers were offered the opportunity to closely collaborate with people who periodically interview firefighters in Montréal (SIM – Service incendie de Montréal), in order to understand how new technology could assist them in their work.
The goal of this study was to support the work of these firefighters in Montréal by managing a swarm of multiple drones using a single AR headset during a fire-related emergency. To do this, we designed an AR interface, using the Magic Leap 1 headset, which can be used to manage a swarm of UAVs in a stressful situation. The goal was to evaluate if AR could be an important tool for the future of critical situations.
The user interface (UI) designed by these researchers presents contextual information related to a fire and its location right in front of a user’s eyes. This information is displayed in the form of a 3D environment, which is projected on top of what a user is actually seeing at any given moment (i.e., with his/her real vision).
The system devised by the researchers ultimately allows users to control a swarm of drones in real-time via the Magic Leap 1 headset. To monitor and control the drones during critical missions, users simply need to interact with the 3D environment presented to them via the headset.
The technology we developed can bring a huge flow of information that can overload the user and must thus be filtered in an optimal way, in order to improve the situational awareness of the user and help him/her to understand the current situation effectively.
In contrast with previously proposed solutions for controlling UAVs during critical missions, the system proposed by the researchers is hands-free. This means that it allows users to focus on their vision, rather than having to simultaneously use their hands and visually monitor the situation.
The researchers evaluated their system in a series of experiments, where they asked people to tackle a complex and critical mission either using the headset they provided or a desktop computer. Their findings highlighted the benefits of AR technology in critical situations and confirmed the potential of the UI they developed.
Emergent technology brings new research, tests, and experiments to improve some use cases, in order to reduce the complexity of tasks for humans. AR is common today with mobile for multiple kinds of marketing or entertainment purpose. Using it with headsets in the field or for practical applications could be a huge improvement. For example, the U.S.A. wants to equip its military resources with this device.
In the future, this AR-based system developed could assist human agents during a wide range of critical missions, allowing them to control UAV swarms with their vision, without having to type on a computer or use conventional controllers. If combined with infrared technology, it could also be used by firefighters or armed forces to monitor their surroundings and control UAVs when they are unable to view their environment (e.g., when they are surrounded by smoke originating from a fire or explosion).
Unfortunately, due to the current pandemic situation, the test and deployment of our application with firefighters were postponed, so the team was forced to adapt these experiments to the current situation. Once the pandemic is over, we intend to conduct some tests with firefighters. Of course, we will also continue to research and develop similar applications of AR technology in various fields of applications such as healthcare, surgery, airplane virtual cockpits, and other collaboration environments.
