Augmented reality assists helicopter pilots with demanding flight manoeuvres at sea
February 2, 2021 | Maritime helicopter missions
Augmented reality assists helicopter pilots with demanding flight manoeuvres at sea
DLR research helicopter EC135 FHS on landing approach
DLR has tested commercial augmented reality glasses in flight for the first time. One of the challenges is ensuring that the head-tracking system of the glasses can correctly distinguish the movements of a pilot’s head from the motion of the helicopter. DLR researchers have developed special software that can do exactly this. The software incorporates this capability into the display system to indicate obstacles.
An initial test flight with augmented reality glasses. In addition to an artificial horizon (red line), the helicopter pilot is provided with key information from the instrument display within the cockpit, directly in his immediate field of vision. This includes the first limit indicator (FLI), which displays the torque, turbine speed and turbine temperature at the same time.
As part of the HEDELA project, DLR and the aviation service of the German Federal Police are researching assistance systems to facilitate helicopter landings on ship decks.
Augmented reality glasses take the pressure off of pilots in poor visibility conditions, superimposing landmarks for orientation and other important information directly onto their field of vision.
Successful flight tests have been carried out with DLR's FHS research helicopter.
Focus: Aviation, security, helicopter research
Whether they are facing fog, rain or darkness, a high-sea landing on a ship deck under poor visibility conditions is a major challenge, even for experienced helicopter pilots. As part of the Helicopter Deck Landing Assistance (HEDELA) project, the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt; DLR) is working with the aviation service of the German Federal Police to develop flight assistance systems that help pilots land safely and accurately on ships. Augmented reality glasses have now been used for the first time in flight tests with DLR’s FHS research helicopter.
Helicopters play an important role in rapidly expanding offshore industry. Their ability to take off and land vertically, hover in one spot and reach high flight speeds means that they are well suited for a variety of applications at sea. Problems arise, however, under poor visibility conditions, often caused by sea fog, precipitation and darkness. "Out at sea, pilots lack fixed reference points for orienting themselves," says Stefan Levedag of the DLR Institute of Flight Systems. "All they see around them is the open sea, and the moving ship offers a relatively small space on which to land." This is compounded by the wake turbulence on the ship's deck, which can make it even more difficult for the helicopter to safely set down.
Visual assistance system helps pilots during landing approach
As part of the HEDELA project, DLR has developed and validated special displays for landings on ship decks. The aim is to support pilots throughout the entire flight, including providing obstacle-free flight path planning (trajectories) for approach, transfer and takeoff. Augmented reality (AR) glasses take the pressure off pilots performing these tasks in poor visibility conditions, superimposing landmarks for orientation and other important information directly onto their field of vision. In thick cloud, for instance, the floating ship is outlined in bright green. Important information about flight speed, altitude and position, plus an artificial horizon are also made visible within the pilot's field of vision. "The superimposed symbols are incredibly beneficial for helping pilots find their bearings," says Project Manager Malte-Jörn Maibach. "The major benefit is that the pilot doesn’t have to be constantly switching his gaze between the instruments in the helicopter cockpit and the view outside. That allows him to keep his eyes on the landing point." This takes the pressure off the pilot and increases his situational awareness. In doing so, it also helps to optimise the operational capability and safety of maritime helicopter missions.
Current helicopter cockpit developments focus on reducing and combining display devices. One example is the first limit indicator (FLI), which presents the most important data relating to the status of the propulsion system (torque, turbine speed and turbine outlet temperature) using a single monitoring instrument. This type of flight assistance system, which was also implemented in the tested head-mounted display (HMD) described below, can significantly improve the general flight characteristics and handling properties of a helicopter.
The holographic display remains aligned with the world even during dynamic manoeuvres.
Looking through augmented reality glasses during an in-flight system test
Your consent to the storage of data ('cookies') is required for the playback of this video on Quickchannel.com. You can view and change your current data storage settings at any time under privacy.
Looking through augmented reality glasses during an in-flight system test
The holographic display remains aligned with the world even during dynamic manoeuvres.
The researchers developed the design of the display system during a predecessor project, Helicopter Flight Safety in Maritime Operations (HELMA), and tested it successfully using a simulator. DLR has now trialled the use of commercial augmented reality glasses in flight for the first time during tests with its Active Control Technology / Flying Helicopter Simulator (ACT/FHS) research helicopter, a modified EC135. HMDs – helmets with integrated displays worn by pilots – are already widely used in the military sector and offer a number of benefits. However, such systems are very expensive. As part of their work, the researchers are therefore drawing upon commercial systems and developing them with the specific requirements of maritime helicopter missions in mind.
"The advantage is that commercial systems are cheaper to procure, while still making use of very advanced technology," says Maibach. The head-tracking feature of AR glasses presents a notorious challenge, however, as it is not designed for use in moving systems, such as by the driver of a vehicle. "The system must be able to distinguish the movements of a pilot's head from the motion of the helicopter, which causes the outside world to appear to move," says Aerospace Engineer Christian Walko of the Institute of Flight Systems. For this purpose, DLR researchers have developed software that recognises the position of the pilot’s head in relation to the moving helicopter and takes it into account when displaying the holographic symbols. The researchers are currently analysing data from the flight tests. They are also investigating how the AR glasses performed in the air in conjunction with the newly developed head-tracking algorithm and attempting to determine the impact of helicopter vibrations on the entire system configuration.
New display systems are of considerable interest to helicopter pilots, who have to perform highly complex flight manoeuvres that require them to keep the helicopter under control while maintaining a constant awareness of the mechanical flight parameters. Innovative displays make it possible to integrate the relevant information with standard interfaces and help make staying within these limits as intuitive as possible.
The challenge of landing on a ship deck – three questions for the Director of the DLR Institute of Flight Systems, Professor Stefan Levedag
Why is the DLR Institute of Flight Systems conducting research into helicopter landings on ship decks?
Landing on a ship deck is one of the most challenging flight manoeuvres for a helicopter pilot. Besides having to cope with a moving landing platform, they are often beset by strong winds, turbulence caused by the contours of the ship, a lack of reference points for gauging position and speed, and poor visibility with no horizon. Providing pilots with an optimal system for coping with such extreme conditions is a major challenge, but DLR researchers are using cutting-edge technology to develop effective solutions.
Such assistance systems are already used in the military sector. Why are there are not yet any affordable alternatives for civil aviation? When might such a system become available for civilian operators?
Military systems are subject to very stringent additional requirements, such as compatibility with night vision technology, and are therefore more costly. Integrating new features is more feasible in that type of environment than with a commercial product, for which there is a pressure to keep costs as low as possible.
New systems, such as the HoloLens, also have to be qualified for future use in aircraft. They have to be able to withstand heavy vibrations, temperature differences and electromagnetic fields, for instance, without compromising on accuracy. Major obstacles often arise when attempting to deploy innovative solutions from the field of consumer electronics, and overcoming them takes time and considerable resources. As a result, it is difficult to predict when such technology might successfully transition into an aviation product. The hurdles are particularly high given the difficult period that the aviation sector is currently experiencing.
What is special about the ACT/FHS research helicopter?
The EC135 ACT/FHS helicopter is a unique instrument for research. It has open interfaces that make it possible to feed all of the essential flight status data into experimental systems, including new displays or even visualisations such as the HoloLens. Due to the exceptional architecture of the system, these new elements do not have to be approved, as would be necessary with a commercial application. This even applies to new systems and functions that feed control commands into the helicopter.
Security research at DLR
DLR's security research and development activities are planned and monitored in conjunction with partners in government, academia, industry and international organisations. As such, the interdisciplinary field of security research combines the core strengths of established DLR programmes in the fields of aviation, space, energy and transport. In total, over 20 DLR institutes and facilities are contributing to the development, testing and evaluation of technologies, systems and concepts within their security-related work, and are also conducting analysis and assessments of security-related applications.
