Gaze tracking and pulse monitoring: how MAI Avionics Center development will help pilots

MAI Avionics Center appeared at the end of 2020: today, engineers with extensive experience in the aviation industry and students of the institute work here. Together they develop avionics, software, simulators, and prepare design documentation. The importance of this work is undeniable, because the active development of avionics requires the introduction of new technologies and the use of new solutions for standardized processes. To get to know the work of the Center better, to talk about avionics and new technologies in the industry, we met with Gleb Boyarsky, the leading engineer of MAI Avionics Center.

Gleb, tell us about your team.

We can divide it into three groups: the first is engaged in software development, the second is the development of hardware parts and rigs, and designers are in the third group. We employ both experienced professionals, recent graduates, and current students. We create conditions for the development of the students' competencies in avionics and are very interested in the fact that, after studying at the institute or even graduating from graduate school, they could continue their path in the industry, promote the chosen direction, becoming highly qualified specialists.

Avionics is…

This is a complex of electronic and radio-electronic devices that are on the board of an aircraft: an airplane, a helicopter, a tiltrotor… The term "avionics" is part of the avionics — avionics, which is responsible for the transmission and reception of radio waves of different characteristics, and the onboard equipment, which is engaged in receiving information, its processing and management. Avionics is also responsible for navigation, radio exchange, control of components and assemblies of the aircraft, security, special tasks, etc.

If we compare modern avionics and the avionics of the beginning of the century, how serious would be the changes that occurred?

The first and most obvious change easily seen by an amateur is the number of aircraft crew members. Earlier the crew consisted of 5-6 people: pilots, flight engineer, radio operator, navigator. And now we have come to the conclusion that one person is able to control the devices of some class, some - two, and in rare cases – three. These changes are due to the automation of a significant part of the tasks on board and, of course, technical means on the ground.

And the development of unmanned aerial systems followed consequently. Previously, for technical and financial reasons, it was impossible to create computers, sensors and means of radio exchange with such mass-dimensional and functional characteristics to equip small UAVs with them. Now unmanned aerial vehicles in range from 15 grams to several tons are designed, developed, produced and applied. At the same time, it should be noted that unmanned aerial vehicles as a class have existed for quite a long time, but the complexes of the current generation allow them to perform complex tasks in an automated mode, since they have become robotic in the broad sense of the word. The difference in the masses, dimensions and energy consumption of systems performing similar tasks at the beginning of the century and at the moment is calculated by orders of magnitude.

By the way, even devices available to the mass consumer are equipped with systems that are close to large onboard complexes in their functions: they have satellite communications, are able to organize radio communication between devices, carry optical systems, miniature lidars, have a developed monitoring and self-diagnosis system, etc.  

Can we mention flight safety improvement in connection with the development of avionics?

One of the main quality criteria in civil (primarily) aviation is safety. Therefore, any change in the composition of the complex, including the change in the number of crew members, receive the confirmation that the flight safety has not been affected.

A person as a component of a control system is inferior to electronics in solving typical tasks, but has the ability to adapt to changing external conditions, which are almost impossible to assume and take into account in the software, hardware or procedural part. Therefore, in the coming years, a pilot controlling a commercial aircraft will not disappear anywhere.

And when will unmanned aerial vehicles appear in commercial aviation?

Not before the 2050s-2060s. Because it's not just about proving that it's safe. Multiple conditions are to be met, including the non-trivial task of overcoming the psychological barrier, because passengers are used to the fact that there is a professional and reliable crew in the cockpit.

And what, from your point of view, will the change by 2051 in avionics be?

It is difficult to predict, because at least two directions of development are seen: increasing the intellectualization of the board and solving problems on board and organizing remote computing with reliable communication with control centers.

Despite all the abovementioned, still gradually the on-board equipment will take functions from the pilot, as cars equipped with an autopilot system already do. Moreover, even now a pilot of a commercial aircraft is more of an operator of a complex technical system that is responsible for control, communication, security, but he works with an automated system.

Assistive technologies will be developed to improve security: synthetic vision, augmented reality, virtual reality…

Artificial intelligence: how is it being introduced into avionics today?

Despite all the hype around neural networks, machine learning, and so on, in general, these are software complexes that solve narrow special tasks. Individual tools that are associated with artificial intelligence are used. And there are also prerequisites for the transition to more multitasking neural networks that can be embedded in the computing complexes of the aircraft.

And why do you need a synthetic vision?

Imagine a situation when a pilot, for example, cannot control the space around him due to fog. You can focus on the devices. But there are such cases when there is a need to know the position of potential obstacles or a runway. And that's when the synthetic vision system comes into operation, it helps to build a virtual image using a virtual camera based on terrain and objects located in the area where the board is located.

Creating such a system is a difficult task. It should add to situational awareness and spatial orientation, improve flight safety. Some avionics manufacturers are already implementing it into their products. We are working on a system and methods of displaying information for a synthetic vision system from the point of view of psychophysiology, ergonomics and are looking for the best ways to present information.

In the Center you are also working on an objective control system…

When a commercial product is being developed, be it a computer, an operating system, or just a website, the manufacturer uses a tracking system to make sure that everything is set up correctly, attention is held exactly where it is needed by the developer, and so on. But for some reason, this tool is not widely used in the aviation industry. After all, the issue of attention distribution in an aircraft is much more important than, for example, on a furniture sales website. We believe that this is an omission, and we propose a system that helps to evaluate not only the gaze direction, but also other physical indicators of a person that can be used in an objective assessment of his condition and reaction to an irritant.

This system allows to register how a person reacts to the interface, and helps to assess the quality of the developed technical complex. It can also help to assess the quality of human interaction with the system, for example, in aviation simulators, which are designed to form pilots' skills of action in certain conditions.

The assessment of the crew's activities is carried out by an instructor now. And although video and audio recording of the process is being conducted, the human factor cannot be completely excluded from the preparation process. An impartial link in this case is an objective control system that evaluates not only the observance of the trajectory, but also the sequence of actions of the pilot: did he look at the indicators before performing a certain action, or did he do it unconsciously? Did he take into account the flight parameters? Is the operator's action erroneous and potentially dangerous? Mistakes of this nature in real life can end with an accident or a disaster. And on the simulator, using an objective control system, it is possible to assess whether the actions were performed consciously and meaningfully or not.

At what stage is the development?

We have all the hardware and software complexes ready: an encephalograph, gaze tracking, a heart rate meter, and so on. Software for integration and synchronization of modules is currently being developed.

When can the implementation on simulators be expected?

In order for the system to be implemented, aviation training centers, airlines and regulatory authorities should be interested in this, and the use of this system in evaluation should be normalized. And now we are looking for partners from the developers of simulator systems who will agree to integrate it into the simulator. We believe that this system will give an advantage in the simulator market for the company that agrees to integrate it.

With the integration of such a system into stands everything is much easier, because the stands do not have the same requirements as simulators. And we plan to implement this system and offer it to customers.

What other developments of the Center can you call the most promising?

In addition to the objective control system, we are engaged in the development of prototyping stands that allow us to shape not only the cockpit of the aircraft, but also other operator seats, the development of automated testing systems, input/output control stands, troubleshooting in the cable network, testing systems for on-board indicators and much more.

We develop software, create tool stands for serial plants (input and output control stands), design prototyping stands, system status control stands. We plan to continue the development of this direction, expanding the stand functionality.