Project Details

HAAWAII - Highly Automated Air Traffic Controller Workstations with Artificial Intelligence Integration

Project Period: 1. 6. 2020 – 30. 11. 2022

Project Type: grant

Code: H2020-SESAR-2019-2

Agency: Evropská unie

Program: Horizon 2020

Czech title
HAAWAII - Highly Automated Air Traffic Controller Workstations with Artificial Intelligence Integration
Type
grant
Keywords

Artificial Intelligence , Machine Learning, Air-Traffic Control, Natural Language
Processing, Automatic Speech Recognition, 

Abstract


Advanced automation support developed in Wave 1 of SESAR IR includes using of
automatic speech recognition (ASR) to
reduce the amount of manual data inputs
by air-traffic controllers. Evaluation of controllers feedback has been subdued
due
to the limited recognition performance of the commercial of the shell ASR
engines that were used, even in laboratory
conditions. The reasons for the
unsatisfactory conclusions include e.g. inability to distinguish controllers
accents, deviations
from standard phraseology and limited real-time
recognition performance. Past exploratory research funded project
MALORCA,
however, has shown (on restricted use-cases) that satisfactory performance can be
reached with novel datadriven
machine learning approaches.
Based on the
results of MALORCA HAAWAII project aims to research and develop a reliable, error
resilient and adaptable
solution to automatically transcribe voice commands
issued by both air-traffic controllers and pilots. The project will build
on
very large collection of data, organized with a minimum expert effort to
develop a new set of models for complex
environments of Icelandic en-route and
London TMA. HAAWAII aims to perform proof-of-concept trials in
challenging
environments, i.e. to be directly connected with real-life data
from ops room. As pilot read-back error detection is the main
application,
HAAWAII aims to significantly enhance the validity of the speech recognition
models. The proposed work goes
far beyond the work planned for the Wave 2 IR
programme and will improve both safety and reduce controllers workload.
The
digitization of controller and pilot voice utterances can be used for a wide
variety of safety and performance related
benefits including, but not limiting
to pre-fill entries into electronic flight strips and CPDLC messages. Another
application
demonstrated during proof-of-concept will be to objectively
estimate controllers workload utilising digitized voice recordings
of the
complex London TMA.

Team members
Smrž Pavel, doc. RNDr., Ph.D. (DCGM) – research leader
Doležal Jan, Ing. (DCGM)
Dytrych Jaroslav, Ing., Ph.D. (DCGM)
Hradiš Michal, Ing., Ph.D. (DCGM)
Jírovec Martin, Ing. (DFIT-Dean)
Musil Martin, Ing., Ph.D. (DCGM)
Otrusina Lubomír, Ing. (DCGM)
Veselý Karel, Ing., Ph.D. (DCGM)
Publication Results

2023

  • MOTLÍČEK, P.; PRASAD, A.; NIGMATULINA, I.; HELMKE, H.; OHNEISER, O.; KLEINERT, M. Automatic Speech Analysis Framework for ATC Communication in HAAWAII. In SESAR Innovation Days. SESAR Innovation Days. Seville: SESAR Joint Undertaking, 2023. no. 11, p. 1-9. ISSN: 0770-1268. Detail
  • ZULUAGA-GOMEZ, J.; SARFJOO, S.; PRASAD, A.; NIGMATULINA, I.; MOTLÍČEK, P.; ONDŘEJ, K.; OHNEISER, O.; HELMKE, H. BERTraffic: BERT-based Joint Speaker Role and Speaker Change Detection for Air Traffic Control Communications. In IEEE Spoken Language Technology Workshop, SLT 2022 - Proceedings. Doha: IEEE Signal Processing Society, 2023. p. 633-640. ISBN: 978-1-6654-7189-3. Detail

2022

  • HELMKE, H.; ONDŘEJ, K.; SHETTY, S.; KLEINERT, M.; OHNEISER, O.; EHR, H.; ZULUAGA-GOMEZ, J.; SMRŽ, P. Readback Error Detection by Automatic Speech Recognition and Understanding - Results of HAAWAII project for Isavia's Enroute Airspace. SESAR Innovation Days 2022. Budapest: 2022. p. 1-9. Detail
  • KOCOUR, M.; ŽMOLÍKOVÁ, K.; ONDEL YANG, L.; ŠVEC, J.; DELCROIX, M.; OCHIAI, T.; BURGET, L.; ČERNOCKÝ, J. Revisiting joint decoding based multi-talker speech recognition with DNN acoustic model. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. Proceedings of Interspeech. Incheon: International Speech Communication Association, 2022. no. 9, p. 4955-4959. ISSN: 1990-9772. Detail
  • NIGMATULINA, I.; ZULUAGA-GOMEZ, J.; PRASAD, A.; SARFJOO, S.; MOTLÍČEK, P. A Two-Step Approach to Leverage Contextual Data: Speech Recognition in Air-Traffic Communications. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. Singapore: IEEE Signal Processing Society, 2022. p. 6282-6286. ISBN: 978-1-6654-0540-9. Detail
  • PRASAD, A.; ZULUAGA-GOMEZ, J.; MOTLÍČEK, P.; SARFJOO, S.; NIGMATULINA, I.; OHNEISER, O.; HELMKE, H. Grammar Based Speaker Role Identification for Air Traffic Control Speech Recognition. Proceedings of the 12th SESAR Innovation Days. Budapest: 2022. p. 1-9. Detail
  • PRASAD, A.; ZULUAGA-GOMEZ, J.; MOTLÍČEK, P.; SARFJOO, S.; NIGMATULINA, I.; VESELÝ, K. Speech and Natural Language Processing Technologies for Pseudo-Pilot Simulator. Proceedings of the 12th SESAR Innovation Days. Budapest: 2022. p. 1-9. Detail

2021

  • HELMKE, H.; KLEINERT, M.; SHETTY, S.; OHNEISER, O.; EHR, H.; PRASAD, A.; MOTLÍČEK, P.; VESELÝ, K.; ONDŘEJ, K.; SMRŽ, P.; HARFMANN, J.; WINDISCH, C. Readback Error Detection by Automatic Speech Recognition to Increase ATM Safety. In Proceedings of ATM Seminar. on-line: EUROPEAN ORGANISATION FOR THE SAFETY OF AIR NAVIGATION, 2021. p. 1-10. Detail
  • HELMKE, H.; SHETTY, S.; KLEINERT, M.; OHNEISER, O.; EHR, H.; MOTLÍČEK, P.; PRASAD, A.; WINDISCH, C. Measuring Speech Recognition And Understanding Performance in Air Traffic Control Domain Beyond Word Error Rates. In Proceedings of 11th SESAR Innovation Days 2021. Belgie: 2021. p. 1-8. Detail
  • KLEINERT, M.; HELMKE, H.; SHETTY, S.; OHNEISER, O.; EHR, H.; PRASAD, A.; MOTLÍČEK, P.; HARFMANN, J. Automated Interpretation of Air Traffic Control Communication: The Journey from Spoken Words to a Deeper Understanding of the Meaning. In Proceedings of DASC 2021. San Antonio, Texas: Institute of Electrical and Electronics Engineers, 2021. p. 1-9. ISBN: 978-1-6654-3420-1. Detail
  • KOCOUR, M.; VESELÝ, K.; BLATT, A.; ZULUAGA-GOMEZ, J.; SZŐKE, I.; ČERNOCKÝ, J.; KLAKOW, D.; MOTLÍČEK, P. Boosting of Contextual Information in ASR for Air-Traffic Call-Sign Recognition. In Proceedings Interspeech 2021. Proceedings of Interspeech. Brno: International Speech Communication Association, 2021. no. 8, p. 3301-3305. ISSN: 1990-9772. Detail
  • SZŐKE, I.; KESIRAJU, S.; NOVOTNÝ, O.; KOCOUR, M.; VESELÝ, K.; ČERNOCKÝ, J. Detecting English Speech in the Air Traffic Control Voice Communication. In Proceedings Interspeech 2021. Proceedings of Interspeech. Brno: International Speech Communication Association, 2021. no. 8, p. 3286-3290. ISSN: 1990-9772. Detail
  • ZULUAGA-GOMEZ, J.; NIGMATULINA, I.; PRASAD, A.; MOTLÍČEK, P.; VESELÝ, K.; KOCOUR, M.; SZŐKE, I. Contextual Semi-Supervised Learning: An Approach to Leverage Air-Surveillance and Untranscribed ATC Data in ASR Systems. In Proceedings Interspeech 2021. Proceedings of Interspeech. Brno: International Speech Communication Association, 2021. no. 8, p. 3296-3300. ISSN: 1990-9772. Detail
Back to top