10a.005.UL_TAU - Improving Drowsiness and Fatigue Prediction of Drivers with Multi-Modal Representation Learning and Information Fusion: Application to Traffic Safety

Project - Summary

According to the National Highway Traffic Safety Administration, drowsiness is a traffic safety hazard, which results in about 100,000 police-reported crashes each year. These crashes result in more than 1,550 fatalities and 71,000 injuries. Unobtrusive drowsiness detection methods can prevent catastrophic crashes by warning or assisting the drivers [1]. Recently, there have been several efforts at investigating drowsiness based on the facial features of the drivers [2]. Early drowsiness detection is still a challenge. There are no publicly available datasets for early driver drowsiness detection.

We designed and developed a driving simulator for multi-modal driver drowsiness detection and prediction system by fusing video and various other biometric signals (i.e. posture, wrist wearable and pressure sensors). The system includes equipment, hardware and software components that includes (driving simulator, multi-modal sensing modules, software for emotion and driver drowsiness detection). We also developed IRB application for conducting the study, and evaluated the simulator.

Project - Team

Team Member Role Email Phone Number Academic Site/IAB
Raju Gottumukkala PI (318) 680-5886 UL Lafayette
Moncef Gabbouj Co-PI 358 40 073 6613 Tampere University
Satya Katragadda Researcher N/A N/A UL Lafayette
RaviTeja Bhupatiraju Researcher N/A N/A UL Lafayette
Seyed Majid Hosseini Student N/A N/A UL Lafayette
Iftikar Ahmad Mentor N/A N/A TietoEvry

Project - Novelty of Approach

The proposed simulator collects the data of several sensors in real-time.

Conventional drowsiness detection models detect the drowsiness using video features (e.g. nodding, blink rate, etc.) that are highly influenced by the environmental conditions. The effect of condition and the data collection and sensor problem can disable the model or decrease the accuracy of the system. The proposed models employ the physiological signals along with video and behavioral data (posture and driving behaviors) to improve the performance of drowsiness detection modules. The proposed model is robust in the absence of one or some modalities.

Moreover, the existing drowsiness systems are detecting the drowsiness using facial signs (e.g. nodding, low blink rate). However, the need for early drowsiness detection is felt. In one hand, early drowsiness prediction facilitates the route management for the drivers. On the other hand, the early drowsiness prediction reduces the human error probability that is responsible for 90 percent of accidents investigated.

The proposed model fuses video and biometric signals using multi-headed attention that is robust to noise, artifacts, and missing data. The proposed model fuses video and different signals separately that studies the effect of each modality on drowsiness detection without increasing the cost drastically.

Project - Deliverables

1 Data collection of 40 participants
2 Investigate methods to integrate behavioral, physiological signals and video features [RL + Information fusion] on the dataset
3 Implementation of prototype for early drowsiness detection
4 Publish our analysis and the laboratory datasets
5 Share our results with IAB partners

Project - Benefits to IAB

Project - Documents

projects/year10/10a.005.ul.txt · Last modified: 2022/10/07 08:45 by sally.johnson