NeuroHybridNet: A Hybrid EEG Classification Model with Autoencoder-Enhanced Fractal-Wavelet Features

Electroencephalography (EEG) signals offer a rich but complex source of information for assessing cognitive states, particularly in dynamic and high-stakes environments such as driver attention monitoring, where rapid and accurate detection of mental fatigue or distraction is critical for safety and performance.This study proposes WaveFrac-AE, a hybrid EEG classification model that combines time-frequency analysis, fractal dynamics, and nonlinear descriptors with Autoencoder-based dimensionality reduction. EEG signals representing three cognitive states—focused, distracted, and drowsy—are transformed into a compact latent space capturing both temporal and structural complexity. By fusing Continuous Wavelet Transform features with fractal measures (Petrosian, Hurst, DFA), the model achieves robust representation of non-stationary EEG dynamics. The Autoencoder component enhances generalizability by filtering noise and redundancy, enabling accurate and scalable mental state classification for real-time neuroergonomic applications. This hybrid approach has been tested with various machine learning algorithms—namely XGBoost, LightGBM, CatBoost, Random Forest, and Support Vector Machines (SVM). In subject-specific analyses, SVM achieved an average accuracy exceeding 97%, while the aggregated dataset—combining all subjects—yielded an accuracy surpassing 92%. Comparisons with contemporary studies suggest that this method occupies a competitive position and, in numerous cases, demonstrates higher performance. Consequently, the proposed model offers high-performance solution in driver attentiveness monitoring systems, thereby showing substantial potential for the development of early warning systems integrated into smart automobiles.