Real-Time Eye Tracking and Head Pose Estimation Using Computer Vision and MediaPipe

Introduction

Eye tracking and head pose estimation have become increasingly important in human-computer interaction, healthcare monitoring, and attention analysis. This paper presents a comprehensive system that combines real-time eye blink detection with head pose estimation using computer vision techniques and the MediaPipe framework. The system provides a non-invasive method for monitoring visual attention and head orientation, with applications ranging from driver drowsiness detection to human behavior analysis.

Methodology

1. System Architecture

The system comprises three main components:

• Facial landmark detection using MediaPipe Face Mesh
• Eye blink detection and tracking
• Head pose estimation using 3D-to-2D point correspondence

2. Eye Blink Detection

2.1 Landmark Detection

• Utilizes MediaPipe's Face Mesh to detect 468 facial landmarks
• Specifically tracks 16 landmarks for each eye (RIGHT_EYE and LEFT_EYE)
• Converts normalized coordinates to image space coordinates

2.2 Blink Analysis

• Calculates Eye Aspect Ratio (EAR) using:
  • Horizontal distance between eye corners
  • Vertical distance between upper and lower eyelids
• Implements a ratio-based threshold system
• Uses temporal consistency check (CLOSED_EYES_FRAME = 3) to reduce false positives

3. Head Pose Estimation

3.1 3D Model Construction

• Creates a simplified 3D facial model using six key landmarks
• Maps 2D image coordinates to 3D space
• Establishes camera matrix using focal length and image dimensions

3.2 Pose Calculation

• Employs Perspective-n-Point (PnP) algorithm
• Extracts rotation matrix using Rodrigues' rotation formula
• Decomposes rotation matrix to obtain Euler angles (pitch, yaw, roll)

3.3 Direction Classification

• Implements threshold-based classification for head orientation:
  • Looking Left: y < -10°
  • Looking Right: y > 10°
  • Looking Down: x < -10°
  • Looking Up: x > 10°
  • Forward: within ±10° range

Implementation Details

1. Technical Requirements

• OpenCV (cv2) for image processing
• MediaPipe for facial landmark detection
• NumPy for numerical computations
• Real-time video capture capabilities

2. Performance Optimization

• Implements frame resizing for processing efficiency
• Maintains real-time performance through optimized calculations
• Includes FPS monitoring for performance assessment

Applications

1. Driver Monitoring Systems

• Drowsiness detection through blink rate analysis
• Distraction detection via head pose tracking
• Real-time alertness monitoring

2. Human-Computer Interaction

• Gaze-based interface control
• Attention tracking in virtual environments
• Accessibility solutions for mobility-impaired users

3. Healthcare and Research

• Patient monitoring systems
• Attention deficit disorder assessment
• Visual behavior studies

4. Educational Technology

• Student attention monitoring
• Engagement analysis in remote learning
• Adaptive learning systems

Future Improvements

1. Integration of machine learning for improved accuracy
2. Addition of gaze direction estimation
3. Implementation of fatigue detection algorithms
4. Enhanced robustness under varying lighting conditions
5. Multi-face tracking capabilities

Conclusion

This system demonstrates the successful integration of eye tracking and head pose estimation in a real-time application. The combination of MediaPipe's facial landmark detection with custom geometric calculations provides a robust solution for human attention monitoring. The system's versatility makes it suitable for various applications while maintaining real-time performance.

Github: https://github.com/VK-Ant/Computervision_Exploration/tree/main/Head_estimation_faceswap

All my computer vision exploration project: https://github.com/VK-Ant/Computervision_Exploration/tree/main