This project introduces an eye-controlled mouse system that works with a fun bee game. The goal is to create a simple and affordable way for people, especially those with physical disabilities, to interact with computers. The system uses a webcam to track eye movements, moving the mouse pointer and clicking based on where you look. The bee game shows how this system can be used for both practical tasks and games.
Computers play a big role in our daily lives, but not everyone can use a mouse or keyboard easily. This is especially true for people with motor disabilities who may face significant challenges using traditional input devices. Eye-controlled systems offer a promising solution, allowing users to interact with a computer simply by moving their eyes. However, many of these systems are either expensive or too complicated for everyday use.
This project focuses on creating a low-cost, user-friendly eye-controlled mouse system using widely available hardware like a webcam and free, open-source software. The system is designed to be accessible to everyone and can be used for both serious tasks and entertainment. To demonstrate its potential, we included a simple bee game where players control a bee by moving their eyes and blinking to collect nectar. This fun addition shows that the system can be applied in creative and engaging ways.
By combining accessibility with ease of use, this project aims to make technology more inclusive for people with disabilities while also inspiring new uses for eye-tracking technology.
The methodology involves integrating accessible hardware with powerful software tools to create a reliable and intuitive system.
The system’s hardware consists of a standard webcam to capture the user’s facial movements and a computer to process the data. The software framework includes three primary libraries: OpenCV for image processing, MediaPipe for detecting facial landmarks, and PyAutoGUI for simulating mouse actions.
The workflow begins with the webcam capturing real-time video of the user. MediaPipe then processes this video to identify facial landmarks, focusing on the eyes and blinks. The system calculates the user’s gaze position and translates it into mouse pointer movements. Blinks are detected as mouse clicks. These functionalities are further applied in the bee game, where users control the bee’s movements and actions, such as collecting nectar.
This combination of hardware and software ensures affordability, ease of use, and wide applicability across different settings.
The system is implemented using Python with the following tools:
The webcam captures video frames, which are processed to extract RGB images for Mediapipe. The facial landmarks are scaled to match screen dimensions, enabling cursor movement. Intentional blinks are detected using the pre-defined thresholds and trigger mouse clicks.
The system was evaluated through a series of experiments to measure its performance, usability, and overall effectiveness.
Experiment Design
The experiments focused on three primary metrics: accuracy, response time, and user feedback. Accuracy measured how well the system tracked eye movements, while response time evaluated the speed of pointer movement and blink detection. Feedback was collected from both able-bodied participants and individuals with motor impairments to assess the system’s accessibility and ease of use.
Testing Environment
All experiments were conducted indoors under consistent lighting conditions. Participants used a basic webcam and standard computer setup to perform tasks such as navigating the pointer and playing the bee game. This setup ensured a realistic evaluation of the system’s performance in everyday scenarios.
The Eye-Controlled Mouse System successfully demonstrated its capability to provide precise and responsive cursor control using eye movements. The integration of Mediapipe's Face Mesh for facial landmark detection allowed the system to track eye positions with high accuracy. Cursor movements were smooth, with minimal latency, as the facial landmarks were normalized and mapped to screen coordinates effectively.
The blink detection mechanism, which calculated the vertical distance between two specific eye landmarks (145 and 159), proved reliable in distinguishing natural blinks from intentional commands. This enabled the system to simulate mouse clicks seamlessly. Users could perform click operations with intentional eye blinks, offering a hands-free and efficient alternative to traditional mouse usage.
Furthermore, the system was tested across different screen resolutions and sizes, adapting well to various configurations without compromising performance. The visual feedback provided on the screen, where key landmarks were highlighted in real time, ensured that users could adjust their movements for optimal control. The system's robustness and adaptability in real-time scenarios highlight its potential as a practical and accessible tool for hands-free interaction.
The proposed system has applications in:
This project successfully introduces a practical and affordable eye-controlled mouse system, bridging the gap between accessibility and technological innovation. The addition of the bee game underscores the system’s potential for entertainment and creative applications, making it a versatile tool for various user needs.
By combining cost-effectiveness with user-friendly design, the system offers a promising solution for individuals with disabilities, enabling them to interact with computers more intuitively. Future enhancements, such as improved calibration and lighting adaptability, could further expand its functionality, opening avenues for gaming, virtual reality, and beyond.
This work highlights the transformative potential of eye-tracking technology in enhancing inclusivity and interaction in computing.
Future enhancements include improving the system's accuracy by incorporating machine learning models for advanced gesture recognition. Expanding functionality to include additional mouse actions, such as scrolling, dragging, and zooming, is also planned. Addressing performance under extreme lighting conditions and incorporating multi-user calibration profiles are additional goals for future iterations.
Kazemi, V., & Sullivan, J. (2014). One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1867–1874. https://doi.org/10.1109/CVPR.2014.241
Deng, J., Guo, J., & Zafeiriou, S. (2019). ArcFace: Additive angular margin loss for deep face recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4690–4699. https://doi.org/10.1109/CVPR.2019.00482
Tulyakov, S., Seferbekov, S., Aliev, K., & Kasimov, A. (2018). Real-time eye blink detection using facial landmarks. Proceedings of the International Conference on Digital Image Computing: Techniques and Applications (DICTA), 1–7. https://doi.org/10.1109/DICTA.2018.8615798
Gupta, S., & Morris, D. (2016). Gesture-based interaction and eye tracking for assistive technology. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 24(4), 453–465. https://doi.org/10.1109/TNSRE.2015.2450299
Zhang, X., Sugano, Y., Fritz, M., & Bulling, A. (2017). It's written all over your face: Full-face appearance-based gaze estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2291–2300. https://doi.org/10.1109/CVPR.2017.246
Mehta, D., Sotnychenko, O., Mueller, F., et al. (2017). Monocular 3D human pose estimation in the wild using improved CNN supervision. Proceedings of the International Conference on 3D Vision (3DV), 506–516. https://doi.org/10.1109/3DV.2017.00065
D’Apuzzo, N. (2006). Photogrammetric measurement of human faces and body parts for the customization of products in the apparel and footwear industry. Journal of Applied Optics, 45(2), 203–209. https://doi.org/10.1117/1.2117169
Shi, Y., He, D., & Xu, Z. (2020). Eye tracking technology for human-computer interaction: Applications, challenges, and future directions. Sensors, 20(3), 1–25. https://doi.org/10.3390/s20030761
Pavel, M., & Epelboim, J. (1992). Towards predictive models of eye movement control in human-computer interaction. Proceedings of the ACM Symposium on Eye Tracking Research & Applications (ETRA), 55–60. https://doi.org/10.1145/130255.130257
Google Research. (2021). Mediapipe: A framework for building multimodal applied machine learning pipelines. https://mediapipe.dev