# Abstract PixEagle 2.0 is an all-in-one image processing, following, and tracking solution meticulously crafted for the PX4 ecosystem, with potential scalability to ArduPilot. Leveraging robust technologies such as MAVSDK Python, OpenCV, and optional YOLO, PixEagle delivers precise object tracking and autonomous navigation capabilities. The system's modular and extensible architecture empowers users to integrate custom tracking, detection, and segmentation algorithms seamlessly. A modern web-based React Ground Control Station (GCS), powered by FastAPI, facilitates real-time monitoring and control, enhancing user experience. PixEagle 2.0 boasts advanced tracker robustness, multiple following modes, and seamless integration with companion computers like the Raspberry Pi 5, positioning it as a transformative tool in aerial robotics applications spanning surveillance, agriculture, and beyond. Comprehensive documentation is available on the PixEagle GitHub Repository, and demonstration videos can be accessed through the PixEagle YouTube Playlist. # Introduction Background Drone tracking and following systems are pivotal in a multitude of modern applications, including surveillance, agriculture, autonomous deliveries, and industrial inspections. These systems enable drones to autonomously navigate and monitor specific targets or areas, significantly enhancing operational efficiency and reducing the need for constant human oversight. Problem Statement Existing drone tracking solutions often encounter challenges related to reliability, adaptability, and user-friendliness. Many systems lack the necessary modularity and extensibility, making it difficult to integrate custom algorithms or adapt to varying mission requirements. Additionally, comprehensive real-time monitoring and control interfaces are frequently absent, hindering operational efficiency and user interaction. Solution Overview PixEagle 2.0 addresses these challenges by offering a highly modular and extensible drone tracking and following solution. Designed to seamlessly integrate with the PX4 ecosystem and potentially expand to ArduPilot, PixEagle 2.0 supports real-time image processing and autonomous navigation. The inclusion of a web-based React GCS, powered by FastAPI, provides users with an intuitive interface for real-time monitoring and control. Advanced features such as enhanced tracker robustness and multiple following modes ensure reliable and versatile performance across diverse applications. # Related work Drone tracking and following systems have evolved significantly over recent years, with numerous solutions leveraging computer vision and AI to enhance autonomous capabilities. Notable among these are systems utilizing traditional tracking algorithms like CSRT and modern deep learning-based approaches such as YOLO for object detection. Traditional Tracking Algorithms Algorithms like CSRT (Discriminative Correlation Filter with Channel and Spatial Reliability) have been widely adopted for their balance between accuracy and computational efficiency. However, they often face limitations in dynamic environments where objects undergo rapid changes in appearance or occlusion. Deep Learning-Based Approaches Deep learning models, particularly YOLO (You Only Look Once), have revolutionized object detection with their ability to accurately identify and localize objects in real-time. When integrated with tracking methodologies like Kalman Filters and SORT (Simple Online and Realtime Tracking), these models enhance the robustness and reliability of drone tracking systems. Modular and Extensible Systems Recent advancements emphasize the importance of modularity and extensibility, allowing users to customize and extend system functionalities without overhauling core components. Systems that support the integration of custom algorithms and provide flexible interfaces for real-time monitoring and control are gaining traction for their adaptability to various mission requirements. PixEagle 2.0 builds upon these foundations by combining traditional and deep learning-based tracking algorithms within a modular architecture, offering enhanced flexibility and robustness compared to existing solutions. # Methodology System Architecture PixEagle 2.0's architecture integrates the Pixhawk flight controller with companion computers such as the Raspberry Pi 5, utilizing MAVLink for seamless communication. The system comprises several key components: MAVSDK Python: Facilitates MAVLink communication between the drone and PixEagle. OpenCV: Handles real-time image processing tasks. YOLO (You Only Look Once): Provides smart auto-detection capabilities for object tracking. React & FastAPI: Power the web-based Ground Control Station (GCS) for real-time monitoring and control. MAVLink2REST: Bridges MAVLink communication with RESTful APIs for enhanced integration. Hardware Components Pixhawk Flight Controller: Manages flight operations and communication protocols. Raspberry Pi 5: Serves as the companion computer, handling image processing, streaming, and system control. CSI Cameras: Capture high-resolution video feeds for real-time processing and tracking. Software Stack MAVSDK Python: Facilitates seamless MAVLink communication. OpenCV: Powers image processing and computer vision tasks. YOLO: Enables real-time object detection and tracking. FastAPI: Handles backend server operations for the GCS. React: Provides a responsive and intuitive frontend for the GCS interface. Tracking Algorithms PixEagle 2.0 employs a combination of advanced algorithms to ensure accurate and reliable tracking: Modified CSRT: Enhances object detection accuracy by integrating particle filters for improved robustness. Particle Filters: Improve tracking reliability in dynamic and noisy environments. Template Matching: Facilitates precise object redetection when initial tracking fails. Kalman Filters: Enable accurate object estimation and prediction for smooth tracking. Modularity and Extensibility Designed with a fully modular architecture, PixEagle 2.0 allows users to integrate new tracking, detection, and segmentation algorithms without modifying existing system components. This flexibility encourages continuous improvement and adaptation to evolving mission requirements. # Experiments Setup Initial experiments were conducted using a Raspberry Pi 5 as the companion computer, integrated with a Pixhawk flight controller and CSI cameras. The system was configured to handle real-time image processing, streaming, and control operations, interfacing with the PX4 ecosystem via MAVLink. Tracking Modes PixEagle 2.0 was tested across various tracking modes to evaluate its versatility and robustness: Top-Down Follow Mode: Maintains a consistent altitude and position relative to the target from a top-down perspective. Forward-Looking Follow Mode: Adapts to different mission requirements by adjusting the drone's trajectory based on the target's movement. Chase Mode: Enables dynamic pursuit of fast-moving targets, simulating scenarios such as "dogfight" engagements. Performance Metrics Key performance metrics assessed during the experiments include: Tracking Accuracy: Measured the precision of object localization and consistency in following the target. Latency: Evaluated the system's responsiveness in processing and reacting to target movements. Reliability: Assessed the system's ability to maintain tracking under varying environmental conditions and target dynamics. Testing Environment Experiments were conducted in controlled indoor environments with varying levels of lighting and background complexity to simulate real-world conditions. The system's ability to handle occlusions, rapid target movements, and changes in target appearance was rigorously tested. # Results Tracking Accuracy PixEagle 2.0 demonstrated high tracking accuracy across all tested modes. The integration of modified CSRT with particle filters significantly reduced tracking errors, maintaining precise localization of targets even during rapid movements and occlusions. Latency The system exhibited low latency in processing and responding to target movements, ensuring smooth and responsive drone behavior. Real-time streaming and control operations were maintained consistently, with negligible delays observed during communication between the Raspberry Pi 5 and Pixhawk flight controller. Reliability PixEagle 2.0 maintained reliable tracking under diverse environmental conditions. The implementation of redetection and tracker failure compensation mechanisms ensured continuous operation, even when initial tracking attempts were disrupted. The system's ability to adaptively learn and update features during missions further enhanced its robustness. User Interface Performance The web-based React GCS provided an intuitive and responsive interface, enabling real-time monitoring and control with minimal latency. Features such as drag-and-select target tracking and real-time data visualization contributed to an enhanced user experience. Comparative Analysis Compared to existing solutions, PixEagle 2.0's modular architecture and combination of traditional and deep learning-based tracking algorithms offered superior adaptability and reliability. The ability to integrate custom algorithms without altering core components positioned PixEagle 2.0 as a more flexible and robust solution in the drone tracking landscape. # Discussion Strengths PixEagle 2.0's primary strengths lie in its modular and extensible architecture, allowing seamless integration of custom tracking and detection algorithms. The combination of modified CSRT, particle filters, YOLO, and Kalman filters ensures high tracking accuracy and robustness in dynamic environments. The real-time streaming and responsive GCS interface further enhance operational efficiency and user experience. Challenges During initial testing, challenges were encountered in optimizing the system for real-time performance on the Raspberry Pi 5. Ensuring low latency while handling multiple processing tasks required careful optimization of the software stack and efficient utilization of hardware resources. Additionally, fine-tuning the tracking algorithms to handle diverse target behaviors and environmental conditions was a complex task. Future Improvements Future enhancements will focus on further improving tracking and detection robustness, increasing system autonomy, and conducting extensive real-world testing to validate and refine performance. Integrating multi-sensor data fusion will enhance tracking accuracy and reliability, while expanding compatibility with additional drone platforms will broaden PixEagle 2.0's applicability. Community and Collaboration PixEagle 2.0's open-source nature fosters community collaboration and continuous improvement. Encouraging contributions and feedback from users and developers will drive innovation and adaptability, ensuring PixEagle remains at the forefront of drone tracking and following solutions. # Conclusion PixEagle 2.0 emerges as a comprehensive and advanced drone tracking and following solution, combining a modular design with robust tracking algorithms and a user-friendly interface. Its seamless integration with the PX4 ecosystem and companion computers like the Raspberry Pi 5, coupled with versatile tracking modes and real-time monitoring capabilities, positions PixEagle 2.0 as a transformative tool in aerial robotics. By addressing key challenges in reliability, adaptability, and user experience, PixEagle 2.0 sets a new standard for drone tracking systems, with significant applications across surveillance, agriculture, autonomous deliveries, and industrial inspections. Future developments will continue to enhance its robustness and scalability, ensuring PixEagle 2.0 remains adaptable to evolving mission requirements and technological advancements. # References 1. [PixEagle GitHub Repository](https://github.com/alireza787b/PixEagle) 2. [PixEagle YouTube Playlist](https://www.youtube.com/watch?v=nMThQLC7nBg&list=PLVZvZdBQdm_4oain9--ClKioiZrq64-Ky&index=1&t=0s) 3. [PX4XPlane Repository](https://github.com/alireza787b/px4xplane) 4. [Follow Mode X-Plane Guide](https://github.com/alireza787b/PixEagle/blob/main/docs/XPlane_SITL_Instructions.md) 5. [PixEagle Series on YouTube](https://www.youtube.com/playlist?list=PLVZvZdBQdm_4oain9--ClKioiZrq64-Ky) # Acknowledgements I would like to extend my gratitude to the open-source community for their invaluable contributions, which have significantly influenced the development of PixEagle 2.0. Special thanks to the contributors on GitHub for their support and feedback. Additionally, appreciation is due to the team at Ready Tensor for providing the platform to showcase innovative projects like PixEagle. # Appendix ### *Installation Guide* **Prerequisites:** - **Operating System:** Linux (recommended) or Windows (for X-Plane SITL tests). - **Python:** Version 3.9 or higher with virtual environment setup. - **Additional Tools:** tmux, lsof for system management. **Installation Steps:** 1. Clone the PixEagle repository: ```bash git clone https://github.com/alireza787b/PixEagle.git cd PixEagle ``` 2. Initialize the project: ```bash bash init_pixeagle.sh ``` 3. Install Node.js and npm for the GCS dashboard as per the [official Node.js installation guide](https://nodejs.org/). 4. Configure application and dashboard settings via YAML and environment variable files: ```bash nano configs/config.yaml nano dashboard/.env ``` 5. Run PixEagle: ```bash bash run_pixeagle.sh ``` 6. Access the Dashboard: Open your browser and navigate to `http://127.0.0.1:3001`. For remote access, replace `127.0.0.1` with your machine's IP address. ### *Key Bindings* While in the video window, use the following keys for control: - `t`: Select target - `c`: Cancel selection - `y`: YOLO detection - `f`: Start following (offboard mode) - `d`: Try to re-detect target - `q`: Quit ### *MAVLink Integration* Ensure MAVLink communication is properly set up by following the [MAVLink Router installation guide](https://github.com/alireza787b/mavlink-anywhere). Run MAVLink2REST: ```bash bash ~/PixEagle/src/tools/mavlink2rest/run_mavlink2rest.sh ``` - **Detailed Setup and Usage Guides:** Available in the [PixEagle GitHub Repository](https://github.com/alireza787b/PixEagle). - **Community Engagement:** Share your thoughts and suggestions in the issues section of the [GitHub repository](https://github.com/alireza787b/PixEagle/issues) and join the growing community to contribute and collaborate.