![{7EE9AF9F-8F1D-44EA-B61C-9580F4B6FAA9}](https://github.com/user-attachments/assets/8f77a880-1e40-4052-95bc-3c70b359924f)

Introduction

In the field of autonomous systems, achieving reliable navigation through complex environments is a challenging task. In this project, we focus on designing and implementing an autonomous navigation and track-following system using computer vision techniques and machine learning algorithms. The system employs a Pixy Camera for real-time image capture and processing, which feeds into a computer vision pipeline built on ROS2 (Robot Operating System 2). The project aims to demonstrate the integration of vision-based control with dynamic steering mechanisms for autonomous cars.

This system’s architecture can be divided into two primary layers: the CV and ML layer for image processing and object detection, and the Line Following layer for control of the vehicle’s movement based on visual feedback.

Project Structure

The project is organized into several critical layers, each focusing on a specific functionality:

CV and ML Layer:

Responsible for finding major obstacles/directions/traffic signs and signals.

Contribute to finding a safe global route
Line Following Layer:

Find the bounds for the immediate road.

Contributes to finding a safe local route

The integration of these components ensures that the self-driving car can autonomously follow a track, navigate efficiently, and perform adaptive steering based on visual inputs.

CV and ML Layer

The CV and ML Layer is primarily responsible for image processing, feature extraction, and object recognition using the Pixy Camera to interpret the environment. This layer is key for enabling the system to "see" and respond to the track.

1. Vision Pipeline

At the core of the CV layer is the vision module, which is responsible for processing the raw camera feed from the Pixy camera. The pipeline begins by capturing the camera feed in real-time and converting it into a format suitable for processing by the computer vision algorithms.

Pixy Camera Integration: The Pixy camera, a powerful vision sensor designed for robotics applications, is interfaced with ROS2 to provide high-speed object tracking. It is particularly effective in detecting colored objects, such as the lines on the track.
Image Preprocessing: The raw images captured by the Pixy Camera undergo preprocessing steps to enhance the image quality and extract relevant features. This typically includes:
- Grayscale Conversion: The image is converted to grayscale to simplify processing and reduce computational complexity.
- Thresholding: A binary thresholding technique is applied to highlight the relevant features (e.g., track lines or obstacles) by converting the image into black and white.
- Noise Reduction: Methods like Gaussian blur or median filtering are employed to reduce noise and ensure that the feature extraction is accurate.
Feature Detection: Once the image is preprocessed, we extract key features such as:
- Track Line Detection: Using edge detection algorithms like the Canny Edge Detector or Hough Transform, the system identifies the edges of the track. These edges help determine the curvature and direction of the track, guiding the car's movement.
- Obstacle Detection: In some cases, the system also needs to detect obstacles in the path. This is typically achieved by applying object recognition algorithms to detect features like other vehicles, cones, or any objects that could block the path.

2. Machine Learning (ML) Layer

To enhance the robustness of the vision system, machine learning techniques are employed for adaptive learning and decision-making. The ML models are used to predict the optimal steering angles or vehicle maneuvers based on the visual data.

Supervised Learning: The system is trained using labeled datasets that include images of the track with corresponding steering commands. This enables the model to predict the correct steering direction based on the current visual input.
Decision Models: After processing the image, the system employs a decision-making model (possibly a simple feed-forward neural network) that outputs a steering command to the vehicle. This command adjusts the vehicle's direction in real-time.

Line Following Layer

The Line Following Layer is the second core component of the system, which processes the output from the computer vision layer and converts it into actionable control signals for the vehicle.

1. Line Following Algorithm (Aim Line Follow)

The line-following algorithm is the driving force behind the vehicle’s navigation along the track. The system continuously adjusts the steering based on the detected line to keep the vehicle on course.

Steering Control: The primary task of this layer is to interpret the output from the computer vision system and convert it into motor control signals. The steering angle is dynamically adjusted to ensure the vehicle stays aligned with the track.

The following steps are typically involved:
- Line Centering: The system calculates the offset of the vehicle from the center of the track based on the detected line. The greater the offset, the more aggressive the steering correction.
- PID Control Loop: A PID (Proportional-Integral-Derivative) controller is often used to fine-tune the steering adjustments. This controller helps to smooth out the steering input, reducing oscillations or jerky movements.
- Steering Adjustment: Based on the line’s position in the image, the car adjusts its steering. If the line is drifting to the left or right, the car will adjust its steering to correct for the deviation.

2. Path Prediction and Smooth Steering

To improve the robustness of the line-following system, predictive models are used to forecast the path ahead based on the current steering angle and position of the car on the track. This helps the car to anticipate turns and curves, adjusting the steering smoothly rather than reacting too late.

Integration with ROS2

The system is built on ROS2 (Robot Operating System 2), which facilitates communication between different layers of the system. ROS2 provides a modular framework that allows for the integration of sensor data, control algorithms, and actuator commands.

Node Architecture: The system is broken down into different ROS2 nodes, each responsible for specific tasks, such as image acquisition, processing, and control. These nodes communicate through topics, services, and actions to enable seamless data exchange.
Real-Time Performance: ROS2 offers improved real-time performance compared to its predecessor, ROS1, making it well-suited for real-time control tasks like autonomous navigation and steering adjustments.

![{905CCF0A-3757-4891-A4B2-D73319D2B548}](https://github.com/user-attachments/assets/35a64ac6-7fb8-4460-a671-a0d6d96385e9)

Conclusion

This self-driving car project demonstrates the integration of computer vision and machine learning techniques for autonomous navigation. The use of the Pixy Camera for image capture and the implementation of a ROS2 framework ensures a scalable and real-time system capable of tracking and following a line while adapting to changes in the track. The system can be expanded to include more advanced features, such as obstacle avoidance, path planning, and multi-sensor fusion.