1. Problem statement

The goal of this project is to implement Real-time video filters on an Nvidia Jetson Nano embedded device. The task involves working with various image processing methods and features specifically to achieve individual tasks.

The following video filters were aimed to achieve on our real-time interface:

1. Background Blur

2. Background replacement

3. Face Distortion

4. Face Filter/ Replacement

5. Creative Filter

Through this, we study various aspects related to embedded systems and image processing. During the implementation phase we tried various operations applied to frames in a video to achieve specific visual effects to enhance certain characteristics.

2. Introduction (Hardware and Software Overview)

2.1. Nvidia Jetson Nano - Architecture Details

This project is demonstrated with a Jetson Kit containing all the hardware elements required. We flash a pre-built Nano Image with jetPack OS which is a customized OS containing all the important software libraries.

2.1.1. GPU Architecture

The Jetson Nano is equipped with a powerful Maxwell architecture GPU, featuring 128 CUDA cores. CUDA (Compute Unified Device Architecture) is a parallel computing platform developed by Nvidia, and the inclusion of CUDA cores in the GPU enables parallel processing, a key aspect for handling computationally intensive tasks.

2.1.2. Memory Architecture

The Jetson Nano is equipped with 4GB of LPDDR4 RAM, providing ample memory for data storage and retrieval during processing. The memory architecture is designed to support the high-throughput demands of parallel processing, ensuring efficient handling of large datasets

2.1.3. Pre-built Image

The Pre-built Jet-Pack OS Image that we flashed on an SD card using Balena-Etcher (find here https://etcher.balena.io/) and used the baseline image for our DIY implementation which had environment setup with python 3.6.9, OpenCV 4.5.3 enabled with Cuda, Torch libraries for YOLO-V5 and example scripts.

2.2. Additional Libraries

2.2.1. OpenCV with CUDA Enabled

OpenCV, a widely used computer vision library, is configured with CUDA support on the Nvidia Jetson Nano. CUDA is a parallel computing platform and application programming interface model created by NVidia, and when integrated with OpenCV, it enables accelerated processing on the GPU. This means that image and video processing tasks can leverage the parallel processing capabilities of the Nvidia GPU, resulting in faster and more efficient computations. We implemented the DNN module for our use cases for real time processing

2.2.2. Jetson_Inference

The jetson_inference library is specifically designed for Nvidia Jetson platforms, providing a collection of pre-trained deep learning models and utilities. This library facilitates tasks such as image classification, object detection, and segmentation. By leveraging jetson_inference, developers can streamline the implementation of advanced AI functionalities, saving time and effort in model development

2.3. GStreamer Integration

GStreamer is a powerful and flexible open-source multimedia framework that facilitates the construction of pipelines for handling multimedia data, including audio and video. In this project, GStreamer is utilized to enhance the video processing workflow on the NVidia Jetson Nano.

Let's break down the pipeline string:

1. nvarguscamerasrc: This is the source element for NVidia cameras. It captures video frames from NVidia camera devices.
2. Video/ X-raw(memory NVMM): Specifies the format of the raw video frames in memory as NVMM (Nvidia Memory Management). This format is optimized for NVidia hardware.
3. width, height, format, framerate: Sets the width, height, pixel format (NV12), and framerate of the captured video frames. The values are provided as placeholders to be replaced by actual values during runtime.
4. nvvidconv flip-method: Applies video conversion and a flip operation. flip-method determines the flip operation to be applied to the video frames.
5. Video/X-raw, width, height, format: Specifies the format for the video frames after the flip operation. In this case, it's set to BGRx.
6. videoconvert: Converts the video frames to a different format if necessary.
7. Video/X-raw, format: Sets the final format for the video frames as BGR.
8. Appsink drop: Setting to True Configures the sink element to be an appsink, which allows the application to receive the processed video frames. The drop=True parameter indicates that if the buffer cannot be pushed to the appsink, it should be dropped.

3. Algorithms

3.1. Person Segmentation Mask

3.1.1. Mask R-CNN Implementation

Person segmentation is a critical component within the system, and for this purpose, we employ the OpenCV DNN module in conjunction with the TensorFlow-based Mask R-CNN model. This model, pretrained on the COCO dataset, follows a two-stage architecture extending the Faster R-CNN framework. It predicts both bounding boxes and pixel-level segmentation masks simultaneously. The Inception v2 variant, known as Inception ResNet V2, enhances the architecture by incorporating features from both Inception and ResNet. The implementation utilizes the OpenCV DNN module, enabling seamless integration and efficient deployment.

3.1.2. SegNet Jetson_Inference Integration

As part of our Person segmentation strategy, we explored the implementation of SegNet, a deep learning architecture characterized by a Convolutional Encoder-Decoder structure. SegNet assigns a class label to each pixel in the input image, making it a powerful tool for semantic segmentation. For our application, we utilized the fcn-resnet18-voc-320x320 SegNet model with pretrained weights trained on the Pascal VOC dataset, encompassing 21 classes. The deployment of SegNet is optimized using the Jetson Inference framework, incorporating TensorRT optimization for Jetson devices. This ensures real-time inference capabilities, a crucial aspect for dynamic video streams.

3.1.3. PP-Human Segmentation Model

Central to our system is the PP-Human Segmentation model, a part of the OpenCV DNN module. This model is specifically chosen for its effectiveness in real-time scenarios, accurately delineating individuals from complex backgrounds. Leveraging a DNN-based approach, we efficiently use pre-trained neural network models for rapid and accurate person segmentation within video streams. The PP-Human Segmentation model serves as the foundational step for subsequent background and foreground segmentation. By quantizing the model to ONNX runtime, we achieve a high accuracy of 0.9581 and mIoU of 0.8996, ensuring reliable performance in our system.

3.2. Face Detection

3.2.1. Haarcascade Frontal Face

Object Detection using Haar feature-based cascade classifiers, introduced by Paul Viola and Michael Jones in their 2001 paper, Rapid Object Detection using a Boosted Cascade of Simple Features, is a robust machine learning approach. The method involves training a cascade function with a substantial dataset of positive and negative samples, where Haar features, akin to convolutional kernels, are extracted as the initial step. These Haar features represent single values obtained by subtracting the sum of pixels under a white rectangle from the sum of pixels under a black rectangle. The algorithm explores all possible sizes and locations for each kernel, and integral images are introduced to streamline the process of finding the sum of pixels under different rectangles, ensuring computational efficiency even with a large number of pixels.

With over 160,000 features generated, Adaboost is employed to determine the significance of each feature, addressing the challenge of selecting the most relevant ones. To optimize computational load, a Cascade of Classifiers is introduced, organizing features into different stages. This staged approach allows for the progressive detection of objects, making the system highly efficient, especially for real-time applications. The use of Haar features and integral images contributes to the overall speed and accuracy of the face detection process, making this approach particularly well-suited for effective object detection, especially in scenarios like face detection.

4. Implementation

4.1. BackGround Blur

4.1.1. Color Space Conversion

The input frame is initially converted from the BGR color space to the RGB color space using cv2.cvtColor. This is a common preprocessing step for input the frame image read from the camera to the model.

4.1.2. Image Resizing

The resized image (image) is obtained by resizing the RGB frame to a fixed size of 192x192 pixels using cv2.resize. This step standardizes the input size for further processing.

4.1.3. Inference

The resized image is then fed into a pre-trained PP-Human Segmentation model for inference, performing image segmentation to identify region of interest.

4.1.4. Thresholding

1. The thresholding step using Otsu's method is common for both background replacement and blur.(cv2.threshold(result,100,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)) with (binary_mask) and datatype is converted for faster calculations (astype(np.uint8))
2. Finally using the threshold as binary mask and merging them into 3 dimensional shape.
3. (cv2.merge([binary_mask, binary_mask, binary_mask]))

4.1.5. Gaussian Blur

Prior to applying the thresholded mask, the original frame is subjected to Gaussian blur (cv2.GaussianBlur) using a kernel size of (91, 91). This blurs the entire frame (blur_input), but the effect will be applied selectively based on the binary mask.

4.1.6. Background Blur

The original frame is bitwise ANDed with the binary mask.(cv2.bitwise_and(frame,mask)). A Gaussian-blurred version of the input frame is then created, and the complement of the binary mask is used to extract the regions not covered by the mask (cv2.bitwise_and(blur_input, cv2.bitwise_not(mask))) .The final blurred frame is obtained by combining the two using bitwise AND and ADD operations (cv2.add(result_blur_frame, RS2_blur))

4.2. Background Replacement

Similarly for background Replacement the First 4 steps are repeated the same from Background Blur. Instead of Gaussian Blur, We read the background image using (cv2.imread("beach.jpg")) and use this Image (img_bcg) for our background replacement task where we get the mask from PP-Human Segmentation model inference.

The original frame is then bitwise ANDed with the binary mask(cv2.bitwise_and(frame,mask)), effectively isolating the identified regions. The replacement background (img_bcg) is resized to match the dimensions of the original frame, and the bitwise NOT operation on the mask is used to extract the complementary region(cv2.bitwise_and(img_bcg, cv2.bitwise_not(mask))). The original frame and replacement background are then combined using the bitwise AND and ADD operations, resulting in a new frame with the background replaced (cv2.add(result_frame, RS2))

4.3. Face Distortion

4.3.1. Radial Distortion Function

• Parameters:
  • image: The input image to which radial distortion will be applied.
  • strength: The strength of the radial distortion. A higher value increases the distortion effect.
  • center: The center of the radial distortion. If not provided, it is dynamically calculated based on the image size and a specified radius.
  • radius: The radius of the distortion, determining how far the distortion effect extends from the center.
  • speed: The speed factor that influences the movement of the center over time.
• Center Calculation:
  • If the center is not specified, it is calculated based on the image dimensions and a dynamic position influenced by time (cv2.getTickCount() and cv2.getTickFrequency()). This creates a dynamic center that moves over time in a circular pattern.
• Meshgrid and Distances:
  • Meshgrid coordinates (x, y) are created, and distances (dx and dy) from the calculated center are computed.
• Radial Distance Calculation:
  • The radial distance (r) from the center is computed using the distances in the x and y directions.
• Distortion Factor:
  • A distortion factor (factor) is computed based on the radial distance. The distortion increases with the square of the radial distance. Pixels outside a specified radius remain unaffected.
• Coordinate Mapping:
  • The x and y coordinates are mapped based on the distortion factor, creating new coordinates (map_x and map_y) for remapping the image.
• Remapping:
  • The cv2.remap function is used to apply the distortion to the image using the computed coordinate mappings.

4.3.2. Swirl Distortion Function

• Parameters
  • image: The input image on which the swirl distortion will be applied.
  • center: The center point (x, y) around which the swirl distortion will occur. If not provided, the default center is set to the center of the image.
  • strength: A parameter controlling the overall strength of the swirl distortion. Higher values result in a more pronounced distortion effect. The default value is set to 1.0.
  • radius: The radial distance from the center up to which the swirl distortion will be applied. Pixels beyond this radius remain unaffected. The default radius is set to 100.
• Center Calculation:
  • If the center is not specified, a dynamic center is calculated based on the image dimensions and a circular pattern influenced by time (cv2.getTickCount() and cv2.getTickFrequency()).
• Meshgrid and Distances:
  • Meshgrid coordinates (x, y) are created.
  • Distances (dx and dy) from the calculated center are computed.
• Radial Distance Calculation:
  • The radial distance (r) from the center is computed using the distances in the x and y directions.
• Distortion Factor:
  • A distortion factor (factor) is computed based on the radial distance. The distortion increases with the square of the radial distance. Pixels outside a specified radius remain unaffected.
• Coordinate Mapping:
  • The x and y coordinates are mapped based on the distortion factor, creating new coordinates (map_x and map_y) for remapping the image.
• Remapping:
  • The cv2.remap function is employed to apply the dynamic swirl distortion to the image using the computed coordinate mappings.

4.4. Face Replacement

The transparent Overlay function takes a source image (src), an overlay image (overlay), a position tuple (pos), and a scale factor (scale). It resizes the overlay image, extracts the region of interest (ROI) from the source image, and blends the two images using the alpha channel of the overlay for transparency.

The second function, face_blur, is designed specifically for face blurring. Given the coordinates (x, y, w, h) of a detected face and the original frame (frame), it extracts the region around the face, enlarges it slightly, and applies a transparent overlay of an Iron Man image (ironman). The enlargement is done to ensure that the overlay covers the entire face region adequately.

Here's a brief breakdown of the face_blur function:

4.4.1. Region of Interest (ROI):

The coordinates of the face (x, y, w, h) are used to define a region around the detected face in the original frame (frame).

4.4.2. Overlay Image:

The Ironman image (ironman) is resized to match the dimensions of the enlarged face region.

The Batman image (batman) is resized to match the dimensions of the enlarged face region.

4.4.3. Transparent Overlay:

The transparent Overlay function is called with the face region, IronMan overlay or Batman overlay and the position tuple. This results in blending the IronMan/ Batman image onto the face region with transparency.

4.5. Creative Filter

4.5.1. Frame Cartoon Function

This function aims to create a cartoon-style representation of the input frame. Here are the steps involved:

• Bilateral Filtering: cv2.bilateralFilter is applied to the original frame (frame) to smooth colors while preserving edges. This results in a variable named color.
• Grayscale and Median Blur:
  • The original frame is converted to grayscale, and cv2.medianBlur is used to apply median filtering for further smoothing.
  • cv2.adaptiveThreshold is then applied to create an adaptive binary edge map (edges) from the blurred grayscale image.
• Edge Detection: Adaptive thresholding (cv2.adaptiveThreshold) is used on the blurred grayscale image to identify edges. This creates a binary image highlighting edges.
• Edge to RGB: The binary edge image is converted to an RGB format (cv2.cvtColor(edges, cv2.COLOR_GRAY2RGB)).
• Cartoon Effect: cv2.bitwise_and is used to combine the bilateral-filtered color image (color) with the RGB edge map (frame_edge). This creates a cartoon-like effect where the edges are emphasized.
• Final Blurring: Another bilateral filter (cv2.bilateralFilter) is applied to further smooth the cartoonized image (cartoon), resulting in the final output.

4.5.2. Mask Function

This function appears to generate a mask based on a pre-trained model. Here are the steps:

• Model Inference:
  • The input frame is preprocessed, including resizing, and passed through a pre-trained model (model.infer). The model is expected to generate a result that likely represents some kind of segmentation mask.
  • This segmentation mask is then passed on to the threshold for a binary mask.
• Thresholding: The result from the model is thresholded to create a binary mask (cv2.threshold). This mask is then converted to an RGB format (cv2.merge([binary_mask, binary_mask, binary_mask])).

4.5.3. Main Part

• Function Calls: The frame_carton function is called to generate a cartoon-style frame (frame_cart), and the Mask function is called to obtain a segmentation mask (mask).
• Bitwise AND and Addition: The cartoon-style frame and the segmentation mask are combined using bitwise AND and addition operations (cv2.bitwise_and and cv2.add). This operation replaces the cartoon-style pixels with the original pixels where the mask is active, creating a final result (result_frame).

4.6. Stream-Lit App

The program captures video using a GStreamer pipeline, applies different effects based on user-selected options, and displays the processed video on Streamlit. This Streamlit-based Python script utilizes computer vision techniques for real-time video processing. It offers various functionalities from the above functions, including face distortion (radial and swirl), face masking (Iron Man and Batman), background blur, background replacement, and a creative cartoon filter.Users can choose these functionalities from a Streamlit sidebar, creating an interactive experience.

5. Testing and Results

5.1. Person Segmentation Methodologies

We conducted extensive testing using various person segmentation methodologies to evaluate their performance on the NVidia Jetson Nano. The tested methodologies include:

5.1.1. OpenCV Dnn Module for TensorFlow-based MaskRCNN

• Results: The methodology yielded satisfactory segmentation outputs; however, the real-time segmentation performance on the Jetson Nano is suboptimal.
• Metrics:
  • Accuracy:
  • FPS: 3

5.1.2. SegNet from Jetson Inference

• Results: SegNet demonstrated real-time segmentation capabilities on the Jetson Nano. However, the quality of the segmentation mask did not meet the desired accuracy levels.
• Metrics:
  • Accuracy: 85.9%
  • FPS: 45

5.1.3. OpenCV Dnn Module for ONNX - PP-Human Segmentation

• Results: PP-Human Segmentation using the ONNX module provided satisfactory segmentation outputs in real-time on the Jetson Nano.
• Metrics:
  • Accuracy: 0.9581
  • FPS: 28-30

6. Discussion

6.1. Comparative Analysis of Implementation

• Speed Comparison:
  • OpenCV Dnn (TensorFlow-based MaskRCNN) demonstrated slower real-time segmentation on the Jetson Nano.
  • Two-stage detectors like Mask R-CNN tend to be computationally more demanding than one-stage detectors, they often excel in tasks where precise localization but are computationally expensive
  • SegNet achieved real-time performance but with compromised segmentation quality.
  • PP-Human Segmentation using ONNX outperformed in terms of real-time speed and segmentation accuracy.
• Quality Comparison:
  • Visual inspections reveal that PP-Human Segmentation produced segmentation masks closer to the expected quality compared to the other methodologies.

6.2. Challenges and Observations

• Real-time Performance:
  • OpenCV Dnn (TensorFlow-based MaskRCNN) faced challenges in achieving real-time performance on the Jetson Nano, highlighting the need for optimization.
  • SegNet achieved real-time performance but struggled with the precision of the segmentation mask.
  • PP-Human Segmentation demonstrated superior real-time performance and segmentation quality.
• Optimization Needs:
  • Further optimization is required for methodologies that did not meet real-time performance expectations.

7. Suggested Improvements

• We can use OpenCV Dnn Module for face detection for better face detection bounding boxes for faces titled or slightly angled.
• We can Face Mesh or Face landmarks from dlib or mediapipe for applying face replacement/mask for faces tilted totally
• Using Homography Estimation points mapped from the mask to face landmarks can result in a much better face replacement
• Implementing Face Detection and person segmentation on every alternate frame to reduce computation of Inference models which are resource intensive
• Reading the videoCapture using low resolutions like 1280x720, 960x540 for better performance and real time capabilities
• using Frame Queueing such as library called imutils, Camgear which are implementing deque functionality to reduce the reading of the frames, as OpenCV takes a lot of time for reading frames from the videoCapture object

8. References

1. https://github.com/opencv/opencv_zoo
2. https://docs.opencv.org/4.x/d2/d99/tutorial_js_face_detection.html
3. https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle
4. https://github.com/dusty-nv/jetson-inference
5.https://docs.opencv.org/3.4/d6/d0f/group__dnn.html
6.https://github.com/spmallick/learnopencv/tree/7285b1f8f663edb21c0dc54c47b2dc307c97ba38/Mask-RCNN