Introduction

This project presents a cutting-edge AI-powered solution to monitor and deter smoking in restricted areas. By leveraging YOLOv8 for real-time smoker detection, facial recognition for individual identification, and email integration for instant notifications, the system ensures prompt and effective enforcement of no-smoking policies. This innovative approach addresses public health and safety concerns, offering a robust tool for security teams, facility managers, and public health authorities. With seamless integration of state-of-the-art technologies, the system sets a new standard in automated surveillance and regulation enforcement

Abstraction

This project addresses the critical challenge of monitoring and enforcing no-smoking policies in restricted areas. By leveraging advanced AI technologies, the system detects individuals smoking in unauthorized zones and integrates with a facial recognition system to identify the employee involved. The system automatically retrieves the detected zone information and the manager's address, sending detailed email notifications to the relevant departments. Each notification includes comprehensive details such as the time of the incident, the employee's name, the restricted area, and the manager's contact information, along with an attached image as evidence.This innovative approach enhances accountability, reduces manual monitoring efforts, and provides robust support for regulatory compliance in sensitive environments.

Project Demo

Methodology

This project employs a combination of advanced AI technologies and custom algorithms to detect smokers in restricted areas, identify individuals through facial recognition, and send real-time email alerts to relevant departments.

1.Smoker Detection with YOLOv8
The system utilizes YOLOv8 (You Only Look Once version 8) for real-time object detection to identify smokers in restricted zones. YOLOv8's deep learning model processes video streams or images to detect smoking behavior with high accuracy and speed.

2. Facial Recognition with FaceNet and MTCNN
Upon detecting a smoker, the system uses MTCNN (Multi-task Cascaded Convolutional Networks) to perform facial detection, ensuring that the correct face is isolated from the frame. Next, FaceNet is used to extract and compare facial features to identify the individual. If the person is recognized, their information (such as name) is retrieved from a database.

3. Vector Database with ChromaDB
ChromaDB is utilized as a vector database to store the facial feature vectors generated by FaceNet. This allows the system to efficiently search for and retrieve recognized individuals' information, including their associated employee details and manager contact information.

4.Email Alerts
Once a smoker is identified, the system sends an automated email alert to the required department. The email includes relevant details, such as the time of the incident, the employee’s name, the detected zone, and an image of the smoker as evidence. Python’s smtplib library is used for seamless email integration and communication.

Code Snippets

This section provides the implementation details for the core components of the project. The code is divided into three main steps: building a facial recognition system, preparing a vector database, and training YOLOv8 for smoker detection. Each step outlines the corresponding functionality and demonstrates how the components work together seamlessly.as below :-

1.Create Facial Recognition System

• Use MTCNN to detect faces in images or video frames.
• Extract facial embeddings using FaceNet to uniquely identify individuals.

2. Prepare Vector Database

• Utilize ChromaDB to store facial embeddings as vectors.

• Implement functions to add, query, and retrieve data for efficient identification.
  3.Train YOLOv8 for Smoker Detection

• Train YOLOv8 on a custom dataset to detect smoking behavior in restricted areas.

• Integrate YOLOv8 with the system for real-time detection and trigger subsequent processes

Step 1: Facial Recognition System

MTCNN (Multi-task Cascaded Convolutional Networks)

• MTCNN is a deep learning-based library for detecting and aligning faces in images or video frames. It consists of three cascaded networks:

1. Proposal Network (P-Net): Generates candidate face regions.
2. Refine Network (R-Net): Eliminates false positives and refines the regions.
3. Output Network (O-Net): Provides the final bounding boxes and landmarks.
   Key Features:

• High accuracy in detecting multiple faces within an image.
• Provides facial landmarks like eyes, nose, and mouth for alignment.
• Lightweight and fast for real-time applications.
  Use in this Project:

MTCNN will detect faces in the input images or video frames, which will then be passed to FaceNet for generating embeddings.

• MTCNN initialization

        self.detector = MTCNN()

• Face Net initialization

        self.faceNetModel = FaceNet()

in this step we need to extract faces from earch face and generate Embbeding as below :-

class FACELOADING:
    def __init__(self, directory):
        self.directory = directory
        self.target_size = (160, 160)
        self.X = []
        self.Y = []
        self.EMBEDDED_X = []  # save face embeddings
        self.detector = MTCNN()
        self.faceNetModel = FaceNet()

    def extract_face(self, filename):
        img = cv2.imread(filename)
        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
        x, y, w, h = self.detector.detect_faces(img)[0]['box']
        face = img[y:y+h, x:x+w]
        face_arr = cv2.resize(face, self.target_size)
        return face_arr.astype('float32') 

    def load_faces(self, folderName):
        FACES = []
        for im_name in os.listdir(folderName):
            try:
                path = os.path.join(folderName, im_name)
                single_face = self.extract_face(path)
                FACES.append(single_face)
            except Exception as e:
                pass
        return FACES

    def load_classes(self):
        for sub_dir in os.listdir(self.directory):
            path = os.path.join(self.directory, sub_dir)
            FACES = self.load_faces(path)
            labels = [sub_dir for _ in range(len(FACES))]
            print(f"Loaded successfully: {len(labels)} faces for class '{sub_dir}'")
            self.X.extend(FACES)
            self.Y.extend(labels)
        return np.asarray(self.X), np.asarray(self.Y)

    def get_embedding(self, face_img):
        face_img = np.expand_dims(face_img, axis=0)
        yhat = self.faceNetModel.embeddings(face_img)
        return yhat[0]  # 512D image (1x1x512)

    def run_embedding(self):
        for img in self.X:
            self.EMBEDDED_X.append(self.get_embedding(img))
        return np.asarray(self.EMBEDDED_X)

    def aggregate_embeddings(self):
        embeddings_dict = {}
        for i, label in enumerate(self.Y):
            if label not in embeddings_dict:
                embeddings_dict[label] = []
            embeddings_dict[label].append(self.EMBEDDED_X[i])

        aggregated_embeddings = {}
        for label, embeddings in embeddings_dict.items():
            print(f'label is '+label)
            aggregated_embeddings[label] = np.mean(embeddings, axis=0)
        return aggregated_embeddings

if __name__ == '__main__':
    directory = "D:/openCV/vision_ahmed/projects/enhancedAtendanceSystemFaceNetfullcodewithSpofing/images"
    face_loader = FACELOADING(directory)
    X, Y = face_loader.load_classes()
    face_embedded = face_loader.run_embedding()
    aggregated_embeddings = face_loader.aggregate_embeddings()

Step 2 Preparing Vector Data base

In this project, ChromaDB is utilized as a vector database to efficiently store and manage facial embeddings. Facial embeddings are generated using FaceNet and represent a unique numerical vector for each individual's face. These embeddings are indexed in ChromaDB, enabling fast and accurate retrieval for facial recognition.

import chromadb
chroma_client = chromadb.PersistentClient(path="D:/openCV/vision_ahmed/projects//database")
#collection = chroma_client.get_or_create_collection(name="my_collection")

collection = chroma_client.get_or_create_collection(  
    name="nbeusers",
    metadata={"hnsw:space": "cosine"}
)

Step 3 Train Cusom Data UsingYolo8

To detect cigarettes in real-time, the project leverages a custom-trained YOLOv8 model. The training process involves collecting, annotating, and fine-tuning a dataset specific to cigarette detection. By focusing on this niche category, the model achieves higher accuracy in identifying smoking behavior in restricted zones

step 4 Mail Integration

• setting main mail settings

email_sender = 'your mail'
email_password = 'password'  
email_receiver = 'your mail'

• full Mail Script
  This Python code defines a function, sendingMail, designed to send email notifications when a smoking activity is detected. It uses the smtplib library to handle email communication over the Simple Mail Transfer Protocol (SMTP)

'''


import smtplib
import ssl
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email import encoders
import cv2
import numpy as np
from mailConfiguration.email_setting import *
from datetime import datetime



def sendingMail(from_email=email_sender, to_email=email_receiver,  img=None,personName=None):
    # Get current time and date
    now = datetime.now()
    current_time = now.strftime("%H:%M:%S")
    current_date = now.strftime("%Y-%m-%d")
    
    # Create the email headers and subject
    sent_subject = "Smoking Activity Detected - Immediate Attention Required"

    # Create the email body
    sent_body = (
        f"This is an automated notification from the SmokerX App. A person identified as {personName} has been detected smoking, "
        "as captured in the attached image. Below are the details:\n"
        f"Person Name: {personName}\n"
        f"Time of Detection: {current_time}\n"
        f"Date of Detection: {current_date}\n"
        "Location: Fourth Floor, Building Number 5\n"
        "Please review the attached image for further verification.\n\n"
        "Best regards,\n"
        "SmokerX Monitoring Team"
    )

    # Create a multipart message and set headers
    message = MIMEMultipart()
    message["From"] = from_email
    message["To"] = to_email
    message["Subject"] = sent_subject

    # Add body to email
    message.attach(MIMEText(sent_body, "plain"))

    if img is not None:
        # Encode the frame in JPEG format
        ret, buffer = cv2.imencode('.jpg', img)
        if ret:
            image_data = buffer.tobytes()

            # Create an attachment part
            image_attachment = MIMEBase("application", "octet-stream")
            image_attachment.set_payload(image_data)
            encoders.encode_base64(image_attachment)
            image_attachment.add_header(
                "Content-Disposition",
                "attachment; filename=smoking_detected.jpg",
            )

            # Attach the image to the email
            message.attach(image_attachment)

    # Convert the message to a string
    email_text = message.as_string()

    try:
        # Connect to the SMTP server using SSL
        context = ssl.create_default_context()
        server = smtplib.SMTP_SSL('smtp.gmail.com', 465, context=context)
        server.login(email_sender, email_password)

        # Send the email
        server.sendmail(email_sender, [email_sender, email_receiver], email_text)

        # Close the server connection
        server.quit()
        print('Email sent successfully!')
        return True

    except Exception as exception:
        print(f"Error: {exception}")
        return False

Code Snippet

• code explanation

1. get object cordinates in clear points

def getObjectCoordinates(x_center, y_center, box_w, box_h):
    x1 = int(x_center - box_w / 2)
    y1 = int(y_center - box_h / 2)
    x2 = int(x1 + box_w)
    y2 = int(y1 + box_h)
    return x1, y1, x2, y2

2. Function to handle sending mail asynchronously

def async_send_mail(img, track_id):
    try:
        sendingMail(img=img)
        print(f"Email sent for track ID: {track_id}")
    except Exception as e:
        print(f"Failed to send email for track ID {track_id}: {str(e)}")

3. Function to handle database queries asynchronously

def async_db_query(emddbedVector, track_id):
    try:
        final = collection.query(
            query_embeddings=emddbedVector.tolist(),
            n_results=1
        )
        return final
    except Exception as e:
        print(f"Database query failed for track ID {track_id}: {str(e)}")
        return None

• In this project, two pre-trained models are integrated to achieve dual detection functionality:

1. Person Detection Model: Identifies and locates individuals in the frame.
2. Smoker Detection Model: Identifies smoking behavior or the presence of cigarettes.
   By combining the outputs of these models, the system can effectively pinpoint individuals who are smoking in restricted areas.

Workflow Explanation :-

1.Person Detection:

• A pre-trained object detection model ( YOLOv8) is used to detect people in real-time.
• The model outputs bounding boxes and confidence scores for all detected individuals.
• These detections serve as the input for the smoker detection step.

2.Smoker Detection:

A separate model, fine-tuned for smoking activity or cigarette detection, processes the regions of interest (ROIs) provided by the person detection model.
If smoking behavior is detected within a person’s bounding box, it triggers an alert.

3.Integration:

The detections from both models are merged to localize and identify individuals engaged in smoking.
Identified smokers are logged for further actions, such as facial recognition and email alerts.

4.System Output:

Outputs include:
Annotated frames highlighting smokers.
Details about detected individuals (name, time, location, etc.).
Evidence in the form of captured images or video clips.

while True:
    ret, frame = cap.read()
    if not ret or frame is None:
        print("Failed to capture image")
        break

    #frame = cv2.resize(frame, (640, 480))
    frame=cv2.resize(frame,(1020,500))

    results = model.track(source=frame, persist=True, tracker="botsort.yaml")

    if results and results[0].boxes is not None:
        boxes = results[0].boxes.xywh.cpu()
        class_ids = results[0].boxes.cls.int().cpu().tolist()
        track_ids = results[0].boxes.id

        if track_ids is not None:
            track_ids = track_ids.int().cpu().tolist()
        else:
            print("Warning: No track IDs were returned.")

        for box, class_id, track_id in zip(boxes, class_ids, track_ids or []):
            if class_id in target_class_ids:
                cx, cy, w_, h_ = box
                x1, y1, x2, y2 = getObjectCoordinates(cx, cy, w_, h_)

                person_roi = frame[y1:y2, x1:x2]
                img, bboxs = detector.findFaces(person_roi, draw=False)
                Smokingresults = smoker.predict(frame, stream=True)

                if bboxs:
                    for bbox in bboxs:
                        fx, fy, fw, fh = bbox['bbox']
                        face_x1 = x1 + fx
                        face_y1 = y1 + fy
                        face_x2 = face_x1 + fw
                        face_y2 = face_y1 + fh

                        face = frame[face_y1:face_y2, face_x1:face_x2]
                        if track_id not in personFace:
                            personFace[track_id] = (fx, fy, fw, fh)

                        for r in Smokingresults:
                            boxs = r.boxes
                            for box in boxs:
                                x1_s, y1_s, x2_s, y2_s = box.xyxy[0]
                                x1_s, y1_s, x2_s, y2_s = int(x1_s), int(y1_s), int(x2_s), int(y2_s)
                                w_s, h_s = x2_s - x1_s, y2_s - y1_s
                                conf = math.ceil((box.conf[0] * 100)) / 100
                                cls = int(box.cls[0])
                                currentClass = classNames_[cls]

                                if face_x1 <= x1_s <= face_x2 and face_y1 <= y1_s <= face_y2:
                                    cv2.rectangle(frame, (x1_s, y1_s), (x2_s, y2_s), (0, 0, 255), 2)
                                    cv2.putText(frame, currentClass, (x1_s, y1_s - 20), cv2.FONT_HERSHEY_COMPLEX, 1, (255, 0, 0), 1)
                                    cv2.rectangle(frame, (face_x1, face_y1), (face_x2, face_y2), (0, 0, 255), 2)
                                    if track_id not in personNames:
                                        face_arr = cv2.resize(face, (160, 160))
                                        face_arr = face_arr.astype('float32')
                                        face_img = np.expand_dims(face_arr, axis=0)
                                        yhat = faceNetModel.embeddings(face_img)
                                        yhat = yhat[0]
                                        emddbedVector = np.asarray(yhat)

                                        # Start a new thread for the database query
                                        final = async_db_query(emddbedVector, track_id)

                                        if final:
                                            score = round(1 - final['distances'][0][0], 2)
                                            print(score)
                                            if score >= .40:
                                                text = final['metadatas'][0][0]['username']
                                                personNames[track_id] = text
                                                cv2.putText(frame, str(text), (face_x1, face_y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
                                                if track_id not in smokerMails:
                                                    smokerMails[track_id] = 'email sent'
                                                    # Start a new thread for sending mail
                                                    threading.Thread(target=async_send_mail, args=(face, track_id)).start()
                                                score = f"-{score}%"
                                            else:
                                                text = 'Unknown'
                                                score = '-0%'
                                                personNames[track_id] = text
                                                cv2.putText(frame, str(text), (face_x1, face_y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
                                                if track_id not in smokerMails:
                                                    smokerMails[track_id] = 'email sent'
                                                    # Start a new thread for sending mail
                                                    threading.Thread(target=async_send_mail, args=(face, track_id)).start()

                                    else:
                                        cv2.putText(frame, str(personNames[track_id]), (face_x1, face_y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)

    cvzone.overlayPNG(frame, logo, pos=[0, 426])
    cv2.imshow('Smoker X', frame)

    key = cv2.waitKey(10)
    if key == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Conclusion

This project successfully demonstrates the integration of advanced AI models to address a real-world problem of smoking in restricted areas. By combining the power of object detection, facial recognition, and email integration, the system provides a comprehensive solution for monitoring and reporting unauthorized smoking behavior.

The dual-model architecture ensures accurate detection of individuals and smoking activities, while the use of ChromaDB for facial embeddings enables swift identification of violators. The email alert system adds a layer of immediacy, ensuring that responsible departments are notified promptly with all necessary details, including time, location, and photographic evidence.

This solution is not only robust and scalable but also adaptable to various environments such as offices, schools, hospitals, and public spaces. The modular design allows for easy updates or extensions, such as adding new detection capabilities or enhancing the user interface.

Moving forward, this project can be expanded to incorporate real-time dashboards, SMS alerts, or integration with IoT devices for automatic response mechanisms. It serves as a model for leveraging AI in safety and compliance monitoring, setting the foundation for more intelligent surveillance systems in the future.