Sanjeevan

Sanjeevan is a video calling application designed to bridge the communication gap for people who cannot speak. Our mission is to empower individuals with diverse abilities to express themselves freely and connect with others effortlessly. Through innovative technology and machine learning, we aim to create an inclusive world where everyone's voice is heard, regardless of linguistic or physical barriers.

Though we made the project for the hackathon purpose, which we did win by the way, the purpose was clear: to connect the bridge between a person who can't speak and a person who can't understand sign language.

Short Working video: Youtube

Features

• Real-time Video Communication: Video calling feature to connect users.

• Hand Gesture Recognition: Uses AI to detect and interpret hand gestures, translating them into meaningful communication.

• Sign Language Support: Includes support for multiple gestures, including common phrases like "Thank You," "I Love You," and "Please."

• We didn't stop there. What if you want to learn sign language or more specifically want to learn a single word? In our app, you can write a complete sentence or just a word and get a video representation of the word in sign language.

• Inclusive Design: Tailored for individuals with speech impairments, ensuring easy and accessible communication.

Technologies Used

Frontend

• HTML
• CSS
• JavaScript

Backend

• FastAPI
• Uvicorn

Services

• FireBase
• Amazon Web Services

AI Model

• Random Forest Classifier
  • Built using custom and pre-existing datasets to improve model variation.
  • Trained to recognize hand gestures and convert them into text.
• LSTM (Long Short-Term Memory) Networks
  • For advanced gesture recognition and prediction.
• Generative AI
  • Planned for real-time generation of sign language.

Real-time Communication

• Agora: Facilitates video calls and chat functionalities.

How It Works

The application leverages MediaPipe to capture hand landmarks and process them using a trained RandomForestClassifier to identify gestures. The hand gestures are mapped to alphabets and common phrases using predefined labels, allowing users to communicate through sign language.

Key Components

1. Model: The gesture recognition model is pre-trained and stored as a pickle file (model.p). It predicts hand signs based on MediaPipe's hand landmark detection.

2. WebSocket: Real-time communication via WebSockets enables live video streaming and gesture detection.

3. Video Feed: Uses OpenCV to capture video frames, process hand gestures, and send predictions in real time.

Dataset and Model Training

• Custom Dataset: We have created the hand sign model to achieve maximum accuracy and expanded our dataset, which is custom-made for Indian Sign Language. This was necessary because the dataset for Indian Sign Language is not easily available. We recorded 50+ videos ourselves and used existing videos available on the internet.

Videos that we recorded https://github.com/vaibhavkothari33/speech_to_handsign/tree/main/video

• Data Augmentation: We performed data augmentation to achieve better accuracy, resulting in a dataset of around 80 GBs.

• High-Performance Computing: To process this large dataset, we utilized a DGX supercomputer and a gaming computer with an RTX 6000 provided by our college. This allowed us to train the model effectively.

• Random Forest and LSTM Models: We explored various models including Random Forest and LSTM (Long Short-Term Memory) networks to enhance the project's capabilities.

• Generative AI: We plan to use generative AI to generate real-time sign language for future scalability.

Installation

To get started, follow these steps:

1. Clone the repository:

   git clone https://github.com/vaibhavkothari33/Hackfest.git

2. Install the required dependencies:

   pip install -r requirements.txt

3. Run the FastAPI server:

   uvicorn app:app --reload

Example Code

Here's an example of how gestures are detected and processed:

results = hands.process(frame_rgb)

if results.multi_hand_landmarks:
    for hand_landmarks in results.multi_hand_landmarks:
        # Process landmarks for gesture prediction
        prediction = model.predict([np.asarray(data_aux)])
        predicted_character = labels_dict[int(prediction[0])]

Example code of how speech to text works

def speech_to_text():
    recognizer = sr.Recognizer()
    with sr.Microphone() as source:
        print("Please say something...")
        audio = recognizer.listen(source)
        try:
            text = recognizer.recognize_google(audio)
            print(f"Recognized text: {text}")
            return text
        except sr.UnknownValueError:
            print("Sorry, I did not understand that.")
        except sr.RequestError:
            print("Sorry, I couldn't request results; check your network connection.")
        return None

To play the video in Sequence :(for showing the user video to learn sign language)

def play_videos_in_sequence(video_paths):
    for video_path in video_paths:
        cap = cv2.VideoCapture(video_path)
        
        if not cap.isOpened():
            print(f"Error opening video file {video_path}")
            continue
        
        while cap.isOpened():
            ret, frame = cap.read()
            if not ret:
                break

            frame = cv2.resize(frame, (640, 360))  # Smaller frame size for faster rendering
            cv2.imshow('Video', frame)

            if cv2.waitKey(10) & 0xFF == ord('q'):  # Faster playback
                cap.release()
                cv2.destroyAllWindows()
                return
        
        cap.release()
    
    cv2.destroyAllWindows()

API Endpoints

• video-feed: WebSocket endpoint for real-time video and gesture prediction.

Future Enhancements

-Gen AI: For the real time sign language generation

• Multi-language Support: Expand gesture recognition to support additional sign languages, enabling more users from diverse linguistic backgrounds to communicate seamlessly.

• Voice Synthesis: Implement text-to-speech functionality to convert recognized hand signs into spoken words, enhancing the communication experience for individuals who rely on verbal interactions.

• Improved Model: Further train the model with larger and more diverse datasets to increase gesture recognition accuracy and prediction speed, making communication faster and more reliable.

Demo

You can check out Sanjeevan at Sanjeevan Demo

Notice

The app that we built is not currently live because, as students, it costs us a lot to keep it live and handle such a large dataset. We are sorry for any inconvenience caused. However, we do have a very simple app that is live, which was made by us and solves the same problem. The only difference is that it supports 7 hand signs.

Rest assured, the GitHub links are working. You can visit the site and the video calling application is also running.

Working video calling application : Sanjeevan Demo

Simple Sanjeevan1.0 Sanjeevan

Github repo: Github