Abstract

This paper presents a Text-to-Video Generation System capable of synthesizing videos from textual descriptions, images, or their combination. The system leverages state-of-the-art diffusion models integrated with FastAPI for backend processing and Streamlit for user interaction. This approach showcases the potential of generative AI in bridging the gap between natural language and video synthesis.

Introduction

Video generation from natural language or image inputs represents a challenging task in generative AI due to its complexity and resource requirements. Recent advancements in diffusion models have made significant strides in addressing this challenge. This project implements a practical pipeline to achieve text-to-video generation by combining powerful generative models, such as damo-vilab/text-to-video-ms-1.7b and ali-vilab/i2vgen-xl, with robust backend and frontend solutions. The project aims to provide an easy-to-use system for generating high-quality videos for applications in creative content generation, education, and entertainment.

Methodology

The proposed system integrates a modular backend and frontend architecture:

1. Backend:

   • Implemented using FastAPI, the backend serves as the processing hub.
   • Diffusion pipelines, specifically damo-vilab/text-to-video-ms-1.7b for text input and ali-vilab/i2vgen-xl for image input, are utilized to generate videos.
   • Request validation and task orchestration are performed using Pydantic models.

2. Frontend:

   • A Streamlit application allows users to interact with the backend.
   • Users can select input types (text, image, or both) and view the generated video.

3. Deployment:

   • Models are hosted using Google Colaboratory with ngrok for exposing endpoints.

Experiments

Experiments were conducted using various prompts and image inputs to evaluate the system's ability to generate coherent and contextually accurate videos:

• Text-only Input: Single-sentence prompts describing actions or scenes (e.g., "A sunset over the mountains with clouds rolling in").
• Image-only Input: High-resolution images depicting static scenarios.
• Text and Image Input: Textual descriptions enhancing the context of the provided image.

Performance metrics included inference time, output quality, and user satisfaction.

Results

The system successfully generated videos that matched the input prompts and images. Key findings include:

• Quality: Videos generated from combined text and image inputs exhibited better contextual alignment.
• Efficiency: Average processing time for a 5-second video was approximately 15-20 seconds using ngrok and GPU acceleration in Google Colab.
• Flexibility: The system effectively handled a range of prompts, from abstract descriptions to specific scenes.

Challenges observed included occasional artifacts in video outputs, which can be mitigated by improving the diffusion models or fine-tuning hyperparameters.

Conclusion

The Text-to-Video Generation System demonstrates the practical application of generative AI for video synthesis. By combining advanced diffusion models with an intuitive interface, the project bridges the gap between AI research and real-world usability. Future work will focus on improving video quality, optimizing inference time, and exploring alternative deployment strategies to ensure scalability.

Project Structure

backend/
|── .env
|── main.py
|── models.py
|── requirnments.py
|── s3_utils.py
└── tasks.py

frontend/
|── app.py

Notebooks/
|── Hosting_model_on_Google_Colaboratory.ipynb

Endpoints

• POST /generate-text-video: Generates a video from a text prompt.
• POST /generate-image-video: Generates a video from an uploaded image.
• POST /generate-text-image-video: Generates a video from a text prompt and an uploaded image.

Files

• main.py: Main FastAPI application with endpoints for video generation.
• models.py: Pydantic models for request validation.
• s3_utils.py: Utilities for uploading files to S3.
• tasks.py: Task processing functions for video generation.
• Hosting_model_on_Google_Colaboratory.ipynb: Notebook for hosting deep learning models on Google Colab.

Backend

The backend is implemented using FastAPI and provides endpoints for generating videos from text, images, or both. It also includes utilities for uploading files to S3.

Frontend

The frontend is implemented using Streamlit and provides a user interface for interacting with the backend.

Features

• Select input type: Text, Image, or Text + Image.
• Enter text prompt and/or upload an image.
• Generate video and display the result.

Files

• app.py: Streamlit application for user interaction.

Installation

1. Clone the repository:

   git clone https://github.com/Dineth9D/text-to-video.git
   cd text-to-video

2. Set up the backend:

   cd backend
   python -m venv .venv
   venv\Scripts\activate
   pip install -r requirements.txt

3. Set up the frontend:

   cd ../frontend
   python -m venv .venv
   .venv\Scripts\activate
   pip install -r requirements.txt

4. Upload the Notebook to Google Colab:

   - Open your Google Drive and create a folder for the project (e.g., 'Text_to_Video_Generation').
   - Upload 'Hosting_model_on_Google_Colaboratory.ipynb' to this folder.
   - Open the notebook in Google Colab by right-clicking and selecting Open with > Google Colaboratory.
   

Usage

1. Start the server in Google Colab:

   - Run the notebook to start the FastAPI server.
   - Note the `ngrok` URL provided in the notebook output.
   - Update the backend server URL in `backend/main.py` with the `ngrok` URL.
   

   

2. Start the backend server:

   cd backend
   uvicorn main:app --reload

3. Start the frontend application:

   cd ../frontend
   streamlit run app.py

4. Open your browser and go to http://localhost:8501 to use the application.