AI-VTuber-System

GitHub : AI-VTuber-System

1. Introduction and Goals

• Purpose
  Develop an integrated AI-driven virtual live-streaming host system that enables users to rapidly create and customize an AI VTuber, offering more interactive and diverse live content.

• Key Features

  1. Language Model: Integrates Google Gemini and OpenAI GPT to provide natural dialog.
  2. Text-to-Speech (TTS): Utilizes Microsoft and OpenAI technologies for human-like voice output.
  3. Automatic Speech Recognition (ASR): Employs OpenAI Whisper for accurate voice input from users or viewers.
  4. VTube Studio: Configures a Live2D avatar and controls animation via API.
  5. OBS: Streams audio and video, and displays live chat subtitles using WebSocket.

---

2. System Architecture and Core Flow

2.1 Core Operational Flow

1. Audio/Text Input
   • Collected via the user’s microphone or messages in the chat.
2. AI Processing
   • The chosen large language model (Gemini or GPT) generates a reply.
3. Voice Output (TTS)
   • The text reply is synthesized into a spoken voice response.
4. Virtual Character
   • VTube Studio is used to animate the avatar’s facial expressions and motions; OBS handles video layout.
5. Subtitle Display
   • Subtitles are updated in real time via the OBS WebSocket interface.
     

Conceptual diagram of the process:

[User/Viewer Input] -> [LLM Generates Reply] -> [TTS Voice Synthesis] -> [Virtual Avatar + Subtitles] -> [OBS Streaming]

---

3. Environment Setup and Installation

3.1 Python and Dependencies

1. Python Version: 3.8 or higher
2. Third-Party Packages (requirements.txt)
   • edge_tts==6.1.10
   • google-generativeai==0.5.0
   • keyboard==0.13.5
   • obs-websocket-py==1.0
   • openai==1.13.3
   • openai-whisper==20231117
   • protobuf==4.25.3
   • pyaudio==0.2.14
   • pydub==0.25.1
   • pyside6==6.6.2
   • pytchat==0.5.5
   • pyvts==0.3.2
   • Requests==2.31.0
   • tiktoken==0.6.0
   • transformers==4.38.2
   • twitchio==2.9.1
3. Installation Command:

   pip3 install -r requirements.txt

4. PyTorch Installation (for Whisper GPU support):

   pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

3.2 GPU-Related Installations (if local Whisper GPU inference is required)

• Nvidia GPU Drivers
• CUDA Toolkit 12.1.1
• cuDNN

3.3 Whisper Models (Local Inference)

Place .pt models (e.g. tiny.pt, base.pt, medium.pt, etc.) in OpenAI/whisper/models/.

---

4. AI Vtuber Control Panel Overview

This is a GUI (built via PySide6 + Qt Designer) serving as the main controller of the system:

1. Main

   • Displays real-time conversation history.
   • Switch the speaker role (user, system, assistant, etc.).
   • Toggle microphone on/off; quickly send speech recognition results.
     

2. Setting

   • Specify your User Name.
   • Choose a Character.
   • Adjust window opacity, audio output device, microphone input, etc.
   • Define “What’s doing now” as context for the AI (e.g., current livestream topic).
     

3. LiveChat

   • Connect with YouTube Live / Twitch.
   • Configure whether to respond to chat, 1-to-1 replies, VIP lists, blacklists, etc.
     

4. LLM

   • Select the large language model (Gemini or GPT) and model version.
   • Set tokens limits (Max Input/Output Tokens) and randomness (Temperature).
   • “Instruction Enhance” for extra role/prompt reinforcement.
     

5. TTS

   • Choose voice synthesis engine (Edge TTS or OpenAI TTS).
   • Adjust voice pitch, speed, and volume.
     

6. Whisper

   • Decide local or OpenAI API for speech recognition.
   • Load/unload specific Whisper models; choose language and optional prompt text.
     

7. OBS

   • Update subtitles via OBS WebSocket.
   • Configure text formatting, clearing, display duration, etc.
   • Set filter names and delays for showing/hiding subtitles.
     

8. VTSP

   • Integrate with VTube Studio hotkeys.
   • Optionally enable sentiment analysis to trigger expressions matching AI’s response.
   • If sentiment analysis is off, you can randomly or consistently trigger selected hotkeys.
     

---

5. Recommended Usage Steps

1. Install Python and Prerequisites
   Ensure Python, GPU drivers, CUDA, and cuDNN are correctly installed.
2. Download and Launch AI-VTuber-System
   • Download either the full source code or the All-in-One release from GitHub.
   • Note: The All-in-One version still requires you to install GPU-related components and place the Whisper model of your choice.
3. Configure the Control Panel
   • Setting: Provide user name, mic device, audio output device, etc.
   • LLM: Input needed API keys (if using GPT or Gemini), set tokens and temperature.
   • TTS: Pick a TTS engine (e.g., Edge TTS) and desired voice.
   • Whisper: Load a local model or use the API.
   • LiveChat: Set up YouTube/Twitch channels for chat.
   • OBS: Enable OBS WebSocket, specify a text source name, set subtitle filters/delays.
   • VTSP: Confirm VTube Studio hotkeys, enable or disable sentiment analysis.
4. Start the Livestream Workflow
   • Launch VTube Studio → Enble its API Plugin → Connect with the Control Panel.
   • Launch OBS → Open WebSocket → Connect from the Control Panel.
   • Test conversation or mic input in the “Main” tab to verify correct AI replies and subtitles.
   • Enable YouTube or Twitch chat to confirm the AI automatically responds to viewers.
5. During the Livestream
   • The system processes inputs from the host or viewers, generating replies in real time.
   • OBS displays the AI’s subtitles as the VTube Studio model performs speaking animations.
   • You can adjust settings in the Control Panel mid-broadcast as needed.

---

6. Additional Notes

• Conversation History
  • The directory Text_files/ConversationHistory logs all daily conversations in date-labeled .txt files.
• Token Calculator
  • Use My_Tools/Token_Calculator_GUI.bat to quickly estimate text token usage.
• GPU Check
  • Run My_Tools/check_gpu_torch.bat to verify whether Torch can detect your GPU for Whisper.

---

7. Conclusion

Through the AI-VTuber-System, users can conveniently build a personalized VTuber without extensive Python/AI expertise. The system seamlessly integrates LLM, TTS, ASR, VTube Studio, and OBS, producing both lifelike voice responses and synchronous Live2D animations/subtitles. This setup significantly broadens the possibilities for interactive and engaging livestreams.

For further installation and usage details, or to review the source code, please visit the GitHub repository.