Project Overview

The project is aimed at creating, training, and using a language model tailored for working with lecture materials in PDF format. It encompasses a complete data processing pipeline – from text extraction to model fine-tuning using modern techniques such as LoRA, as well as preparation for semantic search. The speed and quality of learning directly depend on the power of your PC! For example, I have a gtx 1070 and 32 gb of RAM, which is not the most powerful hardware, is it? The only problem I've encountered is very slow learning and response generation.

---

Main Components of the Project

1. Text Extraction from PDFs

• Script: extract_text.py
  This script is responsible for reading PDF files, extracting text using the pdfminer library, and saving it in a text format. This process prepares the data for subsequent processing stages.

2. Splitting Text into Chunks

• Script: split_text.py
  Text files are divided into smaller chunks with a limited length (e.g., 512 characters). This approach ensures the data is conveniently processed during model training.

3. Text Data Augmentation

• Script: augment_text.py
  To increase the volume and diversity of the data, augmentation is applied using the nlpaug library. The script generates several variants of each text file by randomly swapping words.

4. Dataset Creation

• Script: create_dataset.py
  This script aggregates the augmented texts and splits them into training, validation, and test sets using the train_test_split function from scikit-learn. This structuring facilitates efficient model training.

5. Language Model Training

• Script: train_llama.py
  The main training module includes:
  • Loading and combining data from original and augmented texts;
  • Tokenizing texts using the LLaMA model and its tokenizer;
  • Configuring the model with LoRA for efficient parameter fine-tuning;
  • Setting up training via the Trainer from Hugging Face Transformers;
  • Saving the final model and tokenizer for future use.

6. Preparing for semantic search

• Script: create_faiss_index.py
  Although the current version of the script contains basic settings for working with NLTK, it hints at future integration with the FAISS library for fast search through vector representations of texts, which can be used for semantic search or the implementation of question-answer systems.

---

Project Structure

• data/
  Contains:

  • Original PDF files
  • Extracted texts
  • Prepared datasets (divided into train, val, and test)

• scripts/
  This directory contains all scripts for data processing and model training:

  • extract_text.py – text extraction from PDFs
  • split_text.py – splitting text into chunks
  • augment_text.py – text augmentation
  • create_dataset.py – dataset creation and splitting
  • train_llama.py – language model fine-tuning
  • chat_interface_web.py – interacting with the AI in a web browser (locally). Very simple interface, designed for convenience

• models/
  Contains saved models, training logs, and fine-tuning results.

Additional details and installation instructions are provided in the README.md file.

---

Conclusion

The project represents a comprehensive solution for building a language model capable of processing lecture materials. It automates data preparation—from extracting text from PDFs to splitting, augmenting, and structuring the dataset—and then employs modern fine-tuning techniques (such as LoRA) to adapt the LLaMA model. Potential integration with FAISS opens up prospects for implementing semantic search and interactive systems, making the project a valuable tool for educational and research purposes.