AI-Powered Product Analysis Using Qwen2-VL-2B-Instruct

This repository presents an AI-powered product analysis pipeline leveraging the Qwen2-VL-2B-Instruct model for analyzing images of packaged and fresh products. The project aims to extract meaningful details like product names, expiry dates, freshness indices, and shelf life, integrating fine-tuning techniques and innovative methodologies to improve performance and scalability.

---

Abstract

In industries like retail and e-commerce, efficient product analysis is crucial for inventory management and quality control. Traditional Machine Learning (ML) and Deep Learning (DL) pipelines often struggle with multi-modal data and domain-specific generalization. We propose a novel approach that uses the Qwen2-VL-2B-Instruct model, leveraging its large-scale multi-modal learning capabilities.

Our methodology integrates a robust fine-tuning mechanism using QLoRA on a custom dataset, enabling the model to adapt to niche use cases. By introducing a Gradio-based GUI for real-time experimentation and a CLI for automation, the project ensures usability for developers and researchers. While pre-trained models may falter on domain-specific data, our innovations in data preparation, fine-tuning, and validation demonstrate significant impact, offering a reliable AI-driven solution for product analysis.

---

Why Qwen2-VL-2B-Instruct?

Qwen2-VL-2B-Instruct stands out for its exceptional multi-modal capabilities, which are vital for analyzing both textual and visual information. Unlike traditional ML and DL models requiring separate pipelines for images and metadata, Qwen2-VL seamlessly processes both modalities, ensuring:

• High Accuracy: Pre-trained on extensive datasets, it provides a strong baseline for various domains.
• Multi-Modal Support: Processes text, images, and combinations thereof.
• Ease of Customization: Fine-tuning with QLoRA allows domain adaptation without significant computational overhead.

Traditional ML/DL models would necessitate handcrafted feature extraction or cumbersome ensemble architectures, making them less adaptable to dynamic real-world use cases.

---

Methodology

Data Collection and Preprocessing

1. Custom Dataset:

   • Fresh produce images annotated with freshness indices, expected lifespan, and quality metrics.
   • Packaged product images labeled with brand, expiry dates, manufacturing dates, and "best before" durations.

2. Preprocessing:

   • Images were resized to standard dimensions (256x256).
   • Textual metadata was normalized, including date formats and numeric attributes.

Model Workflow

1. Loading the Model:

   • Qwen2-VL-2B-Instruct is loaded with PyTorch and Hugging Face's transformers library.
   • A processor converts inputs (images and prompts) into model-compatible tensors.

2. Fine-Tuning with QLoRA:

   • The QLoRA (Quantized LoRA) technique minimizes GPU memory usage, making fine-tuning feasible on consumer-grade hardware.
   • Fine-tuning involves:
     • Custom Dataset: A blend of domain-specific and general-purpose data.
     • Augmented Prompts: Adding domain-relevant instructions.
     • Validation Set: Ensuring robust performance on unseen data.

3. Product Analysis Pipeline:

   • Packaged Products: Extract brand, expiry date, manufacturing details, and count. Normalize expiry calculations based on "best before" durations.
   • Fresh Produce: Evaluate freshness on a 1-10 scale and estimate shelf life using visual cues.

4. GUI and CLI Integration:

   • Gradio GUI: Simplifies real-time testing with an intuitive interface.
   • CLI: Automates batch processing for enterprise workflows.

---

Results and Observations

• Initial Model Performance:

  • Out-of-the-box performance highlighted challenges with domain-specific data.
  • Before Fine Tuning its not able to extract expiry dates of different formats correctly

• Impact of Fine-Tuning:

  • QLoRA fine-tuning on the custom dataset will improve its packaged product accuracy and freshness evaluation
  • Enhanced adaptability to date format variations and subtle visual cues like color gradients and texture patterns.

• Comparison with Baselines:

  • Traditional ML/DL models (e.g., ResNet + LSTM pipelines) underperformed due to limited contextual understanding.
  • Models like CLIP exhibited slower convergence as compared to Qwen2-VL.

---

Installation and Usage

Dependencies

• Python 3.8+
• PyTorch
• Hugging Face Transformers
• Gradio

I am in learning phase please give your feedbacks and connect with me on below linkedIn the fine tuned model is not ready due to not availablity of paid GPU.

Project Repo: Smart Vision Quality Testing

---

My LinkedIN and Github:
https://www.linkedin.com/in/pranavmittal07/
https://github.com/pranavmittal07