Abstract

This project addresses the challenges in fashion image analysis by developing models for jeans detection and segmentation, multi-label classification of fashion attributes, and attribute embedding using BERT. The methodologies employed include Detectron2 for object detection, Fastai for multi-label classification, and BERT for attribute embedding. The experiments demonstrate the effectiveness of these models in accurately detecting jeans, classifying multiple fashion attributes, and generating meaningful attribute embeddings.

Introduction

Fashion image analysis involves complex tasks such as detecting specific clothing items, classifying multiple attributes, and understanding textual descriptions. This project focuses on:

1 Jeans Detection and Segmentation:

Utilizing Detectron2's Generalized R-CNN models to accurately detect and segment jeans in images.

2 Multi-Label Classification:

Applying Fastai to classify multiple attributes of fashion items, acknowledging that garments often possess several attributes simultaneously.

3 Attribute Embedding with BERT:

Leveraging BERT embeddings to extract features from product descriptions, facilitating tasks like semantic search and information retrieval.

Methodology

1 Jeans Detection and Segmentation:

• Framework: Detectron2, utilizing Generalized R-CNN models for precise detection and segmentation.
• Approach: Fine-tuning with custom jeans datasets for better performance on fashion-specific tasks.

2 Fashion Multi-Label Classification:

• Library: FastAI, known for its ease in handling image classification tasks with transfer learning.
• Strategy: Implementing multi-label classification networks to categorize jeans by fit, color, style, etc.

3 BERT Attributed (Fashion) Embedding:

• BERT (Bidirectional Encoder Representations from Transformers):
  Introduced in late 2018, BERT is a revolutionary model in NLP, trained on a large corpus to understand language context bidirectionally. We use bert-large-uncased (24-layer, 1024-hidden, 16-heads, 340M parameters) for this project.
  Pre-training: BERT leverages tasks like Masked Language Model (MLM) and Next Sentence Prediction (NSP) to understand language nuances.
• BERT Embeddings:
  Feature Extraction: BERT can generate word and sentence embeddings from fashion product descriptions or attributes.
  Applications: These embeddings facilitate keyword/search expansion, semantic search, and information retrieval by capturing the nuanced meaning of fashion-related text. For example, they help in matching customer queries to existing product descriptions or FAQs even if the terminology differs.
  Input/Output:
  Input: Fashion product descriptions or specific attributes.
  Output: 1024-dimensional vectors for words or sentences, representing their semantic content for tasks like clustering, similarity search, or enhancing recommendation engines.

Experiments

1. Jeans_Dectection_Segmentation_v1.ipynb:

Dataset: Custom dataset with jeans annotations for training.
Model: Detectron2 with configurations for model training, including learning rate and batch size.
Evaluation: Visual and potentially quantitative analysis of detection and segmentation accuracy.

2. levis_multilabel_fastaiv1.ipynb:

Dataset: Levi's jeans data with multi-label annotations.
Training: FastAI's data block API for structured data handling, with focus on balancing datasets.
Metrics: Accuracy, F1 scores, etc., for each attribute classification.

3. BERT Embedding.ipynb:

Text Data: Uses fashion descriptions to show how BERT can produce meaningful embeddings.
Process: Tokenization, embedding generation, and visualization of semantic relationships.

4. multilabe_fastai2.ipynb:

FastAI v2: Enhanced capabilities with new features like data augmentation and progressive resizing.
Experimentation: Focused on optimizing multi-label classification for jeans attributes.

Additional Information from Repository:

Code Structure: Includes datasets, model checkpoints, and visualization scripts.
Pretrained Models: Utilization of COCO pre-trained weights for initial model setup.

For detailed code and further information, please refer to the following notebooks:
https://github.com/Pyligent/Fashion-Jeans-Detection-attributes-BERT-embedding-multilabels/tree/master

Results

• Detection and Segmentation: High accuracy in identifying jeans, with segmentation closely matching actual jean shapes.
• Multi-Label Classification: Models show promising results in attribute classification, though specific metrics are not detailed.
• BERT Embeddings: Demonstrate potential in semantic analysis of fashion text, useful for enhancing search and recommendation systems.

Conclusion

This project illustrates the powerful integration of AI technologies in fashion analysis, specifically for jeans. By combining Detectron2, FastAI, and BERT, we provide tools that can significantly improve how fashion items are searched for, recommended, and understood. Future work could aim at expanding datasets, refining model architectures, and applying these methods in real-world fashion tech scenarios.