Overview

Qdrant is a high-performance vector database and similarity search engine built to manage high-dimensional vectors. It drives AI applications through advanced, open-source vector search capabilities, allowing efficient handling of embeddings and neural network encoders for use cases like intelligent search, recommendation engines, retrieval-augmented generation, and anomaly detection. Designed for cloud-native scalability, Qdrant is easy to deploy and cost-effective, featuring built-in compression options. It seamlessly integrates with popular embedding models and frameworks, making it a flexible solution for a wide range of AI-powered applications.

Qdrant

Payload (Metadata) Based Reranking

A payload is additional information stored together with vectors in a vector database. It represents the information linked to each vector (or entity) and can include different data types such as integers, floats, booleans, keywords, geographical coordinates, and datetime values, typically formatted as JSON.

Payload (Metadata) Based ReRanking in Qdrant leverages this metadata to refine and reorder search results. When a search query is executed, vectors are first ranked by semantic similarity. The associated payloads (metadata) are then applied to adjust the rankings based on specific criteria like user preferences, context, or domain-specific filters. This approach enhances vector search by ensuring results are not only semantically relevant but also contextually aligned with the additional metadata.

Multi-Vector Search in Qdrant

MultiVector Search is the ability to perform searches across multiple vector spaces or use several vectors for a single entity simultaneously. This is especially valuable in scenarios where a data item is represented by more than one vector to capture different aspects or modalities, making a single query vector insufficient to express the full search intent. For instance, an image might have one vector representing its visual features and another capturing its textual description. Using multiple vectors allows the search engine to combine these dimensions, improving the relevance and precision of results. Common applications include multimedia content search, image-text retrieval, and recommendation systems.

Key benefits include:

• Higher Search Accuracy: In cases where products are described both visually and textually, MultiVector Search can combine image and text vectors to deliver more accurate matches. For example, a user searching for a “red dress with floral patterns” benefits from both the image vector, which identifies visual traits like color and pattern, and the text vector, which captures semantic details such as “red,” “dress,” and “floral patterns.” Together, these vectors minimize ambiguity and improve contextual relevance.

• Advanced Reranking: On platforms like e-commerce sites, MultiVector Search enhances reranking by combining vector scores with user preferences. For example, when searching for “summer dresses,” the system can integrate visual features (color, style, fabric) with textual descriptions and past user behavior (eco-friendly brands, light fabrics). This multi-layered reranking ensures the final results align with both query intent and user-specific context.

• Richer Data Representation: Using multiple vectors allows for more comprehensive searches across complex datasets. In a news agency database containing articles, images, videos, and audio clips, MultiVector Search can combine different modalities to return a wide variety of relevant content for journalists and creators.

• Scalability and Performance: Modern vector databases such as Qdrant are designed to efficiently handle multiple vectors per entity. Multi-vector search operations can be parallelized, leveraging distributed systems and multi-core processors to deliver fast responses even on large-scale datasets.

Creating a Qdrant Collection

Creating a collection in Qdrant involves defining a structured storage area within the database to store and manage vector data alongside their associated metadata (payloads). A collection in Qdrant functions similarly to a table in relational databases but is purpose-built for handling high-dimensional vector data and performing efficient similarity searches.

This guide walks through the process of setting up a Qdrant collection, including configuring payloads and demonstrating various payload filtering techniques.

Prerequisite: Before creating a collection, ensure Qdrant is properly set up following the requirements. One key requirement is having Docker installed locally, which can be done by downloading it from Docker’s official website.

Requirements

Docker’s official website

Step 1: Installing Qdrant

After ensuring Docker is installed and running, open your terminal or command prompt. Pull the Qdrant Docker image from Docker Hub using:

   docker pull qdrant/qdrant

Next, start Qdrant by running:

docker run -p 6333:6333 qdrant/qdrant

This command launches a new container and maps port 6333 inside the container to port 6333 on your local machine. Once running, you can access the Qdrant server via http://localhost:6333.

Step 2: Creating the Collection

Here, we define a Qdrant collection for a text retrieval system. The collection will be named "articles" and set up to handle payloads with support for different filtering options.

The code snippet below imports the required modules from the Qdrant Client library and initializes a connection to a Qdrant server running on localhost at port 6333:

from qdrant_client import QdrantClient
from qdrant_client.http import models

# Initialize Qdrant client
client = QdrantClient("localhost", port=6333)

# Define the collection name
collection_name = "articles"

This creates a new collection named “articles” with 384-dimensional vectors using cosine similarity.

Step 3: Defining Appropriate Payloads

For this text retrieval system, the payload will include the following fields:

• title: string
• content: string
• author: string
• date: string
• category: string
• tags: list of strings
• views: integer
• rating: float

To store the data, we will prepare a list of PointStruct objects. Each PointStruct represents a single point in the vector space and contains:

• id: A unique identifier for the point.
• vector: The vector representation of the point (here, using placeholder vectors matching the collection’s dimensions).
• payload: A dictionary containing the metadata (payload fields) associated with that point.

To store the documents in the text retrieval system, we create a list of PointStruct objects containing both the vector representations and their associated metadata.

# Prepare the points with payloads
points = [
    models.PointStruct(
        id=1,
        vector=[0.1] * 384,
        payload={
            "title": "Introduction to Qdrant",
            "content": "Qdrant is a vector database management system...",
            "author": "John Doe",
            "date": "2023-07-01",
            "category": "Technology",
            "tags": ["database", "vector search", "Qdrant"],
            "views": 1000,
            "rating": 4.5
        }
    )
]

Using the client.upsert method, these points are added to the collection. The upsert operation performs the following:

• Inserts new points if their IDs don’t already exist in the collection.

• Updates existing points if their IDs are already present in the collection.

Finally, the status of the upsert operation is printed:

# Upsert the points into the collection
operation_info = client.upsert(
    collection_name=collection_name,
    points=points
)


<!-- RT_DIVIDER -->
# Step 4: Payload Filtering Examples

**Exact Match Filtering**  
This filter retrieves only the articles where the `category` field exactly matches **"Technology"**. It’s useful when you need precise matching on specific fields.

```python
# Exact Match Filtering
results = client.search(
    collection_name="articles",
    query_vector=[0.1] * 384,
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="category",
                match=models.MatchValue(value="Technology")
            )
        ]
    ),
    limit=10
)

Range Filtering
This filter returns articles where the views count is between 500 and 2000. Range filters are ideal for working with numeric fields to find items within a specified range.

# Range Filtering
results = client.search(
    collection_name="articles",
    query_vector=[0.1] * 384,
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="views",
                range=models.Range(gte=500, lt=2000)
            )
        ]
    ),
    limit=10
)

Multiple Condition Filtering
This filter applies multiple conditions simultaneously. For example, it returns articles in the "Technology" category with a rating of 4.0 or higher. It’s useful for applying layered criteria to refine results.

# Multiple Condition Filtering
results = client.search(
    collection_name="articles",
    query_vector=[0.1] * 384,
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="category",
                match=models.MatchValue(value="Technology")
            ),
            models.FieldCondition(
                key="rating",
                range=models.Range(gte=4.0)
            )
        ]
    ),
    limit=10
)

Array Contains Filtering
This filter checks if the tags array contains a specific value, such as "Qdrant". It’s particularly useful when dealing with array fields like tags or categories.

# Array Contains Filtering
results = client.search(
    collection_name="articles",
    query_vector=[0.1] * 384,
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="tags",
                match=models.MatchAny(any=["Qdrant"])
            )
        ]
    ),
    limit=10
)

Combining AND and OR Conditions
This filter combines both AND and OR clauses. For instance, it can return articles with at least 500 views AND authored by either John Doe OR Jane Smith. The must clause acts like an AND condition, while should functions as OR.

# Combining AND and OR Conditions
results = client.search(
    collection_name="articles",
    query_vector=[0.1] * 384,
    query_filter=models.Filter(
        should=[
            models.FieldCondition(
                key="author",
                match=models.MatchValue(value="John Doe")
            ),
            models.FieldCondition(
                key="author",
                match=models.MatchValue(value="Jane Smith")
            )
        ],
        must=[
            models.FieldCondition(
                key="views",
                range=models.Range(gte=500)
            )
        ]
    ),
    limit=10
)

Negative Filtering (NOT condition)
This filter excludes all articles in the "Sports" category. The must_not clause is helpful when you want to remove certain items from your search results.

# Negative Filtering (NOT condition)
results = client.search(
    collection_name="articles",
    query_vector=[0.1] * 384,
    query_filter=models.Filter(
        must_not=[
            models.FieldCondition(
                key="category",
                match=models.MatchValue(value="Sports")
            )
        ]
    ),
    limit=10
)

Prefix Matching
This filter retrieves articles whose titles start with "Intro". Prefix matching is great for partial string matches, especially for searching the beginning of text fields.

# Prefix Matching
results = client.search(
    collection_name="articles",
    query_vector=[0.1] * 384,
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="title",
                match=models.MatchText(text="Intro")
            )
        ]
    ),
    limit=10
)

These filtering techniques provide a comprehensive toolkit for refining search results in Qdrant. By combining them, you can build complex queries that precisely target the desired data, significantly enhancing the effectiveness of the vector search system.

Performing a Multi-Vector Search

A multi-vector search allows you to query using multiple vectors (query points) simultaneously within the vector database. To implement this with payload filtering and rerank the results using metadata, you follow the same steps for creating a Qdrant collection and defining payloads, but enable Qdrant’s multi-vector indexing and search capabilities.

In the context of a text retrieval system, the process involves two main stages:

1. Executing a multi-vector search with payload filtering – using multiple query vectors to capture different aspects of the data while applying filters based on payload metadata.
2. Reranking the results using payload information – adjusting the ranking based on additional metadata to ensure the final results are both semantically relevant and contextually aligned.

Performing a Multi-Vector Search with Payload Filtering

Before executing a multi-vector search, the data needs to be prepared by converting text into vector representations using word embeddings. A common library for this task is SentenceTransformer. In this example, a Google Colab GPU instance is used to generate the embeddings before proceeding with the Multi-Vector Search.

import pandas as pd
from sentence_transformers import SentenceTransformer

# Load the dataset
data_path = '/path/to/your/text_retrieval_task_dataset.csv'
data = pd.read_csv(data_path)

# Load a pre-trained SentenceTransformer model
model = SentenceTransformer('paraphrase-MiniLM-L6-v2')

# Convert text to vectors and add them as a new column
data['vector'] = data['text'].apply(lambda x: model.encode(x).tolist())

# Optional: Display the first few rows to verify
print(data.head())

After creating the file containing the vector embeddings, start Docker and import the required libraries. The Qdrant Client library is used to interact with Qdrant, while additional libraries handle data loading and processing. Finally, load the dataset for use in the multi-vector search.

import json
from qdrant_client import QdrantClient
from qdrant_client.http import models
import pandas as pd
from io import StringIO
import csv

# Initialize Qdrant client
client = QdrantClient("localhost", port=6333)

# Specify the file path to the dataset
file_path = r'c:\Users\ADMIN\Documents\Data_Science\vectors.csv'

# Read the CSV file contents
with open(file_path, 'r') as file:
    content = file.read()

# Use StringIO to treat the CSV content as a file-like object
csv_data = StringIO(content)

# Create a CSV reader to parse the data
reader = csv.DictReader(csv_data)

# Convert the CSV data into a list of dictionaries for easy processing
data = list(reader)

A new collection named "quotes" is created in Qdrant. If a collection with the same name already exists, it is deleted first to ensure a clean setup. The collection is configured to store 384-dimensional vectors and uses cosine distance for similarity search.

The script iterates through the CSV data to generate PointStruct objects for each record. Each point contains:

• ID: A unique identifier
• Vector: The embedding representation
• Payload: Metadata fields such as text, category, length, and sentiment.

At the same time, SearchRequest objects are created for each point to define filtering conditions based on category and sentiment. Finally, the prepared points are inserted into the Qdrant collection.

# Create a new collection
collection_name = "quotes"

# Delete the collection if it already exists
if client.get_collection(collection_name) is not None:
    client.delete_collection(collection_name)

# Create the collection
client.create_collection(
    collection_name=collection_name,
    vectors_config=models.VectorParams(size=384, distance=models.Distance.COSINE)
)

# Prepare points and search requests
points = []
search_requests = []

for item in data:
    vector = json.loads(item['vector'])
    points.append(
        models.PointStruct(
            id=int(item['id']),
            vector=vector,
            payload={
                "text": item['text'],
                "category": item['category'],
                "length": int(item['Length']),
                "sentiment": item['sentiment']
            }
        )
    )

    search_requests.append(
        models.SearchRequest(
            vector=vector,
            filter=models.Filter(
                must=[
                    models.FieldCondition(
                        key="category",
                        match=models.MatchValue(value=["inspirational"])
                    ),
                    models.FieldCondition(
                        key="sentiment",
                        match=models.MatchValue(value=item['sentiment'])
                    )
                ]
            ),
            with_payload=True,
            limit=2
        )
    )

# Insert the points into the collection
client.upsert(
    collection_name=collection_name,
    points=points
)

The multi-vector search is carried out using the client.search_batch() method. This approach applies exact match conditions to ensure the retrieved results align closely with the specified attributes of the data points. Each search request includes filters, such as category and sentiment, to match the original point's metadata.

Using search_batch enables the execution of multiple search requests in a single call, making it more efficient than submitting individual searches, especially when handling a large number of queries. For every request, you can define payload filters to narrow down results based on metadata like category, sentiment, or any other attributes associated with the vectors.

#Perform multi-vector search with payload filtering
search_result = client.search_batch(
    collection_name=collection_name,
    requests=search_requests
)

This method ensures efficient and precise multi-vector searches while leveraging payload metadata to refine results.

Once the batch search is completed, the results are merged, duplicates are removed, and the final output is sorted based on the initial similarity scores.

#Combine and deduplicate results
combined_results = {}
for batch in search_result:
    for hit in batch:
        if hit.id not in combined_results or hit.score > combined_results[hit.id].score:
            combined_results[hit.id] = hit

#Convert combined results to a list and sort by score
results = list(combined_results.values())
results.sort(key=lambda x: x.score, reverse=True)

This ensures that the returned results are unique and ordered by their highest similarity scores.

Rerank Results Using Payload Information

To refine the search output, you can implement a custom reranking function, rerank_score(), which adjusts the initial similarity score by incorporating metadata from the payload. This function combines the base similarity score with additional factors such as text length and sentiment.

# Rerank results using payload information
def rerank_score(hit):
    payload = hit.payload
    base_score = hit.score

    # Calculate adjustment based on text length
    length_score = 1 - abs(payload['length'] - 45) / 45  # 45 is considered the ideal length

    # Adjust based on sentiment
    sentiment_score = 1 if payload['sentiment'] == 'positive' else 0.5

    # Combine the base score with length and sentiment factors
    final_score = base_score * (0.5 + length_score * 0.25 + sentiment_score * 0.25)
    return final_score

Apply the reranking function to the search results and sort them using the updated scores:

# Apply reranking
reranked_results = sorted(results, key=rerank_score, reverse=True)

Finally, you can display the top-ranked results, showing both the original similarity scores and the adjusted reranked scores for comparison.

Display Top 2 Results After Reranking

The following code snippet prints the top two results after applying the reranking function. For each result, it displays the point ID, the associated text, the original similarity score, and the adjusted reranked score.

# Print top 2 results after reranking
for i, hit in enumerate(reranked_results[:2], 1):
    print(f"{i}. ID: {hit.id}, Text: {hit.payload['text']}, "
          f"Original Score: {hit.score:.4f}, "
          f"Reranked Score: {rerank_score(hit):.4f}")

1. ID: 2, Text: A journey of a thousand miles begins with a single step, Original Score: 1.0000, Reranked Score: 0.9500
2. ID: 5, Text: The only thing we have to fear is fear itself, Original Score: 1.0000, Reranked Score: 0.8694

Result

The search returned the top two results from the “Inspirational” category as specified by the filter conditions. Both entries initially had the same similarity score of 1.0000, indicating they were equally close to the query vectors based on vector similarity alone. After applying reranking, their scores diverged:

• "A journey of a thousand miles begins with a single step" received a reranked score of 0.9500.
• "The only thing we have to fear is fear itself" received a reranked score of 0.8694.

The reranking process incorporated additional payload attributes, such as text length and sentiment. The higher reranked score of the first result suggests it had a more favorable combination of these factors according to the rerank_score() function.

This illustrates how reranking can distinguish between results that initially appear equally relevant under vector similarity. By leveraging metadata, the ranking becomes more nuanced. The weights in the rerank_score() function can be adjusted to emphasize different factors for example, prioritizing sentiment over length or vice versa.

This approach combines the strengths of vector similarity search with structured payload information. The initial search uses vector embeddings and basic payload filters to find relevant matches, while the reranking stage fine-tunes the ordering based on additional metadata that can better reflect relevance or quality.

In this example, the reranking slightly favored the "A journey of a thousand miles" quote, possibly due to its length being closer to the ideal value or a more positive sentiment. This demonstrates how Qdrant supports advanced search workflows that merge semantic similarity with metadata-based custom ranking for more accurate and context-aware results.

Conclusion

This article explored Qdrant’s capabilities as a high-performance vector database designed for advanced AI applications. Key features such as Multi-Vector Search and Payload (Metadata) Based Reranking play a significant role in improving search functionality and boosting result accuracy.

By supporting detailed payload configurations and concurrent multi-vector searches, Qdrant addresses complex data scenarios where multiple representations are required for comprehensive analysis. The step-by-step walkthrough on setting up a Qdrant environment covering collection creation, payload definition, and filtering provides practical guidance on effectively leveraging Qdrant’s capabilities.

Overall, Qdrant offers a robust solution that not only streamlines AI-driven workflows but also creates an environment where search precision is enhanced by combining vector similarity with metadata insights.

*The code and dataset referenced in this tutorial can be found on my GitHub page.

Github