Abstract

This project presents an AI-powered search engine developed using Streamlit, LangChain, and Groq to enhance information retrieval and interaction. Leveraging the LangChain framework for retrieval-augmented generation (RAG) and integrating Groq-powered LLM capabilities, the application dynamically pulls information from multiple online sources, including Arxiv, Wikipedia, and DuckDuckGo. The search engine can interpret user queries, retrieve real-time information, and respond interactively. The system pulls and processes content in response to user queries, effectively creating a “live dataset” from these sources for each session.

Methodology

1. Architecture
   The system architecture integrates multiple components:

• Frontend (Streamlit): Provides an interactive user interface where users can input queries, select data sources, and view AI responses in real-time.
• Backend (LangChain and ChatGroq): Responsible for processing the queries and integrating data from Wikipedia, DuckDuckGo, and Arxiv.
• LangChain’s AgentType.ZERO_SHOT_REACT_DESCRIPTION method enables efficient, context-aware retrieval across diverse datasets.
• Environment Configuration: Environment variables and API keys are securely managed through Python’s dotenv package for authenticated access to Groq’s language model.

2. Workflow

• User Input and Session Management: User queries are stored in session state using Streamlit’s st.session_state, allowing users to view the conversation history.
• Query Routing: Users select a data source, determining the relevant LangChain tool (Wikipedia, DuckDuckGo, or Arxiv) used for retrieval.
• Response Generation: The selected tool retrieves the requested data, which ChatGroq processes to produce a coherent, contextual response.
• Error Handling and Logging: The system includes robust error handling, catching issues like API connection errors or encoding issues in responses. These are handled gracefully, with feedback provided to users.

3. Technologies and Libraries

• Streamlit: Used to create the application’s web interface.
• LangChain: Provides wrappers and tools for integrating Wikipedia, DuckDuckGo, and Arxiv.
• ChatGroq: Groq’s LLM is used for generating natural language responses.
• Arxiv, WikipediaAPIWrapper: Tools specifically designed to interact with the Arxiv and Wikipedia APIs for streamlined search capabilities.

import os
import streamlit as st
from dotenv import load_dotenv
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_groq import ChatGroq  
from langchain_community.utilities import ArxivAPIWrapper, WikipediaAPIWrapper
from langchain_community.tools import ArxivQueryRun, WikipediaQueryRun, DuckDuckGoSearchRun
from langchain.agents import initialize_agent, AgentType
from langchain.callbacks import StreamlitCallbackHandler

# Load environment variables (if needed)
load_dotenv()

# Initializing the Tools
arxiv_wrapper = ArxivAPIWrapper(top_k_results=1, doc_content_chars_max=200)
arxiv = ArxivQueryRun(api_wrapper=arxiv_wrapper)

api_wrapper = WikipediaAPIWrapper(top_k_results=1, doc_content_chars_max=200)
wiki = WikipediaQueryRun(api_wrapper=api_wrapper)

search = DuckDuckGoSearchRun(name='search')

# Streamlit UI setup
st.title("Langchain - Search Engine")
st.markdown("""
In this example, we are using the 'StreamlitCallbackHandler' to display the thought and actions of an agent in an interactive Streamlit app.
Try more Streamlit Agent examples at [GitHub].
""")

# Sidebar for Groq API-KEY
st.sidebar.title("Settings")
api_key = st.sidebar.text_input("Enter your Groq API-Key:", type="password")
if st.sidebar.button("Submit API Key"):
    if api_key:
        st.sidebar.success("API Key submitted successfully!")
    else:
        st.sidebar.error("Please enter a valid API Key.")


# Initialize session state for messages
if "messages" not in st.session_state:
    st.session_state["messages"] = [
        {"role": "assistant", "content": "Hi, I am a chatbot who can search for you. How can I help you?"}
    ]

# Display previous messages
for msg in st.session_state.messages:
    st.chat_message(msg["role"]).write(msg['content'])

# User input and processing
if prompt := st.chat_input("What would you like to search for?"):
    st.session_state.messages.append({"role": "user", "content": prompt})
    st.chat_message("user").write(prompt)

    llm = ChatGroq(groq_api_key=api_key, model_name="Llama3-8b-8192", streaming=True)
    tools = [search, arxiv, wiki]

    search_agent = initialize_agent(tools, llm, AgentType.ZERO_SHOT_REACT_DESCRIPTION, handling_parsing_errors=True)

    with st.chat_message('assistant'):
        st_cb = StreamlitCallbackHandler(st.container(), expand_new_thoughts=False)
        try:
            response = search_agent.run(st.session_state.messages, callbacks=[st_cb])
            st.session_state.messages.append({"role": "assistant", "content": response})
            st.write("### Response:")
            st.write(response)
        except Exception as e:
            st.error(f"An error occurred: {str(e)}")
            st.session_state.messages.append({"role": "assistant", "content": "Sorry, I encountered an error while processing your request."})

# Add a footer for additional information# Add a footer with emojis for Langchain and Streamlit
st.markdown("---")
st.markdown("Made with 🛠️ Langchain and 📊 Streamlit")

Further Advancement

The current application can be extended in the following ways:

• Integration with More Data Sources: Additional APIs like PubMed for healthcare research or LexisNexis for legal cases can be incorporated.
• Enhanced Response Summarization: Incorporate NLP techniques to summarize long documents, providing users with concise responses without losing essential information.
• Fine-Tuned User Feedback Mechanism: A feedback loop can be implemented to improve response accuracy based on user feedback over time.
• Advanced User Profiles: Adding user profiles could enable personalized information retrieval, adapting results based on past queries or preferences.

Application in Different Industries

The application’s versatility allows it to be used across various industries:

1. Business
   In business intelligence, this tool can help teams analyze current trends, competitive analysis, and emerging market data. Integrating DuckDuckGo and Wikipedia provides insights into business competitors and global events impacting industries.

2 Healthcare
By incorporating databases like Arxiv and possibly extending to PubMed, the tool can serve as a valuable resource for healthcare professionals, researchers, and medical students to quickly access scientific studies, reviews, and updates on new treatments and technologies.

3 Education and Research
Educational institutions can deploy this tool for students and researchers who need quick access to scientific literature, historical data, or current events. This project can be adapted to function as a research assistant, helping students locate primary sources or recent studies in seconds.

4 Legal and Compliance
With the addition of industry-specific databases, this application could assist legal professionals in accessing case laws, statutes, and regulatory documents, aiding in case preparation and compliance management.