Abstract

The AI Web Search Agent is a tool designed to automate data retrieval and extraction tasks by leveraging web search capabilities. Given a dataset (in CSV or Google Sheets format), the AI agent reads specific entities in a chosen column, performs web searches for each, and uses a language model (LLM) to parse relevant information based on user-defined queries. The structured output can then be viewed in a simple dashboard interface and downloaded for further analysis and also added to the google sheet.

Methodology

1. Dashboard for File Upload and Google Sheets Connection -
   ○ Allow users to upload a CSV file or connect to a Google Sheet directly for data input.
   ○ Display the available columns from the CSV or Google Sheet, allowing the user to select the main column {company name}
   ○ Show a preview of the uploaded data.
   ○ Integrated with the Google Sheets API for real-time Google Sheet access, allowing users to authenticate and pull data from their sheets.

2. Dynamic Query Input with Prompt Template
   ○Allow users to specify the type of information they want to retrieve for each entity in the main column.
   ○ A text input box where users can define a custom prompt, such as "Get me the email address of {company}"

3. Automated Web Search for Information Retrieval
   ○Perform web searches for each entity using the custom prompt and gather relevant web results.
   ○ For each entity in the selected column, agent conducts a web search (e.g., “Get me the email address of {company}”).
   ○ Gathered and stored search results (e.g., URLs, snippets) for each entity.
   ○ using SerpAPI as web scraping api
   ○ Implemented logic to handle rate limiting and avoid potential blocking.
   ○ Stored each entity's results in a structured format, ready for further processing by the LLM.

4. Passing Results to an LLM for Parsing and Information Extraction
   ○ Used an LLM to extract specific information based on the user-defined prompt and web results.
   ○ Sent each entity’s search results to the LLM, along with a backend prompt like “Extract the email address of {company} from the following web results.”. This prompt could be asked from the user as well.
   ○ Implemented LLM integration using GROQ API for processing the data.

5. Displaying and Storing the Extracted Information
   ○ Showed extracted data in a user-friendly format and provide an option to download the data.
   ○ Display the extracted data in a table format within the dashboard, organized by entity and extracted information.
   ○ Offer an option to download the results as a CSV or update a connected Google Sheet with the extracted information.
   ○ Provide a “Download CSV” button and an option to update the Google Sheet.

some additional features

1. created database with name search_results.db using sqlite to cache the search results
2. Provides users information about validation success and errors (if any) regarding failed API calls or LLM queries.
3. Added google sheet output intergation by making a new sheet in the link
4. We can ask for multiple info in the query template providing advanced query search

github code repo- https://github.com/manyac24/AI-Web-Search-Agent