In the paper "AgentRec: Agent Recommendation Using Sentence Embeddings Aligned to Human Feedback," we introduce a novel architecture designed to recommend the most suitable large language model (LLM) agent for a given task based on a natural language prompt. This system extends the Sentence-BERT (SBERT) encoder model to classify prompts efficiently and accurately.

By doing this, we are the first to find a knowledge representation that rigorously encodes the capability of a given LLM agent, which is necessary to improve the safety of LLM applications by finding ways to ensure LLMs do not confidently answer prompts they are provably unable to provide an answer to.

Furthermore, we provide a framework for visualizing the capabilities of LLM agents as these encodings are locations in a 768 dimensional space that can be projected into a 3-dimensional space through methods such as t-SNE or PCA.

For a more rigorous write-up, see https://arxiv.org/abs/2501.13333

Key Contributions:

• Efficient Agent Selection: AgentRec encodes natural language prompts into sentence embeddings, capturing semantic content pertinent to agent selection. The model minimizes the distance between embeddings corresponding to the same agent through fine-tuning and aligns them with human values via reinforcement learning from human feedback. This approach enables prompt classification based on nearest neighbors by measuring cosine similarity between embeddings.
• Synthetic Dataset Generation: Due to the scarcity of real-world data, the authors generated a synthetic dataset using the Llama-3.1-8B-Instruct model. They employed techniques like top-k sampling and nucleus sampling to ensure the generation of coherent and diverse prompts, addressing challenges related to repetitiveness in language model outputs.
• Performance Metrics: On test data, AgentRec achieved a top-1 accuracy of 92.2%, with each classification taking less than 300 milliseconds. This demonstrates the model's efficiency and effectiveness in real-time agent recommendation scenarios.
• Open-Source Availability: We have open-sourced both the synthetic dataset and the code for the AgentRec recommendation system, available at https://github.com/joshprk/agentrec.

This work addresses the challenge of selecting appropriate agents in multi-agent systems by leveraging sentence embeddings aligned with human feedback, offering a scalable and interpretable solution for agent recommendation tasks.