Ebiose: evolving multi-LLM agent systems

November 2023 · Hackathon (Inria Hackatech) · Team of 4

LLMsEvolutionary AlgorithmsMulti-Agent Systems

Introduction

Ebiose was born during a 2-day hackathon at Inria Bordeaux, and later turned into a startup: Ebiose. The idea: solve tasks with a small system of LLM agents, each agent being an LLM (here GPT-3.5) paired with a specific prompt. The system follows a fixed sequence: one agent answers a question, another critiques the answer, and a last one synthesizes both. This setup is inspired by the AutoGen framework. We call such a group of agents an Individual.

To improve these Individuals, we use an evolutionary algorithm on their prompts, based on EvoPrompt. We create a population of Individuals, evaluate them on a dataset to get their fitness, and use the best ones as parents to generate offspring through crossovers and mutations of their prompts. These text operations are themselves performed by an LLM, instructed to fuse and modify prompts.

Method

We tested the method on GSM8K (grade-school math questions), evolving a population of 25 Individuals for 20 iterations. Each Individual has three GPT-3.5 agents: actor, critic and synthesis. Initial prompts were generated with GPT-4, for example:

Actor: "Generate 25 different prompts optimized for solving mathematical problems that I can apply to solve any math problem."

Synthesis: "Continue this list of example prompts: Analyze the proposed solution and its critique to obtain a better solution, Summarize the results provided in a synthetic manner."

To evaluate an Individual on a question, the actor answers, the critic assesses the answer, and the synthesis agent reads the whole conversation to produce the final answer (plus a formatting prompt to extract it). Fitness is the accuracy over the dataset.

Parents are selected based on fitness, and their prompts are mutated and crossed with EvoPrompt (using GPT-4 for the evolution step, GPT-3.5 for answering). We applied the operators pairwise between two parents: Actor 1 × Actor 2, Critic 1 × Critic 2, Synthesis 1 × Synthesis 2.

Example of an EvoPrompt crossover and mutation prompt — Example of prompt crossover + mutation with EvoPrompt.

The population size stays fixed at 25: after evaluating the offspring, only the top 25 Individuals across parents and offspring are kept. This loop (evaluate, select, create offspring, update) is repeated for the desired number of iterations.

Results

Box plots of population scores over 20 iterations, rising from about 0.75 to 0.83 median — Distribution of population accuracy over evolution iterations.

Prompts of the best Individual on the train set:

Actor: "Delve into this mathematical puzzle by merging the study of series or sequences with a strategic and theoretical approach. Explore the depths of the problem's complexity by examining the role of logical sequences and recurring patterns, and how they shape the solutions in diverse situations."

Critic: "Investigate different problem-solving approaches with an emphasis on retaining critical nuances without sacrificing speed, aiming to find a balanced method that offers thorough, expeditious resolutions to complicated problems."

Synthesis: "Develop a refined strategy for tackling mathematical challenges by iteratively improving the problem-solving process, utilizing feedback and critical analysis from earlier attempts to ensure a clear and superior resolution is achieved."

Bar chart of train and test accuracy for GPT-3.5, the 25 evolved individuals, and GPT-4 — Train/test accuracy of the final population vs. GPT-3.5 and GPT-4 on GSM8K.

The evolved systems clearly improve over plain GPT-3.5 (~0.55 → ~0.8 test accuracy), and the best Individual gets close to GPT-4, a much larger model.