Learning DSPy with Ollama and Mistral-NeMo

In the realm of artificial intelligence, the ability to process and understand human language is paramount. One of the most promising advancements in this area is the emergence of large language models like Mistral NeMo, which excel at complex tasks such as question answering. This blog post will explore how to optimize the performance of a ReAct agent using Mistral NeMo in conjunction with Ollama and DSPy. For further insights into the evolving landscape of AI and the significance of frameworks like DSPy, check out our previous blog discussing the future of prompt engineering here.

What is Mistral NeMo?

Mistral NeMo is a state-of-the-art language model developed in partnership with NVIDIA. With 12 billion parameters, it offers impressive capabilities in reasoning, world knowledge, and coding accuracy. One of its standout features is its large context window, which can handle up to 128,000 tokens of text—this allows it to process and understand long passages, making it particularly useful for complex queries and dialogues (NVIDIA).

Key Features of Mistral NeMo

1. Large Context Window: This allows Mistral NeMo to analyze and respond to extensive texts, accommodating intricate questions and discussions.
2. State-of-the-Art Performance: The model excels in reasoning tasks, providing accurate and relevant answers.
3. Collaboration with NVIDIA: By leveraging NVIDIA’s advanced technology, Mistral NeMo incorporates optimizations that enhance its performance.

Challenges in Optimization

While Mistral NeMo is a powerful tool, there are challenges when it comes to optimizing and fine-tuning ReAct agents. One significant issue is that the current documentation does not provide clear guidelines on implementing few-shot learning techniques effectively. This can affect the adaptability and overall performance of the agent in real-world applications (Hugging Face).

What is a ReAct Agent?

Before diving deeper, let’s clarify what a ReAct agent is. ReAct, short for "Reasoning and Acting," refers to AI systems designed to interact with users by answering questions and performing tasks based on user input. These agents can be applied in various fields, from customer service to educational tools (OpenAI).

Integrating DSPy for Optimization

To overcome the challenges mentioned above, we can use DSPy, a framework specifically designed to optimize ReAct agents. Here are some of the key functionalities DSPy offers:

• Simulating Traces: This feature allows developers to inspect data and simulate traces through the program, helping to generate both good and bad examples.
• Refining Instructions: DSPy can propose or refine instructions based on performance feedback, making it easier to improve the agent’s effectiveness.

Setting Up a ReAct Agent with Mistral NeMo and DSPy

Now that we have a good understanding of Mistral NeMo and DSPy, let’s look at how to set up a simple ReAct agent using these technologies. Below, you’ll find a code example that illustrates how to initialize the Mistral NeMo model through Ollama and optimize it using DSPy.

Code Example

Here’s a sample code that Uses a dataset called HotPotQA and ColBertV2 a Dataset Retrieval model to test and optimise a ReAct Agent that is using mistral-nemo-latest as the llm

Step-by-Step Breakdown of the Code

1. Importing Libraries configuring Datasets:

First We will import DSpy libraries evaluate,datasets,teleprompt.
The first one is used to check the performance of a dspy agent.
The second one is used to load inbuilt datasets to evaluate the performance of the LLms
The third one is used as an optimisation framework for training and tuning the prompts that are provided to the LLMs

import dspy
from dspy.evaluate import Evaluate
from dspy.datasets.hotpotqa import HotPotQA
from dspy.teleprompt import BootstrapFewShotWithRandomSearch

ollama=dspy.OllamaLocal(model='mistral-nemo:latest')
colbert = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')
dspy.configure(lm=ollama, rm=colbert)

2. Loading some data:

We will now load the Data and segment to into training data, testing data and development data

dataset = HotPotQA(train_seed=1, train_size=200, eval_seed=2023, dev_size=300, test_size=0)
trainset = [x.with_inputs('question') for x in dataset.train[0:150]]
valset = [x.with_inputs('question') for x in dataset.train[150:200]]
devset = [x.with_inputs('question') for x in dataset.dev]

# show an example datapoint; it's just a question-answer pair
trainset[23]

3. Creating a ReAct Agent:

First we will make a default (Dumb 😂) ReAct agent

agent = dspy.ReAct("question -> answer", tools=[dspy.Retrieve(k=1)])

4. Evaluting the agent:

Set up an evaluator on the first 300 examples of the devset.

config = dict(num_threads=8, display_progress=True, display_table=25)
evaluate = Evaluate(devset=devset, metric=dspy.evaluate.answer_exact_match, **config)

evaluate(agent)

5. Optimizing the ReAct Agent:

Now we will (try to) put some brains into the dumb agent by training it

config = dict(max_bootstrapped_demos=2, max_labeled_demos=0, num_candidate_programs=5, num_threads=8)
tp = BootstrapFewShotWithRandomSearch(metric=dspy.evaluate.answer_exact_match, **config)
optimized_react = tp.compile(agent, trainset=trainset, valset=valset)

6. Testing the Agent:

Now we will check if the agents have become smart (enough)

evaluate(optimized_react)

Conclusion

Integrating MistralNeMo with Ollama and DSPy presents a powerful framework for developing and optimizing question-answering ReAct agents. By leveraging the model’s extensive capabilities, including its large context window tool calling capabilities and advanced reasoning skills, developers can create AI agents that efficiently handle complex queries with high accuracy in a local setting.

However, it’s essential to address the gaps in current documentation regarding optimization techniques for Local and opensource models and agents. By understanding these challenges and utilizing tools like DSPy, developers can significantly enhance the performance of their AI projects.

As AI continues to evolve, the integration of locally running models like Mistral NeMo will play a crucial role in creating intelligent systems capable of understanding and responding to human needs. With the right tools and strategies, developers can harness the full potential of these technologies, ultimately leading to more sophisticated and effective AI applications.

By following the guidance provided in this blog post, you can start creating your own optimized question-answering agents using Mistral NeMo, Ollama, and DSPy. Happy coding!

References

1. Creating ReAct AI Agents with Mistral-7B/Mixtral and Ollama using … Creating ReAct AI Agents with Mistral-7B/Mixtral a…

2. Mistral NeMo – Hacker News Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, …

3. Lack of Guidance on Optimizing/Finetuning ReAct Agent with Few … The current ReAct documentation lacks clear instructions on optimizing or fine…

4. Introducing Mistral NeMo – Medium Mistral NeMo is an advanced 12 billion parameter model developed in co…

5. Optimizing Multi-Agent Systems with Mistral Large, Nemo … – Zilliz Agents can handle complex tasks with minimal human intervention. Learn how to bu…

6. mistral-nemo – Ollama Mistral NeMo is a 12B model built in collaboration with NVIDIA. Mistra…

7. Mistral NeMo : THIS IS THE BEST LLM Right Now! (Fully … – YouTube … performance loss. Multilingual Support: The new Tekken t…

8. dspy/README.md at main · stanfordnlp/dspy – GitHub Current DSPy optimizers can inspect your data, simulate traces …

9. Is Prompt Engineering Dead? DSPy Says Yes! AI&U

   ---

   Your thoughts matter—share them with us on LinkedIn here.

   Want the latest updates? Visit AI&U for more in-depth articles now.