We now support VLMs in smolagents!

You hypocrite, first take the log out of your own eye, and then you will see clearly to take the speck out of your brother's eye. Matthew 7, 3-5

TL;DR

We have added vision support to smolagents, which unlocks the use of vision language models in agentic pipelines natively.

Table of Contents

• Overview
• How we gave sight to smolagents
• How to create a Web browsing agent with vision
• Next Steps

Overview

In the agentic world, many capabilities are hidden behind a vision wall. A common example is web browsing: web pages feature rich visual content that you never fully recover by simply extracting their text, be it the relative position of objects, messages transmitted through color, specific icons… In this case, vision is a real superpower for agents. So we just added this capability to our smolagents!

Teaser of what this gives: an agentic browser that navigates the web in complete autonomy!

Here's an example of what it looks like:

How we gave sight to smolagents

🤔 How do we want to pass images to agents? Passing an image can be done in two ways:

1. You can have images directly available to the agent at start. This is often the case for Document AI.
2. Sometimes, images need to be added dynamically. A good example is when a web browser just performed an action, and needs to see the impact on its viewports.

1. Pass images once at agent start

For the case where we want to pass images at once, we added the possibility to pass a list of images to the agent in the run method: agent.run("Describe these images:", images=[image_1, image_2]) .

These image inputs are then stored in the task_images attribute of TaskStep along with the prompt of the task that you'd like to accomplish.

When running the agent, they will be passed to the model. This comes in handy with cases like taking actions based on long PDFs that include visual elements.

2. Pass images at each step ⇒ use a callback

How to dynamically add images into the agent’s memory?

To find out, we first need to understand how our agents work.