Releasing Outlines-core 0.1.0: structured generation in Rust and Python

dottxt and Hugging Face are excited to announce that we have been collaborating on outlines-core, a Rust port of outlines’s core algorithms for structured generation. On top of getting reliable output from LLMs with outlines, this Rust port offers several further benefits to users of outlines:

• Speed: Users can expect to see an 2x improvement in index compilation.
• Separation of Concerns: It's now easier to incorporate structured generation into other libraries. outlines-core is very lightweight.
• Portability: Having core algorithms in Rust allows binding for languages other than Python.

These improvements should not only improve the performance for existing outlines users, but also dramatically increase the ways users can incorporate structured generation into their LLM workflows. outlines-core is now public, integrated in outlines, and the version 0.1.0 of the Python bindings are out. You can find the repo here.

A quick primer on structured generation 🧑‍🎓

How it works

Structured generation means that your LLM is guaranteed to follow a desired format. This could be JSON, a Pydantic Model, a regular expression or a context-free grammar. The key is that structured generation forbids the 'wrong' tokens from being generated.

Let’s take an extremely simple example. The LLM should generate a boolean, “true” or “false”. And nothing more. For the sake of illustration, let’s say that LLMs generate characters instead of tokens. So the first character is ", we can just skip the forward pass. For the second, we don’t need to sample from all possible characters. The LLM should just choose between t or f.

Why it’s important

These are just a few examples of what structured generation enables:

• Generating synthetic data, (there's also an integration with Distilabel for this)
• Extracting information from documents and images.
• Function calling/building agents
• Chain of Thought
• Making sure your LLM outputs a valid tic-tac-toe board
• Or ever generating virtual worlds!