Making sense of this mess

When I joined Hugging Face nearly 3 years ago, the Transformers documentation was very different from its current form today. It focused on text models and how to train or use them for inference on natural language tasks (text classification, summarization, language modeling, etc.).

The main version of the Transformers documentation today compared to version 4.10.0 from nearly 3 years ago.

As transformer models increasingly became the default way to approach AI, the documentation expanded significantly to include new models and new usage patterns. But new content was added incrementally without really considering how the audience and the Transformers library have evolved.

I think that's the reason why the documentation experience (DocX) feels disjointed, difficult to navigate, and outdated. Basically, a mess.

This is why a Transformers documentation redesign is necessary to make sense of this mess. The goal is to:

1. Write for developers interested in building products with AI.
2. Allow organic documentation structure and growth that scales naturally, instead of rigidly adhering to a predefined structure.
3. Create a more unified documentation experience by integrating content rather than amending it to the existing documentation.

A new audience

IMO companies that will understand that AI is not just APIs to integrate but a new paradigm to build all tech and who develop this muscle internally (aka build, train their own models) will be able to build 100x better than others and get the differentiation and long-term value

— clem 🤗 (@ClementDelangue) March 3, 2023

The Transformers documentation was initially written for machine learning engineers and researchers, model tinkerers.

Now that AI is more mainstream and mature, and not just a fad, developers are growing interested in learning how to build AI into products. This means realizing developers interact with documentation differently than machine learning engineers and researchers do.

Two key distinctions are: