Snorkel AI x Hugging Face: unlock foundation models for enterprises

This article is a cross-post from an originally published post on April 6, 2023 in Snorkel's blog, by Friea Berg .

As OpenAI releases GPT-4 and Google debuts Bard in beta, enterprises around the world are excited to leverage the power of foundation models. As that excitement builds, so does the realization that most companies and organizations are not equipped to properly take advantage of foundation models.

Foundation models pose a unique set of challenges for enterprises. Their larger-than-ever size makes them difficult and expensive for companies to host themselves, and using off-the-shelf FMs for production use cases could mean poor performance or substantial governance and compliance risks.

Snorkel AI bridges the gap between foundation models and practical enterprise use cases and has yielded impressive results for AI innovators like Pixability. We’re teaming with Hugging Face, best known for its enormous repository of ready-to-use open-source models, to provide enterprises with even more flexibility and choice as they develop AI applications.

Foundation models in Snorkel Flow

The Snorkel Flow development platform enables users to adapt foundation models for their specific use cases. Application development begins by inspecting the predictions of a selected foundation model “out of the box” on their data. These predictions become an initial version of training labels for those data points. Snorkel Flow helps users to identify error modes in that model and correct them efficiently via programmatic labeling, which can include updating training labels with heuristics or prompts. The base foundation model can then be fine-tuned on the updated labels and evaluated once again, with this iterative “detect and correct” process continuing until the adapted foundation model is sufficiently high quality to deploy.

How does Hugging Face help?

– Snorkel CTO and Co-founder Braden Hancock

How does this help Snorkel customers?

Clement Delangue, co-founder and CEO, Hugging Face