Hugging Face + PyCharm

It’s a Tuesday morning. As a Transformers maintainer, I’m doing the same thing I do most weekday mornings: Opening PyCharm, loading up the Transformers codebase and gazing lovingly at the chat template documentation while ignoring the 50 user issues I was pinged on that day. But this time, something feels different:

Something is… wait! Computer! Enhance!
Is that..?

Those user issues are definitely not getting responses today. Let’s talk about the Hugging Face integration in PyCharm.

The Hugging Face Is Inside Your House

I could introduce this integration by just listing features, but that’s boring and there’s documentation for that. Instead, let’s walk through how we’d use it all in practice. Let’s say I’m writing a Python app, and I decide I want the app to be able to chat with users. Not just text chat, though – we want the users to be able to paste in images too, and for the app to naturally chat about them as well.

If you’re not super-familiar with the current state-of-the-art in machine learning, this might seem like a terrifying demand, but don’t fear. Simply right click in your code, and select “Insert HF Model”. You’ll get a dialog box:
Chatting with both images and text is called “image-text-to-text”: the user can supply images and text, and the model outputs text. Scroll down on the left until you find it. By default, the model list will be sorted by Likes – but remember, older models often have a lot of likes built up even if they’re not really the state of the art anymore. We can check how old models are by seeing the date they were last updated, just under the model name. Let’s pick something that’s both recent and popular: microsoft/Phi-3.5-vision-instruct.

These models can be large! If you’re getting memory errors, try using a GPU with more memory, or try reducing the 20 in the sample code. You can also remove device_map="cuda" to put the model in CPU memory instead, at the cost of speed.