Deep Learning with Proteins

I have two audiences in mind while writing this. One is biologists who are trying to get into machine learning, and the other is machine learners who are trying to get into biology. If you’re not familiar with either biology or machine learning then you’re still welcome to come along, but you might find it a bit confusing at times! And if you’re already familiar with both, then you probably don’t need this post at all - you can just skip straight to our example notebooks to see these models in action:

• Fine-tuning protein language models (PyTorch, TensorFlow)
• Protein folding with ESMFold (PyTorch only for now because of openfold dependencies)

Introduction for biologists: What the hell is a language model?

The models used to handle proteins are heavily inspired by large language models like BERT and GPT. So to understand how these models work we’re going to go back in time to 2016 or so, before they existed. Donald Trump hasn’t been elected yet, Brexit hasn’t yet happened, and Deep Learning (DL) is the hot new technique that’s breaking new records every day. The key to DL’s success is that it uses artificial neural networks to learn complex patterns in data. DL has one critical problem, though - it needs a lot of data to work well, and on many tasks that data just isn’t available.

Let’s say that you want to train a DL model to take a sentence in English as input and decide if it’s grammatically correct or not. So you assemble your training data, and it looks something like this:

Text	Label
The judge told the jurors to think carefully.	Correct
The judge told that the jurors to think carefully.	Incorrect
…	…

To understand just how difficult that is, pretend you’re a machine learning model and I’m giving you some training data for a task I want you to learn. Here it is:

Text	Label
Is í an stiúrthóir is fearr ar domhan!	1
Is fuath liom an scannán seo.	0
Scannán den scoth ab ea é.	1
D’fhág mé an phictiúrlann tar éis fiche nóiméad!	0