Simple considerations for simple people building fancy neural networks

As machine learning continues penetrating all aspects of the industry, neural networks have never been so hyped. For instance, models like GPT-3 have been all over social media in the past few weeks and continue to make headlines outside of tech news outlets with fear-mongering titles.

An article from The Guardian

At the same time, deep learning frameworks, tools, and specialized libraries democratize machine learning research by making state-of-the-art research easier to use than ever. It is quite common to see these almost-magical/plug-and-play 5 lines of code that promise (near) state-of-the-art results. Working at Hugging Face 🤗, I admit that I am partially guilty of that. 😅 It can give an inexperienced user the misleading impression that neural networks are now a mature technology while in fact, the field is in constant development.

In reality, building and training neural networks can often be an extremely frustrating experience:

• It is sometimes hard to understand if your performance comes from a bug in your model/code or is simply limited by your model’s expressiveness.
• You can make tons of tiny mistakes at every step of the process without realizing at first, and your model will still train and give a decent performance.

In this post, I will try to highlight a few steps of my mental process when it comes to building and debugging neural networks. By “debugging”, I mean making sure you align what you have built and what you have in mind. I will also point out things you can look at when you are not sure what your next step should be by listing the typical questions I ask myself.

A lot of these thoughts stem from my experience doing research in natural language processing but most of these principles can be applied to other fields of machine learning.

1. 🙈 Start by putting machine learning aside

2. 📚 Continue as if you just started machine learning

3. 🦸‍♀️ Don’t be afraid to look under the hood of these 5-liners templates

Some common errors include: