Three Mighty Alerts Supporting Hugging Face’s Production Infrastructure

The Infrastructure team at Hugging Face is excited to share a behind-the-scenes look at the inner workings of Hugging Face's production infrastructure, which we’ve had the privilege of helping to build and maintain. Our team's dedication to designing and implementing a robust monitoring and alerting system has been instrumental in ensuring the stability and scalability of our platforms. We’re constantly reminded of the impact that our alerts have on our ability to identify and respond to potential issues before they become major incidents.

In this blog post, we’ll dive into the details of three mighty alerts that play their unique role in supporting our production infrastructure, and explore how they've helped us maintain the high level of performance and uptime that our community relies on.

High NAT Gateway Throughput

Oftentimes, with cloud computing architectures, where data flows between private and public networks, implementing a NAT (Network Address Translation) gateway stands as a steadfast best practice. This gateway acts as a strategic gatekeeper, monitoring and facilitating all outbound traffic towards the public Internet. By centralizing egress traffic, the NAT gateway offers a strategic vantage point for comprehensive visibility. Our team can easily query and analyze this traffic, making it an invaluable asset when working through security, cost optimization or various other investigative scenarios.

For instance, when integrating third-party security and autoscaling tools, we've observed increased telemetry data egress from our nodes, triggering the alert and prompting us to optimize our configurations.

As a final note, while we have configuration-as-code ensuring the desired state is always in effect, having an additional layer of alerting around helps in case mistakes are made when expressing the desired state through code.