Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face

Intel and Hugging Face collaborated to demonstrate the real-world value of upgrading to Google’s latest C4 Virtual Machine (VM) running on Intel® Xeon® 6 processors (codenamed Granite Rapids (GNR)). We specifically wanted to benchmark improvements in the text generation performance of OpenAI GPT OSS Large Language Model(LLM).

The results are in, and they are impressive, demonstrating a 1.7x improvement in Total Cost of Ownership(TCO) over the previous-generation Google C3 VM instances. The Google Cloud C4 VM instance further resulted in:

• 1.4x to 1.7x TPOT throughput/vCPU/dollar
• Lower price per hour over C3 VM

Introduction

GPT OSS is a common name for an open-source Mixture of Experts (MoE) model released by OpenAI. An MoE model is a deep neural network architecture that uses specialized “expert” sub-networks and a “gating network” to decide which experts to use for a given input. MoE models allow you to scale your model capacity efficiently without linearly scaling compute costs. They also allow for specialization, where different “experts” learn different skills, allowing them to adapt to diverse data distributions.

Even with very large parameters, only a small subset of experts is activated per token, making CPU inference viable.

Intel and Hugging Face collaborated to merge an expert execution optimization (PR #40304) to eliminate redundant computation where every expert processed all tokens to transformers. This optimization directed each expert to run only on the tokens it is routed to, removing FLOPs waste and improving utilization.

Benchmark Scope & Hardware

Configuration Summary

• Model: unsloth/gpt-oss-120b-BF16
• Precision: bfloat16
• Task: Text generation
• Input length: 1024 tokens (left‑padded)
• Output length: 1024 tokens
• Batch sizes: 1, 2, 4, 8, 16, 32, 64
• Enabled features:
  • Static KV cache
  • SDPA attention backend
• Reported metrics:
  • Throughput (Total generated tokens per second aggregated over the batch)