Code Llama: Llama 2 learns to code

Introduction

Code Llama is a family of state-of-the-art, open-access versions of Llama 2 specialized on code tasks, and we’re excited to release integration in the Hugging Face ecosystem! Code Llama has been released with the same permissive community license as Llama 2 and is available for commercial use.

Today, we’re excited to release:

• Models on the Hub with their model cards and license
• Transformers integration
• Integration with Text Generation Inference for fast and efficient production-ready inference
• Integration with Inference Endpoints
• Integration with VS Code extension
• Code benchmarks

Code LLMs are an exciting development for software engineers because they can boost productivity through code completion in IDEs, take care of repetitive or annoying tasks like writing docstrings, or create unit tests.

Table of Contents

• Introduction
• Table of Contents
• What’s Code Llama?
• How to use Code Llama?
  • Demo
  • Transformers
    • A Note on dtypes
    • Code Completion
    • Code Infilling
    • Conversational Instructions
    • 4-bit Loading
  • Using text-generation-inference and Inference Endpoints
  • Using VS Code extension
• Evaluation
• Additional Resources

What’s Code Llama?

The Code Llama release introduces a family of models of 7, 13, and 34 billion parameters. The base models are initialized from Llama 2 and then trained on 500 billion tokens of code data. Meta fine-tuned those base models for two different flavors: a Python specialist (100 billion additional tokens) and an instruction fine-tuned version, which can understand natural language instructions.

The models show state-of-the-art performance in Python, C++, Java, PHP, C#, TypeScript, and Bash. The 7B and 13B base and instruct variants support infilling based on surrounding content, making them ideal for use as code assistants.

Code Llama was trained on a 16k context window. In addition, the three model variants had additional long-context fine-tuning, allowing them to manage a context window of up to 100,000 tokens.