Introduction to ggml

ggml is a machine learning (ML) library written in C and C++ with a focus on Transformer inference. The project is open-source and is being actively developed by a growing community. ggml is similar to ML libraries such as PyTorch and TensorFlow, though it is still in its early stages of development and some of its fundamentals are still changing rapidly.

Over time, ggml has gained popularity alongside other projects like llama.cpp and whisper.cpp. Many other projects also use ggml under the hood to enable on-device LLM, including ollama, jan, LM Studio, GPT4All.

The main reasons people choose to use ggml over other libraries are:

1. Minimalism: The core library is self-contained in less than 5 files. While you may want to include additional files for GPU support, it's optional.
2. Easy compilation: You don't need fancy build tools. Without GPU support, you only need GCC or Clang!
3. Lightweight: The compiled binary size is less than 1MB, which is tiny compared to PyTorch (which usually takes hundreds of MB).
4. Good compatibility: It supports many types of hardware, including x86_64, ARM, Apple Silicon, CUDA, etc.
5. Support for quantized tensors: Tensors can be quantized to save memory (similar to JPEG compression) and in certain cases to improve performance.
6. Extremely memory efficient: Overhead for storing tensors and performing computations is minimal.

However, ggml also comes with some disadvantages that you need to keep in mind when using it (this list may change in future versions of ggml):

• Not all tensor operations are supported on all backends. For example, some may work on CPU but won't work on CUDA.
• Development with ggml may not be straightforward and may require deep knowledge of low-level programming.
• The project is in active development, so breaking changes are expected.

Getting started

Great, so how do you start?

Expected output:

mul mat (4 x 3) (transposed result):
[ 60.00 55.00 50.00 110.00
 90.00 54.00 54.00 126.00
 42.00 29.00 28.00 64.00 ]

Terminology and concepts

Run it: