Repolode
Submit
ServiceNow

ServiceNow/Fast-LLM

Open-source library for training large language models with 3D parallelism, ZeRO memory optimization and Hugging Face integration, built on PyTorch and Triton.

View on GitHubWebsite
Stars
337
Forks
53
Open issues
56
Last commit
6 days ago

What is it?

Fast-LLM is an open-source library for training large language models, built on PyTorch and Triton. It targets models from about 1B parameters up to clusters training 70B+ parameter models, and it runs from a single GPU to multi-node clusters managed by Slurm or Kubernetes with KubeFlow.

The library supports distributed training with data, tensor and pipeline parallelism, sequence length parallelism, and ZeRO-1, ZeRO-2 and ZeRO-3 for memory efficiency. Models are configured through YAML files and launched from a command-line interface. It also includes a dropless Mixture-of-Experts implementation and integrates with Hugging Face Transformers.

The project is licensed under Apache 2.0, and pre-built Docker images are published for quick deployment. Its documentation and examples cover the Mistral-7B training setup used in the README.

Key features

  • 01Distributed training with data, tensor, and pipeline (3D) parallelism.
  • 02ZeRO-1, ZeRO-2, and ZeRO-3 memory optimization.
  • 03Dropless Mixture-of-Experts (MoE) implementation.
  • 04Mixed precision training and gradient accumulation.
  • 05YAML-based configuration with a command-line interface.
  • 06Integration with Hugging Face Transformers.

Use cases

  • Pretraining a Mistral-style or GPT-like model on a multi-node GPU cluster managed by Slurm.
  • Running large-scale LLM training jobs on Kubernetes with KubeFlow PyTorchJob.
  • Training custom MoE models with an open-source, Apache 2.0 licensed codebase.

Quick start

  1. 1

    Deploy the nvcr.io/nvidia/pytorch:25.11-py3 Docker image to all nodes (recommended)

  2. 2

    Install Fast-LLM on all nodes with pip

    pip install --no-cache-dir "fast-llm[CORE,OPTIONAL] @ git+https://github.com/ServiceNow/Fast-LLM.git"
  3. 3

    Submit the example Slurm job script to the cluster

    sbatch examples/fast-llm.sbat

Things to know

  • Training at the scale shown needs a multi-node GPU cluster, such as 4 nodes with 8 A100 or H100 GPUs each.
  • It requires CUDA 12.1 or higher plus PyTorch, Triton and Apex installed on every node.
  • The README lists some features such as customizable optimizers and tutorials as still in progress.

Topics

#llm-training#pytorch#distributed-training#deep-learning#model-parallelism#zero-optimizer#mixture-of-experts

FAQ

What is Fast-LLM?
Open-source library for training large language models with 3D parallelism, ZeRO memory optimization and Hugging Face integration, built on PyTorch and Triton. Fast-LLM is an open-source library for training large language models, built on PyTorch and Triton. It targets models from about 1B parameters up to clusters training 70B+ parameter models, and it runs from a single GPU to multi-node clusters managed by Slurm or Kubernetes with KubeFlow.
Is Fast-LLM free to use? What license does it have?
Fast-LLM is open source under the Other license. Check the license terms before using it in commercial products.
Is Fast-LLM still maintained?
Actively maintained. The last commit to Fast-LLM was on Oct 15, 2026.
What can I use Fast-LLM for?
Pretraining a Mistral-style or GPT-like model on a multi-node GPU cluster managed by Slurm. Running large-scale LLM training jobs on Kubernetes with KubeFlow PyTorchJob. Training custom MoE models with an open-source, Apache 2.0 licensed codebase.
How do I install Fast-LLM?
Deploy the nvcr.io/nvidia/pytorch:25.11-py3 Docker image to all nodes (recommended) Install Fast-LLM on all nodes with pip: pip install --no-cache-dir "fast-llm[CORE,OPTIONAL] @ git+https://github.com/ServiceNow/Fast-LLM.git" Submit the example Slurm job script to the cluster: sbatch examples/fast-llm.sbat
Where can I get Fast-LLM?
The source code and installation instructions are on GitHub: https://github.com/ServiceNow/Fast-LLM

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects