Repolode
Submit
arcee-ai

arcee-ai/mergekit

A toolkit for merging pre-trained large language models by combining their weights with many merge algorithms, run on CPU or GPU.

View on GitHub
Stars
7,439
Forks
803
Open issues
276
Last commit
7 days ago

What is it?

mergekit merges the weights of pre-trained language models into a single model without additional training. It is built to work out-of-core, loading tensors lazily so that complex merges can run on CPU or on a GPU with as little as 8 GB of VRAM.

Merges are described in YAML configuration files. A config picks a merge method, selects source models or layer slices, and sets per-tensor parameters such as weights and densities. The toolkit supports a broad set of methods, including linear, SLERP, TIES, DARE, DELLA, and Model Stock, along with layer-stacking (passthrough) merges, LoRA extraction, mixture-of-experts assembly, and tokenizer alignment across models with different vocabularies.

Beyond the main `mergekit-yaml` command, the project includes tools for multi-stage merge workflows, merging raw PyTorch checkpoints, and transplanting tokenizers, plus an in-memory Python API for applying merge methods to tensors or state dicts.

Key features

  • 01Many merge methods, including linear, SLERP, TIES, DARE, DELLA, and Model Stock
  • 02Lazy tensor loading that runs on CPU or on GPUs with about 8 GB VRAM
  • 03YAML-driven configuration with per-tensor parameter control
  • 04Layer-stacking (passthrough) merges and mixture-of-experts assembly
  • 05LoRA extraction into PEFT-compatible adapters
  • 06Tokenizer alignment for models with different vocabularies

Use cases

  • Combining several fine-tuned checkpoints of the same base model into one versatile model
  • Building experimental frankenmerges by stacking layers from different models
  • Producing draft models for speculative decoding via tokenizer transplantation
  • Extracting LoRA adapters from finetuned models relative to their base

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/arcee-ai/mergekit.git
  2. 2

    Enter the project directory

    cd mergekit
  3. 3

    Install the package in editable mode

    pip install -e .
  4. 4

    Run a merge with your YAML configuration file

    mergekit-yaml path/to/your/config.yml ./output-model-directory

Things to know

  • Licensed under LGPL-3.0, a copyleft license whose terms apply to modifications of the library code.
  • The README gives only a source install with pip install -e ., with no published package install shown.

Topics

#llm#model-merging#pytorch#hugging-face#fine-tuning#lora#machine-learning#llama

FAQ

What is mergekit?
A toolkit for merging pre-trained large language models by combining their weights with many merge algorithms, run on CPU or GPU. mergekit merges the weights of pre-trained language models into a single model without additional training. It is built to work out-of-core, loading tensors lazily so that complex merges can run on CPU or on a GPU with as little as 8 GB of VRAM.
Is mergekit free to use? What license does it have?
mergekit is open source under the LGPL-3.0 license. Check the license terms before using it in commercial products.
Is mergekit still maintained?
Actively maintained. The last commit to mergekit was on Oct 31, 2026.
What can I use mergekit for?
Combining several fine-tuned checkpoints of the same base model into one versatile model Building experimental frankenmerges by stacking layers from different models Producing draft models for speculative decoding via tokenizer transplantation Extracting LoRA adapters from finetuned models relative to their base
How do I install mergekit?
Clone the repository: git clone https://github.com/arcee-ai/mergekit.git Enter the project directory: cd mergekit Install the package in editable mode: pip install -e . Run a merge with your YAML configuration file: mergekit-yaml path/to/your/config.yml ./output-model-directory
Where can I get mergekit?
The source code and installation instructions are on GitHub: https://github.com/arcee-ai/mergekit

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects