Repolode
Submit
georgian-io

georgian-io/LLM-Finetuning-Toolkit

A config-driven CLI for running and comparing LLM fine-tuning experiments across prompts, open-source models, and LoRA settings, with built-in output tests.

View on GitHub
Stars
870
Forks
107
Open issues
16
Last commit
15 days ago

What is it?

LLM Finetuning Toolkit is a CLI tool that drives a series of LLM fine-tuning experiments from a single YAML file. The config controls the prompt template, the base model, the optimization setup (such as LoRA rank, dropout, and 4-bit quantization through bitsandbytes), and the post-training quality checks. Any open-source model available on Hugging Face can be specified, and datasets can come from Hugging Face or local JSON and CSV files.

The main use is ablation studies. Lists of prompt templates, model checkpoints, or LoRA hyperparameters can be placed in the config, and the toolkit runs each combination. Each configuration is hashed, so an interrupted run resumes from the cached dataset instead of starting over. Outputs are saved in separate directories for the dataset, model weights, prediction results, and QA test results.

The project includes built-in tests such as a length test and a word-overlap test, which check whether generated text behaves as expected. It can be installed with pipx or pip as the package llm-toolkit.

Key features

  • 01Defines prompts, models, LoRA settings, and QA tests in one YAML file.
  • 02Runs ablation studies over lists of prompt templates, model checkpoints, and LoRA parameters.
  • 03Loads datasets from Hugging Face, JSON, or CSV files.
  • 04Supports 4-bit quantization and Flash Attention 2 for supported models.
  • 05Caches results by configuration hash so interrupted runs can resume.
  • 06Includes built-in QA tests such as length and word-overlap checks.

Use cases

  • Comparing several open-source base models on the same fine-tuning dataset.
  • Testing which prompt template or LoRA configuration gives the best results for a task.
  • Checking that a summarization model's outputs are shorter than their inputs.
  • Running reproducible fine-tuning experiments from a versioned config file.

Quick start

  1. 1

    Install the toolkit with pipx (recommended)

    pipx install llm-toolkit
  2. 2

    Generate a starter config file

    llmtune generate config
  3. 3

    Run the fine-tuning experiment with the config

    llmtune run ./config.yml

Things to know

  • Flash Attention 2 needs a separate manual install step and a compatible GPU.
  • Fine-tuning models such as Llama-2 7B is resource-intensive and depends on Hugging Face model access.

In collections

Testing frameworks & test automation toolsNatural language processing tools & libraries

Topics

#fine-tuning#llm#lora#qlora#ablation-study#nlp#hugging-face#testing#classification#falcon

FAQ

What is LLM-Finetuning-Toolkit?
A config-driven CLI for running and comparing LLM fine-tuning experiments across prompts, open-source models, and LoRA settings, with built-in output tests. LLM Finetuning Toolkit is a CLI tool that drives a series of LLM fine-tuning experiments from a single YAML file. The config controls the prompt template, the base model, the optimization setup (such as LoRA rank, dropout, and 4-bit quantization through bitsandbytes), and the post-training quality checks. Any open-source model available on Hugging Face can be specified, and datasets can come from Hugging Face or local JSON and CSV files.
Is LLM-Finetuning-Toolkit free to use? What license does it have?
LLM-Finetuning-Toolkit is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is LLM-Finetuning-Toolkit still maintained?
Actively maintained. The last commit to LLM-Finetuning-Toolkit was on Oct 22, 2026.
What can I use LLM-Finetuning-Toolkit for?
Comparing several open-source base models on the same fine-tuning dataset. Testing which prompt template or LoRA configuration gives the best results for a task. Checking that a summarization model's outputs are shorter than their inputs. Running reproducible fine-tuning experiments from a versioned config file.
How do I install LLM-Finetuning-Toolkit?
Install the toolkit with pipx (recommended): pipx install llm-toolkit Generate a starter config file: llmtune generate config Run the fine-tuning experiment with the config: llmtune run ./config.yml
Where can I get LLM-Finetuning-Toolkit?
The source code and installation instructions are on GitHub: https://github.com/georgian-io/LLM-Finetuning-Toolkit

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects