Repolode
Submit
run-llama

run-llama/semtools

Command-line tools for parsing documents into markdown, running local semantic keyword search, and asking questions over files with an AI agent.

View on GitHub
Stars
1,878
Forks
153
Open issues
6
Last commit
7 months ago

What is it?

SemTools is a Rust command-line toolkit with four subcommands: parse, search, ask, and workspace. The parse command converts documents such as PDF and DOCX into markdown, using the LlamaParse API by default. The search command runs fully local semantic keyword matching with multilingual model2vec embeddings, returning matching lines with surrounding context and a configurable distance threshold.

The ask command runs an AI agent that uses search and read tools to answer questions over a set of files. It defaults to OpenAI models and can be pointed at any OpenAI-compatible endpoint through the config file. The workspace command stores computed embeddings under ~/.semtools/workspaces/, so repeated searches over large collections re-use cached results and changed files are re-embedded automatically.

All tools follow Unix conventions, accepting stdin and writing to stdout, so they can be chained with grep, xargs, and find. The parse and ask features require API keys, while search and workspace work offline.

Key features

  • 01Parses PDF, DOCX, PPTX and other documents into markdown via LlamaParse.
  • 02Runs local semantic search with multilingual model2vec embeddings and per-line context.
  • 03Lets an AI agent answer questions over files, with any OpenAI-compatible API supported.
  • 04Caches embeddings in workspaces and re-embeds files when they change.
  • 05Works with stdin and stdout for use in shell pipelines.

Use cases

  • Searching a folder of text or converted reports for concepts rather than exact keywords.
  • Converting PDFs to markdown as a preprocessing step before search or LLM use.
  • Asking questions over a set of research papers or notes from the terminal.
  • Feeding parsed document content into coding agents through pipelines.

Quick start

  1. 1

    Install semtools via npm

    npm i -g @llamaindex/semtools
  2. 2

    Run a local keyword search over text files

    semtools search "some keywords" *.txt --max-distance 0.3 --n-lines 5
  3. 3

    Parse documents into markdown (requires a LlamaIndex Cloud API key)

    semtools parse my_dir/*.pdf

Things to know

  • The parse and ask commands need external API keys: LlamaParse for parsing and OpenAI for ask.
  • Installing via npm builds Rust binaries locally if no prebuilt binary is available, so Rust and Cargo may be needed.

Topics

#semantic-search#cli#document-parsing#embeddings#rust#rag#pdf-to-markdown#parser#search#semantic

FAQ

What is semtools?
Command-line tools for parsing documents into markdown, running local semantic keyword search, and asking questions over files with an AI agent. SemTools is a Rust command-line toolkit with four subcommands: parse, search, ask, and workspace. The parse command converts documents such as PDF and DOCX into markdown, using the LlamaParse API by default. The search command runs fully local semantic keyword matching with multilingual model2vec embeddings, returning matching lines with surrounding context and a configurable distance threshold.
Is semtools free to use? What license does it have?
semtools is open source under the MIT license. Check the license terms before using it in commercial products.
Is semtools still maintained?
No commits for 6+ months. The last commit to semtools was on Mar 11, 2026.
What can I use semtools for?
Searching a folder of text or converted reports for concepts rather than exact keywords. Converting PDFs to markdown as a preprocessing step before search or LLM use. Asking questions over a set of research papers or notes from the terminal. Feeding parsed document content into coding agents through pipelines.
How do I install semtools?
Install semtools via npm: npm i -g @llamaindex/semtools Run a local keyword search over text files: semtools search "some keywords" *.txt --max-distance 0.3 --n-lines 5 Parse documents into markdown (requires a LlamaIndex Cloud API key): semtools parse my_dir/*.pdf
Where can I get semtools?
The source code and installation instructions are on GitHub: https://github.com/run-llama/semtools

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects