Repolode
Submit
Dicklesworthstone

Dicklesworthstone/fast_vector_similarity

Rust library for fast vector similarity measures, with Python bindings and bootstrapped confidence intervals.

View on GitHubWebsite
Stars
429
Forks
27
Open issues
0
Last commit
5 days ago

What is it?

Fast Vector Similarity is a Rust library for computing similarity and dependence measures between two numeric vectors, with Python bindings published on PyPI. It targets data analysis and statistics work, including comparing high-dimensional text embeddings from LLMs.

It implements six measures: Spearman's rank-order correlation, Kendall's tau, approximate distance correlation, a Jensen-Shannon dependency measure, Hoeffding's D, and normalized mutual information. Several of these detect non-linear or non-monotonic dependence that plain correlation misses. Computation is parallelized with rayon, Kendall's tau counts inversions with merge sort to cut complexity, and distance-matrix work uses vectorized ndarray operations.

The library also offers bootstrapping: it resamples the data repeatedly to estimate the distribution of each measure and produce more robust results. The Python API exposes py_compute_vector_similarity_stats and py_compute_bootstrapped_similarity_stats, both returning JSON strings.

Key features

  • 01Six similarity and dependence measures: Spearman's rho, Kendall's tau, approximate distance correlation, Jensen-Shannon dependency, Hoeffding's D, and normalized mutual information.
  • 02Parallel computation via rayon, scaling with the number of CPU cores.
  • 03Kendall's tau counts inversions with merge sort, reducing complexity from O(n^2) to O(n log n).
  • 04Bootstrapping with configurable sample size and iteration count for robust estimates.
  • 05Python bindings return results as JSON strings for use with Pandas and other Python tooling.
  • 06Handles high-dimensional vectors, including 4096-dimensional LLM embeddings.

Use cases

  • Ranking LLM text embeddings by similarity to a query sentence.
  • Detecting non-linear and non-monotonic dependence between two variables.
  • Producing similarity estimates with bootstrapped confidence intervals that are less sensitive to outliers.
  • Calling the similarity functions directly from a Rust project for statistical analysis.

Quick start

  1. 1

    Install the Python package from PyPI

    pip install fast_vector_similarity
  2. 2

    Add the library to your Rust project's Cargo.toml

  3. 3

    Call py_compute_vector_similarity_stats from Python, as in the README example

Things to know

  • No license is specified, so usage rights are unclear.

In collections

Data science courses & tools

Topics

#rust#vector-similarity#statistics#embeddings#python-bindings#llm#correlation#bootstrap#embedding-vectors#similarity

FAQ

What is fast_vector_similarity?
Rust library for fast vector similarity measures, with Python bindings and bootstrapped confidence intervals. Fast Vector Similarity is a Rust library for computing similarity and dependence measures between two numeric vectors, with Python bindings published on PyPI. It targets data analysis and statistics work, including comparing high-dimensional text embeddings from LLMs.
Is fast_vector_similarity free to use? What license does it have?
fast_vector_similarity has no declared license. Its code is public, but without a license you have no explicit permission to reuse it; ask the author before relying on it.
Is fast_vector_similarity still maintained?
Actively maintained. The last commit to fast_vector_similarity was on Nov 1, 2026.
What can I use fast_vector_similarity for?
Ranking LLM text embeddings by similarity to a query sentence. Detecting non-linear and non-monotonic dependence between two variables. Producing similarity estimates with bootstrapped confidence intervals that are less sensitive to outliers. Calling the similarity functions directly from a Rust project for statistical analysis.
How do I install fast_vector_similarity?
Install the Python package from PyPI: pip install fast_vector_similarity Add the library to your Rust project's Cargo.toml Call py_compute_vector_similarity_stats from Python, as in the README example
Where can I get fast_vector_similarity?
The source code and installation instructions are on GitHub: https://github.com/Dicklesworthstone/fast_vector_similarity

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects