snowflakedb/ArcticInference

What is it?

Key features

• 01Shift Parallelism and Ulysses sequence parallelism for LLM serving
• 02Speculative decoding via Arctic Speculator and Suffix Decoding
• 03SwiftKV model-level optimization
• 04Optimized embedding inference on vLLM
• 05Works with existing vLLM APIs and CLI, enabled by an environment variable

Use cases

• Serving large LLMs on vLLM with lower latency and higher throughput
• Running high-volume embedding generation pipelines on GPUs
• Reducing inference cost for long-context or reasoning-heavy workloads

Quick start

1. 1

   Install the package with vLLM extras

   pip install arctic-inference[vllm]
2. 2

   Serve a model with Arctic Inference enabled

   ARCTIC_INFERENCE_ENABLED=1 vllm serve Snowflake/Llama-3.1-SwiftKV-8B-Instruct --quantization "fp8" --tensor-parallel-size 1 --ulysses-sequence-parallel-size 2 --enable-shift-parallel --speculative-config '{"method": "arctic", "model":"Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct", "num_speculative_tokens": 3, "enable_suffix_decoding": true, "disable_by_batch_size": 64}'

Things to know

• !Optimized for vLLM and requires a GPU-based serving setup with vLLM installed.
• !Speculative decoding and Shift Parallelism settings are tuned for specific Snowflake models and need configuration.

In collections

Topics

FAQ

What is ArcticInference?

A vLLM plugin that adds Snowflake's inference optimizations (Shift Parallelism, speculative decoding, SwiftKV) for faster, cheaper LLM and embedding serving. Arctic Inference is an open-source plugin for vLLM, the popular LLM serving engine. Once installed, it patches vLLM so that existing vLLM APIs and CLI commands can use Arctic's optimizations without changing application code. The optimizations cover parallelism (Arctic Ulysses sequence parallelism and Shift Parallelism), speculative decoding (Arctic Speculator and Suffix Decoding), model-level techniques such as SwiftKV, and workload-specific paths for embeddings and reasoning.

Is ArcticInference free to use? What license does it have?

ArcticInference is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.

Is ArcticInference still maintained?

Updated within 6 months. The last commit to ArcticInference was on Sep 23, 2026.

What can I use ArcticInference for?

Serving large LLMs on vLLM with lower latency and higher throughput Running high-volume embedding generation pipelines on GPUs Reducing inference cost for long-context or reasoning-heavy workloads

How do I install ArcticInference?

Install the package with vLLM extras: pip install arctic-inference[vllm] Serve a model with Arctic Inference enabled: ARCTIC_INFERENCE_ENABLED=1 vllm serve Snowflake/Llama-3.1-SwiftKV-8B-Instruct --quantization "fp8" --tensor-parallel-size 1 --ulysses-sequence-parallel-size 2 --enable-shift-parallel --speculative-config '{"method": "arctic", "model":"Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct", "num_speculative_tokens": 3, "enable_suffix_decoding": true, "disable_by_batch_size": 64}'

Where can I get ArcticInference?

The source code and installation instructions are on GitHub: https://github.com/snowflakedb/ArcticInference