Skip to content
@jitllm

jitllm

An operating system for LLM inference
README.md

jitllm

Performance: 4.62x llama.cpp's prefill on Qwen3-Next-80B (4x V100); 35 of 35 models decode and 34 of 35 prefill faster than on llama.cpp (V100); up to 4.08x sooner to the first token; up to 5.53x vLLM 0.18.1's batched decode; up to 1.53x llama.cpp's decode on an Apple M4 A simulation of placement: model blocks page from the .jlm onto a GPU and the CPU, and relocate, KV with them, as a GPU is added, the placement changes and the GPU is removed

The JIT: the inner loop of a Q4_K matrix-vector product emitted for an AVX2+VNNI CPU, then for CUDA Expert pages: each expert of a mixture is its own page, so a token reads only the routed experts that are not resident

An operating system for LLM inference: generate the compute for your hardware, page models larger than memory, and move execution without losing the conversation.

jitllm.org · Documentation · Source · Latest release

  • Every kernel JIT-generated at load time for your CPU (AVX2, SSE, NEON) or GPU (CUDA, Vulkan, Metal), specialised to the model's shapes and weight formats. No precompiled kernels or interpreted compute fallback.
  • Disk, RAM and VRAM form the memory system. Weights, routed MoE experts and KV state page as needed. Blocks move between CPU and GPUs at run time, carrying their attention and recurrent state with them.
  • Runs everywhere, depends on nothing. One binary per platform (Linux, macOS, Windows; x86 and Arm) that finds CUDA, Vulkan or Metal at run time and otherwise runs on the CPU.

Serve it as an OpenAI- and Anthropic-compatible API with streaming and tool calling from one standalone binary, jitllmd, or embed it as a Go library. There are desktop and terminal apps too.

Install

Linux and macOS:

curl -fsSL https://jitllm.org/install.sh | sh

Windows (PowerShell):

irm https://jitllm.org/install.ps1 | iex

Popular repositories Loading

  1. jitllm jitllm Public

    Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

    Go 55 5

  2. .github .github Public

    jitllm organization profile

  3. homebrew-jitllm homebrew-jitllm Public

    Homebrew tap for jitllm

    Ruby

Repositories

Showing 3 of 3 repositories

People

This organization has no public members. You must be a member to see who’s a part of this organization.

Top languages

Loading…

Most used topics

Loading…