magnitude
Open source inference engine for agents that optimizes itself for your exact hardware. Compiles and tunes its kernels on your device, so open models run up to 2x faster than llama.cpp. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.
Open source inference engine for agents that optimizes itself for your exact hardware. Compiles and tunes its kernels on your device, so open models run up to 2x faster than llama.cpp. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.
Run open LLMs on your own machine — no API bills, no data leaving your device.
Key features
- 针对本机硬件调优 kernel
- 最高 2 倍于 llama.cpp
- 内存占用低 27%
- 多会话共享前缀缓存
- 一键接入主流 agent
- 全平台：苹果/英伟达/AMD/CPU
Good for
- 本地高速跑开源模型
- agent 推理加速
- 苹果芯片本机推理
- 多 agent 并发会话
Stack & tags
RustMetalCUDAllama.cpp