Model-Optimizer
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
Train, fine-tune, and evaluate models — the infrastructure layer.
Key features
- 量化剪枝蒸馏一整套技术
- 支持 HF/PyTorch/ONNX 输入
- 一键导出 TensorRT-LLM/vLLM
- NVFP4 低精度高精度保持
- 集成 Megatron 训练流程
- 官方持续更新教程
Good for
- 大模型量化部署
- 推理加速降本
- 蒸馏小模型
- 低精度精度恢复
Stack & tags
Quick start
pip install -U nvidia-modelopt[all]From the project README on GitHub.
Similar open-source projects
tensorflow
★ 200,721An Open Source Machine Learning Framework for Everyone
transformers
★ 167,239🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.