Neural Magic

📢 Llama 3.1 is Here, and We're Actively Compressing Them! 📢 Meta unveiled their latest Llama series, featuring an impressive 405 billion parameter model surpassing OpenAI's GPT4o. This milestone significantly boosts open source and the AI community, although the largest model now requires multiple servers (810 GB!). Model compression is crucial! Our (Neural Magic) Llama 3.1 compression project is underway, aiming for cost-effective and sustainable deployments without compromising accuracy. The FP8 quantized Llama 3.1 8B model has already achieved over 99% recovery, with detailed accuracy metrics and deployment guidelines available. Also, we've introduced FP8 model support for all Llama versions in vLLM for immediate use. Explore the latest models here: - Meta-Llama-3.1-8B-Instruct-FP8: https://lnkd.in/dDAcXAAY - Meta-Llama-3.1-8B-Instruct-FP8-dynamic: https://lnkd.in/djtw4GMr For more insights, visit the vLLM Llama 3.1 Blog: - https://lnkd.in/dnZsvKjy Stay tuned for further updates—I'll be sharing more posts in the days ahead! #LLMs #vLLM #AI #MachineLearning #Quantization #NeuralMagic

Our bi-weekly vLLM office hours continue today. Hear from Eldar Kurtić about model quantization for efficient vLLM inference. In two weeks, learn from Roger Wang about multimodal models in vLLM. Link in comments.

📢 FP8 Quantized Llama 3.1 70B Now Available! 📢 Continuing our (Neural Magic) Llama 3.1 compression project, the first versions of the FP8 quantized 70B model are ready, achieving ~100% recovery. This milestone reduces the model size to 70 GB, enabling deployments on a single H100 or A100 GPU instead of two (resulting in a 50% cost reduction!) for the FP16 version. Additionally, it offers ~2X faster inference on the latest NVIDIA Hardware. Explore the Model Links: - FP8 Dynamic Quantization: https://lnkd.in/eKtbAJBv - FP8 Static Quantization: https://lnkd.in/eQDSj_9i For more insights, check out the vLLM Llama 3.1 Blog: - https://lnkd.in/dnZsvKjy Previous Llama 3.1 8B FP8 Post: - https://lnkd.in/eWPXcUVj Stay tuned for further updates—I'll be sharing more posts in the days ahead! #LLMs #AI #MachineLearning #Quantization #vLLM #OpenSource

Join us for bi-weekly vLLM Office Hours this Thursday, July 25! Eldar Kurtić, a model optimization expert, will show us how to quantize LLMs for fast and efficient inference in #vLLM. We'll have ample time for questions and community discussion. See you there!

It's a big day for us - we've joined the Linux Foundation! We are thrilled to collaborate with the greater ecosystem and accelerate #opensource AI innovation.

EXCITING NEWS: Neural Magic and Anyscale contributed FP8 quantization support to vLLM, making LLM inference even more efficient. FP8 reduces latency on NVIDIA GPUs by 2x with >99% accuracy preservation. Thank you to NVIDIA AI for validating our benchmarks! 🔍 What is FP8? FP8 is a modern quantization format that balances precision and efficiency with hardware acceleration on newer GPUs. It reduces memory usage significantly, enabling more cost-effective LLM deployments and higher throughput. 📈 Performance gains: FP8 delivers up to 2x Inter Token Latency (ITL) improvement for Llama 3 70B, 1.6x ITL improvement for Mixtral 8x7B, and up to 3x throughput improvement on 2 NVIDIA H100 GPUs. Memory savings allow for larger batch sizes, boosting performance across various models. Our blog contains specific accuracy details. ✅ Model accuracy: We validated the accuracy preservation of FP8 in vLLM through lm-evaluation-harness comparisons on Open LLM Leaderboard v1 tasks. Most models experience over 99% accuracy preservation compared to the unquantized baseline. 🛠️ Get Started: You can now try out FP8 support in vLLM using a quantized FP8 checkpoint. Access Neural Magic's growing list of accuracy-verified quantized FP8 checkpoints of popular LLMs on our Hugging Face Model Hub. Ready to use with vLLM: https://lnkd.in/gTimN5dZ 🗓️ Learn more: See our blog for more detailed FP8 insights and join our bi-weekly vLLM Office Hours to regularly hear from and give feedback to the vLLM committer community. https://lnkd.in/g2suBKvr 🙏 Thank you for reading and please spread the word about FP8 in vLLM by sharing this post.

🗓 Happening today at 2PM EST! Learn why vLLM is the leading open-source inference server and how Neural Magic works with enterprises to build and scale vLLM-based model services. https://hubs.li/Q02DVnBd0

FP8 quantization is now available in vLLM - check it out! Quantized inference is one of the best ways to reduce the costs of LLM deployments.

Our bi-weekly vLLM Office Hours continue tomorrow. We are excited to bring Philipp Moritz and Cody Yu from Anyscale for a deep dive into FP8 quantization in vLLM. This is an exciting opportunity to give feedback and get your questions answered. Join us: https://lnkd.in/euF8m73q

Are you looking to optimize your #LLM inference for more performance and lower costs? Tune in to hear Eldar Kurtić, our Sr. ML Researcher, break down how quantization can optimize LLM inference and reduce memory footprint without compromising model accuracy.