You can now run a 117B open-weight reasoning model on a single H100: 5.1B active params, 80GB, Apache-2.0.

unsloth/gpt-oss-120b
https://huggingface.co/unsloth/gpt-oss-120b

#AI #LocalLLM #OpenSource #GGUF #Quantization

QATFactory is an open-source framework for quantization-aware training and distillation of LLMs, simulating deployment-time formats like NVFP4 on H100 GPUs without native FP4 support. It supports multiple formats,…

#QATFactory #LLM #Quantization #AIInfrastructure
https://arxiv.org/abs/2609.39223

New arXiv work "Bits Under ZK-LLM" explores how quantization choices affect proving cost and constraint complexity for zero-knowledge LLM inference, aiming to make private verifiable inference more practical.

#ZKLMs #Quantization #PrivacyPreservingAI
https://arxiv.org/abs/2609.36437

Below 4 bits, the quality drops sharply, and 2-bit models often produce visibly degraded text. If you are choosing a quant, aim for 4-bit to maximize efficiency without sacrificing coherence. #gpuoptimization #batchinference #selfhosting #quantization #llm #modelcompression 2/2

By the numbers:
16-bit to 8-bit: Statistically indistinguishable outputs
4-bit: The knee of the curve where quality loss remains small
3-bit: Noticeably worse performance
2-bit: Visibly degraded text

#gpuoptimization #batchinference #selfhosting #quantization #llm #modelcompression

4-bit quantization doesn't just shrink models—it silently decides which private data leaks. Benchmarks never measure PII, and no one's #AIprivacy #Quantization #MachineLearning #LLM

https://freegardner.com/synapse/quantization-silently-decides-which-private-data-leaks.html

Model Quantization & Compression Techniques: A Deep Dive for Faster AI

Discover practical model quantization and compression methods, real‑world use cases, tools, and best practic…

https://ai-blog-seven-wine.vercel.app/en/posts/2026-09-24-am-k6ir6

#model-compression #quantization #AI-optimization

If you see IQ, it stands for 'importance-aware' quantization, which generally performs better than standard K-quants when you're forced to go to very low bit rates like 3-bit. #selfhosting #batchinference #llm #quantization #gguf #localai 2/2

Below 4 bits, the quality drops sharply, and 2-bit models often produce visibly degraded text. If you are choosing a quant, aim for 4-bit to maximize efficiency without sacrificing coherence. #gpuoptimization #batchinference #selfhosting #quantization #llm #modelcompression 2/2

eDQA: Efficient Deep Quantization of DNN Activations on Edge Devices

Wenhao Hu, Jude Haris, Paul Henderson, José Cano

Action editor: Min Wu

https://openreview.net/forum?id=SEIBCdgE5W

#quantization #quantizing #dnn

VQEL: Enabling Self-Play in Emergent Language Games via Agent Internal Vector Quantization

Mahdi Samiei, Mehdi Jamalkhah, Mahdieh Soleymani Baghshah

Action editor: Baoxiang Wang

https://openreview.net/forum?id=5nqQlGWlsW

#vqel #emergent #quantization

😥:

“So You Want To Use OpenRouter?”, Mohamed Moustafa (mmoustafa.com/blog/so-you-...).

Via HN: news.ycombinator.com/item?id=4962...

#AI #LLMs #OpenRouter #Quantization #ModelProviders #Reliability #DeepSeek

Benford’s Law as a Distributional Prior for Post-Training Quantization of Large Language Models

Arthur Negrão de Faria Martins da Costa, Pedro Silva, Vander L. S. Freitas et al.

Action editor: Naigang Wang

https://openreview.net/forum?id=YiLcQY4Nje

#quantization #tensors #memory

🎧 Listen www.buzzsprout.com/2405788/epis...

📖 Read helioxpodcast.substack.com/p/folding-gi...

📻 Available for Broadcast on PRX exchange.prx.org/p/633381

#AI #TechSky #Science #MachineLearning #LLM #Quantization #Podcast