New #TMLR-Paper-with-Video:

Benford’s Law as a Distributional Prior for Post-Training Quantization of Large Language Models

Arthur Negrão de Faria Martins da Costa, Pedro Silva, Vander L. S. Freitas et al.

https://tmlr.infinite-conf.org/paper_pages/YiLcQY4Nje

#quantization #tensors

Why smaller AI files write faster
https://youtube.com/shorts/TQsJjMYOhk0

#AI #LLM #Quantization

New arXiv work presents the first unified evaluation of weight-only quantization strategies for LLM inference, measuring energy use across Llama 2 7B, Phi 3.5 Mini Instruct, OLMo 1B, Qwen 2.5 7B, and Qwen 2.5 14B on…

#GreenAI #LLMs #Quantization #AIInfrastructure
https://arxiv.org/abs/2610.17531

Four bits: nearly free. Two bits: broken
https://youtube.com/shorts/w-Llcgf8ndU

#AI #LLM #Quantization

A new arXiv paper introduces algebraic integer quantization (AIQ), a compression framework that reduces to error-free integer arithmetic and enables efficient matrix multiplication on modern hardware accelerators.

#AI #MachineLearning #Hardware #Quantization
https://arxiv.org/abs/2610.12494

TAH-Quant: Effective Activation Quantization in Pipeline Parallelism over Slow Network

Guangxin He, Yuan Cao, Yutong He, Tianyi Bai, Kai Chen, Kun Yuan, Binhang Yuan

Action editor: Reza Babanezhad Harikandeh

https://openreview.net/forum?id=6ysPGq2RVD

#quantization #throughput

New #TMLR-Paper-with-Video:

LoraQuant: Mixed-Precision Quantization of LoRA to Ultra- Low Bits for LLM Customization

Amir Reza Mirzaei, Yuqiao Wen, Yanshuai Cao, Lili Mou

https://tmlr.infinite-conf.org/paper_pages/71svCWi178

#quantization #quantizing #quantize

Quadratic integers reduce to pure integer arithmetic, enabling error-free multiply-accumulate operations, per an arXiv preprint. The abstract gives no accuracy figures against MX. #LLM #quantization

https://bymachine.news/algebraic-integer-quantization-llm-inference

#selfhosting #batchinference #gpuoptimization #gguf #quantization #llm 2/2

A 16 GB AI model, rounded to 4 bits: 5 GB, nearly 3x faster, barely worse. Rounded to 2 bits: smaller, slower, and its score more than doubles. I measured all four sizes on my MacBook. https://youtu.be/FIPZCMYqMzs

What would you do with the memory you save?

#AI #LLM #Quantization

#batchinference #selfhosting #gpuoptimization #llm #quantization #reasoning 2/2

New arXiv work challenges a core assumption in low-bit LLM weight quantization, showing lower reconstruction loss can actually hurt performance when input distributions shift, and proposes Distributionally Robust Quantization to…

#LLM #Quantization #AIResearch
https://arxiv.org/abs/2610.11226

By the numbers:
16-bit to 8-bit: Statistically indistinguishable outputs
4-bit: The knee of the curve where quality loss remains small
3-bit: Noticeably worse
2-bit: Visibly degraded text

#gpuoptimization #batchinference #quantization #llm #modelcompression #aiperformance

27.4B agent decision model now runs local from 17.67 GB instead of being trapped on giant servers.

ggml-org/OpenJev-GGUF
https://huggingface.co/ggml-org/OpenJev-GGUF

#AI #LocalLLM #OpenSource #GGUF #Quantization

NVFP4 quantization of video VAE latents can preserve reconstruction yet still collapse downstream policy success, a trade-off the LatentQuant method targets by maintaining the policy-facing latent contract.

#Robotics #AI #WorldModels #Quantization
https://arxiv.org/abs/2610.03959

New arXiv paper audits 27 quantized open-weight language models and finds that aggressive GGUF Q2_K quantization often breaks rare-word knowledge in sub-2B models while larger ones mostly hold up. Useful caution for anyone…

#AI #LLM #Quantization #OnDeviceAI
https://arxiv.org/abs/2610.04403

#selfhosting #batchinference #gpuoptimization #gguf #quantization #llm 2/2

#batchinference #selfhosting #gpuoptimization #llm #quantization #reasoning 2/2

New work introduces a 2.7-bit quantization framework for running vision-language models on mobile hardware, compressing Llama 3.2 11B Vision Instruct to 3.7 GB for Arm CPU inference.

#Llama #VLM #MobileAI #Quantization
https://arxiv.org/abs/2608.21134

🤖 Powerful AI doesn't always need huge models.

Quantization can reduce model precision to shrink size, lower memory use, speed up inference and cut deployment costs.

The challenge: balance efficiency with accuracy.

#AI #Quantization #AIEngineering