Model Quantization

Summary

Reducing the numerical precision of model weights (e.g. FP16 → INT8/INT4) to shrink memory footprint and speed up inference, with a small accuracy trade-off.

• 1Quantize a model with a library like `bitsandbytes` and measure the memory savings.
• 2Compare output quality before and after quantization on the same prompts.
• 3Learn the difference between post-training quantization and quantization-aware training.