ProjectAIResearch
Quantization × Interpretability | what quantization does to sparse autoencoders
How model quantization affects sparse autoencoders.
What I Built
- Sparse autoencoders trained on Qwen3-30B and StarCoder2-15B at four precisions, then compared across them.
- Code generation breaks at INT4 while knowledge retrieval holds. Undercomplete SAEs transfer better than overcomplete ones.
Stats
168SAEs trained
99%sample correlation across precisions, BF16 to INT4
−50%code generation at INT4
2.3×better transfer for undercomplete SAEs
The Stack
ToolsPython, PyTorch, SAELens, bitsandbytes, Transformers