MIT 2022 - TinyML EfficientML Course (Prof Song Han)
Created: =dateformat(this.file.ctime,"dd MMM yyyy, hh:mm a") | Modified: =dateformat(this.file.mtime,"dd MMM yyyy, hh:mm a")
Tags: TinyML
Annotations
Quantization Dynamic Quantization Static Quantization
- Brain Float (BF16) * show annotation
developed by google recently, increase the range of FP16, while reducing the precision.
allows for training weights to be of larger range, since the precision not as important
- Nvidia TensorFloat (TF32) * show annotation
another good tradeoff between range and precision again
- ==== * show annotation
AI/ML research leading to alot of advances in number representation, such as in FP8 (floating point 8)
https://semiengineering.com/will-floating-point-8-solve-ai-ml-overhead/
FP8 Formats for Deep Learning https://arxiv.org/abs/2209.05433