Optimising PyTorch in General

---

Created: 29 Nov 2022, 06:06 PM | Modified: =dateformat(this.file.mtime,"dd MMM yyyy, hh:mm a") Tags: knowledge, GeneralDL

---

https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html

https://twitter.com/karpathy/status/1299921324333170689

PyTorch Performance Tuning Guide - Szymon Migacz, NVIDIA

Faster Deep Learning Training with PyTorch — a 2021 Guide

https://efficientdl.com/faster-deep-learning-in-pytorch-a-guide/

1. Consider using a different learning rate schedule.
2. Use multiple workers and pinned memory in DataLoader.
3. Max out the batch size.
4. Use Automatic Mixed Precision (AMP).
5. Consider using a different optimizer.
6. Turn on cudNN benchmarking.
7. Beware of frequently transferring data between CPUs and GPUs.
8. Use gradient/activation checkpointing.
9. Use gradient accumulation.
10. Use DistributedDataParallel for multi-GPU training.
11. Set gradients to None rather than 0.
12. Use .as_tensor rather than .tensor()
13. Turn off debugging APIs if not needed.
14. Use gradient clipping.
15. Turn off bias before BatchNorm.
16. Turn off gradient computation during validation.
17. Use input and batch normalization.