Improvements due to “software-for-hardware”

Innovations in compilers and other low-level improvements have helped increase the utilisation rate of GPUs and improve training efficiency.
Make a list of such improvements and how much did they improve performance overall for tasks such as training a Neural Network.