We extend the ideas of from Automatic Differentiation with Scala to include the reverse accumulation. Reverse accumulation is a non-obvious improvement to automatic differentiation that can in many cases vastly speed up calculations of gradients.
As the tables, diagrams and equations do not translate well into HTML, our full article is available here in PDF: https://github.com/WinVector/Examples/tree/main/dfiles/ReverseAccumulation.pdf.
The purpose of our article is to explain reverse accumulation automatic differentiation clearly (and to release some sample code and timing results). A side effect of the article is to make sense of the following two diagrams:
If the following is picture of standard or forward differentiation:
then the following is a picture of reverse accumulation:
Example code now distributed from: github.com/WinVector/AutoDiff.
Data Scientist and trainer at Win Vector LLC. One of the authors of Practical Data Science with R.