We extend the ideas of from Automatic Differentiation with Scala to include the reverse accumulation. Reverse accumulation is a non-obvious improvement to automatic differentiation that can in many cases vastly speed up calculations of gradients.
As the tables, diagrams and equations do not translate well into HTML, our full article is available here in PDF: https://github.com/WinVector/Examples/tree/main/dfiles/ReverseAccumulation.pdf.

The purpose of our article is to explain reverse accumulation automatic differentiation clearly (and to release some sample code and timing results). A side effect of the article is to make sense of the following two diagrams:

If the following is picture of standard or forward differentiation:

then the following is a picture of reverse accumulation:

Example code now distributed from: github.com/WinVector/AutoDiff.

Categories: Applications Coding Exciting Techniques Mathematics Tutorials

Tagged as: Automatic Differentiation Conjugate Gradient Gradient math programming Mathematical Bedside Reading Optimization Programming Reverse Accumulation Scala

jmount

Data Scientist and trainer at Win Vector LLC. One of the authors of Practical Data Science with R.