Menu Home

Gradients via Reverse Accumulation

We extend the ideas of from Automatic Differentiation with Scala to include the reverse accumulation. Reverse accumulation is a non-obvious improvement to automatic differentiation that can in many cases vastly speed up calculations of gradients.
As the tables, diagrams and equations do not translate well into HTML, our full article is available here in PDF: https://github.com/WinVector/Examples/tree/main/dfiles/ReverseAccumulation.pdf.

The purpose of our article is to explain reverse accumulation automatic differentiation clearly (and to release some sample code and timing results). A side effect of the article is to make sense of the following two diagrams:

If the following is picture of standard or forward differentiation:

cutFwd.png

then the following is a picture of reverse accumulation:

cutRev.png


Example code now distributed from: github.com/WinVector/AutoDiff.


Categories: Applications Coding Exciting Techniques Mathematics Tutorials

Tagged as:

jmount

Data Scientist and trainer at Win Vector LLC. One of the authors of Practical Data Science with R.

%d bloggers like this: