Algorithmic Differentiation, Functional Programming, and Iterate-to-Fixedpoint
Algorithmic differentiation (AD) transforms straight lin e numeric code so that it calculates the derivative of the function originally calculated (Corliss et al., 2001). The re are two varieties of AD: forward mode (Wengert, 1964), in which the transformed program calculates the product of the Jacobian of the original program with an input vector, and reverse mode (Speelpenning, 1980; Rall, 1981; Rumelhart et al., 1986), in which it calculates the product of the transpose of the Jacobian with an input vector. The functions produced by the forward and reverse mode AD of a functionf can be characterized in terms of the standard derivativeDf x = (dfi(x)/dxj)ij by