1.(An attempt of)Introduction to ad-3.4,an automatic differentiation library in Haskell3/31/2013 Ekmett study meetingin Shibuya, Tokyoby NebutaAny comments or correction to the material are welcome

2. About myselfNebuta (@nebutalab) https://github.com/nebutaMy interest in softwares: Programming languages (Haskell, Scala, Ruby, etc) Image processing, data visualization, web design Brainstorming and lifehack methods that take advantage of IT, etc.My research areas: A graduate student, studying biophysical chemistry and quantitative biology (2010) Imaging live cells, analyzing microscopy images by Scala on ImageJWhere my interest in Haskell came from: MATLABImageJ (2010) MATLAB,Java Scala2011 Haskell 2011 3. ad-3.4, an automatic differentiation libraryWhat you can do Differentiation of arbitrary mathematical functions Taylor expansion Calculation of gradient, Jacobian, and Hessian, etc.Dependencies array (0.2 & > diff sin 0trigonometric function1.0>> :m + Debug.SimpleReflect>> diff sin x -- x :: Expr is defined in Debug.SimpleReflectcos x * 1 Derivative with a symbol!>> diff (x -> if x >= 0 then 1 else 0) 00.0 Not delta function nor undefined.Gradient>> grad ([x,y] -> exp (x * y)) [x,y][0 + (0 + y * (0 + exp (x * y) * 1)),0 + (0 + x * (0 + exp (x* y) * 1))]>> grad ([x,y] -> exp (x * y)) [1,1][2.718281828459045,2.718281828459045] 5. How to use (continued)Taylor expansionPrelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp 0 d[exp 0 * 1,1 * exp 0 * (1 * d / 1),(0 * exp 0 + 1 * exp 0 * 1) *(1 * d / 1 * d / (1 + 1))]Prelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp x d[exp x * 1,1 * exp x * (1 * d / 1),(0 * exp x + 1 * exp x * 1) *(1 * d / 1 * d / (1 + 1))]Prelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp x 0[exp x * 1,1 * exp x * (1 * 0 / 1),(0 * exp x + 1 * exp x * 1) *(1 * 0 / 1 * 0 / (1 + 1))]Taylor expansion is an infinite list! Taylor expansion (general)No simplification, and slow in higher order terms Exponential function 6. How to use (continued)Equality of functions>> sin x == sin xTrue>> diff sin xcos x * 1>> diff sin x == cos x * 1True>> diff sin x == cos x * 0.5 * 2FalseCool! (no simplification, though...)And so on. 7. Cf. Mechanism of automatic differentiationRead a Wikipedia article http://en.wikipedia.org/wiki/Automatic_differentiationI dont understand it yet.Whats the difference from symbolic differentiation? (f + g) = f + g It seems to be mechanical, successive application of rules of differentiation for composite functions. 8. Ill try to appreciate the type and class organizationGithubhttps://github.com/ekmett/ad4.0Hackage(http://hackage.haskell.org/package/ad-3.4)cabal unpack ad-3.4 ver. 3.4HackageIll use ad-3.4 on Hackage, not ad-4.0 on Github 9. Package structure http://new-hackage.haskell.org/package/ad-3.4 10. The starting point for exploration: diff functionNumeric.AD.Mode.Forward{-# LANGUAGE Rank2Types #-}diff :: Num a => (forall s. Mode s => AD s a -> AD s a) -> a -> adiff f a = tangent $ apply f a f(AD s a->AD s a)Numeric.AD.Internal.Forward{-# LANGUAGE Rank2Types, TypeFamilies, DeriveDataTypeable,TemplateHaskell, UndecidableInstances, BangPatterns #-}tangent :: Num a => AD Forward a -> atangent (AD (Forward _ da)) = datangent _ = 0bundle :: a -> a -> AD Forward abundle a da = AD (Forward a da)apply :: Num a => (AD Forward a -> b) -> a -> bapply f a = f (bundle a 1) AD Forward aAD 11. AD typeNumeric.AD.TypesAn instance of Mode class determinesthe behavior. e.g.) AD (Forward 10 1) :: Num a => AD Forward a{-# LANGUAGE CPP #-}{-# LANGUAGE Rank2Types, GeneralizedNewtypeDeriving,TemplateHaskell, FlexibleContexts, FlexibleInstances,MultiParamTypeClasses, UndecidableInstances #-}{-# ANN module "HLint: ignore Eta reduce" #-}newtype AD f a = AD { runAD :: f a } deriving (Iso (f a), Lifted,Mode, Primal) 12. Classes that have AD type as an instanceNumeric.AD.Types http://hackage.haskell.org/packages/archive/ad/3.4/doc/ html/Numeric-AD-Types.html#t:AD y = f(x)fLifted &&aFloatingAD f aFloatingdiffAD s a -> AD s a Floating a => a -> a 13. Lets look at some examples of values with AD typeadtest.hsimport Numeric.ADimport Numeric.AD.Typesimport Numeric.AD.Internal.Classesimport Numeric.AD.Internal.Forwardf x = x + 3g x | x > 0 = x| otherwise = 0d = diff (g . f)GHCi*Main> :t (g . f)(g . f) :: (Num c, Ord c) => c -> c*Main> :t (g . f) :: (Num a, Ord a, Mode s) => AD s a -> AD s a(g . f) :: (Num a, Ord a, Mode s) => AD s a -> AD s a:: (Num a, Ord a, Mode s) => AD s a -> AD s a*Main> :t ff :: Num a => a -> a*Main> :t f :: (Num a, Lifted s) => AD s a -> AD s af :: (Num a, Lifted s) => AD s a -> AD s a:: (Num a, Lifted s) => AD s a -> AD s a*Main> :t gg :: (Num a, Ord a) => a -> a*Main> :t dd :: Integer -> Integer 14. Lets play around with apply and tangentGHCi> :l adtest.hs[1 of 1] Compiling Main ( adtest.hs, interpreted )Ok, modules loaded: Main.[*Main]> :t applyapply :: Num a => (AD Forward a -> b) -> a -> b[*Main]> :t apply fapply f :: Num a => a -> AD Forward a Since f :: Num t0 => t0 -> t0,[*Main]b in the type of apply is restricted to> :t apply f 0 AD Forward aapply f 0 :: Num a => AD Forward a[*Main]> apply f 03[*Main]> :t tangenttangent :: Num a => AD Forward a -> a[*Main]> :t tangent $ apply f 0tangent $ apply f 0 :: Num a => a[*Main]> tangent $ apply f 01 15. Lifted class and Mode classChain rule of differentiationhttp://hackage.haskell.org/packages/archive/ad/3.4/doc/Wrap a (Num a => a) value into t html/Numeric-AD-Internal-Classes.html#t:Mode Definition of different modes. Mode is a subclass of Liftedhttp://hackage.haskell.org/packages/archive/ad/3.4/doc/html/Numeric-AD-Internal-Classes.html#t:Liftede.g.) Forward is an instance of Lifted and Mode 16. Lifted class defines chain rules of differentiationhttp://hackage.haskell.org/packages/archive/ad/3.4/doc/html/src/Numeric-AD-Internal-Classes.html#deriveLiftedderiveLifted (in Numeric.AD.Internal.Classes)TemplateHaskellLiftedA part of deriveLifted exp1= lift1_ exp const log1= lift1 log recip1 sin1= lift1 sin cos1 cos1= lift1 cos $ negate1 . sin1 tan1= lift1 tan $ recip1 . square1 . cos1 17. deriveNumeric generates instances of Num, etc. http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/ src/Numeric-AD-Internal-Classes.html#deriveNumeric deriveNumeric (in Numeric.AD.Internal.Classes) -- | @deriveNumeric f g@ provides the following instances: -- -- > instance (Lifted $f, Num a, Enum a) => Enum ($g a) -- > instance (Lifted $f, Num a, Eq a) => Eq ($g a) -- > instance (Lifted $f, Num a, Ord a) => Ord ($g a) -- > instance (Lifted $f, Num a, Bounded a) => Bounded ($g a) -- -- > instance (Lifted $f, Show a) => Show ($g a) -- > instance (Lifted $f, Num a) => Num ($g a) -- > instance (Lifted $f, Fractional a) => Fractional ($g a) -- > instance (Lifted $f, Floating a) => Floating ($g a) -- > instance (Lifted $f, RealFloat a) => RealFloat ($g a) -- > instance (Lifted $f, RealFrac a) => RealFrac ($g a) -- > instance (Lifted $f, Real a) => Real ($g a) 18. Definition of deriveNumericderiveNumeric :: ([Q Pred] -> [Q Pred]) -> Q Type -> Q [Dec]deriveNumeric f t = domembers [Q Pred]) -> Q Type -> Name -> Q DeclowerInstance p f t n = do#ifdef OldClassIClassI (ClassD _ _ _ _ ds) > :t diff ((**2) :: Floating a => a -> a)diff ((**2) :: Floating a => a -> a) :: Floating a => a -> a>> :t diff ((**2) :: Double -> Double):1:7:Couldnt match expected type `ad-3.4:Numeric.AD.Internal.Types.AD s a0with actual type `DoubleExpected type: ad-3.4:Numeric.AD.Internal.Types.AD s a0 -> ad-3.4:Numeric.AD.Internal.Types.AD s a0Actual type: Double -> DoubleIn the first argument of `diff, namely`((** 2) :: Double -> Double)In the expression: diff ((** 2) :: Double -> Double)You need to keep functions polymorphic for differentiation.import Numeric.ADfunc x = x ** 2func2 x = x ** 3fs = [func, func2]test = map (f -> diff f 1.0) fsSo, this does not compile. (there is a GHC extension to accept this, isnt there..?) 20. SummaryAn interesting library that shows how to use types andtypeclasses effectively and beautifully.Sorry, but Im still unclear about how Lifted actuallyexecutes differentiation It seems to be done by Num instance of AD f a, via instance (Lifted f, Num a) => Num (AD f a)...Things that might be improved for application Simplification of terms Improving the performance of higher order taylor expansion. Is automatic integration possible, maybe?These might be a good practice for seasoned Haskellers (Im not...) 21. Haskell Ekmett GHC