From 7067e3b12065f50ace352ae2ede3947441718599 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Sun, 30 Jul 2023 12:47:59 +0000 Subject: [PATCH] build based on 2ac75a0 --- previews/PR64/api/index.html | 6 +++--- previews/PR64/examples/0_intro/index.html | 2 +- previews/PR64/examples/1_basic/index.html | 2 +- previews/PR64/examples/2_advanced/index.html | 2 +- previews/PR64/examples/3_tricks/index.html | 2 +- previews/PR64/faq/index.html | 4 ++-- previews/PR64/index.html | 2 +- previews/PR64/search/index.html | 2 +- 8 files changed, 11 insertions(+), 11 deletions(-) diff --git a/previews/PR64/api/index.html b/previews/PR64/api/index.html index 360e753..1e78f01 100644 --- a/previews/PR64/api/index.html +++ b/previews/PR64/api/index.html @@ -1,4 +1,4 @@ -API reference · ImplicitDifferentiation.jl
View on GitHub

API reference

Public

ImplicitDifferentiation.ImplicitFunction — Type
ImplicitFunction{FF,CC,LS}

Differentiable wrapper for an implicit function defined by a forward mapping and a set of conditions.

Constructors

You can construct an ImplicitFunction from a forward mapping f and conditions c, both of which must be callables (function-like objects). While f does not not need to be compatible with automatic differentiation, c has to be.

ImplicitFunction(f, c[, HandleByproduct()])
-ImplicitFunction(f, c, linear_solver[, HandleByproduct()])

Callable behavior

An ImplicitFunction object implicit behaves like a function, and every call to it is differentiable.

implicit(x::AbstractArray[, ReturnByproduct()]; kwargs...)

Details

  • By default, we assume that the forward mapping is x -> y(x) and the conditions are c(x,y(x)) = 0.
  • If HandleByproduct() is passed as an argument to the constructor, we assume instead that the forward mapping is x -> (y(x),z(x)) and the conditions are c(x,y(x),z(x)) = 0. In this case, z(x) can contain additional information generated by the forward mapping, but beware that we consider it constant for differentiation purposes.

Given x ∈ ℝⁿ and y ∈ ℝᵈ, we need as many conditions as output dimensions: c(x,y,z) ∈ ℝᵈ. We can then compute the Jacobian of y(⋅) using the implicit function theorem:

∂₂c(x,y(x),z(x)) * ∂y(x) = -∂₁c(x,y(x),z(x))

This requires solving a linear system A * J = -B, where A ∈ ℝᵈˣᵈ, B ∈ ℝᵈˣⁿ and J ∈ ℝᵈˣⁿ.

Fields

source
ChainRulesCore.rrule — Function
rrule(rc, implicit, x; kwargs...)
-rrule(rc, implicit, x, ReturnByproduct(); kwargs...)

Custom reverse rule for an ImplicitFunction, to ensure compatibility with reverse mode autodiff.

This is only available if ChainRulesCore.jl is loaded (extension), except on Julia < 1.9 where it is always available.

  • By default, this returns a single output y(x) with a pullback accepting a single cotangent dy.
  • If ReturnByproduct() is passed as an argument, this returns a couple of outputs (y(x),z(x)) with a pullback accepting a couple of cotangents (dy, dz) (remember that z(x) is not differentiated so its cotangent is ignored).

We compute the vector-Jacobian product Jᵀv by solving Aᵀu = v and setting Jᵀv = -Bᵀu (see ImplicitFunction for the definition of A and B). Keyword arguments are given to both implicit.forward and implicit.conditions.

source

Internals

ImplicitDifferentiation.Forward — Type
Forward{byproduct,F}

Callable wrapper for a forward mapping f::F, which ensures that a byproduct z(x) is always returned in addition to y(x).

The type parameter byproduct is a boolean stating whether or not f natively returns z(x).

source
ImplicitDifferentiation.Conditions — Type
Conditions{byproduct,C}

Callable wrapper for conditions c::C, which ensures that a byproduct z is always accepted in addition to x and y.

The type parameter byproduct is a boolean stating whether or not c natively accepts z.

source
ImplicitDifferentiation.AbstractLinearSolver — Type
AbstractLinearSolver

All linear solvers used within an ImplicitFunction must satisfy this interface.

Required methods

  • presolve(linear_solver, A, y): return a matrix-like object A for which it is cheaper to solve several linear systems with different vectors b (a typical example would be to perform LU factorization).
  • solve(linear_solver, A, b): return a tuple (x, stats) where x satisfies Ax = b and stats.solved ∈ {true, false}.
source
ImplicitDifferentiation.PushforwardMul! — Type
PushforwardMul!{P,N}

Callable structure wrapping a pushforward with N-dimensional inputs into an in-place multiplication for vectors.

Fields

  • pushforward::P: the pushforward function
  • input_size::NTuple{N,Int}: the array size of the function input
source
ImplicitDifferentiation.PullbackMul! — Type
PullbackMul!{P,N}

Callable structure wrapping a pullback with N-dimensional outputs into an in-place multiplication for vectors.

Fields

  • pullback::P: the pullback of the function
  • output_size::NTuple{N,Int}: the array size of the function output
source

Index

+API reference · ImplicitDifferentiation.jl
View on GitHub

API reference

Public

ImplicitDifferentiation.ImplicitFunction — Type
ImplicitFunction{FF,CC,LS}

Differentiable wrapper for an implicit function defined by a forward mapping and a set of conditions.

Constructors

You can construct an ImplicitFunction from a forward mapping f and conditions c, both of which must be callables (function-like objects). While f does not not need to be compatible with automatic differentiation, c has to be.

ImplicitFunction(f, c[, HandleByproduct()])
+ImplicitFunction(f, c, linear_solver[, HandleByproduct()])

Callable behavior

An ImplicitFunction object implicit behaves like a function, and every call to it is differentiable.

implicit(x::AbstractArray[, ReturnByproduct()]; kwargs...)

Details

  • By default, we assume that the forward mapping is x -> y(x) and the conditions are c(x,y(x)) = 0.
  • If HandleByproduct() is passed as an argument to the constructor, we assume instead that the forward mapping is x -> (y(x),z(x)) and the conditions are c(x,y(x),z(x)) = 0. In this case, z(x) can contain additional information generated by the forward mapping, but beware that we consider it constant for differentiation purposes.

Given x ∈ ℝⁿ and y ∈ ℝᵈ, we need as many conditions as output dimensions: c(x,y,z) ∈ ℝᵈ. We can then compute the Jacobian of y(⋅) using the implicit function theorem:

∂₂c(x,y(x),z(x)) * ∂y(x) = -∂₁c(x,y(x),z(x))

This requires solving a linear system A * J = -B, where A ∈ ℝᵈˣᵈ, B ∈ ℝᵈˣⁿ and J ∈ ℝᵈˣⁿ.

Fields

source
ChainRulesCore.rrule — Function
rrule(rc, implicit, x; kwargs...)
+rrule(rc, implicit, x, ReturnByproduct(); kwargs...)

Custom reverse rule for an ImplicitFunction, to ensure compatibility with reverse mode autodiff.

This is only available if ChainRulesCore.jl is loaded (extension), except on Julia < 1.9 where it is always available.

  • By default, this returns a single output y(x) with a pullback accepting a single cotangent dy.
  • If ReturnByproduct() is passed as an argument, this returns a couple of outputs (y(x),z(x)) with a pullback accepting a couple of cotangents (dy, dz) (remember that z(x) is not differentiated so its cotangent is ignored).

We compute the vector-Jacobian product Jᵀv by solving Aᵀu = v and setting Jᵀv = -Bᵀu (see ImplicitFunction for the definition of A and B). Keyword arguments are given to both implicit.forward and implicit.conditions.

source

Internals

ImplicitDifferentiation.Forward — Type
Forward{byproduct,F}

Callable wrapper for a forward mapping f::F, which ensures that a byproduct z(x) is always returned in addition to y(x).

The type parameter byproduct is a boolean stating whether or not f natively returns z(x).

source
ImplicitDifferentiation.Conditions — Type
Conditions{byproduct,C}

Callable wrapper for conditions c::C, which ensures that a byproduct z is always accepted in addition to x and y.

The type parameter byproduct is a boolean stating whether or not c natively accepts z.

source
ImplicitDifferentiation.AbstractLinearSolver — Type
AbstractLinearSolver

All linear solvers used within an ImplicitFunction must satisfy this interface.

Required methods

  • presolve(linear_solver, A, y): return a matrix-like object A for which it is cheaper to solve several linear systems with different vectors b (a typical example would be to perform LU factorization).
  • solve(linear_solver, A, b): return a tuple (x, stats) where x satisfies Ax = b and stats.solved ∈ {true, false}.
source
ImplicitDifferentiation.PushforwardMul! — Type
PushforwardMul!{P,N}

Callable structure wrapping a pushforward with N-dimensional inputs into an in-place multiplication for vectors.

Fields

  • pushforward::P: the pushforward function
  • input_size::NTuple{N,Int}: the array size of the function input
source
ImplicitDifferentiation.PullbackMul! — Type
PullbackMul!{P,N}

Callable structure wrapping a pullback with N-dimensional outputs into an in-place multiplication for vectors.

Fields

  • pullback::P: the pullback of the function
  • output_size::NTuple{N,Int}: the array size of the function output
source

Index

diff --git a/previews/PR64/examples/0_intro/index.html b/previews/PR64/examples/0_intro/index.html index a887990..afe5899 100644 --- a/previews/PR64/examples/0_intro/index.html +++ b/previews/PR64/examples/0_intro/index.html @@ -25,4 +25,4 @@ return c end
conditions (generic function with 1 method)

Finally, we construct a wrapper implicit around the previous objects. By default, forward is assumed to return a single output and conditions is assumed to accept 2 arguments.

implicit = ImplicitFunction(forward, conditions)
ImplicitFunction(forward, conditions, IterativeLinearSolver())

What does this wrapper do? When we call it as a function, it just falls back on first ∘ implicit.forward, so unsurprisingly we get the first output $y(x)$.

implicit(x) ≈ sqrt.(x)
true

And when we try to compute its Jacobian, the implicit function theorem is applied in the background to circumvent the lack of differentiablility of the forward mapping.

Forward and reverse mode autodiff

Now ForwardDiff.jl works seamlessly.

ForwardDiff.jacobian(implicit, x) ≈ J
true

And so does Zygote.jl. Hurray!

Zygote.jacobian(implicit, x)[1] ≈ J
true

Second derivative

We can even go higher-order by mixing the two packages (forward-over-reverse mode). The only technical requirement is to switch the linear solver to something that can handle dual numbers:

implicit_higher_order = ImplicitFunction(forward, conditions, DirectLinearSolver())
ImplicitFunction(forward, conditions, DirectLinearSolver())

Then the Jacobian itself is differentiable.

h = rand(2)
 J_Z(t) = Zygote.jacobian(implicit_higher_order, x .+ t .* h)[1]
-ForwardDiff.derivative(J_Z, 0) ≈ Diagonal((-0.25 .* h) ./ (x .^ 1.5))
true

This page was generated using Literate.jl.

+ForwardDiff.derivative(J_Z, 0) ≈ Diagonal((-0.25 .* h) ./ (x .^ 1.5))
true

This page was generated using Literate.jl.

diff --git a/previews/PR64/examples/1_basic/index.html b/previews/PR64/examples/1_basic/index.html index edc08c6..068a9bf 100644 --- a/previews/PR64/examples/1_basic/index.html +++ b/previews/PR64/examples/1_basic/index.html @@ -71,4 +71,4 @@ Zygote.jacobian(_x -> forward_fixedpoint(_x; iterations=10), x)[1] catch e e -end
ErrorException("Mutating arrays is not supported -- called copyto!(Vector{Float64}, ...)\nThis error occurs when you ask Zygote to differentiate operations that change\nthe elements of arrays in place (e.g. setting values with x .= ...)\n\nPossible fixes:\n- avoid mutating operations (preferred)\n- or read the documentation and solutions for this error\n  https://fluxml.ai/Zygote.jl/latest/limitations\n")

This page was generated using Literate.jl.

+end
ErrorException("Mutating arrays is not supported -- called copyto!(Vector{Float64}, ...)\nThis error occurs when you ask Zygote to differentiate operations that change\nthe elements of arrays in place (e.g. setting values with x .= ...)\n\nPossible fixes:\n- avoid mutating operations (preferred)\n- or read the documentation and solutions for this error\n  https://fluxml.ai/Zygote.jl/latest/limitations\n")

This page was generated using Literate.jl.

diff --git a/previews/PR64/examples/2_advanced/index.html b/previews/PR64/examples/2_advanced/index.html index 3189874..d9f8d29 100644 --- a/previews/PR64/examples/2_advanced/index.html +++ b/previews/PR64/examples/2_advanced/index.html @@ -38,4 +38,4 @@ Zygote.jacobian(forward_cstr_optim, x)[1] catch e e -end
Zygote.CompileError(Tuple{typeof(Optim.optimize), NLSolversBase.OnceDifferentiable{Float64, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Optim.Fminbox{Optim.GradientDescent{LineSearches.InitialPrevious{Float64}, LineSearches.HagerZhang{Float64, Base.RefValue{Bool}}, Nothing, Optim.var"#13#15"}, Float64, Optim.var"#49#51"}, Optim.Options{Float64, Nothing}}, ErrorException("try/catch is not supported.\nRefer to the Zygote documentation for fixes.\nhttps://fluxml.ai/Zygote.jl/latest/limitations\n"))

This page was generated using Literate.jl.

+end
Zygote.CompileError(Tuple{typeof(Optim.optimize), NLSolversBase.OnceDifferentiable{Float64, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Optim.Fminbox{Optim.GradientDescent{LineSearches.InitialPrevious{Float64}, LineSearches.HagerZhang{Float64, Base.RefValue{Bool}}, Nothing, Optim.var"#13#15"}, Float64, Optim.var"#49#51"}, Optim.Options{Float64, Nothing}}, ErrorException("try/catch is not supported.\nRefer to the Zygote documentation for fixes.\nhttps://fluxml.ai/Zygote.jl/latest/limitations\n"))

This page was generated using Literate.jl.

diff --git a/previews/PR64/examples/3_tricks/index.html b/previews/PR64/examples/3_tricks/index.html index 0d684ec..6f0f8c2 100644 --- a/previews/PR64/examples/3_tricks/index.html +++ b/previews/PR64/examples/3_tricks/index.html @@ -44,4 +44,4 @@ implicit_cointoss(x)
2-element Vector{Float64}:
  2.0
- 2.0

Or if you also need the byproduct, you can do

implicit_cointoss(x, ReturnByproduct())
([0.5, 0.5], false)

But whatever you choose, the byproduct is taken into account during differentiation!

Zygote.withjacobian(implicit_cointoss, x)
(val = [0.5, 0.5], grad = ([0.5 0.0; 0.0 0.5],))

This page was generated using Literate.jl.

+ 2.0

Or if you also need the byproduct, you can do

implicit_cointoss(x, ReturnByproduct())
([0.5, 0.5], false)

But whatever you choose, the byproduct is taken into account during differentiation!

Zygote.withjacobian(implicit_cointoss, x)
(val = [0.5, 0.5], grad = ([0.5 0.0; 0.0 0.5],))

This page was generated using Literate.jl.

diff --git a/previews/PR64/faq/index.html b/previews/PR64/faq/index.html index 3d4af1e..948b9dd 100644 --- a/previews/PR64/faq/index.html +++ b/previews/PR64/faq/index.html @@ -1,5 +1,5 @@ -FAQ · ImplicitDifferentiation.jl
View on GitHub

Frequently Asked Questions

Supported autodiff backends

ModeBackendSupport
Forward[ForwardDiff.jl]yes
Reverse[ChainRules.jl]-compatible ([Zygote.jl], [ReverseDiff.jl])yes
Forward[ChainRules.jl]-compatible ([Diffractor.jl])soon
Both[Enzyme.jl]someday

[ForwardDiff.jl]: https://github.com/JuliaDiff/ForwardDiff.jl [ChainRules.jl]: https://github.com/JuliaDiff/ChainRules.jl [Zygote.jl]: https://github.com/FluxML/Zygote.jl [ReverseDiff.jl]: https://github.com/JuliaDiff/ReverseDiff.jl [Enzyme.jl]: https://github.com/EnzymeAD/Enzyme.jl [Diffractor.jl]: https://github.com/JuliaDiff/Diffractor.jl

Writing conditions

We recommend that the conditions themselves do not involve calls to autodiff, even when they describe a gradient. Otherwise, you will need to make sure that nested autodiff works well in your case. For instance, if you're differentiating your implicit function in reverse mode with Zygote.jl, you may want to use Zygote.forwarddiff to wrap the conditions and differentiate them with ForwardDiff.jl instead.

Matrices and higher-order arrays

For simplicity, our examples only display functions that eat and spit out vectors. However, arbitrary array shapes are supported, as long as the forward mapping and conditions return similar arrays. Beware however, sparse arrays will be densified in the differentiation process.

Scalars

Functions that eat or spit out a single number are not supported. The forward mapping and conditions need arrays: for example, instead of returning val you should return [val] (a 1-element Vector).

Multiple inputs / outputs

In this package, implicit functions can only take a single input array x and output a single output array y (plus the byproduct z). But sometimes, your forward mapping or conditions may require multiple input arrays, say a and b:

function f(a, b)
+FAQ · ImplicitDifferentiation.jl
View on GitHub
Frequently Asked Questions
Supported autodiff backends
ModeBackendSupport
Forward[ForwardDiff.jl]yes
Reverse[ChainRules.jl]-compatible ([Zygote.jl], [ReverseDiff.jl])yes
Forward[ChainRules.jl]-compatible ([Diffractor.jl])soon
Both[Enzyme.jl]someday
[ForwardDiff.jl]: https://github.com/JuliaDiff/ForwardDiff.jl [ChainRules.jl]: https://github.com/JuliaDiff/ChainRules.jl [Zygote.jl]: https://github.com/FluxML/Zygote.jl [ReverseDiff.jl]: https://github.com/JuliaDiff/ReverseDiff.jl [Enzyme.jl]: https://github.com/EnzymeAD/Enzyme.jl [Diffractor.jl]: https://github.com/JuliaDiff/Diffractor.jl
Writing conditions
We recommend that the conditions themselves do not involve calls to autodiff, even when they describe a gradient. Otherwise, you will need to make sure that nested autodiff works well in your case. For instance, if you're differentiating your implicit function in reverse mode with Zygote.jl, you may want to use Zygote.forwarddiff to wrap the conditions and differentiate them with ForwardDiff.jl instead.
Matrices and higher-order arrays
For simplicity, our examples only display functions that eat and spit out vectors. However, arbitrary array shapes are supported, as long as the forward mapping and conditions return similar arrays. Beware however, sparse arrays will be densified in the differentiation process.
Scalars
Functions that eat or spit out a single number are not supported. The forward mapping and conditions need arrays: for example, instead of returning val you should return [val] (a 1-element Vector).
Multiple inputs / outputs
In this package, implicit functions can only take a single input array x and output a single output array y (plus the byproduct z). But sometimes, your forward mapping or conditions may require multiple input arrays, say a and b:
function f(a, b)
     # do stuff
     return y, z
-end
In that case, you should gather the inputs inside a single ComponentVector from ComponentArrays.jl and define a new method:
f(x::ComponentVector) = f(x.a, x.b)
The same trick works for multiple outputs.
Using byproducts
Why would the forward mapping return a byproduct z in addition to y? It is mainly useful when the solution procedure creates objects such as Jacobians, which we want to reuse when computing or differentiating the conditions. In that case, you may want to write the differentiation rules yourself for the conditions. A more advanced application is given by DifferentiableFrankWolfe.jl.
Keep in mind that derivatives of z will not be computed: the byproduct is considered constant during differentiation (unlike the case of multiple outputs outlined above).
Performance tips
If you work with small arrays (say, less than 100 elements), consider using StaticArrays.jl if you seek increased performance.
Modeling tips
To express constrained optimization problems as implicit functions, you might need differentiable projections or proximal operators to write the optimality conditions. See Efficient and modular implicit differentiation for precise formulations.
In case these operators are too complicated to code them yourself, here are a few places you can look:
An alternative is differentiating through the KKT conditions, which is exactly what DiffOpt.jl does for JuMP models.

+end

In that case, you should gather the inputs inside a single ComponentVector from ComponentArrays.jl and define a new method:

f(x::ComponentVector) = f(x.a, x.b)

The same trick works for multiple outputs.

Using byproducts

Why would the forward mapping return a byproduct z in addition to y? It is mainly useful when the solution procedure creates objects such as Jacobians, which we want to reuse when computing or differentiating the conditions. In that case, you may want to write the differentiation rules yourself for the conditions. A more advanced application is given by DifferentiableFrankWolfe.jl.

Keep in mind that derivatives of z will not be computed: the byproduct is considered constant during differentiation (unlike the case of multiple outputs outlined above).

Performance tips

If you work with small arrays (say, less than 100 elements), consider using StaticArrays.jl if you seek increased performance.

Modeling tips

To express constrained optimization problems as implicit functions, you might need differentiable projections or proximal operators to write the optimality conditions. See Efficient and modular implicit differentiation for precise formulations.

In case these operators are too complicated to code them yourself, here are a few places you can look:

An alternative is differentiating through the KKT conditions, which is exactly what DiffOpt.jl does for JuMP models.

diff --git a/previews/PR64/index.html b/previews/PR64/index.html index 9b65e5f..cb30c52 100644 --- a/previews/PR64/index.html +++ b/previews/PR64/index.html @@ -1,2 +1,2 @@ -Home · ImplicitDifferentiation.jl
View on GitHub

ImplicitDifferentiation.jl

ImplicitDifferentiation.jl is a package for automatic differentiation of functions defined implicitly, i.e., forward mappings

\[f: x \in \mathbb{R}^n \longmapsto y(x) \in \mathbb{R}^m\]

whose output is defined by conditions

\[c(x,y(x)) = 0 \in \mathbb{R}^m\]

Background

Implicit differentiation is useful to differentiate through two types of functions:

  • Those for which automatic differentiation fails. Reasons can vary depending on your backend, but the most common include calls to external solvers, mutating operations or type restrictions.
  • Those for which automatic differentiation is very slow. A common example is iterative procedures like fixed point equations or optimization algorithms.

Please refer to Efficient and modular implicit differentiation for an introduction to the underlying theory.

Getting started

To install the stable version, open a Julia REPL and run:

julia> using Pkg; Pkg.add("ImplicitDifferentiation")

For the latest version, run this instead:

julia> using Pkg; Pkg.add(url="https://github.com/gdalle/ImplicitDifferentiation.jl")
  • DiffOpt.jl: differentiation of convex optimization problems
  • InferOpt.jl: approximate differentiation of combinatorial optimization problems
  • NonconvexUtils.jl: contains the original implementation from which this package drew inspiration
+Home · ImplicitDifferentiation.jl
View on GitHub

ImplicitDifferentiation.jl

ImplicitDifferentiation.jl is a package for automatic differentiation of functions defined implicitly, i.e., forward mappings

\[f: x \in \mathbb{R}^n \longmapsto y(x) \in \mathbb{R}^m\]

whose output is defined by conditions

\[c(x,y(x)) = 0 \in \mathbb{R}^m\]

Background

Implicit differentiation is useful to differentiate through two types of functions:

  • Those for which automatic differentiation fails. Reasons can vary depending on your backend, but the most common include calls to external solvers, mutating operations or type restrictions.
  • Those for which automatic differentiation is very slow. A common example is iterative procedures like fixed point equations or optimization algorithms.

Please refer to Efficient and modular implicit differentiation for an introduction to the underlying theory.

Getting started

To install the stable version, open a Julia REPL and run:

julia> using Pkg; Pkg.add("ImplicitDifferentiation")

For the latest version, run this instead:

julia> using Pkg; Pkg.add(url="https://github.com/gdalle/ImplicitDifferentiation.jl")
  • DiffOpt.jl: differentiation of convex optimization problems
  • InferOpt.jl: approximate differentiation of combinatorial optimization problems
  • NonconvexUtils.jl: contains the original implementation from which this package drew inspiration
diff --git a/previews/PR64/search/index.html b/previews/PR64/search/index.html index 40a0c75..3d03bca 100644 --- a/previews/PR64/search/index.html +++ b/previews/PR64/search/index.html @@ -1,2 +1,2 @@ -Search · ImplicitDifferentiation.jl

Loading search...

    +Search · ImplicitDifferentiation.jl

    Loading search...