From 0b8edc99a1be39876ebb14345c353f71bc687c94 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Mon, 22 Jul 2024 16:39:33 +0200 Subject: [PATCH 01/13] Bug fix in tutorial 'A more verbose getting started' --- ...rees.ipynb => abstract_syntax_trees.ipynb} | 0 docs/src/tutorials/abstract_syntax_trees.md | 435 ++++++++++++++++++ .../tutorials/getting_started_with_herb.ipynb | 149 +++++- 3 files changed, 565 insertions(+), 19 deletions(-) rename docs/src/tutorials/{syntax-trees.ipynb => abstract_syntax_trees.ipynb} (100%) create mode 100644 docs/src/tutorials/abstract_syntax_trees.md diff --git a/docs/src/tutorials/syntax-trees.ipynb b/docs/src/tutorials/abstract_syntax_trees.ipynb similarity index 100% rename from docs/src/tutorials/syntax-trees.ipynb rename to docs/src/tutorials/abstract_syntax_trees.ipynb diff --git a/docs/src/tutorials/abstract_syntax_trees.md b/docs/src/tutorials/abstract_syntax_trees.md new file mode 100644 index 0000000..85b1e31 --- /dev/null +++ b/docs/src/tutorials/abstract_syntax_trees.md @@ -0,0 +1,435 @@ +# Herb tutorial: Abstract syntax trees + +In this tutorial, you will learn + +- How to represent a computer program as an abstract syntax tree in Herb. +- How to replace parts of the tree to modify the program. + +## Abstract syntax trees + +The syntactic structure of a computer program can be represented in a hierarchical tree structure, a so-called _Abstract Syntax Tree (AST)_. The syntax of a programming language is typically defined using a formal grammar, a set of rules on how valid programs can be constructed. ASTs are derived from the grammar, but are abstractions in the sense that they omit details such as parenthesis, semicolons, etc. and only retain what's necessary to capture the program structure. + +In the context of program synthesis, ASTs are often used to define the space of all possible programs which is searched to find one that satisfies the given specifications. During the search process, different ASTs, each corresponding to a different program, are generated and evaluated until a suitable one is found. + +Each _node_ of the AST represents a construct in the program (e.g., a variable, an operator, a statement, or a function) and this construct corresponds to a rule in the formal grammar. +An _edge_ describes the relationship between constructs, and the tree structure captures the nesting of constructs. + +## A simple example program + +We first consider the simple program 5*(x+3). We will define a grammar that is sufficient to represent this program and use it to construct a AST for our program. + +### Define the grammar + + +```julia +using HerbCore, HerbGrammar, HerbInterpret + +grammar = @csgrammar begin + Number = |(0:9) + Number = x + Number = Number + Number + Number = Number * Number +end + +``` + + + 1: Number = 0 + 2: Number = 1 + 3: Number = 2 + 4: Number = 3 + 5: Number = 4 + 6: Number = 5 + 7: Number = 6 + 8: Number = 7 + 9: Number = 8 + 10: Number = 9 + 11: Number = x + 12: Number = Number + Number + 13: Number = Number * Number + + + +### Construct the syntax tree + +The AST of this program is shown in the diagram below. The number in each node refers to the index of the corresponding rule in our grammar. + +```mermaid + flowchart + id1((13)) --- + id2((6)) + id1 --- id3((12)) + id4((11)) + id5((4)) + id3 --- id4 + id3 --- id5 +``` + +In `Herb.jl`, the `HerbCore.RuleNode` is used to represent both an individual node, but also entire ASTs or sub-trees. This is achieved by nesting instances of `RuleNode`. A `RuleNode` can be instantiated by providing the index of the grammar rule that the node represents and a vector of child nodes. + + +```julia +syntaxtree = RuleNode(13, [RuleNode(6), RuleNode(12, [RuleNode(11), RuleNode(4)])]) +``` + + + 13{6,12{11,4}} + + +We can confirm that our AST is correct by displaying it in a more human-readable way, using `HerbGrammar.rulenode2expr` and by testing it on a few input examples using `HerbInterpret.execute_on_input`. + + +```julia +program = rulenode2expr(syntaxtree, grammar) +println(program) +``` + + 5 * (x + 3) + + + +```julia +# test solution on inputs +output = execute_on_input(grammar, syntaxtree, Dict(:x => 10)) +``` + + + 65 + + +## Another example: FizzBuzz + +Let's look at a more interesting example. +The program `fizbuzz()` is based on the popular _FizzBuzz_ problem. Given an integer number, the program simply returns a `String` of that number, but replace numbers divisible by 3 with `"Fizz"`, numbers divisible by 5 with `"Buzz"`, and number divisible by both 3 and 5 with `"FizzBuzz"`. + + +```julia +function fizzbuzz(x) + if x % 5 == 0 && x % 3 == 0 + return "FizzBuzz" + else + if x % 3 == 0 + return "Fizz" + else + if x % 5 == 0 + return "Buzz" + else + return string(x) + end + end + end +end +``` + + + fizzbuzz (generic function with 1 method) + + +### Define the grammar + +Let's define a grammar with all the rules that we need. + + +```julia +grammar_fizzbuzz = @csgrammar begin + Int = input1 + Int = 0 | 3 | 5 + String = "Fizz" | "Buzz" | "FizzBuzz" + String = string(Int) + Return = String + Int = Int % Int + Bool = Int == Int + Int = Bool ? Int : Int + Bool = Bool && Bool +end +``` + + + 1: Int = input1 + 2: Int = 0 + 3: Int = 3 + 4: Int = 5 + 5: String = Fizz + 6: String = Buzz + 7: String = FizzBuzz + 8: String = string(Int) + 9: Return = String + 10: Int = Int % Int + 11: Bool = Int == Int + 12: Int = if Bool + Int + else + Int + end + 13: Bool = Bool && Bool + + + +### Construct the syntax tree + +Given the grammar, the AST of `fizzbuzz()` looks like this: + +```mermaid +flowchart + id1((12)) --- id21((13)) + id1--- id22((9)) + id1--- id23((12)) + + id21 --- id31((11)) + id21 --- id32((11)) + + id31 --- id41((10)) + id31 --- id42((2)) + + id41 --- id51((1)) + id41 --- id52((4)) + + id32 --- id43((10)) + id32 --- id44((2)) + + id43 --- id53((1)) + id43 --- id54((3)) + + id22 --- id33((7)) + id23 --- id34((11)) + + id34 --- id45((10)) + id34 --- id46((2)) + + id45 --- id55((1)) + id45 --- id56((3)) + + id23 --- id35((9)) + id35 --- id47((5)) + + id23 --- id36((12)) + id36 --- id48((11)) + id48 --- id57((10)) + id57 --- id61((1)) + id57 --- id62((4)) + id48 --- id58((2)) + + id36 --- id49((9)) + id49 --- id59((6)) + + id36 --- id410((9)) + id410 --- id510((8)) + id510 --- id63((1)) + + + +``` + +As before, we use nest instanced of `RuleNode` to implement the AST. + + +```julia +fizzbuzz_syntaxtree = RuleNode(12, [ + RuleNode(13, [ + RuleNode(11, [ + RuleNode(10, [ + RuleNode(1), + RuleNode(4) + ]), + RuleNode(2) + ]), + RuleNode(11, [ + RuleNode(10, [ + RuleNode(1), + RuleNode(3) + ]), + RuleNode(2) + ]) + ]), + RuleNode(9, [ + RuleNode(7) + + ]), + RuleNode(12, [ + RuleNode(11, [ + RuleNode(10, [ + RuleNode(1), + RuleNode(3), + ]), + RuleNode(2) + ]), + RuleNode(9, [ + RuleNode(5) + ]), + RuleNode(12, [ + RuleNode(11, [ + RuleNode(10, [ + RuleNode(1), + RuleNode(4) + ]), + RuleNode(2) + ]), + RuleNode(9, [ + RuleNode(6) + ]), + RuleNode(9, [ + RuleNode(8, [ + RuleNode(1) + ]) + ]) + ]) + ]) + ]) +``` + + + 12{13{11{10{1,4}2}11{10{1,3}2}}9{7}12{11{10{1,3}2}9{5}12{11{10{1,4}2}9{6}9{8{1}}}}} + + +And we check our syntax tree is correct: + + +```julia +program = rulenode2expr(fizzbuzz_syntaxtree, grammar_fizzbuzz) +println(program) +``` + + if input1 % 5 == 0 && input1 % 3 == 0 + "FizzBuzz" + else + if input1 % 3 == 0 + "Fizz" + else + if input1 % 5 == 0 + "Buzz" + else + string(input1) + end + end + end + + + +```julia +# test solution on inputs +input = [Dict(:input1 => 3), Dict(:input1 => 5), Dict(:input1 =>15), Dict(:input1 => 22)] +output = execute_on_input(grammar_fizzbuzz, fizzbuzz_syntaxtree, input) +``` + + + 4-element Vector{Any}: + "Fizz" + "Buzz" + "FizzBuzz" + "22" + + +### Modify the AST/program + +There are several ways to modify an AST and hence, a program. You can + +- directly replace a node with `HerbCore.swap_node()` +- insert a rule node with `insert!` + +Let's modify our example such that if the input number is divisible by 3, the program returns "Buzz" instead of "Fizz". +We use `swap_node()` to replace the node of the AST that corresponds to rule 5 in the grammar (`String = Fizz`) with rule 6 (`String = Buzz`). To do so, `swap_node()` needs the tree that contains the node we want to modify, the new node we want to replace the node with, and the path to that node. + +Note that `swap_node()` modifies the tree, hence we make a deep copy of it first. + + +```julia +modified_fizzbuzz_syntaxtree = deepcopy(fizzbuzz_syntaxtree) +newnode = RuleNode(6) +path = [3, 2, 1] +swap_node(modified_fizzbuzz_syntaxtree, newnode, path) +``` + + + 6, + + +Let's confirm that we modified the AST, and hence the program, correctly: + + +```julia +program = rulenode2expr(modified_fizzbuzz_syntaxtree, grammar_fizzbuzz) +println(program) +``` + + if input1 % 5 == 0 && input1 % 3 == 0 + "FizzBuzz" + else + if input1 % 3 == 0 + "Buzz" + else + if input1 % 5 == 0 + "Buzz" + else + string(input1) + end + end + end + + + +```julia +# test solution on same inputs as before +output = execute_on_input(grammar_fizzbuzz, modified_fizzbuzz_syntaxtree, input) +``` + + + 4-element Vector{Any}: + "Buzz" + "Buzz" + "FizzBuzz" + "22" + + +An alternative way to modify the AST is by using `insert!()`. This requires to provide the location of the node that we want to as `NodeLoc`. `NodeLoc` points to a node in the tree and consists of the parent and the child index of the node. +Again, we make a deep copy of the original AST first. + + +```julia +modified_fizzbuzz_syntaxtree = deepcopy(fizzbuzz_syntaxtree) +# get the node we want to modify and instantiate a NodeLoc from it. +node = get_node_at_location(modified_fizzbuzz_syntaxtree, [3, 2, 1]) +nodeloc = NodeLoc(node, 0) +# replace the node +insert!(node, nodeloc, newnode) +``` + + + 6, + + +Again, we check that we modified the program as intended: + + +```julia +program = rulenode2expr(modified_fizzbuzz_syntaxtree, grammar_fizzbuzz) +println(program) +``` + + if input1 % 5 == 0 && input1 % 3 == 0 + "FizzBuzz" + else + if input1 % 3 == 0 + "Buzz" + else + if input1 % 5 == 0 + "Buzz" + else + string(input1) + end + end + end + + + +```julia +# test on same inputs as before +output = execute_on_input(grammar_fizzbuzz, modified_fizzbuzz_syntaxtree, input) +``` + + + 4-element Vector{Any}: + "Buzz" + "Buzz" + "FizzBuzz" + "22" + diff --git a/docs/src/tutorials/getting_started_with_herb.ipynb b/docs/src/tutorials/getting_started_with_herb.ipynb index 45abcbb..f9690e0 100644 --- a/docs/src/tutorials/getting_started_with_herb.ipynb +++ b/docs/src/tutorials/getting_started_with_herb.ipynb @@ -22,7 +22,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": {}, "outputs": [], "source": [ @@ -53,9 +53,33 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "1: Real = 0\n", + "2: Real = 1\n", + "3: Real = 2\n", + "4: Real = 3\n", + "5: Real = 4\n", + "6: Real = 5\n", + "7: Real = 6\n", + "8: Real = 7\n", + "9: Real = 8\n", + "10: Real = 9\n", + "11: Real = x\n", + "12: Real = Real + Real\n", + "13: Real = Real - Real\n", + "14: Real = Real * Real\n" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "g = HerbGrammar.@cfgrammar begin\n", " Real = |(0:9)\n", @@ -92,9 +116,25 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "5-element Vector{IOExample}:\n", + " IOExample(Dict{Symbol, Any}(:x => 1), 8)\n", + " IOExample(Dict{Symbol, Any}(:x => 2), 11)\n", + " IOExample(Dict{Symbol, Any}(:x => 3), 14)\n", + " IOExample(Dict{Symbol, Any}(:x => 4), 17)\n", + " IOExample(Dict{Symbol, Any}(:x => 5), 20)" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# Create input-output examples\n", "examples = [HerbSpecification.IOExample(Dict(:x => x), 3x+5) for x ∈ 1:5]" @@ -112,9 +152,20 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "Problem{Vector{IOExample}}(\"example\", IOExample[IOExample(Dict{Symbol, Any}(:x => 1), 8), IOExample(Dict{Symbol, Any}(:x => 2), 11), IOExample(Dict{Symbol, Any}(:x => 3), 14), IOExample(Dict{Symbol, Any}(:x => 4), 17), IOExample(Dict{Symbol, Any}(:x => 5), 20)])" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "problem = HerbSpecification.Problem(\"example\", examples)" ] @@ -145,18 +196,54 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "BFSIterator(GenericSolver(1: Real = 0\n", + "2: Real = 1\n", + "3: Real = 2\n", + "4: Real = 3\n", + "5: Real = 4\n", + "6: Real = 5\n", + "7: Real = 6\n", + "8: Real = 7\n", + "9: Real = 8\n", + "10: Real = 9\n", + "11: Real = x\n", + "12: Real = Real + Real\n", + "13: Real = Real - Real\n", + "14: Real = Real * Real\n", + ", SolverState(hole[Bool[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]],, Set{AbstractLocalConstraint}(), true), DataStructures.PriorityQueue{AbstractLocalConstraint, Int64, Base.Order.ForwardOrdering}(), nothing, true, false, 9223372036854775807, 9223372036854775807))" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "iterator = BFSIterator(g, :Real)" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "(12{14{4,11}6}, optimal_program)" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "synth(problem, iterator)" ] @@ -185,12 +272,20 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 9, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(14{14{4,9}8}, optimal_program)" + ] + } + ], "source": [ "problem = HerbSpecification.Problem(\"example2\", [HerbSpecification.IOExample(Dict(:x => x), 168) for x ∈ 1:5])\n", - "iterator = HerbSearch.BFSIterator(g, :Real, max_depth=4, max_size=30, max_time=180)\n", + "iterator = HerbSearch.BFSIterator(g, :Real, max_depth=4, max_size=30)\n", "expr = HerbSearch.synth(problem, iterator)\n", "print(expr)" ] @@ -207,9 +302,25 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 10, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33m\u001b[1m┌ \u001b[22m\u001b[39m\u001b[33m\u001b[1mWarning: \u001b[22m\u001b[39mUniformSolver is iterating over more than 1000000 solutions...\n", + "\u001b[33m\u001b[1m└ \u001b[22m\u001b[39m\u001b[90m@ HerbSearch ~/.julia/packages/HerbSearch/PIeQW/src/uniform_iterator.jl:96\u001b[39m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(12{13{6,1}14{14{3,10}10}}, optimal_program)" + ] + } + ], "source": [ "problem = HerbSpecification.Problem(\"example3\", [HerbSpecification.IOExample(Dict(:x => x), 167) for x ∈ 1:5])\n", "expr = HerbSearch.synth(problem, iterator)\n", @@ -219,15 +330,15 @@ ], "metadata": { "kernelspec": { - "display_name": "Julia 1.9.4", + "display_name": "Julia 1.10.3", "language": "julia", - "name": "julia-1.9" + "name": "julia-1.10" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", - "version": "1.9.4" + "version": "1.10.3" } }, "nbformat": 4, From fd5d48cf42af26ac10e195e8c8524b2a28fafda4 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Tue, 23 Jul 2024 08:05:02 +0200 Subject: [PATCH 02/13] Convert tutorials Verbose getting started and Getting started with Constraints into pluto notebooks. --- docs/make.jl | 33 +- .../getting_started_with_constraints.html | 17 + .../getting_started_with_constraints.jl | 1098 +++++++++++++++++ .../tutorials/getting_started_with_herb.html | 17 + .../tutorials/getting_started_with_herb.jl | 843 +++++++++++++ 5 files changed, 1998 insertions(+), 10 deletions(-) create mode 100644 docs/src/tutorials/getting_started_with_constraints.html create mode 100644 docs/src/tutorials/getting_started_with_constraints.jl create mode 100644 docs/src/tutorials/getting_started_with_herb.html create mode 100644 docs/src/tutorials/getting_started_with_herb.jl diff --git a/docs/make.jl b/docs/make.jl index b5c74d2..402bc80 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -1,5 +1,7 @@ using Documenter +using Pluto, PlutoSliderServer + using Herb using HerbConstraints @@ -9,18 +11,28 @@ using HerbInterpret using HerbCore using HerbSpecification -# Use jupyter.nbconver to convert notebooks to markdown -using PyCall -jupyter = pyimport("jupyterlab") -nbconvert = pyimport("nbconvert") -all_notebooks = readdir("docs/src/tutorials/") -for file in all_notebooks - if occursin("ipynb", file) - path = "docs/src/tutorials/" * file - run(`jupyter nbconvert --to markdown $path`) +# Create md file for tutorial that embeds html file +basedir = joinpath(@__DIR__, "src", "tutorials") +html_files = filter!(f -> occursin(r"\.html$", f), readdir(basedir)) # assumes all html file in directory are tutorials + +for f in html_files + html_path = joinpath(basedir, f) + filename = replace(f, ".html" => "") + md_path = joinpath(basedir, "$filename.md") + content = """ + # Tutorial $filename + + + """ + open(md_path, "w") do file + write( + file, + content + ) end end + makedocs( modules=[HerbConstraints, HerbSearch, HerbGrammar, HerbSpecification, HerbInterpret, HerbCore], authors="PONYs", @@ -38,7 +50,8 @@ makedocs( "Advanced Search Procedures" => "tutorials/advanced_search.md", "Top Down Iterator" => "tutorials/TopDown.md", "Getting started with Constraints" => "tutorials/getting_started_with_constraints.md", - "Working with custom interpreters" => "tutorials/working_with_interpreters.md" + "Working with custom interpreters" => "tutorials/working_with_interpreters.md", + "Abstract Syntax Trees" => "tutorials/abstract_syntax_trees.md", ], "Sub-Modules" => [ "HerbCore.jl" => "HerbCore/index.md", diff --git a/docs/src/tutorials/getting_started_with_constraints.html b/docs/src/tutorials/getting_started_with_constraints.html new file mode 100644 index 0000000..a16e59f --- /dev/null +++ b/docs/src/tutorials/getting_started_with_constraints.html @@ -0,0 +1,17 @@ + + + + + + + +
\ No newline at end of file diff --git a/docs/src/tutorials/getting_started_with_constraints.jl b/docs/src/tutorials/getting_started_with_constraints.jl new file mode 100644 index 0000000..b359985 --- /dev/null +++ b/docs/src/tutorials/getting_started_with_constraints.jl @@ -0,0 +1,1098 @@ +### A Pluto.jl notebook ### +# v0.19.43 + +using Markdown +using InteractiveUtils + +# ╔═╡ c5cb3782-c9af-4cf5-9e28-e1892c6442a2 +using HerbCore, HerbGrammar, HerbConstraints, HerbSearch + +# ╔═╡ c5509a19-43bc-44d3-baa9-9af83717b6e6 +md""" +# Getting started with HerbConstraints + +When enumerating programs using a grammar, we will encounter many redundant programs. For example, `x`, `-(-x)` and `1 * x` are syntactically different programs, but they have the same semantics. Grammar constraints aim to speed up synthesis by eliminating such redundant programs and thereby reducing the size of the program space. +""" + +# ╔═╡ 864bc658-9612-4439-9024-74668ba3f971 +md""" +### Setup + +For this tutorial, we need to import the following modules of the Herb.jl framework: + +* `HerbCore` for the necessary data strucutes, like `Hole`s and `RuleNode`s +* `HerbGrammar` to define the grammar +* `HerbConstraints` to define the constraints +* `HerbSearch` to execute a constrained enumeration + +We will also redefine the simple arithmetic grammar from the previous tutorial. +""" + +# ╔═╡ aacedef3-08d6-4bc0-a2c3-dd9a9aac340d +grammar = @cfgrammar begin + Int = 1 + Int = x + Int = - Int + Int = Int + Int + Int = Int * Int +end + +# ╔═╡ 1ff2dd58-01df-4cab-93d9-aabe1d07cc85 +md""" +### Working with constraints + +To show the effects of constraints, we will first enumerate all programs without constraints (up to a maximum size of 3 AST nodes). + +(To make sure the grammar doesn't have any constraints, we can clear the constraints using `clearconstraints!`. This is not needed at this point, but could come in handy if your REPL holds a reference to a constrained version of the grammar) +""" + +# ╔═╡ 62e41031-6866-46fc-b160-fb2f81f75c04 +begin + clearconstraints!(grammar) + iter_1 = BFSIterator(grammar, :Int, max_size=3) + + for program ∈ iter_1 + println(rulenode2expr(program, grammar)) + end + +end + +# ╔═╡ 76478bc7-e1b0-44f7-b1e0-95573db9f0e3 +md""" +Upon inspection, we can already see some redundant programs, like `1 * 1` and `-(-1)`. To eliminate these redundant programs, we will set up some constraints that prevent these patterns from appearing. Then we will create another iteratator to enumerate all programs that satisfy the defined grammar constraints. + +To make the forbidden pattern constraint general, we will use a special type of rulenode: `VarNode(:A)`. This node matches with any subtree and can be used to forbid multiple forbidden patterns using a single constraint. For example, `Forbidden(RuleNode(minus, [RuleNode(minus, [VarNode(:A)])])))` forbids: + +* `-(-1)` +* `-(-X)` +* `-(-(1 + 1))` +* `1 + -(-(1 + 1))` +* etc +""" + +# ╔═╡ 4d94057f-0b69-4d7d-8bde-900e6eb3e53f +begin + one = 1 + x = 2 + minus = 3 + plus = 4 + times = 5 + + addconstraint!(grammar, Forbidden(RuleNode(times, [RuleNode(one), VarNode(:A)]))) # forbid 1*A + addconstraint!(grammar, Forbidden(RuleNode(minus, [RuleNode(minus, [VarNode(:A)])]))) # forbid -(-A) + + iter_2 = BFSIterator(grammar, :Int, max_size=3) + + for program ∈ iter_2 + println(rulenode2expr(program, grammar)) + end +end + +# ╔═╡ 1f0ba3c1-d160-44f5-85c4-4089cbd6f284 +md""" +### Forbidden Constraint + +The `Forbidden` constraint forbids any subtree in the program that matches a given template tree. Such a template tree can consist of 3 node types: +* `RuleNode(1)`. Matches exactly the given rule. +* `DomainRuleNode(BitVector((0, 0, 0, 1, 1)), children)`. Matches any rule in its bitvector domain. In this case, rule 4 and 5. +* `VarNode(:A)`. Matches any subtree. If another VarNode of the same name is used, the subtrees have to be the same. +""" + +# ╔═╡ 030a150e-84ac-4aa6-9a08-5639538fb981 +begin + #this constraint forbids A+A and A*A + constraint_1 = Forbidden(DomainRuleNode(BitVector((0, 0, 0, 1, 1)), [VarNode(:A), VarNode(:A)])) + + # Without this constraint, we encounter 154 programs + clearconstraints!(grammar) + iter_3 = BFSIterator(grammar, :Int, max_size=5) + println(length(iter_3)) + + # With this constraint, we encounter 106 programs + clearconstraints!(grammar) + addconstraint!(grammar, constraint_1) + iter_4 = BFSIterator(grammar, :Int, max_size=5) + println(length(iter_4)) + +end + +# ╔═╡ 4e1e76b3-b6b2-4a1d-a749-9226a283522d +md""" +### Contains Constraint + +The `Contains` constraint enforces that a given rule appears in the program tree at least once. + +In the arithmetic grammar, this constraint can be used to ensure the input symbol `x` is used in the program. Otherwise, the program is just a constant. +""" + +# ╔═╡ 92eb7f60-88df-4212-a50b-b31bf386c720 +begin + clearconstraints!(grammar) + addconstraint!(grammar, Contains(2)) #rule 2 should be used in the program + iter_5 = BFSIterator(grammar, :Int, max_size=3) + + for program ∈ iter_5 + println(rulenode2expr(program, grammar)) + end +end + +# ╔═╡ 6c6becfb-1a7f-403a-9b04-6c4334c9f489 +md""" +### Contains Subtree Constraint + +Similarly to the `Contains` constraint, the `ContainsSubtree` can be used to enforce a given template tree is used in the program at least once. +""" + +# ╔═╡ ada60cc5-7aa1-41c4-828b-3d73c29fd087 +begin + clearconstraints!(grammar) + addconstraint!(grammar, ContainsSubtree(RuleNode(times, [RuleNode(x), RuleNode(x)]))) #x*x should be in the program tree + iter_6 = BFSIterator(grammar, :Int, max_size=4) + + for program ∈ iter_6 + println(rulenode2expr(program, grammar)) + end +end + +# ╔═╡ 3f42462a-0be0-42bd-a428-a666769054dd +md""" +### Ordered Constraint + +The `Ordered` constraint enforces an `<=` ordering on a provided list of variables. With this constraint, we can break symmetries based on commutativity. For example, `1+x` and `x+1` are semantically equivalent. By imposing an `Ordered` constraint, we can eliminate one of the symmetric variants. + +To define an `Ordered` constraint, we need to provide it with a template tree including at least two differently named `VarNode`s. And additionally, an ordering of the variables in the tree. + +In the upcoming example we will set up a template tree representing `a+b` and `a*b`. +Then, we will impose an ordering `a<=b` on all the subtrees that match the template. + +The result is that our iterator skips the redundant programs `x+1` and `x*1`, as they are already represented by `1+x` and `1*x`. + +""" + +# ╔═╡ 4e76bb87-c429-4547-accc-e304d4220f2d +begin + clearconstraints!(grammar) + + template_tree = DomainRuleNode(BitVector((0, 0, 0, 1, 1)), [VarNode(:a), VarNode(:b)]) + order = [:a, :b] + + addconstraint!(grammar, Ordered(template_tree, order)) + iter_7 = BFSIterator(grammar, :Int, max_size=3) + + for program ∈ iter_7 + println(rulenode2expr(program, grammar)) + end + +end + +# ╔═╡ 00f62b26-a15d-4525-bef0-857d37bb0d85 +md""" +### Forbidden Sequence Constraint + +The `ForbiddenSequence` constraints forbids a given sequence of rule nodes in a vertical path of the tree. + +An optional second argument, `ignore_if`, can be used to overrule the constraint in case any of the rules on the `ignore_if` list are present. + +Below we will define the constraint `ForbiddenSequence([plus, one], ignore_if=[times])`. It forbids an `1` after an `+` unless an `*` disrupts the sequence. + +This constraint will **forbid** the following programs: + +* x + 1 +* x + -1 +* x + -(-1) +* x + (x + 1) +* x * (x + 1) + +But it will **allow** the following program (as * disrupts the sequence): + +* x + (x * 1) + +""" + +# ╔═╡ 19d0a61b-46a3-4d53-b1d2-2b0e2650b56a +begin + constraint_2 = ForbiddenSequence([plus, one], ignore_if=[times]) + addconstraint!(grammar, constraint_2) + iter_8 = BFSIterator(grammar, :Int, max_size=3) + + for program ∈ iter_8 + println(rulenode2expr(program, grammar)) + end + +end + +# ╔═╡ 2bea7a99-c46a-4200-9ba8-1227a5806f2f +md""" +### Custom Constraint + +To implement a new constraint, we need to define two structs: an `AbstractGrammarConstraint` and an `AbstractLocalConstraint`. + +A **grammar constraint** is a high-level constraint on the grammar itself and does not refer to a location in the tree. For example, the `Forbidden` constraint is responsible for forbidding a template tree everywhere in the tree. To divide the work of constraint propagation, the grammar constraint will post several local constraints that are responsible for propagating the constraint at each particular location. + +A **local constraint** is a rooted version of a grammar constraint. Each local constraint holds a `path` field that points to a location in the tree where this constraint applies. +""" + +# ╔═╡ f4c15b60-5d45-4b75-9100-a7be2969d7ca +md""" +Suppose we want to implement a simple custom constraint that forbids a given `rule` twice in a row. + +Each time a new AST node is added to a tree, the `on_new_node` function is called to notify that an unseen node has been added to the tree at path `path`. Our grammar constraint has the opportunity to react to this event. In this example, we will post a new local constraint at the new location using the `post!` function. + +(Don't worry about the `HerbConstraints.` prefixes. Normally, constraints are defined within the HerbConstraints repository, so there is no need to specify the namespace) +""" + +# ╔═╡ bacc917b-2706-412d-9b85-deb4b6685323 +md""" +Next, we will define our local constraint. This constraint is responsible for propagating the constraint at a given path. The `propagate!` method can use several solver functions to manipulate the tree. The following **tree manipulations** can be used to remove rules from the domain of a hole at a given path: + +* `remove!(solver::Solver, path::Vector{Int}, rule_index::Int)` +* `remove!(solver::Solver, path::Vector{Int}, rules::Vector{Int})` +* `remove_all_but!(solver::Solver, path::Vector{Int}, new_domain::BitVector)` +* `remove_above!(solver::Solver, path::Vector{Int}, rule_index::Int)` +* `remove_below!(solver::Solver, path::Vector{Int}, rule_index::Int)` +* `make_equal!(solver::Solver, node1::AbstractRuleNode, node2::AbstractRuleNode)` (a high level manipulation that requires `node1` and `node2` to be in the tree) + +In addition to tree manipulations, the following solver functions can be used to communicate new information to the solver: + +* `set_infeasible!(solver)`. If a propagator detects an inconsistency, the solver should be notified and cancel any other scheduled propagators. +* `deactivate!(solver, constraint)`. If a constraint is satisfied, it should deactivate itself to prevent re-propagation. +* `post!(solver, constraint)` A constraint is allowed to post new local constraints. This might be helpful if a constraint can be reduced to a smaller constraint. + +The solver manages all constraints and the program tree we propagate on. Applying tree manipulations might cause a chain reaction of other propagators, so the shape of the tree might update as we propagate. The get the latest information about the tree, we should use the following getter functions: + +* `get_tree(solver)` returns the root node of the current (partial) program tree +* `isfeasible(solver)` returns the a flag indicating if the solver is not violating any (other) constriants. +* `get_path(solver, node)` returns the path at which the node is located. +* `get_node_at_location(solver, path)` returns the node that is currently at the given path (be aware that this instance might be replaced by manipulations). +* `get_hole_at_location(solver, path)` same as get node at location, but asserts the node is a hole (domain size >= 2). + +To get information about a node, we can use the following getter functions: + +* `isfilled(node)`. Returns true if the node is a `RuleNode` or has domain size 1. +* `get_rule(node)`. Get the rule of a filled node. +* `get_children(node)`. Get the children of a node. +* `node.domain[rule]`. Given the node is a hole, return true if `rule` is in the domain. + +Finally, another useful function for propagators is `pattern_match(node1, node2)`. This function compares two trees and returns a `PatternMatchResult` that indicates if the nodes match, and potentially indicate which holes need to be filled to complete the match. + +""" + +# ╔═╡ fef62621-716a-4f8a-85b4-d92e48b30bc6 +begin + """ + Forbids the consecutive application of the specified rule at path `path`. + """ + struct LocalForbidConsecutive <: AbstractLocalConstraint + path::Vector{Int} + rule::Int + end + + """ + Propagates the constraints by preventing a consecutive application of the specified rule. + """ + function HerbConstraints.propagate!(solver::Solver, constraint::LocalForbidConsecutive) + node = get_node_at_location(solver, constraint.path) + if isfilled(node) + if get_rule(node) == constraint.rule + #the specified rule is used, make sure the rule will not be used by any of the children + for (i, child) ∈ enumerate(get_children(node)) + if isfilled(child) + if get_rule(child) == constraint.rule + #the specified rule was used twice in a row, which is violating the constraint + set_infeasible!(solver) + return + end + elseif child.domain[constraint.rule] + child_path = push!(copy(constraint.path), i) + remove!(solver, child_path, constraint.rule) # remove the rule from the domain of the child + end + end + end + elseif node.domain[constraint.rule] + #our node is a hole with the specified rule in its domain + #we will now check if any of the children already uses the specified rule + softfail = false + for (i, child) ∈ enumerate(get_children(node)) + if isfilled(child) + if get_rule(child) == constraint.rule + #the child holds the specified rule, so the parent cannot have this rule + remove!(solver, constraint.path, constraint.rule) + end + elseif child.domain[constraint.rule] + #the child is a hole and contains the specified node. since there are 2 holes involved, we will softfail. + softfail = true + end + end + if softfail + #we cannot deactivate the constraint, because it needs to be repropagated + return + end + end + + #the constraint is satisfied and can be deactivated + HerbConstraints.deactivate!(solver, constraint) + end +end + +# ╔═╡ b00b039b-63ee-40d0-8f4b-d25539e596d5 +begin + """ + Forbids the consecutive application of the specified rule. + For example, CustomConstraint(4) forbids the tree 4(1, 4(1, 1)) as it applies rule 4 twice in a row. + """ + struct ForbidConsecutive <: AbstractGrammarConstraint + rule::Int + end + + """ + Post a local constraint on each new node that appears in the tree + """ + function HerbConstraints.on_new_node(solver::Solver, constraint::ForbidConsecutive, path::Vector{Int}) + HerbConstraints.post!(solver, LocalForbidConsecutive(path, constraint.rule)) + end +end + +# ╔═╡ e40e09fc-c697-4c83-90eb-2b758254128e +md""" +Posting a local constraint will trigger the initial propagation. To re-propagate, the constraint needs to be rescheduled for propagation. + +Whenever the tree is manipulated, we will make a `shouldschedule` check to see if our constraint needs to be rescheduled for propagation based on the manipulation. + +In our case, we want to repropagate if either: +* a tree manipulation occured at the `constraint.path` +* a tree manipulation occured at the child of the `constraint.path` +""" + +# ╔═╡ 4e0989a8-7e63-45eb-80c9-a3a2f97c357c + +""" +Gets called whenever an tree manipulation occurs at the given `path`. +Returns true iff the `constraint` should be rescheduled for propagation. +""" +function HerbConstraints.shouldschedule(solver::Solver, constraint::LocalForbidConsecutive, path::Vector{Int})::Bool + return (path == constraint.path) || (path == constraint.path[1:end-1]) +end + + +# ╔═╡ ce9ae2a2-f3e3-4693-9e6d-60e91128de34 +md""" +With all the components implemented, we can do a constrained enumeration using our new `ForbidConsecutive` constraint. +""" + +# ╔═╡ 89c165c6-3e04-4887-924f-364b25b21bcd +begin + clearconstraints!(grammar) + + addconstraint!(grammar, ForbidConsecutive(minus)) + addconstraint!(grammar, ForbidConsecutive(plus)) + addconstraint!(grammar, ForbidConsecutive(times)) + + iter = BFSIterator(grammar, :Int, max_size=6) + + for program ∈ iter + println(rulenode2expr(program, grammar)) + end +end + +# ╔═╡ 00000000-0000-0000-0000-000000000001 +PLUTO_PROJECT_TOML_CONTENTS = """ +[deps] +HerbConstraints = "1fa96474-3206-4513-b4fa-23913f296dfc" +HerbCore = "2b23ba43-8213-43cb-b5ea-38c12b45bd45" +HerbGrammar = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" +HerbSearch = "3008d8e8-f9aa-438a-92ed-26e9c7b4829f" + +[compat] +HerbConstraints = "~0.2.2" +HerbCore = "~0.3.0" +HerbGrammar = "~0.3.0" +HerbSearch = "~0.3.0" +""" + +# ╔═╡ 00000000-0000-0000-0000-000000000002 +PLUTO_MANIFEST_TOML_CONTENTS = """ +# This file is machine-generated - editing it directly is not advised + +julia_version = "1.10.4" +manifest_format = "2.0" +project_hash = "f550d76e5fc2109cce82f8527132b9929cc7271f" + +[[deps.AbstractTrees]] +git-tree-sha1 = "2d9c9a55f9c93e8887ad391fbae72f8ef55e1177" +uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" +version = "0.4.5" + +[[deps.ArgTools]] +uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" +version = "1.1.1" + +[[deps.ArnoldiMethod]] +deps = ["LinearAlgebra", "Random", "StaticArrays"] +git-tree-sha1 = "d57bd3762d308bded22c3b82d033bff85f6195c6" +uuid = "ec485272-7323-5ecc-a04f-4719b315124d" +version = "0.4.0" + +[[deps.Artifacts]] +uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" + +[[deps.Base64]] +uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" + +[[deps.Bzip2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "9e2a6b69137e6969bab0152632dcb3bc108c8bdd" +uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0" +version = "1.0.8+1" + +[[deps.Cairo_jll]] +deps = ["Artifacts", "Bzip2_jll", "CompilerSupportLibraries_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "a2f1c8c668c8e3cb4cca4e57a8efdb09067bb3fd" +uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a" +version = "1.18.0+2" + +[[deps.CommonSubexpressions]] +deps = ["MacroTools", "Test"] +git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7" +uuid = "bbf7d656-a473-5ed7-a52c-81e309532950" +version = "0.3.0" + +[[deps.Compat]] +deps = ["TOML", "UUIDs"] +git-tree-sha1 = "b1c55339b7c6c350ee89f2c1604299660525b248" +uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" +version = "4.15.0" +weakdeps = ["Dates", "LinearAlgebra"] + + [deps.Compat.extensions] + CompatLinearAlgebraExt = "LinearAlgebra" + +[[deps.CompilerSupportLibraries_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" +version = "1.1.1+0" + +[[deps.DataAPI]] +git-tree-sha1 = "abe83f3a2f1b857aac70ef8b269080af17764bbe" +uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" +version = "1.16.0" + +[[deps.DataStructures]] +deps = ["Compat", "InteractiveUtils", "OrderedCollections"] +git-tree-sha1 = "1d0a14036acb104d9e89698bd408f63ab58cdc82" +uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" +version = "0.18.20" + +[[deps.Dates]] +deps = ["Printf"] +uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" + +[[deps.Distributed]] +deps = ["Random", "Serialization", "Sockets"] +uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b" + +[[deps.DocStringExtensions]] +deps = ["LibGit2"] +git-tree-sha1 = "2fb1e02f2b635d0845df5d7c167fec4dd739b00d" +uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae" +version = "0.9.3" + +[[deps.Downloads]] +deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"] +uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" +version = "1.6.0" + +[[deps.Expat_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "1c6317308b9dc757616f0b5cb379db10494443a7" +uuid = "2e619515-83b5-522b-bb60-26c02a35a201" +version = "2.6.2+0" + +[[deps.FileWatching]] +uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee" + +[[deps.Fontconfig_jll]] +deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Zlib_jll"] +git-tree-sha1 = "db16beca600632c95fc8aca29890d83788dd8b23" +uuid = "a3f928ae-7b40-5064-980b-68af3947d34b" +version = "2.13.96+0" + +[[deps.FreeType2_jll]] +deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "5c1d8ae0efc6c2e7b1fc502cbe25def8f661b7bc" +uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7" +version = "2.13.2+0" + +[[deps.Gettext_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "XML2_jll"] +git-tree-sha1 = "9b02998aba7bf074d14de89f9d37ca24a1a0b046" +uuid = "78b55507-aeef-58d4-861c-77aaff3498b1" +version = "0.21.0+0" + +[[deps.Glib_jll]] +deps = ["Artifacts", "Gettext_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE2_jll", "Zlib_jll"] +git-tree-sha1 = "7c82e6a6cd34e9d935e9aa4051b66c6ff3af59ba" +uuid = "7746bdde-850d-59dc-9ae8-88ece973131d" +version = "2.80.2+0" + +[[deps.Graphite2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "344bf40dcab1073aca04aa0df4fb092f920e4011" +uuid = "3b182d85-2403-5c21-9c21-1e1f0cc25472" +version = "1.3.14+0" + +[[deps.Graphs]] +deps = ["ArnoldiMethod", "Compat", "DataStructures", "Distributed", "Inflate", "LinearAlgebra", "Random", "SharedArrays", "SimpleTraits", "SparseArrays", "Statistics"] +git-tree-sha1 = "4f2b57488ac7ee16124396de4f2bbdd51b2602ad" +uuid = "86223c79-3864-5bf0-83f7-82e725a168b6" +version = "1.11.0" + +[[deps.HarfBuzz_ICU_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "6ccbc4fdf65c8197738c2d68cc55b74b19c97ac2" +uuid = "655565e8-fb53-5cb3-b0cd-aec1ca0647ea" +version = "2.8.1+0" + +[[deps.HarfBuzz_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "129acf094d168394e80ee1dc4bc06ec835e510a3" +uuid = "2e76f6c2-a576-52d4-95c1-20adfe4de566" +version = "2.8.1+1" + +[[deps.HerbConstraints]] +deps = ["DataStructures", "HerbCore", "HerbGrammar", "MLStyle"] +git-tree-sha1 = "2e54da1d19119847b242d1ceda212b180cca36a9" +uuid = "1fa96474-3206-4513-b4fa-23913f296dfc" +version = "0.2.2" + +[[deps.HerbCore]] +git-tree-sha1 = "923877c2715b8166d7ba9f9be2136d70eed87725" +uuid = "2b23ba43-8213-43cb-b5ea-38c12b45bd45" +version = "0.3.0" + +[[deps.HerbGrammar]] +deps = ["AbstractTrees", "DataStructures", "HerbCore", "Serialization", "TreeView"] +git-tree-sha1 = "b4cbf9712dbb3ab281ff4ed517d3cd6bc12f3078" +uuid = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" +version = "0.3.0" + +[[deps.HerbInterpret]] +deps = ["HerbCore", "HerbGrammar", "HerbSpecification"] +git-tree-sha1 = "9e19b4ee5f29eb8bb9b1049524728b38e878eca2" +uuid = "5bbddadd-02c5-4713-84b8-97364418cca7" +version = "0.1.3" + +[[deps.HerbSearch]] +deps = ["DataStructures", "HerbConstraints", "HerbCore", "HerbGrammar", "HerbInterpret", "HerbSpecification", "Logging", "MLStyle", "Random", "StatsBase"] +git-tree-sha1 = "472e3f427c148f334dde3837b0bb1549897ed00a" +uuid = "3008d8e8-f9aa-438a-92ed-26e9c7b4829f" +version = "0.3.0" + +[[deps.HerbSpecification]] +git-tree-sha1 = "5385b81e40c3cd62aeea591319896148036863c9" +uuid = "6d54aada-062f-46d8-85cf-a1ceaf058a06" +version = "0.1.0" + +[[deps.ICU_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "20b6765a3016e1fca0c9c93c80d50061b94218b7" +uuid = "a51ab1cf-af8e-5615-a023-bc2c838bba6b" +version = "69.1.0+0" + +[[deps.Inflate]] +git-tree-sha1 = "ea8031dea4aff6bd41f1df8f2fdfb25b33626381" +uuid = "d25df0c9-e2be-5dd7-82c8-3ad0b3e990b9" +version = "0.1.4" + +[[deps.InteractiveUtils]] +deps = ["Markdown"] +uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" + +[[deps.IrrationalConstants]] +git-tree-sha1 = "630b497eafcc20001bba38a4651b327dcfc491d2" +uuid = "92d709cd-6900-40b7-9082-c6be49f344b6" +version = "0.2.2" + +[[deps.JLLWrappers]] +deps = ["Artifacts", "Preferences"] +git-tree-sha1 = "7e5d6779a1e09a36db2a7b6cff50942a0a7d0fca" +uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210" +version = "1.5.0" + +[[deps.JpegTurbo_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "c84a835e1a09b289ffcd2271bf2a337bbdda6637" +uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8" +version = "3.0.3+0" + +[[deps.LERC_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "bf36f528eec6634efc60d7ec062008f171071434" +uuid = "88015f11-f218-50d7-93a8-a6af411a945d" +version = "3.0.0+1" + +[[deps.LLVMOpenMP_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "d986ce2d884d49126836ea94ed5bfb0f12679713" +uuid = "1d63c593-3942-5779-bab2-d838dc0a180e" +version = "15.0.7+0" + +[[deps.LZO_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "70c5da094887fd2cae843b8db33920bac4b6f07d" +uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac" +version = "2.10.2+0" + +[[deps.LaTeXStrings]] +git-tree-sha1 = "50901ebc375ed41dbf8058da26f9de442febbbec" +uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" +version = "1.3.1" + +[[deps.LibCURL]] +deps = ["LibCURL_jll", "MozillaCACerts_jll"] +uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" +version = "0.6.4" + +[[deps.LibCURL_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"] +uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" +version = "8.4.0+0" + +[[deps.LibGit2]] +deps = ["Base64", "LibGit2_jll", "NetworkOptions", "Printf", "SHA"] +uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" + +[[deps.LibGit2_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll"] +uuid = "e37daf67-58a4-590a-8e99-b0245dd2ffc5" +version = "1.6.4+0" + +[[deps.LibSSH2_jll]] +deps = ["Artifacts", "Libdl", "MbedTLS_jll"] +uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" +version = "1.11.0+1" + +[[deps.Libdl]] +uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" + +[[deps.Libffi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "0b4a5d71f3e5200a7dff793393e09dfc2d874290" +uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490" +version = "3.2.2+1" + +[[deps.Libgcrypt_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgpg_error_jll"] +git-tree-sha1 = "9fd170c4bbfd8b935fdc5f8b7aa33532c991a673" +uuid = "d4300ac3-e22c-5743-9152-c294e39db1e4" +version = "1.8.11+0" + +[[deps.Libgpg_error_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "fbb1f2bef882392312feb1ede3615ddc1e9b99ed" +uuid = "7add5ba3-2f88-524e-9cd5-f83b8a55f7b8" +version = "1.49.0+0" + +[[deps.Libiconv_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "f9557a255370125b405568f9767d6d195822a175" +uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531" +version = "1.17.0+0" + +[[deps.Libmount_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "0c4f9c4f1a50d8f35048fa0532dabbadf702f81e" +uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9" +version = "2.40.1+0" + +[[deps.Libtiff_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "LERC_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"] +git-tree-sha1 = "3eb79b0ca5764d4799c06699573fd8f533259713" +uuid = "89763e89-9b03-5906-acba-b20f662cd828" +version = "4.4.0+0" + +[[deps.Libuuid_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "5ee6203157c120d79034c748a2acba45b82b8807" +uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700" +version = "2.40.1+0" + +[[deps.LinearAlgebra]] +deps = ["Libdl", "OpenBLAS_jll", "libblastrampoline_jll"] +uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + +[[deps.LittleCMS_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pkg"] +git-tree-sha1 = "110897e7db2d6836be22c18bffd9422218ee6284" +uuid = "d3a379c0-f9a3-5b72-a4c0-6bf4d2e8af0f" +version = "2.12.0+0" + +[[deps.LogExpFunctions]] +deps = ["DocStringExtensions", "IrrationalConstants", "LinearAlgebra"] +git-tree-sha1 = "18144f3e9cbe9b15b070288eef858f71b291ce37" +uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688" +version = "0.3.27" + + [deps.LogExpFunctions.extensions] + LogExpFunctionsChainRulesCoreExt = "ChainRulesCore" + LogExpFunctionsChangesOfVariablesExt = "ChangesOfVariables" + LogExpFunctionsInverseFunctionsExt = "InverseFunctions" + + [deps.LogExpFunctions.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + ChangesOfVariables = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0" + InverseFunctions = "3587e190-3f89-42d0-90ee-14403ec27112" + +[[deps.Logging]] +uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" + +[[deps.MLStyle]] +git-tree-sha1 = "bc38dff0548128765760c79eb7388a4b37fae2c8" +uuid = "d8e11817-5142-5d16-987a-aa16d5891078" +version = "0.4.17" + +[[deps.MacroTools]] +deps = ["Markdown", "Random"] +git-tree-sha1 = "2fa9ee3e63fd3a4f7a9a4f4744a52f4856de82df" +uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" +version = "0.5.13" + +[[deps.Markdown]] +deps = ["Base64"] +uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" + +[[deps.MbedTLS_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" +version = "2.28.2+1" + +[[deps.Missings]] +deps = ["DataAPI"] +git-tree-sha1 = "ec4f7fbeab05d7747bdf98eb74d130a2a2ed298d" +uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28" +version = "1.2.0" + +[[deps.Mmap]] +uuid = "a63ad114-7e13-5084-954f-fe012c677804" + +[[deps.MozillaCACerts_jll]] +uuid = "14a3606d-f60d-562e-9121-12d972cd8159" +version = "2023.1.10" + +[[deps.NetworkOptions]] +uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" +version = "1.2.0" + +[[deps.OpenBLAS_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] +uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" +version = "0.3.23+4" + +[[deps.OpenJpeg_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libtiff_jll", "LittleCMS_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "76374b6e7f632c130e78100b166e5a48464256f8" +uuid = "643b3616-a352-519d-856d-80112ee9badc" +version = "2.4.0+0" + +[[deps.OpenSSL_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "a12e56c72edee3ce6b96667745e6cbbe5498f200" +uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95" +version = "1.1.23+0" + +[[deps.OrderedCollections]] +git-tree-sha1 = "dfdf5519f235516220579f949664f1bf44e741c5" +uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" +version = "1.6.3" + +[[deps.PCRE2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "efcefdf7-47ab-520b-bdef-62a2eaa19f15" +version = "10.42.0+1" + +[[deps.Pixman_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "LLVMOpenMP_jll", "Libdl"] +git-tree-sha1 = "35621f10a7531bc8fa58f74610b1bfb70a3cfc6b" +uuid = "30392449-352a-5448-841d-b1acce4e97dc" +version = "0.43.4+0" + +[[deps.Pkg]] +deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] +uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" +version = "1.10.0" + +[[deps.Poppler_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "Glib_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "OpenJpeg_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "02148a0cb2532f22c0589ceb75c110e168fb3d1f" +uuid = "9c32591e-4766-534b-9725-b71a8799265b" +version = "21.9.0+0" + +[[deps.PrecompileTools]] +deps = ["Preferences"] +git-tree-sha1 = "5aa36f7049a63a1528fe8f7c3f2113413ffd4e1f" +uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a" +version = "1.2.1" + +[[deps.Preferences]] +deps = ["TOML"] +git-tree-sha1 = "9306f6085165d270f7e3db02af26a400d580f5c6" +uuid = "21216c6a-2e73-6563-6e65-726566657250" +version = "1.4.3" + +[[deps.Printf]] +deps = ["Unicode"] +uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" + +[[deps.REPL]] +deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] +uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" + +[[deps.Random]] +deps = ["SHA"] +uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" + +[[deps.Requires]] +deps = ["UUIDs"] +git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7" +uuid = "ae029012-a4dd-5104-9daa-d747884805df" +version = "1.3.0" + +[[deps.SHA]] +uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" +version = "0.7.0" + +[[deps.Serialization]] +uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" + +[[deps.SharedArrays]] +deps = ["Distributed", "Mmap", "Random", "Serialization"] +uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383" + +[[deps.SimpleTraits]] +deps = ["InteractiveUtils", "MacroTools"] +git-tree-sha1 = "5d7e3f4e11935503d3ecaf7186eac40602e7d231" +uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d" +version = "0.9.4" + +[[deps.Sockets]] +uuid = "6462fe0b-24de-5631-8697-dd941f90decc" + +[[deps.SortingAlgorithms]] +deps = ["DataStructures"] +git-tree-sha1 = "66e0a8e672a0bdfca2c3f5937efb8538b9ddc085" +uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c" +version = "1.2.1" + +[[deps.SparseArrays]] +deps = ["Libdl", "LinearAlgebra", "Random", "Serialization", "SuiteSparse_jll"] +uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" +version = "1.10.0" + +[[deps.StaticArrays]] +deps = ["LinearAlgebra", "PrecompileTools", "Random", "StaticArraysCore"] +git-tree-sha1 = "9ae599cd7529cfce7fea36cf00a62cfc56f0f37c" +uuid = "90137ffa-7385-5640-81b9-e52037218182" +version = "1.9.4" + + [deps.StaticArrays.extensions] + StaticArraysChainRulesCoreExt = "ChainRulesCore" + StaticArraysStatisticsExt = "Statistics" + + [deps.StaticArrays.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" + +[[deps.StaticArraysCore]] +git-tree-sha1 = "36b3d696ce6366023a0ea192b4cd442268995a0d" +uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" +version = "1.4.2" + +[[deps.Statistics]] +deps = ["LinearAlgebra", "SparseArrays"] +uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" +version = "1.10.0" + +[[deps.StatsAPI]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "1ff449ad350c9c4cbc756624d6f8a8c3ef56d3ed" +uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0" +version = "1.7.0" + +[[deps.StatsBase]] +deps = ["DataAPI", "DataStructures", "LinearAlgebra", "LogExpFunctions", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"] +git-tree-sha1 = "5cf7606d6cef84b543b483848d4ae08ad9832b21" +uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91" +version = "0.34.3" + +[[deps.SuiteSparse_jll]] +deps = ["Artifacts", "Libdl", "libblastrampoline_jll"] +uuid = "bea87d4a-7f5b-5778-9afe-8cc45184846c" +version = "7.2.1+1" + +[[deps.TOML]] +deps = ["Dates"] +uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" +version = "1.0.3" + +[[deps.Tar]] +deps = ["ArgTools", "SHA"] +uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" +version = "1.10.0" + +[[deps.Test]] +deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] +uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[[deps.TikzGraphs]] +deps = ["Graphs", "LaTeXStrings", "TikzPictures"] +git-tree-sha1 = "e8f41ed9a2cabf6699d9906c195bab1f773d4ca7" +uuid = "b4f28e30-c73f-5eaf-a395-8a9db949a742" +version = "1.4.0" + +[[deps.TikzPictures]] +deps = ["LaTeXStrings", "Poppler_jll", "Requires", "tectonic_jll"] +git-tree-sha1 = "79e2d29b216ef24a0f4f905532b900dcf529aa06" +uuid = "37f6aa50-8035-52d0-81c2-5a1d08754b2d" +version = "3.5.0" + +[[deps.TreeView]] +deps = ["CommonSubexpressions", "Graphs", "MacroTools", "TikzGraphs"] +git-tree-sha1 = "41ddcefb625f2ab0f4d9f2081c2da1af2ccbbf8b" +uuid = "39424ebd-4cf3-5550-a685-96706a953f40" +version = "0.5.1" + +[[deps.UUIDs]] +deps = ["Random", "SHA"] +uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" + +[[deps.Unicode]] +uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" + +[[deps.XML2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Zlib_jll"] +git-tree-sha1 = "52ff2af32e591541550bd753c0da8b9bc92bb9d9" +uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a" +version = "2.12.7+0" + +[[deps.XSLT_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgcrypt_jll", "Libgpg_error_jll", "Libiconv_jll", "Pkg", "XML2_jll", "Zlib_jll"] +git-tree-sha1 = "91844873c4085240b95e795f692c4cec4d805f8a" +uuid = "aed1982a-8fda-507f-9586-7b0439959a61" +version = "1.1.34+0" + +[[deps.Xorg_libX11_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libxcb_jll", "Xorg_xtrans_jll"] +git-tree-sha1 = "afead5aba5aa507ad5a3bf01f58f82c8d1403495" +uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc" +version = "1.8.6+0" + +[[deps.Xorg_libXau_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "6035850dcc70518ca32f012e46015b9beeda49d8" +uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec" +version = "1.0.11+0" + +[[deps.Xorg_libXdmcp_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "34d526d318358a859d7de23da945578e8e8727b7" +uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05" +version = "1.1.4+0" + +[[deps.Xorg_libXext_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "d2d1a5c49fae4ba39983f63de6afcbea47194e85" +uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3" +version = "1.3.6+0" + +[[deps.Xorg_libXrender_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "47e45cd78224c53109495b3e324df0c37bb61fbe" +uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa" +version = "0.9.11+0" + +[[deps.Xorg_libpthread_stubs_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "8fdda4c692503d44d04a0603d9ac0982054635f9" +uuid = "14d82f49-176c-5ed1-bb49-ad3f5cbd8c74" +version = "0.1.1+0" + +[[deps.Xorg_libxcb_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "XSLT_jll", "Xorg_libXau_jll", "Xorg_libXdmcp_jll", "Xorg_libpthread_stubs_jll"] +git-tree-sha1 = "b4bfde5d5b652e22b9c790ad00af08b6d042b97d" +uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b" +version = "1.15.0+0" + +[[deps.Xorg_xtrans_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e92a1a012a10506618f10b7047e478403a046c77" +uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10" +version = "1.5.0+0" + +[[deps.Zlib_jll]] +deps = ["Libdl"] +uuid = "83775a58-1f1d-513f-b197-d71354ab007a" +version = "1.2.13+1" + +[[deps.Zstd_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e678132f07ddb5bfa46857f0d7620fb9be675d3b" +uuid = "3161d3a3-bdf6-5164-811a-617609db77b4" +version = "1.5.6+0" + +[[deps.libblastrampoline_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" +version = "5.8.0+1" + +[[deps.libpng_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "d7015d2e18a5fd9a4f47de711837e980519781a4" +uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f" +version = "1.6.43+1" + +[[deps.nghttp2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" +version = "1.52.0+1" + +[[deps.p7zip_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" +version = "17.4.0+2" + +[[deps.tectonic_jll]] +deps = ["Artifacts", "Fontconfig_jll", "FreeType2_jll", "Graphite2_jll", "HarfBuzz_ICU_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "OpenSSL_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "54867b00af20c70b52a1f9c00043864d8b926a21" +uuid = "d7dd28d6-a5e6-559c-9131-7eb760cdacc5" +version = "0.13.1+0" +""" + +# ╔═╡ Cell order: +# ╟─c5509a19-43bc-44d3-baa9-9af83717b6e6 +# ╟─864bc658-9612-4439-9024-74668ba3f971 +# ╠═c5cb3782-c9af-4cf5-9e28-e1892c6442a2 +# ╠═aacedef3-08d6-4bc0-a2c3-dd9a9aac340d +# ╟─1ff2dd58-01df-4cab-93d9-aabe1d07cc85 +# ╠═62e41031-6866-46fc-b160-fb2f81f75c04 +# ╟─76478bc7-e1b0-44f7-b1e0-95573db9f0e3 +# ╠═4d94057f-0b69-4d7d-8bde-900e6eb3e53f +# ╟─1f0ba3c1-d160-44f5-85c4-4089cbd6f284 +# ╠═030a150e-84ac-4aa6-9a08-5639538fb981 +# ╟─4e1e76b3-b6b2-4a1d-a749-9226a283522d +# ╠═92eb7f60-88df-4212-a50b-b31bf386c720 +# ╟─6c6becfb-1a7f-403a-9b04-6c4334c9f489 +# ╠═ada60cc5-7aa1-41c4-828b-3d73c29fd087 +# ╟─3f42462a-0be0-42bd-a428-a666769054dd +# ╠═4e76bb87-c429-4547-accc-e304d4220f2d +# ╟─00f62b26-a15d-4525-bef0-857d37bb0d85 +# ╠═19d0a61b-46a3-4d53-b1d2-2b0e2650b56a +# ╟─2bea7a99-c46a-4200-9ba8-1227a5806f2f +# ╟─f4c15b60-5d45-4b75-9100-a7be2969d7ca +# ╠═b00b039b-63ee-40d0-8f4b-d25539e596d5 +# ╟─bacc917b-2706-412d-9b85-deb4b6685323 +# ╠═fef62621-716a-4f8a-85b4-d92e48b30bc6 +# ╟─e40e09fc-c697-4c83-90eb-2b758254128e +# ╠═4e0989a8-7e63-45eb-80c9-a3a2f97c357c +# ╟─ce9ae2a2-f3e3-4693-9e6d-60e91128de34 +# ╠═89c165c6-3e04-4887-924f-364b25b21bcd +# ╟─00000000-0000-0000-0000-000000000001 +# ╟─00000000-0000-0000-0000-000000000002 diff --git a/docs/src/tutorials/getting_started_with_herb.html b/docs/src/tutorials/getting_started_with_herb.html new file mode 100644 index 0000000..990d2a6 --- /dev/null +++ b/docs/src/tutorials/getting_started_with_herb.html @@ -0,0 +1,17 @@ + + + + + + + +
\ No newline at end of file diff --git a/docs/src/tutorials/getting_started_with_herb.jl b/docs/src/tutorials/getting_started_with_herb.jl new file mode 100644 index 0000000..83adbd8 --- /dev/null +++ b/docs/src/tutorials/getting_started_with_herb.jl @@ -0,0 +1,843 @@ +### A Pluto.jl notebook ### +# v0.19.43 + +using Markdown +using InteractiveUtils + +# ╔═╡ 1defafc5-ce65-42f0-90cd-de9e8895ec90 +using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret, HerbConstraints + +# ╔═╡ adabb7f1-7952-43c4-88a9-54225b40aaf0 +md""" +# Search + +This notebook describes how you can search a program space as defined by a grammar. +Specifically, we will look at example-based search, where the goal is to find a program that is able to transform the inputs of every example to the corresponding output. +""" + +# ╔═╡ 841f097d-a389-4dd2-9ad3-1a2292568634 +md""" +### Setup +First, we start with the setup. We need to access to all the function in the Herb.jl framework. +""" + +# ╔═╡ db7fe47b-ab3e-4705-b6ac-2733b9e81434 +md""" +### Defining the program space + +Next, we start by creating a grammar. We define a context-free grammar (cfg) as a [`HerbGrammar.ContextSpecificGrammar`](@ref) without any constraints. A cfg is just a simple set of production rules for defining combinations of terminal symbols (in our case real numbers). + +Contrary, we could define a context-sensitive grammar, when the production rules only hold in a certain context. However, for more information on this, please see our tutorial on [defining grammars](defining_grammars.md). + +For now, we specify a simple grammar for dealing with integers and explain all the rules individually: + +1. First, we specify our interval `[0:9]` on real numbers and also constrain them to be integer. +2. Then, we can also use the variable `x` to hold an integer. +3. The third rule determines we can add two integers. +4. The fourth rule determines we can subtract an integer from another. +5. Finally, we also allow the multiplication of two integers. + +If you run this cell, you can see all the rules rolled out. +""" + +# ╔═╡ 763b378b-66f9-481e-a3da-ca37825eb255 +g = HerbGrammar.@cfgrammar begin + Real = |(0:9) + Real = x + Real = Real + Real + Real = Real - Real + Real = Real * Real +end + +# ╔═╡ 6d01dfe8-9048-4696-916c-b33fbc97268b +md""" +### Defining the problem +""" + +# ╔═╡ 56a63f9e-b484-4d85-af3e-de2cc4476e09 +md""" +As mentioned before, we are looking at example-based search. +This means that the problem is defined by a set of input-output examples. +A single example hence consists of an input and an output. +The input is defined as a dictionary, with a value assigned to each variable in the grammar. +It is important to write the variable name as a `Symbol` instead of a string. +A `Symbol` in Julia is written with a colon prefix, i.e. `:x`. +The output of the input-output example is just a single value for this specific grammar, but could possibly relate to e.g. arrays of values, too. + +In the cell below we automatically generate some examples for `x` assigning values `1-5`. +""" + +# ╔═╡ 8bf48b7a-0ff5-4015-81d3-ed2eeeceff1c +# Create input-output examples +examples = [HerbSpecification.IOExample(Dict(:x => x), 3x+5) for x ∈ 1:5] + +# ╔═╡ 2baa7f33-c86d-40e2-9253-720ec19e4c43 +md""" +Now that we have some input-output examples, we can define the problem. +Next to the examples, a problem also contains a name meant to link to the file path, which can be used to keep track of current examples. +For now, this is irrelevant, and you can give the program any name you like. +""" + +# ╔═╡ 059306d1-a45a-4727-ab01-1b5b80187999 +problem_1 = HerbSpecification.Problem("example", examples) + +# ╔═╡ 0f090666-9007-417e-a801-8231fffa19f3 +md""" +### Searching + +Now that we have defined the search space and the goal of the search, we can start the search. + +Of course, our problem is underdefined as there might be multiple programs that satisfy our examples. +Let us consider the case where we also have a ternary if-then-else operator and standard boolean operators in our grammar: we could synthesize the program `x ≤ 5 ? 3x+5 : 0`. +This program satisfies all our examples, but we don't expect it to generalize very well. + +To search through a program space, we first need to define a [`HerbSearch.ProgramIterator`](@ref), which can be instantiated with different iterators, for now we use a simple [`HerbSearch.BFSIterator`](@ref). For more advanced search methods check out our tutorial on [advanced search](.advanced_search.md). For more information about iterators, check out our tutorial on [working with interpreters](.working_with_interpreters.md). + +In general, we assume that a smaller program is more general than a larger program. +Therefore, we search for the smallest program in our grammar that satisfies our examples. +This can be done using a breadth-first search over the program/search space. + +This search is very basic; it makes use of an enumeration technique, where we enumerate programs one-by-one until we find a program that matches our examples. The search procedure has a built-in default evaluator to verify the candidate programs with the given input. The search procedure also has a built-in search procedure using breadth-first search. + +So, we only need to give our grammar and the problem to our search procedure, along with a starting `Symbol`, in our case a `Real`. +""" + +# ╔═╡ d553f37b-bc8a-4426-a98b-fb195ed994d9 +iterator_1 = BFSIterator(g, :Real) + +# ╔═╡ e1910236-9783-4989-a014-c3f7ccdf33d3 +synth(problem_1, iterator_1) + +# ╔═╡ 4c9f6236-2a84-4e76-86ab-c1fd1c7a1ba1 +md""" +As you can see, the search procedure found the correct program! +""" + +# ╔═╡ 8f87eed7-dfa1-47e4-a9b3-8e2a8a966207 +md""" +### Defining the search procedure + +In the previous case, we used the built-ins of the search procedure. However, we can also give a custom enumerator to the search procedure and define a few more values. + +We first define a new problem to test with, we are looking for the programs that can compute the value `167`. We immediately pass the examples to the problem and then set up the new search. + +Search is done by passing the grammar, the problem and the starting point like before. We now also specify the enumeration function to be used, and now we use depth-first search. Then, we give the maximum depth of the programs we want to search for `(3)`, the maximum number of nodes in the Abstract Syntax Tree that exists during search `(10)`, and the maximum time in seconds allowed for the search. +""" + +# ╔═╡ cdab3f55-37e4-4aee-bae1-14d3475cbdcd +begin + problem_2 = HerbSpecification.Problem("example2", [HerbSpecification.IOExample(Dict(:x => x), 168) for x ∈ 1:5]) + iterator_2 = HerbSearch.BFSIterator(g, :Real, max_depth=4, max_size=30) + expr_2 = HerbSearch.synth(problem_2, iterator_2) + print(expr_2) +end + +# ╔═╡ 5ad86beb-eb25-4bae-b0c2-a33d1a38581a +md""" +We see that our synthesizer can find a program to construct the value `168`, though a fun experiment would be trying to get the value `167`, what do you think would happen? You can try below, using the same iterator. + +In any case, this concludes our first introduction to the `Herb.jl` program synthesis framework. You can see more examples in this repository, or explore yourself. Enjoy! +""" + +# ╔═╡ c06d09a5-138a-4821-8a60-074fa7ec026d +begin + problem_3 = HerbSpecification.Problem("example3", [HerbSpecification.IOExample(Dict(:x => x), 167) for x ∈ 1:5]) + expr_3 = HerbSearch.synth(problem_3, iterator_2) + print(expr_3) +end + +# ╔═╡ 00000000-0000-0000-0000-000000000001 +PLUTO_PROJECT_TOML_CONTENTS = """ +[deps] +HerbConstraints = "1fa96474-3206-4513-b4fa-23913f296dfc" +HerbGrammar = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" +HerbInterpret = "5bbddadd-02c5-4713-84b8-97364418cca7" +HerbSearch = "3008d8e8-f9aa-438a-92ed-26e9c7b4829f" +HerbSpecification = "6d54aada-062f-46d8-85cf-a1ceaf058a06" + +[compat] +HerbConstraints = "~0.2.2" +HerbGrammar = "~0.3.0" +HerbInterpret = "~0.1.3" +HerbSearch = "~0.3.0" +HerbSpecification = "~0.1.0" +""" + +# ╔═╡ 00000000-0000-0000-0000-000000000002 +PLUTO_MANIFEST_TOML_CONTENTS = """ +# This file is machine-generated - editing it directly is not advised + +julia_version = "1.10.4" +manifest_format = "2.0" +project_hash = "ec293deacc6b3fb0969b68da9eec7f6d9738e0e4" + +[[deps.AbstractTrees]] +git-tree-sha1 = "2d9c9a55f9c93e8887ad391fbae72f8ef55e1177" +uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" +version = "0.4.5" + +[[deps.ArgTools]] +uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" +version = "1.1.1" + +[[deps.ArnoldiMethod]] +deps = ["LinearAlgebra", "Random", "StaticArrays"] +git-tree-sha1 = "d57bd3762d308bded22c3b82d033bff85f6195c6" +uuid = "ec485272-7323-5ecc-a04f-4719b315124d" +version = "0.4.0" + +[[deps.Artifacts]] +uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" + +[[deps.Base64]] +uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" + +[[deps.Bzip2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "9e2a6b69137e6969bab0152632dcb3bc108c8bdd" +uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0" +version = "1.0.8+1" + +[[deps.Cairo_jll]] +deps = ["Artifacts", "Bzip2_jll", "CompilerSupportLibraries_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "a2f1c8c668c8e3cb4cca4e57a8efdb09067bb3fd" +uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a" +version = "1.18.0+2" + +[[deps.CommonSubexpressions]] +deps = ["MacroTools", "Test"] +git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7" +uuid = "bbf7d656-a473-5ed7-a52c-81e309532950" +version = "0.3.0" + +[[deps.Compat]] +deps = ["TOML", "UUIDs"] +git-tree-sha1 = "b1c55339b7c6c350ee89f2c1604299660525b248" +uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" +version = "4.15.0" +weakdeps = ["Dates", "LinearAlgebra"] + + [deps.Compat.extensions] + CompatLinearAlgebraExt = "LinearAlgebra" + +[[deps.CompilerSupportLibraries_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" +version = "1.1.1+0" + +[[deps.DataAPI]] +git-tree-sha1 = "abe83f3a2f1b857aac70ef8b269080af17764bbe" +uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" +version = "1.16.0" + +[[deps.DataStructures]] +deps = ["Compat", "InteractiveUtils", "OrderedCollections"] +git-tree-sha1 = "1d0a14036acb104d9e89698bd408f63ab58cdc82" +uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" +version = "0.18.20" + +[[deps.Dates]] +deps = ["Printf"] +uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" + +[[deps.Distributed]] +deps = ["Random", "Serialization", "Sockets"] +uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b" + +[[deps.DocStringExtensions]] +deps = ["LibGit2"] +git-tree-sha1 = "2fb1e02f2b635d0845df5d7c167fec4dd739b00d" +uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae" +version = "0.9.3" + +[[deps.Downloads]] +deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"] +uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" +version = "1.6.0" + +[[deps.Expat_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "1c6317308b9dc757616f0b5cb379db10494443a7" +uuid = "2e619515-83b5-522b-bb60-26c02a35a201" +version = "2.6.2+0" + +[[deps.FileWatching]] +uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee" + +[[deps.Fontconfig_jll]] +deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Zlib_jll"] +git-tree-sha1 = "db16beca600632c95fc8aca29890d83788dd8b23" +uuid = "a3f928ae-7b40-5064-980b-68af3947d34b" +version = "2.13.96+0" + +[[deps.FreeType2_jll]] +deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "5c1d8ae0efc6c2e7b1fc502cbe25def8f661b7bc" +uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7" +version = "2.13.2+0" + +[[deps.Gettext_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "XML2_jll"] +git-tree-sha1 = "9b02998aba7bf074d14de89f9d37ca24a1a0b046" +uuid = "78b55507-aeef-58d4-861c-77aaff3498b1" +version = "0.21.0+0" + +[[deps.Glib_jll]] +deps = ["Artifacts", "Gettext_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE2_jll", "Zlib_jll"] +git-tree-sha1 = "7c82e6a6cd34e9d935e9aa4051b66c6ff3af59ba" +uuid = "7746bdde-850d-59dc-9ae8-88ece973131d" +version = "2.80.2+0" + +[[deps.Graphite2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "344bf40dcab1073aca04aa0df4fb092f920e4011" +uuid = "3b182d85-2403-5c21-9c21-1e1f0cc25472" +version = "1.3.14+0" + +[[deps.Graphs]] +deps = ["ArnoldiMethod", "Compat", "DataStructures", "Distributed", "Inflate", "LinearAlgebra", "Random", "SharedArrays", "SimpleTraits", "SparseArrays", "Statistics"] +git-tree-sha1 = "4f2b57488ac7ee16124396de4f2bbdd51b2602ad" +uuid = "86223c79-3864-5bf0-83f7-82e725a168b6" +version = "1.11.0" + +[[deps.HarfBuzz_ICU_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "6ccbc4fdf65c8197738c2d68cc55b74b19c97ac2" +uuid = "655565e8-fb53-5cb3-b0cd-aec1ca0647ea" +version = "2.8.1+0" + +[[deps.HarfBuzz_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "129acf094d168394e80ee1dc4bc06ec835e510a3" +uuid = "2e76f6c2-a576-52d4-95c1-20adfe4de566" +version = "2.8.1+1" + +[[deps.HerbConstraints]] +deps = ["DataStructures", "HerbCore", "HerbGrammar", "MLStyle"] +git-tree-sha1 = "2e54da1d19119847b242d1ceda212b180cca36a9" +uuid = "1fa96474-3206-4513-b4fa-23913f296dfc" +version = "0.2.2" + +[[deps.HerbCore]] +git-tree-sha1 = "923877c2715b8166d7ba9f9be2136d70eed87725" +uuid = "2b23ba43-8213-43cb-b5ea-38c12b45bd45" +version = "0.3.0" + +[[deps.HerbGrammar]] +deps = ["AbstractTrees", "DataStructures", "HerbCore", "Serialization", "TreeView"] +git-tree-sha1 = "b4cbf9712dbb3ab281ff4ed517d3cd6bc12f3078" +uuid = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" +version = "0.3.0" + +[[deps.HerbInterpret]] +deps = ["HerbCore", "HerbGrammar", "HerbSpecification"] +git-tree-sha1 = "9e19b4ee5f29eb8bb9b1049524728b38e878eca2" +uuid = "5bbddadd-02c5-4713-84b8-97364418cca7" +version = "0.1.3" + +[[deps.HerbSearch]] +deps = ["DataStructures", "HerbConstraints", "HerbCore", "HerbGrammar", "HerbInterpret", "HerbSpecification", "Logging", "MLStyle", "Random", "StatsBase"] +git-tree-sha1 = "472e3f427c148f334dde3837b0bb1549897ed00a" +uuid = "3008d8e8-f9aa-438a-92ed-26e9c7b4829f" +version = "0.3.0" + +[[deps.HerbSpecification]] +git-tree-sha1 = "5385b81e40c3cd62aeea591319896148036863c9" +uuid = "6d54aada-062f-46d8-85cf-a1ceaf058a06" +version = "0.1.0" + +[[deps.ICU_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "20b6765a3016e1fca0c9c93c80d50061b94218b7" +uuid = "a51ab1cf-af8e-5615-a023-bc2c838bba6b" +version = "69.1.0+0" + +[[deps.Inflate]] +git-tree-sha1 = "ea8031dea4aff6bd41f1df8f2fdfb25b33626381" +uuid = "d25df0c9-e2be-5dd7-82c8-3ad0b3e990b9" +version = "0.1.4" + +[[deps.InteractiveUtils]] +deps = ["Markdown"] +uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" + +[[deps.IrrationalConstants]] +git-tree-sha1 = "630b497eafcc20001bba38a4651b327dcfc491d2" +uuid = "92d709cd-6900-40b7-9082-c6be49f344b6" +version = "0.2.2" + +[[deps.JLLWrappers]] +deps = ["Artifacts", "Preferences"] +git-tree-sha1 = "7e5d6779a1e09a36db2a7b6cff50942a0a7d0fca" +uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210" +version = "1.5.0" + +[[deps.JpegTurbo_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "c84a835e1a09b289ffcd2271bf2a337bbdda6637" +uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8" +version = "3.0.3+0" + +[[deps.LERC_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "bf36f528eec6634efc60d7ec062008f171071434" +uuid = "88015f11-f218-50d7-93a8-a6af411a945d" +version = "3.0.0+1" + +[[deps.LLVMOpenMP_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "d986ce2d884d49126836ea94ed5bfb0f12679713" +uuid = "1d63c593-3942-5779-bab2-d838dc0a180e" +version = "15.0.7+0" + +[[deps.LZO_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "70c5da094887fd2cae843b8db33920bac4b6f07d" +uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac" +version = "2.10.2+0" + +[[deps.LaTeXStrings]] +git-tree-sha1 = "50901ebc375ed41dbf8058da26f9de442febbbec" +uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" +version = "1.3.1" + +[[deps.LibCURL]] +deps = ["LibCURL_jll", "MozillaCACerts_jll"] +uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" +version = "0.6.4" + +[[deps.LibCURL_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"] +uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" +version = "8.4.0+0" + +[[deps.LibGit2]] +deps = ["Base64", "LibGit2_jll", "NetworkOptions", "Printf", "SHA"] +uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" + +[[deps.LibGit2_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll"] +uuid = "e37daf67-58a4-590a-8e99-b0245dd2ffc5" +version = "1.6.4+0" + +[[deps.LibSSH2_jll]] +deps = ["Artifacts", "Libdl", "MbedTLS_jll"] +uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" +version = "1.11.0+1" + +[[deps.Libdl]] +uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" + +[[deps.Libffi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "0b4a5d71f3e5200a7dff793393e09dfc2d874290" +uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490" +version = "3.2.2+1" + +[[deps.Libgcrypt_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgpg_error_jll"] +git-tree-sha1 = "9fd170c4bbfd8b935fdc5f8b7aa33532c991a673" +uuid = "d4300ac3-e22c-5743-9152-c294e39db1e4" +version = "1.8.11+0" + +[[deps.Libgpg_error_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "fbb1f2bef882392312feb1ede3615ddc1e9b99ed" +uuid = "7add5ba3-2f88-524e-9cd5-f83b8a55f7b8" +version = "1.49.0+0" + +[[deps.Libiconv_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "f9557a255370125b405568f9767d6d195822a175" +uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531" +version = "1.17.0+0" + +[[deps.Libmount_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "0c4f9c4f1a50d8f35048fa0532dabbadf702f81e" +uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9" +version = "2.40.1+0" + +[[deps.Libtiff_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "LERC_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"] +git-tree-sha1 = "3eb79b0ca5764d4799c06699573fd8f533259713" +uuid = "89763e89-9b03-5906-acba-b20f662cd828" +version = "4.4.0+0" + +[[deps.Libuuid_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "5ee6203157c120d79034c748a2acba45b82b8807" +uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700" +version = "2.40.1+0" + +[[deps.LinearAlgebra]] +deps = ["Libdl", "OpenBLAS_jll", "libblastrampoline_jll"] +uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + +[[deps.LittleCMS_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pkg"] +git-tree-sha1 = "110897e7db2d6836be22c18bffd9422218ee6284" +uuid = "d3a379c0-f9a3-5b72-a4c0-6bf4d2e8af0f" +version = "2.12.0+0" + +[[deps.LogExpFunctions]] +deps = ["DocStringExtensions", "IrrationalConstants", "LinearAlgebra"] +git-tree-sha1 = "18144f3e9cbe9b15b070288eef858f71b291ce37" +uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688" +version = "0.3.27" + + [deps.LogExpFunctions.extensions] + LogExpFunctionsChainRulesCoreExt = "ChainRulesCore" + LogExpFunctionsChangesOfVariablesExt = "ChangesOfVariables" + LogExpFunctionsInverseFunctionsExt = "InverseFunctions" + + [deps.LogExpFunctions.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + ChangesOfVariables = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0" + InverseFunctions = "3587e190-3f89-42d0-90ee-14403ec27112" + +[[deps.Logging]] +uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" + +[[deps.MLStyle]] +git-tree-sha1 = "bc38dff0548128765760c79eb7388a4b37fae2c8" +uuid = "d8e11817-5142-5d16-987a-aa16d5891078" +version = "0.4.17" + +[[deps.MacroTools]] +deps = ["Markdown", "Random"] +git-tree-sha1 = "2fa9ee3e63fd3a4f7a9a4f4744a52f4856de82df" +uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" +version = "0.5.13" + +[[deps.Markdown]] +deps = ["Base64"] +uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" + +[[deps.MbedTLS_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" +version = "2.28.2+1" + +[[deps.Missings]] +deps = ["DataAPI"] +git-tree-sha1 = "ec4f7fbeab05d7747bdf98eb74d130a2a2ed298d" +uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28" +version = "1.2.0" + +[[deps.Mmap]] +uuid = "a63ad114-7e13-5084-954f-fe012c677804" + +[[deps.MozillaCACerts_jll]] +uuid = "14a3606d-f60d-562e-9121-12d972cd8159" +version = "2023.1.10" + +[[deps.NetworkOptions]] +uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" +version = "1.2.0" + +[[deps.OpenBLAS_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] +uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" +version = "0.3.23+4" + +[[deps.OpenJpeg_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libtiff_jll", "LittleCMS_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "76374b6e7f632c130e78100b166e5a48464256f8" +uuid = "643b3616-a352-519d-856d-80112ee9badc" +version = "2.4.0+0" + +[[deps.OpenSSL_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "a12e56c72edee3ce6b96667745e6cbbe5498f200" +uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95" +version = "1.1.23+0" + +[[deps.OrderedCollections]] +git-tree-sha1 = "dfdf5519f235516220579f949664f1bf44e741c5" +uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" +version = "1.6.3" + +[[deps.PCRE2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "efcefdf7-47ab-520b-bdef-62a2eaa19f15" +version = "10.42.0+1" + +[[deps.Pixman_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "LLVMOpenMP_jll", "Libdl"] +git-tree-sha1 = "35621f10a7531bc8fa58f74610b1bfb70a3cfc6b" +uuid = "30392449-352a-5448-841d-b1acce4e97dc" +version = "0.43.4+0" + +[[deps.Pkg]] +deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] +uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" +version = "1.10.0" + +[[deps.Poppler_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "Glib_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "OpenJpeg_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "02148a0cb2532f22c0589ceb75c110e168fb3d1f" +uuid = "9c32591e-4766-534b-9725-b71a8799265b" +version = "21.9.0+0" + +[[deps.PrecompileTools]] +deps = ["Preferences"] +git-tree-sha1 = "5aa36f7049a63a1528fe8f7c3f2113413ffd4e1f" +uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a" +version = "1.2.1" + +[[deps.Preferences]] +deps = ["TOML"] +git-tree-sha1 = "9306f6085165d270f7e3db02af26a400d580f5c6" +uuid = "21216c6a-2e73-6563-6e65-726566657250" +version = "1.4.3" + +[[deps.Printf]] +deps = ["Unicode"] +uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" + +[[deps.REPL]] +deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] +uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" + +[[deps.Random]] +deps = ["SHA"] +uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" + +[[deps.Requires]] +deps = ["UUIDs"] +git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7" +uuid = "ae029012-a4dd-5104-9daa-d747884805df" +version = "1.3.0" + +[[deps.SHA]] +uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" +version = "0.7.0" + +[[deps.Serialization]] +uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" + +[[deps.SharedArrays]] +deps = ["Distributed", "Mmap", "Random", "Serialization"] +uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383" + +[[deps.SimpleTraits]] +deps = ["InteractiveUtils", "MacroTools"] +git-tree-sha1 = "5d7e3f4e11935503d3ecaf7186eac40602e7d231" +uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d" +version = "0.9.4" + +[[deps.Sockets]] +uuid = "6462fe0b-24de-5631-8697-dd941f90decc" + +[[deps.SortingAlgorithms]] +deps = ["DataStructures"] +git-tree-sha1 = "66e0a8e672a0bdfca2c3f5937efb8538b9ddc085" +uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c" +version = "1.2.1" + +[[deps.SparseArrays]] +deps = ["Libdl", "LinearAlgebra", "Random", "Serialization", "SuiteSparse_jll"] +uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" +version = "1.10.0" + +[[deps.StaticArrays]] +deps = ["LinearAlgebra", "PrecompileTools", "Random", "StaticArraysCore"] +git-tree-sha1 = "9ae599cd7529cfce7fea36cf00a62cfc56f0f37c" +uuid = "90137ffa-7385-5640-81b9-e52037218182" +version = "1.9.4" + + [deps.StaticArrays.extensions] + StaticArraysChainRulesCoreExt = "ChainRulesCore" + StaticArraysStatisticsExt = "Statistics" + + [deps.StaticArrays.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" + +[[deps.StaticArraysCore]] +git-tree-sha1 = "36b3d696ce6366023a0ea192b4cd442268995a0d" +uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" +version = "1.4.2" + +[[deps.Statistics]] +deps = ["LinearAlgebra", "SparseArrays"] +uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" +version = "1.10.0" + +[[deps.StatsAPI]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "1ff449ad350c9c4cbc756624d6f8a8c3ef56d3ed" +uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0" +version = "1.7.0" + +[[deps.StatsBase]] +deps = ["DataAPI", "DataStructures", "LinearAlgebra", "LogExpFunctions", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"] +git-tree-sha1 = "5cf7606d6cef84b543b483848d4ae08ad9832b21" +uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91" +version = "0.34.3" + +[[deps.SuiteSparse_jll]] +deps = ["Artifacts", "Libdl", "libblastrampoline_jll"] +uuid = "bea87d4a-7f5b-5778-9afe-8cc45184846c" +version = "7.2.1+1" + +[[deps.TOML]] +deps = ["Dates"] +uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" +version = "1.0.3" + +[[deps.Tar]] +deps = ["ArgTools", "SHA"] +uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" +version = "1.10.0" + +[[deps.Test]] +deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] +uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[[deps.TikzGraphs]] +deps = ["Graphs", "LaTeXStrings", "TikzPictures"] +git-tree-sha1 = "e8f41ed9a2cabf6699d9906c195bab1f773d4ca7" +uuid = "b4f28e30-c73f-5eaf-a395-8a9db949a742" +version = "1.4.0" + +[[deps.TikzPictures]] +deps = ["LaTeXStrings", "Poppler_jll", "Requires", "tectonic_jll"] +git-tree-sha1 = "79e2d29b216ef24a0f4f905532b900dcf529aa06" +uuid = "37f6aa50-8035-52d0-81c2-5a1d08754b2d" +version = "3.5.0" + +[[deps.TreeView]] +deps = ["CommonSubexpressions", "Graphs", "MacroTools", "TikzGraphs"] +git-tree-sha1 = "41ddcefb625f2ab0f4d9f2081c2da1af2ccbbf8b" +uuid = "39424ebd-4cf3-5550-a685-96706a953f40" +version = "0.5.1" + +[[deps.UUIDs]] +deps = ["Random", "SHA"] +uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" + +[[deps.Unicode]] +uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" + +[[deps.XML2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Zlib_jll"] +git-tree-sha1 = "52ff2af32e591541550bd753c0da8b9bc92bb9d9" +uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a" +version = "2.12.7+0" + +[[deps.XSLT_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgcrypt_jll", "Libgpg_error_jll", "Libiconv_jll", "Pkg", "XML2_jll", "Zlib_jll"] +git-tree-sha1 = "91844873c4085240b95e795f692c4cec4d805f8a" +uuid = "aed1982a-8fda-507f-9586-7b0439959a61" +version = "1.1.34+0" + +[[deps.Xorg_libX11_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libxcb_jll", "Xorg_xtrans_jll"] +git-tree-sha1 = "afead5aba5aa507ad5a3bf01f58f82c8d1403495" +uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc" +version = "1.8.6+0" + +[[deps.Xorg_libXau_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "6035850dcc70518ca32f012e46015b9beeda49d8" +uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec" +version = "1.0.11+0" + +[[deps.Xorg_libXdmcp_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "34d526d318358a859d7de23da945578e8e8727b7" +uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05" +version = "1.1.4+0" + +[[deps.Xorg_libXext_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "d2d1a5c49fae4ba39983f63de6afcbea47194e85" +uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3" +version = "1.3.6+0" + +[[deps.Xorg_libXrender_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "47e45cd78224c53109495b3e324df0c37bb61fbe" +uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa" +version = "0.9.11+0" + +[[deps.Xorg_libpthread_stubs_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "8fdda4c692503d44d04a0603d9ac0982054635f9" +uuid = "14d82f49-176c-5ed1-bb49-ad3f5cbd8c74" +version = "0.1.1+0" + +[[deps.Xorg_libxcb_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "XSLT_jll", "Xorg_libXau_jll", "Xorg_libXdmcp_jll", "Xorg_libpthread_stubs_jll"] +git-tree-sha1 = "b4bfde5d5b652e22b9c790ad00af08b6d042b97d" +uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b" +version = "1.15.0+0" + +[[deps.Xorg_xtrans_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e92a1a012a10506618f10b7047e478403a046c77" +uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10" +version = "1.5.0+0" + +[[deps.Zlib_jll]] +deps = ["Libdl"] +uuid = "83775a58-1f1d-513f-b197-d71354ab007a" +version = "1.2.13+1" + +[[deps.Zstd_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e678132f07ddb5bfa46857f0d7620fb9be675d3b" +uuid = "3161d3a3-bdf6-5164-811a-617609db77b4" +version = "1.5.6+0" + +[[deps.libblastrampoline_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" +version = "5.8.0+1" + +[[deps.libpng_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "d7015d2e18a5fd9a4f47de711837e980519781a4" +uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f" +version = "1.6.43+1" + +[[deps.nghttp2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" +version = "1.52.0+1" + +[[deps.p7zip_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" +version = "17.4.0+2" + +[[deps.tectonic_jll]] +deps = ["Artifacts", "Fontconfig_jll", "FreeType2_jll", "Graphite2_jll", "HarfBuzz_ICU_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "OpenSSL_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "54867b00af20c70b52a1f9c00043864d8b926a21" +uuid = "d7dd28d6-a5e6-559c-9131-7eb760cdacc5" +version = "0.13.1+0" +""" + +# ╔═╡ Cell order: +# ╟─adabb7f1-7952-43c4-88a9-54225b40aaf0 +# ╟─841f097d-a389-4dd2-9ad3-1a2292568634 +# ╠═1defafc5-ce65-42f0-90cd-de9e8895ec90 +# ╟─db7fe47b-ab3e-4705-b6ac-2733b9e81434 +# ╠═763b378b-66f9-481e-a3da-ca37825eb255 +# ╟─6d01dfe8-9048-4696-916c-b33fbc97268b +# ╟─56a63f9e-b484-4d85-af3e-de2cc4476e09 +# ╠═8bf48b7a-0ff5-4015-81d3-ed2eeeceff1c +# ╟─2baa7f33-c86d-40e2-9253-720ec19e4c43 +# ╠═059306d1-a45a-4727-ab01-1b5b80187999 +# ╟─0f090666-9007-417e-a801-8231fffa19f3 +# ╠═d553f37b-bc8a-4426-a98b-fb195ed994d9 +# ╠═e1910236-9783-4989-a014-c3f7ccdf33d3 +# ╟─4c9f6236-2a84-4e76-86ab-c1fd1c7a1ba1 +# ╟─8f87eed7-dfa1-47e4-a9b3-8e2a8a966207 +# ╠═cdab3f55-37e4-4aee-bae1-14d3475cbdcd +# ╟─5ad86beb-eb25-4bae-b0c2-a33d1a38581a +# ╠═c06d09a5-138a-4821-8a60-074fa7ec026d +# ╟─00000000-0000-0000-0000-000000000001 +# ╟─00000000-0000-0000-0000-000000000002 From 115187a9c144045c7e0c11be517bef849bcf1868 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Tue, 23 Jul 2024 08:07:11 +0200 Subject: [PATCH 03/13] Pluot notebook for AST tutorial --- docs/src/tutorials/abstract_syntax_trees.jl | 1051 +++++++++++++++++++ 1 file changed, 1051 insertions(+) create mode 100644 docs/src/tutorials/abstract_syntax_trees.jl diff --git a/docs/src/tutorials/abstract_syntax_trees.jl b/docs/src/tutorials/abstract_syntax_trees.jl new file mode 100644 index 0000000..07f8c7b --- /dev/null +++ b/docs/src/tutorials/abstract_syntax_trees.jl @@ -0,0 +1,1051 @@ +### A Pluto.jl notebook ### +# v0.19.43 + +using Markdown +using InteractiveUtils + +# ╔═╡ c784cbc3-19fc-45c9-b344-db10cf7a81fa +begin + using PlutoUI + + using HerbCore + using HerbGrammar + using HerbInterpret +end + +# ╔═╡ 65fbf850-74ae-4ea4-85f0-683095c73fba +md""" +# Herb tutorial: Abstract syntax trees""" + +# ╔═╡ 2493e9db-8b8e-4ef6-8379-f50ad26aab88 +md""" +In this tutorial, you will learn + +- How to represent a computer program as an abstract syntax tree in Herb. +- How to replace parts of the tree to modify the program.""" + +# ╔═╡ 8ff96964-e39e-4762-ae03-e9166e163fca +md""" +## Abstract syntax trees + +The syntactic structure of a computer program can be represented in a hierarchical tree structure, a so-called _Abstract Syntax Tree (AST)_. The syntax of a programming language is typically defined using a formal grammar, a set of rules on how valid programs can be constructed. ASTs are derived from the grammar, but are abstractions in the sense that they omit details such as parenthesis, semicolons, etc. and only retain what's necessary to capture the program structure. + +In the context of program synthesis, ASTs are often used to define the space of all possible programs which is searched to find one that satisfies the given specifications. During the search process, different ASTs, each corresponding to a different program, are generated and evaluated until a suitable one is found. + +Each _node_ of the AST represents a construct in the program (e.g., a variable, an operator, a statement, or a function) and this construct corresponds to a rule in the formal grammar. +An _edge_ describes the relationship between constructs, and the tree structure captures the nesting of constructs. """ + +# ╔═╡ bff155ab-ff0e-452b-a8f5-fe744e41a30f +md""" +## A simple example program + +We first consider the simple program 5*(x+3). We will define a grammar that is sufficient to represent this program and use it to construct a AST for our program.""" + +# ╔═╡ caa3446e-c5df-4dac-905a-20515f681074 +md""" +### Define the grammar""" + +# ╔═╡ 9f54f013-e8b9-4e0d-8bac-9867f5d1a393 +grammar = @csgrammar begin + Number = |(0:9) + Number = x + Number = Number + Number + Number = Number * Number + end + +# ╔═╡ 46fbbe87-ee6e-4874-9708-20a42347ff18 +md""" +### Construct the syntax tree""" + +# ╔═╡ 5dc6be9c-e4d9-4fdb-90bf-f4c59bb66a70 +md""" +The AST of this program is shown in the diagram below. The number in each node refers to the index of the corresponding rule in our grammar. """ + +# ╔═╡ 64c2a6ce-5e3b-413b-bcb7-84936137439f +md""" +```mermaid + flowchart + id1((13)) --- + id2((6)) + id1 --- id3((12)) + id4((11)) + id5((4)) + id3 --- id4 + id3 --- id5 +``` +""" + +# ╔═╡ 64f6f1e3-5cbc-4e37-a806-d4fd45c20855 +md""" +```mermaid + flowchart + id1((13)) --- + id2((6)) + id1 --- id3((12)) + id4((11)) + id5((4)) + id3 --- id4 + id3 --- id5 +```""" + +# ╔═╡ 29b37a82-d022-453e-bf65-672aa94e4c87 +md""" +In `Herb.jl`, the `HerbCore.RuleNode` is used to represent both an individual node, but also entire ASTs or sub-trees. This is achieved by nesting instances of `RuleNode`. A `RuleNode` can be instantiated by providing the index of the grammar rule that the node represents and a vector of child nodes. """ + +# ╔═╡ 822d9601-284d-4d30-9551-605684f83d90 +syntaxtree = RuleNode(13, [RuleNode(6), RuleNode(12, [RuleNode(11), RuleNode(4)])]) + +# ╔═╡ 351210d1-20b6-4695-b9fe-f1136d4447d5 +md""" +We can confirm that our AST is correct by displaying it in a more human-readable way, using `HerbGrammar.rulenode2expr` and by testing it on a few input examples using `HerbInterpret.execute_on_input`.""" + +# ╔═╡ dc882fd5-d0fd-4a7d-8d5b-40516f3a3bcb +rulenode2expr(syntaxtree, grammar) + +# ╔═╡ cbfaa1b6-f3c5-490a-9e54-006262d0c727 +# test solution on inputs +execute_on_input(grammar, syntaxtree, Dict(:x => 10)) + +# ╔═╡ 6e018fd3-7626-48b2-b56a-240ae62a1ac4 +md""" +## Another example: FizzBuzz + +Let's look at a more interesting example. +The program `fizbuzz()` is based on the popular _FizzBuzz_ problem. Given an integer number, the program simply returns a `String` of that number, but replace numbers divisible by 3 with `\"Fizz\"`, numbers divisible by 5 with `\"Buzz\"`, and number divisible by both 3 and 5 with `\"FizzBuzz\"`.""" + +# ╔═╡ 3fd0895e-7f1f-4ecd-855f-95c69a466dde +function fizzbuzz(x) + if x % 5 == 0 && x % 3 == 0 + return "FizzBuzz" + else + if x % 3 == 0 + return "Fizz" + else + if x % 5 == 0 + return "Buzz" + else + return string(x) + end + end + end +end + +# ╔═╡ b302e44c-29a9-4851-bb11-e95c0dfbacdb +md""" +### Define the grammar + +Let's define a grammar with all the rules that we need.""" + +# ╔═╡ 59444d63-8b2a-4b76-9af3-b92b4abd4a98 +grammar_fizzbuzz = @csgrammar begin + Int = input1 + Int = 0 | 3 | 5 + String = "Fizz" | "Buzz" | "FizzBuzz" + String = string(Int) + Return = String + Int = Int % Int + Bool = Int == Int + Int = Bool ? Int : Int + Bool = Bool && Bool +end + +# ╔═╡ e389cf25-5f0b-4d94-bab0-7cf85ee0e6e0 +md""" +### Construct the syntax tree""" + +# ╔═╡ 8fa5cbbc-ad25-42ff-9ec8-284590fe1084 +md""" +Given the grammar, the AST of `fizzbuzz()` looks like this:""" + +# ╔═╡ 6a663bce-155b-4c0d-94ec-7dc5fbba348a +md""" +```mermaid +flowchart + id1((12)) --- id21((13)) + id1--- id22((9)) + id1--- id23((12)) + + id21 --- id31((11)) + id21 --- id32((11)) + + id31 --- id41((10)) + id31 --- id42((2)) + + id41 --- id51((1)) + id41 --- id52((4)) + + id32 --- id43((10)) + id32 --- id44((2)) + + id43 --- id53((1)) + id43 --- id54((3)) + + id22 --- id33((7)) + id23 --- id34((11)) + + id34 --- id45((10)) + id34 --- id46((2)) + + id45 --- id55((1)) + id45 --- id56((3)) + + id23 --- id35((9)) + id35 --- id47((5)) + + id23 --- id36((12)) + id36 --- id48((11)) + id48 --- id57((10)) + id57 --- id61((1)) + id57 --- id62((4)) + id48 --- id58((2)) + + id36 --- id49((9)) + id49 --- id59((6)) + + id36 --- id410((9)) + id410 --- id510((8)) + id510 --- id63((1)) + + + +```""" + +# ╔═╡ d9272c48-a7da-4ca0-af15-98c0fe4a3f24 +md""" +As before, we use nest instanced of `RuleNode` to implement the AST.""" + +# ╔═╡ 6a268dbb-e884-4b1f-b0c3-4ae1d36064a3 +fizzbuzz_syntaxtree = RuleNode(12, [ + RuleNode(13, [ + RuleNode(11, [ + RuleNode(10, [ + RuleNode(1), + RuleNode(4) + ]), + RuleNode(2) + ]), + RuleNode(11, [ + RuleNode(10, [ + RuleNode(1), + RuleNode(3) + ]), + RuleNode(2) + ]) + ]), + RuleNode(9, [ + RuleNode(7) + + ]), + RuleNode(12, [ + RuleNode(11, [ + RuleNode(10, [ + RuleNode(1), + RuleNode(3), + ]), + RuleNode(2) + ]), + RuleNode(9, [ + RuleNode(5) + ]), + RuleNode(12, [ + RuleNode(11, [ + RuleNode(10, [ + RuleNode(1), + RuleNode(4) + ]), + RuleNode(2) + ]), + RuleNode(9, [ + RuleNode(6) + ]), + RuleNode(9, [ + RuleNode(8, [ + RuleNode(1) + ]) + ]) + ]) + ]) + ]) + +# ╔═╡ 61b27735-25ba-4995-b506-7982db8c50b5 +md""" +And we check our syntax tree is correct:""" + +# ╔═╡ 3692d164-4deb-4da2-834c-fc2eb8ac3fa0 +rulenode2expr(fizzbuzz_syntaxtree, grammar_fizzbuzz) + +# ╔═╡ 0d5aaa64-ae46-4b88-ac05-df8910da1648 +begin + # test solution on inputs + input = [Dict(:input1 => 3), Dict(:input1 => 5), Dict(:input1 =>15), Dict(:input1 => 22)] + output1 = execute_on_input(grammar_fizzbuzz, fizzbuzz_syntaxtree, input) + output1 +end + +# ╔═╡ c9a57153-96f8-4bcb-905b-dc46bc1f7765 +md""" +### Modify the AST/program + +There are several ways to modify an AST and hence, a program. You can + +- directly replace a node with `HerbCore.swap_node()` +- insert a rule node with `insert!` + +Let's modify our example such that if the input number is divisible by 3, the program returns \"Buzz\" instead of \"Fizz\". +We use `swap_node()` to replace the node of the AST that corresponds to rule 5 in the grammar (`String = Fizz`) with rule 6 (`String = Buzz`). To do so, `swap_node()` needs the tree that contains the node we want to modify, the new node we want to replace the node with, and the path to that node. + +Note that `swap_node()` modifies the tree, hence we make a deep copy of it first.""" + +# ╔═╡ 6e0bed41-6f06-47e2-a659-ec61fa9c0d40 +begin + modified_fizzbuzz_syntaxtree = deepcopy(fizzbuzz_syntaxtree) + newnode = RuleNode(6) + path = [3, 2, 1] + swap_node(modified_fizzbuzz_syntaxtree, newnode, path) + rulenode2expr(modified_fizzbuzz_syntaxtree, grammar_fizzbuzz) +end + +# ╔═╡ f9e4ec58-ded3-4bf8-9207-a05596d15586 +md""" +Let's confirm that we modified the AST, and hence the program, correctly:""" + +# ╔═╡ d9543762-e438-492c-a0b1-632c4c25c58b +# test solution on same inputs as before +execute_on_input(grammar_fizzbuzz, modified_fizzbuzz_syntaxtree, input) + +# ╔═╡ 44cc6617-9262-44a5-8cec-90713819d03a +md""" +An alternative way to modify the AST is by using `insert!()`. This requires to provide the location of the node that we want to as `NodeLoc`. `NodeLoc` points to a node in the tree and consists of the parent and the child index of the node. +Again, we make a deep copy of the original AST first.""" + +# ╔═╡ 0b595e4a-c6a3-4986-b311-f09bb53ee189 +begin + anothermodified_fizzbuzz_syntaxtree = deepcopy(fizzbuzz_syntaxtree) + # get the node we want to modify and instantiate a NodeLoc from it. + node = get_node_at_location(anothermodified_fizzbuzz_syntaxtree, [3, 2, 1]) + nodeloc = NodeLoc(node, 0) + # replace the node + insert!(node, nodeloc, newnode) + rulenode2expr(anothermodified_fizzbuzz_syntaxtree, grammar_fizzbuzz) +end + +# ╔═╡ e40403aa-cb27-4bfb-b581-1795fc1cce41 +md""" +Again, we check that we modified the program as intended:""" + +# ╔═╡ 8847f8b8-bb52-4a95-86f1-6483e5e0ab85 +# test on same inputs as before +execute_on_input(grammar_fizzbuzz, anothermodified_fizzbuzz_syntaxtree, input) + +# ╔═╡ 00000000-0000-0000-0000-000000000001 +PLUTO_PROJECT_TOML_CONTENTS = """ +[deps] +HerbCore = "2b23ba43-8213-43cb-b5ea-38c12b45bd45" +HerbGrammar = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" +HerbInterpret = "5bbddadd-02c5-4713-84b8-97364418cca7" +PlutoUI = "7f904dfe-b85e-4ff6-b463-dae2292396a8" + +[compat] +HerbCore = "~0.2.0" +HerbGrammar = "~0.2.1" +HerbInterpret = "~0.1.2" +PlutoUI = "~0.7.59" +""" + +# ╔═╡ 00000000-0000-0000-0000-000000000002 +PLUTO_MANIFEST_TOML_CONTENTS = """ +# This file is machine-generated - editing it directly is not advised + +julia_version = "1.10.4" +manifest_format = "2.0" +project_hash = "0ce6f08e629fb0ab6945e23b691d12a7e72d4c7a" + +[[deps.AbstractPlutoDingetjes]] +deps = ["Pkg"] +git-tree-sha1 = "6e1d2a35f2f90a4bc7c2ed98079b2ba09c35b83a" +uuid = "6e696c72-6542-2067-7265-42206c756150" +version = "1.3.2" + +[[deps.AbstractTrees]] +git-tree-sha1 = "2d9c9a55f9c93e8887ad391fbae72f8ef55e1177" +uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" +version = "0.4.5" + +[[deps.ArgTools]] +uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" +version = "1.1.1" + +[[deps.ArnoldiMethod]] +deps = ["LinearAlgebra", "Random", "StaticArrays"] +git-tree-sha1 = "d57bd3762d308bded22c3b82d033bff85f6195c6" +uuid = "ec485272-7323-5ecc-a04f-4719b315124d" +version = "0.4.0" + +[[deps.Artifacts]] +uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" + +[[deps.Base64]] +uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" + +[[deps.Bzip2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "9e2a6b69137e6969bab0152632dcb3bc108c8bdd" +uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0" +version = "1.0.8+1" + +[[deps.Cairo_jll]] +deps = ["Artifacts", "Bzip2_jll", "CompilerSupportLibraries_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "a2f1c8c668c8e3cb4cca4e57a8efdb09067bb3fd" +uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a" +version = "1.18.0+2" + +[[deps.ColorTypes]] +deps = ["FixedPointNumbers", "Random"] +git-tree-sha1 = "b10d0b65641d57b8b4d5e234446582de5047050d" +uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f" +version = "0.11.5" + +[[deps.CommonSubexpressions]] +deps = ["MacroTools", "Test"] +git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7" +uuid = "bbf7d656-a473-5ed7-a52c-81e309532950" +version = "0.3.0" + +[[deps.Compat]] +deps = ["TOML", "UUIDs"] +git-tree-sha1 = "b1c55339b7c6c350ee89f2c1604299660525b248" +uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" +version = "4.15.0" +weakdeps = ["Dates", "LinearAlgebra"] + + [deps.Compat.extensions] + CompatLinearAlgebraExt = "LinearAlgebra" + +[[deps.CompilerSupportLibraries_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" +version = "1.1.1+0" + +[[deps.DataStructures]] +deps = ["Compat", "InteractiveUtils", "OrderedCollections"] +git-tree-sha1 = "1d0a14036acb104d9e89698bd408f63ab58cdc82" +uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" +version = "0.18.20" + +[[deps.Dates]] +deps = ["Printf"] +uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" + +[[deps.Distributed]] +deps = ["Random", "Serialization", "Sockets"] +uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b" + +[[deps.Downloads]] +deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"] +uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" +version = "1.6.0" + +[[deps.Expat_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "1c6317308b9dc757616f0b5cb379db10494443a7" +uuid = "2e619515-83b5-522b-bb60-26c02a35a201" +version = "2.6.2+0" + +[[deps.FileWatching]] +uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee" + +[[deps.FixedPointNumbers]] +deps = ["Statistics"] +git-tree-sha1 = "05882d6995ae5c12bb5f36dd2ed3f61c98cbb172" +uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93" +version = "0.8.5" + +[[deps.Fontconfig_jll]] +deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Zlib_jll"] +git-tree-sha1 = "db16beca600632c95fc8aca29890d83788dd8b23" +uuid = "a3f928ae-7b40-5064-980b-68af3947d34b" +version = "2.13.96+0" + +[[deps.FreeType2_jll]] +deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "5c1d8ae0efc6c2e7b1fc502cbe25def8f661b7bc" +uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7" +version = "2.13.2+0" + +[[deps.Gettext_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "XML2_jll"] +git-tree-sha1 = "9b02998aba7bf074d14de89f9d37ca24a1a0b046" +uuid = "78b55507-aeef-58d4-861c-77aaff3498b1" +version = "0.21.0+0" + +[[deps.Glib_jll]] +deps = ["Artifacts", "Gettext_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE2_jll", "Zlib_jll"] +git-tree-sha1 = "7c82e6a6cd34e9d935e9aa4051b66c6ff3af59ba" +uuid = "7746bdde-850d-59dc-9ae8-88ece973131d" +version = "2.80.2+0" + +[[deps.Graphite2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "344bf40dcab1073aca04aa0df4fb092f920e4011" +uuid = "3b182d85-2403-5c21-9c21-1e1f0cc25472" +version = "1.3.14+0" + +[[deps.Graphs]] +deps = ["ArnoldiMethod", "Compat", "DataStructures", "Distributed", "Inflate", "LinearAlgebra", "Random", "SharedArrays", "SimpleTraits", "SparseArrays", "Statistics"] +git-tree-sha1 = "4f2b57488ac7ee16124396de4f2bbdd51b2602ad" +uuid = "86223c79-3864-5bf0-83f7-82e725a168b6" +version = "1.11.0" + +[[deps.HarfBuzz_ICU_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "6ccbc4fdf65c8197738c2d68cc55b74b19c97ac2" +uuid = "655565e8-fb53-5cb3-b0cd-aec1ca0647ea" +version = "2.8.1+0" + +[[deps.HarfBuzz_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "129acf094d168394e80ee1dc4bc06ec835e510a3" +uuid = "2e76f6c2-a576-52d4-95c1-20adfe4de566" +version = "2.8.1+1" + +[[deps.HerbCore]] +git-tree-sha1 = "f3312458fa882d4adaeecadf8f4b5721ec6e3322" +uuid = "2b23ba43-8213-43cb-b5ea-38c12b45bd45" +version = "0.2.0" + +[[deps.HerbGrammar]] +deps = ["AbstractTrees", "DataStructures", "HerbCore", "Serialization", "TreeView"] +git-tree-sha1 = "aa70e06c5cec398294cc65c16e18f2ebdc3a7348" +uuid = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" +version = "0.2.1" + +[[deps.HerbInterpret]] +deps = ["HerbCore", "HerbGrammar", "HerbSpecification"] +git-tree-sha1 = "3723e5ace26a2f2cd342b28689dea543491687c6" +uuid = "5bbddadd-02c5-4713-84b8-97364418cca7" +version = "0.1.2" + +[[deps.HerbSpecification]] +git-tree-sha1 = "5385b81e40c3cd62aeea591319896148036863c9" +uuid = "6d54aada-062f-46d8-85cf-a1ceaf058a06" +version = "0.1.0" + +[[deps.Hyperscript]] +deps = ["Test"] +git-tree-sha1 = "179267cfa5e712760cd43dcae385d7ea90cc25a4" +uuid = "47d2ed2b-36de-50cf-bf87-49c2cf4b8b91" +version = "0.0.5" + +[[deps.HypertextLiteral]] +deps = ["Tricks"] +git-tree-sha1 = "7134810b1afce04bbc1045ca1985fbe81ce17653" +uuid = "ac1192a8-f4b3-4bfe-ba22-af5b92cd3ab2" +version = "0.9.5" + +[[deps.ICU_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "20b6765a3016e1fca0c9c93c80d50061b94218b7" +uuid = "a51ab1cf-af8e-5615-a023-bc2c838bba6b" +version = "69.1.0+0" + +[[deps.IOCapture]] +deps = ["Logging", "Random"] +git-tree-sha1 = "8b72179abc660bfab5e28472e019392b97d0985c" +uuid = "b5f81e59-6552-4d32-b1f0-c071b021bf89" +version = "0.2.4" + +[[deps.Inflate]] +git-tree-sha1 = "ea8031dea4aff6bd41f1df8f2fdfb25b33626381" +uuid = "d25df0c9-e2be-5dd7-82c8-3ad0b3e990b9" +version = "0.1.4" + +[[deps.InteractiveUtils]] +deps = ["Markdown"] +uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" + +[[deps.JLLWrappers]] +deps = ["Artifacts", "Preferences"] +git-tree-sha1 = "7e5d6779a1e09a36db2a7b6cff50942a0a7d0fca" +uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210" +version = "1.5.0" + +[[deps.JSON]] +deps = ["Dates", "Mmap", "Parsers", "Unicode"] +git-tree-sha1 = "31e996f0a15c7b280ba9f76636b3ff9e2ae58c9a" +uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6" +version = "0.21.4" + +[[deps.JpegTurbo_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "c84a835e1a09b289ffcd2271bf2a337bbdda6637" +uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8" +version = "3.0.3+0" + +[[deps.LERC_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "bf36f528eec6634efc60d7ec062008f171071434" +uuid = "88015f11-f218-50d7-93a8-a6af411a945d" +version = "3.0.0+1" + +[[deps.LLVMOpenMP_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "d986ce2d884d49126836ea94ed5bfb0f12679713" +uuid = "1d63c593-3942-5779-bab2-d838dc0a180e" +version = "15.0.7+0" + +[[deps.LZO_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "70c5da094887fd2cae843b8db33920bac4b6f07d" +uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac" +version = "2.10.2+0" + +[[deps.LaTeXStrings]] +git-tree-sha1 = "50901ebc375ed41dbf8058da26f9de442febbbec" +uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" +version = "1.3.1" + +[[deps.LibCURL]] +deps = ["LibCURL_jll", "MozillaCACerts_jll"] +uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" +version = "0.6.4" + +[[deps.LibCURL_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"] +uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" +version = "8.4.0+0" + +[[deps.LibGit2]] +deps = ["Base64", "LibGit2_jll", "NetworkOptions", "Printf", "SHA"] +uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" + +[[deps.LibGit2_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll"] +uuid = "e37daf67-58a4-590a-8e99-b0245dd2ffc5" +version = "1.6.4+0" + +[[deps.LibSSH2_jll]] +deps = ["Artifacts", "Libdl", "MbedTLS_jll"] +uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" +version = "1.11.0+1" + +[[deps.Libdl]] +uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" + +[[deps.Libffi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "0b4a5d71f3e5200a7dff793393e09dfc2d874290" +uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490" +version = "3.2.2+1" + +[[deps.Libgcrypt_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgpg_error_jll"] +git-tree-sha1 = "9fd170c4bbfd8b935fdc5f8b7aa33532c991a673" +uuid = "d4300ac3-e22c-5743-9152-c294e39db1e4" +version = "1.8.11+0" + +[[deps.Libgpg_error_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "fbb1f2bef882392312feb1ede3615ddc1e9b99ed" +uuid = "7add5ba3-2f88-524e-9cd5-f83b8a55f7b8" +version = "1.49.0+0" + +[[deps.Libiconv_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "f9557a255370125b405568f9767d6d195822a175" +uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531" +version = "1.17.0+0" + +[[deps.Libmount_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "0c4f9c4f1a50d8f35048fa0532dabbadf702f81e" +uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9" +version = "2.40.1+0" + +[[deps.Libtiff_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "LERC_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"] +git-tree-sha1 = "3eb79b0ca5764d4799c06699573fd8f533259713" +uuid = "89763e89-9b03-5906-acba-b20f662cd828" +version = "4.4.0+0" + +[[deps.Libuuid_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "5ee6203157c120d79034c748a2acba45b82b8807" +uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700" +version = "2.40.1+0" + +[[deps.LinearAlgebra]] +deps = ["Libdl", "OpenBLAS_jll", "libblastrampoline_jll"] +uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + +[[deps.LittleCMS_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pkg"] +git-tree-sha1 = "110897e7db2d6836be22c18bffd9422218ee6284" +uuid = "d3a379c0-f9a3-5b72-a4c0-6bf4d2e8af0f" +version = "2.12.0+0" + +[[deps.Logging]] +uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" + +[[deps.MIMEs]] +git-tree-sha1 = "65f28ad4b594aebe22157d6fac869786a255b7eb" +uuid = "6c6e2e6c-3030-632d-7369-2d6c69616d65" +version = "0.1.4" + +[[deps.MacroTools]] +deps = ["Markdown", "Random"] +git-tree-sha1 = "2fa9ee3e63fd3a4f7a9a4f4744a52f4856de82df" +uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" +version = "0.5.13" + +[[deps.Markdown]] +deps = ["Base64"] +uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" + +[[deps.MbedTLS_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" +version = "2.28.2+1" + +[[deps.Mmap]] +uuid = "a63ad114-7e13-5084-954f-fe012c677804" + +[[deps.MozillaCACerts_jll]] +uuid = "14a3606d-f60d-562e-9121-12d972cd8159" +version = "2023.1.10" + +[[deps.NetworkOptions]] +uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" +version = "1.2.0" + +[[deps.OpenBLAS_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] +uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" +version = "0.3.23+4" + +[[deps.OpenJpeg_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libtiff_jll", "LittleCMS_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "76374b6e7f632c130e78100b166e5a48464256f8" +uuid = "643b3616-a352-519d-856d-80112ee9badc" +version = "2.4.0+0" + +[[deps.OpenSSL_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "a12e56c72edee3ce6b96667745e6cbbe5498f200" +uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95" +version = "1.1.23+0" + +[[deps.OrderedCollections]] +git-tree-sha1 = "dfdf5519f235516220579f949664f1bf44e741c5" +uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" +version = "1.6.3" + +[[deps.PCRE2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "efcefdf7-47ab-520b-bdef-62a2eaa19f15" +version = "10.42.0+1" + +[[deps.Parsers]] +deps = ["Dates", "PrecompileTools", "UUIDs"] +git-tree-sha1 = "8489905bcdbcfac64d1daa51ca07c0d8f0283821" +uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0" +version = "2.8.1" + +[[deps.Pixman_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "LLVMOpenMP_jll", "Libdl"] +git-tree-sha1 = "35621f10a7531bc8fa58f74610b1bfb70a3cfc6b" +uuid = "30392449-352a-5448-841d-b1acce4e97dc" +version = "0.43.4+0" + +[[deps.Pkg]] +deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] +uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" +version = "1.10.0" + +[[deps.PlutoUI]] +deps = ["AbstractPlutoDingetjes", "Base64", "ColorTypes", "Dates", "FixedPointNumbers", "Hyperscript", "HypertextLiteral", "IOCapture", "InteractiveUtils", "JSON", "Logging", "MIMEs", "Markdown", "Random", "Reexport", "URIs", "UUIDs"] +git-tree-sha1 = "ab55ee1510ad2af0ff674dbcced5e94921f867a9" +uuid = "7f904dfe-b85e-4ff6-b463-dae2292396a8" +version = "0.7.59" + +[[deps.Poppler_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "Glib_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "OpenJpeg_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "02148a0cb2532f22c0589ceb75c110e168fb3d1f" +uuid = "9c32591e-4766-534b-9725-b71a8799265b" +version = "21.9.0+0" + +[[deps.PrecompileTools]] +deps = ["Preferences"] +git-tree-sha1 = "5aa36f7049a63a1528fe8f7c3f2113413ffd4e1f" +uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a" +version = "1.2.1" + +[[deps.Preferences]] +deps = ["TOML"] +git-tree-sha1 = "9306f6085165d270f7e3db02af26a400d580f5c6" +uuid = "21216c6a-2e73-6563-6e65-726566657250" +version = "1.4.3" + +[[deps.Printf]] +deps = ["Unicode"] +uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" + +[[deps.REPL]] +deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] +uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" + +[[deps.Random]] +deps = ["SHA"] +uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" + +[[deps.Reexport]] +git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b" +uuid = "189a3867-3050-52da-a836-e630ba90ab69" +version = "1.2.2" + +[[deps.Requires]] +deps = ["UUIDs"] +git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7" +uuid = "ae029012-a4dd-5104-9daa-d747884805df" +version = "1.3.0" + +[[deps.SHA]] +uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" +version = "0.7.0" + +[[deps.Serialization]] +uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" + +[[deps.SharedArrays]] +deps = ["Distributed", "Mmap", "Random", "Serialization"] +uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383" + +[[deps.SimpleTraits]] +deps = ["InteractiveUtils", "MacroTools"] +git-tree-sha1 = "5d7e3f4e11935503d3ecaf7186eac40602e7d231" +uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d" +version = "0.9.4" + +[[deps.Sockets]] +uuid = "6462fe0b-24de-5631-8697-dd941f90decc" + +[[deps.SparseArrays]] +deps = ["Libdl", "LinearAlgebra", "Random", "Serialization", "SuiteSparse_jll"] +uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" +version = "1.10.0" + +[[deps.StaticArrays]] +deps = ["LinearAlgebra", "PrecompileTools", "Random", "StaticArraysCore"] +git-tree-sha1 = "9ae599cd7529cfce7fea36cf00a62cfc56f0f37c" +uuid = "90137ffa-7385-5640-81b9-e52037218182" +version = "1.9.4" + + [deps.StaticArrays.extensions] + StaticArraysChainRulesCoreExt = "ChainRulesCore" + StaticArraysStatisticsExt = "Statistics" + + [deps.StaticArrays.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" + +[[deps.StaticArraysCore]] +git-tree-sha1 = "36b3d696ce6366023a0ea192b4cd442268995a0d" +uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" +version = "1.4.2" + +[[deps.Statistics]] +deps = ["LinearAlgebra", "SparseArrays"] +uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" +version = "1.10.0" + +[[deps.SuiteSparse_jll]] +deps = ["Artifacts", "Libdl", "libblastrampoline_jll"] +uuid = "bea87d4a-7f5b-5778-9afe-8cc45184846c" +version = "7.2.1+1" + +[[deps.TOML]] +deps = ["Dates"] +uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" +version = "1.0.3" + +[[deps.Tar]] +deps = ["ArgTools", "SHA"] +uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" +version = "1.10.0" + +[[deps.Test]] +deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] +uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[[deps.TikzGraphs]] +deps = ["Graphs", "LaTeXStrings", "TikzPictures"] +git-tree-sha1 = "e8f41ed9a2cabf6699d9906c195bab1f773d4ca7" +uuid = "b4f28e30-c73f-5eaf-a395-8a9db949a742" +version = "1.4.0" + +[[deps.TikzPictures]] +deps = ["LaTeXStrings", "Poppler_jll", "Requires", "tectonic_jll"] +git-tree-sha1 = "79e2d29b216ef24a0f4f905532b900dcf529aa06" +uuid = "37f6aa50-8035-52d0-81c2-5a1d08754b2d" +version = "3.5.0" + +[[deps.TreeView]] +deps = ["CommonSubexpressions", "Graphs", "MacroTools", "TikzGraphs"] +git-tree-sha1 = "41ddcefb625f2ab0f4d9f2081c2da1af2ccbbf8b" +uuid = "39424ebd-4cf3-5550-a685-96706a953f40" +version = "0.5.1" + +[[deps.Tricks]] +git-tree-sha1 = "eae1bb484cd63b36999ee58be2de6c178105112f" +uuid = "410a4b4d-49e4-4fbc-ab6d-cb71b17b3775" +version = "0.1.8" + +[[deps.URIs]] +git-tree-sha1 = "67db6cc7b3821e19ebe75791a9dd19c9b1188f2b" +uuid = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4" +version = "1.5.1" + +[[deps.UUIDs]] +deps = ["Random", "SHA"] +uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" + +[[deps.Unicode]] +uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" + +[[deps.XML2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Zlib_jll"] +git-tree-sha1 = "52ff2af32e591541550bd753c0da8b9bc92bb9d9" +uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a" +version = "2.12.7+0" + +[[deps.XSLT_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgcrypt_jll", "Libgpg_error_jll", "Libiconv_jll", "Pkg", "XML2_jll", "Zlib_jll"] +git-tree-sha1 = "91844873c4085240b95e795f692c4cec4d805f8a" +uuid = "aed1982a-8fda-507f-9586-7b0439959a61" +version = "1.1.34+0" + +[[deps.Xorg_libX11_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libxcb_jll", "Xorg_xtrans_jll"] +git-tree-sha1 = "afead5aba5aa507ad5a3bf01f58f82c8d1403495" +uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc" +version = "1.8.6+0" + +[[deps.Xorg_libXau_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "6035850dcc70518ca32f012e46015b9beeda49d8" +uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec" +version = "1.0.11+0" + +[[deps.Xorg_libXdmcp_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "34d526d318358a859d7de23da945578e8e8727b7" +uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05" +version = "1.1.4+0" + +[[deps.Xorg_libXext_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "d2d1a5c49fae4ba39983f63de6afcbea47194e85" +uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3" +version = "1.3.6+0" + +[[deps.Xorg_libXrender_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "47e45cd78224c53109495b3e324df0c37bb61fbe" +uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa" +version = "0.9.11+0" + +[[deps.Xorg_libpthread_stubs_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "8fdda4c692503d44d04a0603d9ac0982054635f9" +uuid = "14d82f49-176c-5ed1-bb49-ad3f5cbd8c74" +version = "0.1.1+0" + +[[deps.Xorg_libxcb_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "XSLT_jll", "Xorg_libXau_jll", "Xorg_libXdmcp_jll", "Xorg_libpthread_stubs_jll"] +git-tree-sha1 = "b4bfde5d5b652e22b9c790ad00af08b6d042b97d" +uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b" +version = "1.15.0+0" + +[[deps.Xorg_xtrans_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e92a1a012a10506618f10b7047e478403a046c77" +uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10" +version = "1.5.0+0" + +[[deps.Zlib_jll]] +deps = ["Libdl"] +uuid = "83775a58-1f1d-513f-b197-d71354ab007a" +version = "1.2.13+1" + +[[deps.Zstd_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e678132f07ddb5bfa46857f0d7620fb9be675d3b" +uuid = "3161d3a3-bdf6-5164-811a-617609db77b4" +version = "1.5.6+0" + +[[deps.libblastrampoline_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" +version = "5.8.0+1" + +[[deps.libpng_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "d7015d2e18a5fd9a4f47de711837e980519781a4" +uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f" +version = "1.6.43+1" + +[[deps.nghttp2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" +version = "1.52.0+1" + +[[deps.p7zip_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" +version = "17.4.0+2" + +[[deps.tectonic_jll]] +deps = ["Artifacts", "Fontconfig_jll", "FreeType2_jll", "Graphite2_jll", "HarfBuzz_ICU_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "OpenSSL_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "54867b00af20c70b52a1f9c00043864d8b926a21" +uuid = "d7dd28d6-a5e6-559c-9131-7eb760cdacc5" +version = "0.13.1+0" +""" + +# ╔═╡ Cell order: +# ╟─65fbf850-74ae-4ea4-85f0-683095c73fba +# ╟─2493e9db-8b8e-4ef6-8379-f50ad26aab88 +# ╟─8ff96964-e39e-4762-ae03-e9166e163fca +# ╟─bff155ab-ff0e-452b-a8f5-fe744e41a30f +# ╟─caa3446e-c5df-4dac-905a-20515f681074 +# ╠═c784cbc3-19fc-45c9-b344-db10cf7a81fa +# ╠═9f54f013-e8b9-4e0d-8bac-9867f5d1a393 +# ╟─46fbbe87-ee6e-4874-9708-20a42347ff18 +# ╟─5dc6be9c-e4d9-4fdb-90bf-f4c59bb66a70 +# ╠═64c2a6ce-5e3b-413b-bcb7-84936137439f +# ╟─64f6f1e3-5cbc-4e37-a806-d4fd45c20855 +# ╟─29b37a82-d022-453e-bf65-672aa94e4c87 +# ╠═822d9601-284d-4d30-9551-605684f83d90 +# ╟─351210d1-20b6-4695-b9fe-f1136d4447d5 +# ╠═dc882fd5-d0fd-4a7d-8d5b-40516f3a3bcb +# ╠═cbfaa1b6-f3c5-490a-9e54-006262d0c727 +# ╟─6e018fd3-7626-48b2-b56a-240ae62a1ac4 +# ╠═3fd0895e-7f1f-4ecd-855f-95c69a466dde +# ╟─b302e44c-29a9-4851-bb11-e95c0dfbacdb +# ╠═59444d63-8b2a-4b76-9af3-b92b4abd4a98 +# ╟─e389cf25-5f0b-4d94-bab0-7cf85ee0e6e0 +# ╟─8fa5cbbc-ad25-42ff-9ec8-284590fe1084 +# ╠═6a663bce-155b-4c0d-94ec-7dc5fbba348a +# ╟─d9272c48-a7da-4ca0-af15-98c0fe4a3f24 +# ╠═6a268dbb-e884-4b1f-b0c3-4ae1d36064a3 +# ╟─61b27735-25ba-4995-b506-7982db8c50b5 +# ╠═3692d164-4deb-4da2-834c-fc2eb8ac3fa0 +# ╠═0d5aaa64-ae46-4b88-ac05-df8910da1648 +# ╟─c9a57153-96f8-4bcb-905b-dc46bc1f7765 +# ╠═6e0bed41-6f06-47e2-a659-ec61fa9c0d40 +# ╟─f9e4ec58-ded3-4bf8-9207-a05596d15586 +# ╠═d9543762-e438-492c-a0b1-632c4c25c58b +# ╟─44cc6617-9262-44a5-8cec-90713819d03a +# ╠═0b595e4a-c6a3-4986-b311-f09bb53ee189 +# ╟─e40403aa-cb27-4bfb-b581-1795fc1cce41 +# ╠═8847f8b8-bb52-4a95-86f1-6483e5e0ab85 +# ╟─00000000-0000-0000-0000-000000000001 +# ╟─00000000-0000-0000-0000-000000000002 From a067c521cad150fbf673282f1900bda5a173c054 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Tue, 27 Aug 2024 17:00:47 +0200 Subject: [PATCH 04/13] Replace broken constraint. Replace cfgrammar with csgrammar. --- docs/src/tutorials/defining_grammars.ipynb | 69 +++++++++++----------- 1 file changed, 33 insertions(+), 36 deletions(-) diff --git a/docs/src/tutorials/defining_grammars.ipynb b/docs/src/tutorials/defining_grammars.ipynb index 8bbfb79..dec24cb 100644 --- a/docs/src/tutorials/defining_grammars.ipynb +++ b/docs/src/tutorials/defining_grammars.ipynb @@ -36,10 +36,10 @@ "### Creating a simple grammar\n", "\n", "This cell contains a very simple arithmetic grammar. \n", - "The grammar is defined using the `@cfgrammar` macro. \n", + "The grammar is defined using the `@csgrammar` macro. \n", "This macro converts the grammar definition in the form of a Julia expression into Herb's internal grammar representation. \n", "Macro's are executed during compilation.\n", - "If you want to load a grammar during execution, have a look at the `HerbGrammar.expr2cfgrammar` function." + "If you want to load a grammar during execution, have a look at the `HerbGrammar.expr2csgrammar` function." ] }, { @@ -48,7 +48,7 @@ "metadata": {}, "outputs": [], "source": [ - "g₁ = HerbGrammar.@cfgrammar begin\n", + "g₁ = HerbGrammar.@csgrammar begin\n", " Int = 1\n", " Int = 2\n", " Int = 3\n", @@ -71,7 +71,7 @@ "metadata": {}, "outputs": [], "source": [ - "g₂ = HerbGrammar.@cfgrammar begin\n", + "g₂ = HerbGrammar.@csgrammar begin\n", " Int = |(0:9)\n", " Int = Int * Int\n", " Int = Int + Int\n", @@ -91,7 +91,7 @@ "metadata": {}, "outputs": [], "source": [ - "g₃ = HerbGrammar.@cfgrammar begin\n", + "g₃ = HerbGrammar.@csgrammar begin\n", " Int = |([0, 2, 4, 6, 8])\n", " Int = Int * Int\n", " Int = Int + Int\n", @@ -111,7 +111,7 @@ "metadata": {}, "outputs": [], "source": [ - "g₄ = HerbGrammar.@cfgrammar begin\n", + "g₄ = HerbGrammar.@csgrammar begin\n", " Int = |(0:9)\n", " Int = Int * Int\n", " Int = Int + Int\n", @@ -136,7 +136,7 @@ "source": [ "f(a) = a + 1\n", "\n", - "g₅ = HerbGrammar.@cfgrammar begin\n", + "g₅ = HerbGrammar.@csgrammar begin\n", " Int = |(0:9)\n", " Int = Int * Int\n", " Int = Int + Int\n", @@ -161,7 +161,7 @@ "source": [ "×(a, b) = a * b\n", "\n", - "g₆ = HerbGrammar.@cfgrammar begin\n", + "g₆ = HerbGrammar.@csgrammar begin\n", " Int = |(0:9)\n", " Int = a\n", " Int = Int + Int\n", @@ -177,9 +177,9 @@ "### Working with grammars\n", "\n", "If you want to implement something using these grammars, it is useful to know about the functions that you can use to manipulate grammars and extract information. \n", - "This section is not necessarily complete, but it aims to give an overview of the most important functions. \n", + "This section is not complete, but it aims to give an overview of the most important functions. \n", "\n", - "It is recommended to also read up on [Julia metaprogramming](https://docs.julialang.org/en/v1/manual/metaprogramming/) if you are not already familiar with that.\n", + "It is recommended to also read up on [Julia metaprogramming](https://docs.julialang.org/en/v1/manual/metaprogramming/) if you are not already familiar with the concept.\n", "\n", "One of the most important things about grammars is that each rule has an index associated with it:" ] @@ -190,7 +190,7 @@ "metadata": {}, "outputs": [], "source": [ - "g₇ = HerbGrammar.@cfgrammar begin\n", + "g₇ = HerbGrammar.@csgrammar begin\n", " Int = |(0:9)\n", " Int = Int + Int\n", " Int = Int * Int\n", @@ -304,8 +304,7 @@ "source": [ "### Adding rules\n", "\n", - "It is also possible to add rules to a grammar during execution. This can be done using the `add_rule!` function.\n", - "As with most functions in Julia that end with an exclamation mark, this function modifies its argument (the grammar).\n", + "It is also possible to add rules to a grammar during execution. This can be done using the `add_rule!` function. The exclamatin mark is a Julia convention and is appended to name if a function modifies its arguments (in our example the grammar).\n", "\n", "A rule can be provided in the same syntax as is used in the grammar definition.\n", "The rule should be of the `Expr` type, which is a built-in type for representing expressions. \n", @@ -363,8 +362,8 @@ "source": [ "## Context-sensitive grammars\n", "\n", - "Context-sensitive grammars allow additional constraints to be added with respect to context-free grammars.\n", - "The syntax for defining a context-sensitive grammar is identical to defining a context-sensitive grammar:" + "Context-sensitive grammars introduce additional constraints compared to context-free grammars (like the simple grammar examples above).\n", + "As before, we use the `@csgrammar` macro:" ] }, { @@ -386,15 +385,8 @@ "metadata": {}, "source": [ "Constraints can be added using the `addconstraint!` function, which takes a context-sensitive grammar and a constraint and adds the constraint to the grammar.\n", - "Currently, Herb.jl only has propagators constraints. \n", - "These constraints each have a corresponding `propagate` function that removes all options that violate that constraint from the domain. \n", - "At the moment, there are three propagator constraints:\n", "\n", - "- `ComesAfter(rule, predecessors)`: It is only possible to use rule `rule` when `predecessors` are in its path to the root.\n", - "- `Forbidden(sequence)`: Forbids the derivation specified as a path in an expression tree.\n", - "- `Ordered(order)`: Rules have to be used in the specified order. That is, rule at index K can only be used if rules at indices `[1...K-1]` are used in the left subtree of the current expression.\n", - "\n", - "Below, an example is given of a context-sensitive grammar with a `ComesAfter` constraint:" + "For example, we can add a `` constraint to enforce that the input symbol `x` (rule 13) appears at least once in the program, to avoid programs that are just a constant. " ] }, { @@ -403,7 +395,14 @@ "metadata": {}, "outputs": [], "source": [ - "HerbGrammar.addconstraint!(g₈, HerbConstraints.ComesAfter(1, [9]))" + "HerbGrammar.addconstraint!(g₈, Contains(13))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There is a dedicated tutorial for constraints in Herb.jl and how to work with them." ] }, { @@ -423,7 +422,7 @@ "metadata": {}, "outputs": [], "source": [ - "g₉ = HerbGrammar.@pcfgrammar begin\n", + "g₉ = HerbGrammar.@pcsgrammar begin\n", " 0.4 : Int = |(0:9)\n", " 0.2 : Int = Int + Int\n", " 0.1 : Int = Int * Int\n", @@ -458,8 +457,8 @@ "\n", "### Saving & loading context-free grammars\n", "\n", - "If you want to store a grammar on the disk, you can use the `store_cfg`, `read_cfg` and functions to store and read grammars respectively. \n", - "The `store_cfg` grammar can also be used to store probabilistic grammars. Reading probabilistic grammars can be done using `read_pcfg`.\n", + "If you want to store a grammar on the disk, you can use the `store_csg`, `read_csg` and functions to store and read grammars respectively. \n", + "The `store_csg` grammar can also be used to store probabilistic grammars. To read probabilistic grammars, use `read_pcsg`.\n", "The stored grammar files can also be opened using a text editor to be modified, as long as the contents of the file doesn't violate the syntax for defining grammars." ] }, @@ -469,18 +468,16 @@ "metadata": {}, "outputs": [], "source": [ - "HerbGrammar.store_cfg(\"demo.txt\", g₇)" + "HerbGrammar.store_csg(g₇, \"demo.txt\")" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "scrolled": false - }, + "metadata": {}, "outputs": [], "source": [ - "HerbGrammar.read_cfg(\"demo.txt\")" + "HerbGrammar.read_csg(\"demo.txt\")" ] }, { @@ -501,7 +498,7 @@ "metadata": {}, "outputs": [], "source": [ - "HerbGrammar.store_csg(\"demo.grammar\", \"demo.constraints\", g₈)\n", + "HerbGrammar.store_csg( g₈, \"demo.grammar\", \"demo.constraints\")\n", "g₈, g₈.constraints" ] }, @@ -518,15 +515,15 @@ ], "metadata": { "kernelspec": { - "display_name": "Julia 1.9.4", + "display_name": "Julia 1.10.3", "language": "julia", - "name": "julia-1.9" + "name": "julia-1.10" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", - "version": "1.9.4" + "version": "1.10.3" } }, "nbformat": 4, From 9c054afd3b92de50f6d1cec313531ff71d5fa6f1 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Wed, 28 Aug 2024 07:20:40 +0200 Subject: [PATCH 05/13] Pluto notebook for tutorial Defining Grammars. --- docs/src/tutorials/defining_grammars.jl | 981 ++++++++++++++++++++++++ 1 file changed, 981 insertions(+) create mode 100644 docs/src/tutorials/defining_grammars.jl diff --git a/docs/src/tutorials/defining_grammars.jl b/docs/src/tutorials/defining_grammars.jl new file mode 100644 index 0000000..08bb920 --- /dev/null +++ b/docs/src/tutorials/defining_grammars.jl @@ -0,0 +1,981 @@ +### A Pluto.jl notebook ### +# v0.19.43 + +using Markdown +using InteractiveUtils + +# ╔═╡ 84549502-876a-4ffc-8069-1ffc17622f9a +using HerbGrammar, HerbConstraints + +# ╔═╡ 93b96839-9676-477b-aebb-e50d733a6719 +md""" +# Defining Grammars in Herb.jl using HerbGrammar + +The program space in Herb.jl is defined using a grammar. +This notebook demonstrates how such a grammar can be created. +There are multiple kinds of grammars, but they can all be defined in a very similar way. +""" + +# ╔═╡ d863795b-5fea-4b69-8f63-e04352bb970a +md""" +### Setup +First, we import the necessary Herb packages. +""" + +# ╔═╡ 751e1119-fe59-41dc-8785-a3ed0a9cacb9 +md""" +### Creating a simple grammar + +This cell contains a very simple arithmetic grammar. +The grammar is defined using the `@csgrammar` macro. +This macro converts the grammar definition in the form of a Julia expression into Herb's internal grammar representation. +Macro's are executed during compilation. +If you want to load a grammar during execution, have a look at the `HerbGrammar.expr2csgrammar` function. +""" + +# ╔═╡ 1030a96d-b739-4232-820b-f21826512251 +g₁ = HerbGrammar.@csgrammar begin + Int = 1 + Int = 2 + Int = 3 + Int = Int * Int + Int = Int + Int +end + +# ╔═╡ 0224c914-3763-4c55-844c-e9aca430b377 +md""" +Defining every integer one-by-one can be quite tedious. Therefore, it is also possible to use the following syntax that makes use of a Julia iterator: +""" + +# ╔═╡ 4619b92e-2f9e-429f-9e9a-77b97be7edd1 +g₂ = HerbGrammar.@csgrammar begin + Int = |(0:9) + Int = Int * Int + Int = Int + Int +end + +# ╔═╡ f8e82610-b0d0-4763-939d-aadc92336c52 +md""" +You can do the same with lists: +""" + +# ╔═╡ f4d34f0a-6c88-48db-84a0-ed3fb01fb7aa +g₃ = HerbGrammar.@csgrammar begin + Int = |([0, 2, 4, 6, 8]) + Int = Int * Int + Int = Int + Int +end + +# ╔═╡ 8732f232-3069-4fb1-9b6a-0e24b8724451 +md""" +Variables can also be added to the grammar by just using the variable name: +""" + +# ╔═╡ b68090b2-0eed-4e4a-acdc-6df71747a46d +g₄ = HerbGrammar.@csgrammar begin + Int = |(0:9) + Int = Int * Int + Int = Int + Int + Int = x +end + +# ╔═╡ 89e8ab14-7097-4088-937f-6453722da9d0 +md""" +Grammars can also work with functions. +After all, `+` and `*` are just infix operators for Julia's identically-named functions. +You can use functions that are provided by Julia, or functions that you wrote yourself: +""" + +# ╔═╡ 26b9a050-c4b4-42a8-b1d7-e50fb98a7e8a +begin + f(a) = a + 1 + + g₅ = HerbGrammar.@csgrammar begin + Int = |(0:9) + Int = Int * Int + Int = Int + Int + Int = f(Int) + Int = x + end +end + +# ╔═╡ 7ebb2202-8aa1-4404-b1dd-ac1669f9a120 +md""" +Similarly, we can also define the operator times (x) manually. +""" + +# ╔═╡ 9351c7ab-a39f-4f47-a3ee-e1684e2b3c07 +begin + ×(a, b) = a * b + + g₆ = HerbGrammar.@csgrammar begin + Int = |(0:9) + Int = a + Int = Int + Int + Int = Int × Int + end +end + +# ╔═╡ 3f44d67b-bd18-481e-bca8-bb4904f6b418 +md""" +### Working with grammars + +If you want to implement something using these grammars, it is useful to know about the functions that you can use to manipulate grammars and extract information. +This section is not complete, but it aims to give an overview of the most important functions. + +It is recommended to also read up on [Julia metaprogramming](https://docs.julialang.org/en/v1/manual/metaprogramming/) if you are not already familiar with the concept. + +One of the most important things about grammars is that each rule has an index associated with it: +""" + +# ╔═╡ 58af16a2-7f23-4d06-8836-fb737788ada0 +begin + g₇ = HerbGrammar.@csgrammar begin + Int = |(0:9) + Int = Int + Int + Int = Int * Int + Int = x + end + + collect(enumerate(g₇.rules)) +end + +# ╔═╡ 303894cd-a0c2-461c-a671-7330f75b338e +md""" +We can use this index to extract information from the grammar. +""" + +# ╔═╡ 5c103ce3-0cb2-4004-ac44-50613a461153 +md""" +### isterminal + +`isterminal` returns `true` if a rule is terminal, i.e. it cannot be expanded. For example, rule 1 is terminal, but rule 11 is not, since it contains the non-terminal symbol `:Int`. +""" + +# ╔═╡ 38422e77-cfd8-4a2c-8cf2-f1b430f64ad1 +HerbGrammar.isterminal(g₇, 1) + +# ╔═╡ 112d4837-e646-4e24-84c8-771b9855aebf +HerbGrammar.isterminal(g₇, 11) + +# ╔═╡ e8d4890b-db1c-4b74-8a6d-4388597cead8 +md""" +### return_type + +This function is rather obvious; it returns the non-terminal symbol that corresponds to a certain rule. The return type for all rules in our grammar is `:Int`. +""" + +# ╔═╡ 842f6873-1f7f-4785-9be5-336dee828ad8 +HerbGrammar.return_type(g₇, 11) + +# ╔═╡ c894f9c1-d608-4a11-9146-ea4b83b40325 +md""" +### child_types + +`child_types` returns the types of the nonterminal children of a rule in a vector. +If you just want to know how many children a rule has, and not necessarily which types they have, you can use `nchildren` +""" + +# ╔═╡ c4c058fb-28e6-4aa4-8609-5d0e9d29f42e +HerbGrammar.child_types(g₇, 11) + +# ╔═╡ d17660c0-fe52-4b71-9f7e-c77533801e5b +HerbGrammar.nchildren(g₇, 11) + +# ╔═╡ 681ac0f0-b207-48d1-9524-e2bb93f79717 +md""" +### nonterminals + +The `nonterminals` function can be used to obtain a list of all nonterminals in the grammar. +""" + +# ╔═╡ df689c72-deeb-4d94-8c10-702c3621bcba +HerbGrammar.nonterminals(g₇) + +# ╔═╡ 5374925a-46ef-49bc-b3b1-14a5afc5f523 +md""" +### Adding rules + +It is also possible to add rules to a grammar during execution. This can be done using the `add_rule!` function. The exclamatin mark is a Julia convention and is appended to name if a function modifies its arguments (in our example the grammar). + +A rule can be provided in the same syntax as is used in the grammar definition. +The rule should be of the `Expr` type, which is a built-in type for representing expressions. +An easy way of creating `Expr` values in Julia is to encapsulate it in brackets and use a colon as prefix: +""" + +# ╔═╡ d7f632bb-c3a5-4196-b4ce-9fc80ed24034 +HerbGrammar.add_rule!(g₇, :(Int = Int - Int)) + +# ╔═╡ 44aa6111-2374-4261-8616-73925c320b34 +md""" +### Removing rules + +It is also possible to remove rules in Herb.jl, however, this is a bit more involved. +As said before, rules have an index associated with them. +The internal representation of programs that are defined by the grammar makes use of those indices for efficiency. +Blindly removing a rule would shift the indices of other rules, and this could mean that existing programs get a different meaning or become invalid. + +Therefore, there are two functions for removing rules: + +- `remove_rule!` removes a rule from the grammar, but fills its place with a placeholder. Therefore, the indices stay the same, and only programs that use the removed rule become invalid. +- `cleanup_removed_rules!` removes all placeholders and shifts the indices of the other rules. + +""" + +# ╔═╡ f381e96a-a8c6-4ebd-adf2-e6da20ca63de +HerbGrammar.remove_rule!(g₇, 11) + +# ╔═╡ 5e0cb560-2a27-4372-945b-4be44fca1fbc +HerbGrammar.cleanup_removed_rules!(g₇) + +# ╔═╡ eb3bfde8-62c0-4497-b9cd-faba6ec9cf8d +md""" +## Context-sensitive grammars + +Context-sensitive grammars introduce additional constraints compared to context-free grammars (like the simple grammar examples above). +As before, we use the `@csgrammar` macro: +""" + +# ╔═╡ 112ce5fc-560d-438d-b58f-d52991a1651f +g₈ = HerbGrammar.@csgrammar begin + Int = |(0:9) + Int = Int + Int + Int = Int * Int + Int = x +end + +# ╔═╡ 327fcddb-9bdc-4ae8-8908-86a0b0619639 +md""" +Constraints can be added using the `addconstraint!` function, which takes a context-sensitive grammar and a constraint and adds the constraint to the grammar. + +For example, we can add a `` constraint to enforce that the input symbol `x` (rule 13) appears at least once in the program, to avoid programs that are just a constant. +""" + +# ╔═╡ 2b23504e-8e2c-47c5-b9b8-07b7b0063d2e +HerbGrammar.addconstraint!(g₈, Contains(13)) + +# ╔═╡ 4fb800af-7ac7-45b0-a3a0-ce153c7eb464 +md""" +There is a dedicated tutorial for constraints in Herb.jl and how to work with them. +""" + +# ╔═╡ 85724067-ff56-40e2-99e5-e26698a4a444 +md""" +### Probabilistic grammars + +Herb.jl also supports probabilistic grammars. +These grammars allow the user to assign a probability to each rule in the grammar. +A probabilistic grammar can be defined in a very similar way to a standard grammar, but has some slightly different syntax: +""" + +# ╔═╡ 19ef2475-1ebd-4b73-af6b-d7079182f27a +begin + g₉ = HerbGrammar.@pcsgrammar begin + 0.4 : Int = |(0:9) + 0.2 : Int = Int + Int + 0.1 : Int = Int * Int + 0.3 : Int = x + end + + for r ∈ 1:length(g₃.rules) + p = HerbGrammar.probability(g₈, r) + + println("$p : $r") + end +end + +# ╔═╡ 7f78d745-8b16-4308-9191-4dbe8bd11da9 +md""" +The numbers before each rule represent the probability assigned to that rule. +The total probability for each return type should add up to 1.0. +If this isn't the case, Herb.jl will normalize the probabilities. + +If a single line in the grammar definition represents multiple rules, such as `0.4 : Int = |(0:9)`, the probability will be evenly divided over all these rules. +""" + +# ╔═╡ 1f0ef848-d9b0-414d-9e1a-82beb6633c02 +md""" +## File writing + +### Saving & loading context-free grammars + +If you want to store a grammar on the disk, you can use the `store_csg`, `read_csg` and functions to store and read grammars respectively. +The `store_csg` grammar can also be used to store probabilistic grammars. To read probabilistic grammars, use `read_pcsg`. +The stored grammar files can also be opened using a text editor to be modified, as long as the contents of the file doesn't violate the syntax for defining grammars. +""" + +# ╔═╡ 9426a96a-43e3-458d-933e-44b36966db57 +HerbGrammar.store_csg(g₇, "demo.txt") + +# ╔═╡ 5ac3dfc3-54c0-438a-a3b9-d2d24309f532 +HerbGrammar.read_csg("demo.txt") + +# ╔═╡ fc5bd55a-3936-4b93-8f62-b2f708b83a4e +md""" +### Saving & loading context-sensitive grammars + +Saving and loading context-sensitive grammars is very similar to how it is done with context-free grammars. +The only difference is that an additional file is created for the constraints. +The file that contains the grammars can be edited and can also be read using the reader for context-free grammars. +The file that contains the constraints cannot be edited. +""" + +# ╔═╡ 32f1afed-5ccf-4378-8f1d-14e293ff2c1a +HerbGrammar.store_csg( g₈, "demo.grammar", "demo.constraints") + +# ╔═╡ c21e2aa6-55ba-4068-b127-d7b53c17eb45 +g₈, g₈.constraints + +# ╔═╡ 407f8c03-a0ec-4b23-ba2c-22de526becc8 +g₁₀ = HerbGrammar.read_csg("demo.grammar", "demo.constraints") + +# ╔═╡ ad351cf6-8f2b-4cd0-b207-1ecb53eb2e38 +g₁₀, g₁₀.constraints + +# ╔═╡ 00000000-0000-0000-0000-000000000001 +PLUTO_PROJECT_TOML_CONTENTS = """ +[deps] +HerbConstraints = "1fa96474-3206-4513-b4fa-23913f296dfc" +HerbGrammar = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" + +[compat] +HerbConstraints = "~0.2.2" +HerbGrammar = "~0.3.0" +""" + +# ╔═╡ 00000000-0000-0000-0000-000000000002 +PLUTO_MANIFEST_TOML_CONTENTS = """ +# This file is machine-generated - editing it directly is not advised + +julia_version = "1.10.4" +manifest_format = "2.0" +project_hash = "7e68dadf6c36ec756dee195b1ce6f2abc419c446" + +[[deps.AbstractTrees]] +git-tree-sha1 = "2d9c9a55f9c93e8887ad391fbae72f8ef55e1177" +uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" +version = "0.4.5" + +[[deps.ArgTools]] +uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" +version = "1.1.1" + +[[deps.ArnoldiMethod]] +deps = ["LinearAlgebra", "Random", "StaticArrays"] +git-tree-sha1 = "d57bd3762d308bded22c3b82d033bff85f6195c6" +uuid = "ec485272-7323-5ecc-a04f-4719b315124d" +version = "0.4.0" + +[[deps.Artifacts]] +uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" + +[[deps.Base64]] +uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" + +[[deps.Bzip2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "9e2a6b69137e6969bab0152632dcb3bc108c8bdd" +uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0" +version = "1.0.8+1" + +[[deps.Cairo_jll]] +deps = ["Artifacts", "Bzip2_jll", "CompilerSupportLibraries_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "a2f1c8c668c8e3cb4cca4e57a8efdb09067bb3fd" +uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a" +version = "1.18.0+2" + +[[deps.CommonSubexpressions]] +deps = ["MacroTools", "Test"] +git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7" +uuid = "bbf7d656-a473-5ed7-a52c-81e309532950" +version = "0.3.0" + +[[deps.Compat]] +deps = ["TOML", "UUIDs"] +git-tree-sha1 = "b1c55339b7c6c350ee89f2c1604299660525b248" +uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" +version = "4.15.0" +weakdeps = ["Dates", "LinearAlgebra"] + + [deps.Compat.extensions] + CompatLinearAlgebraExt = "LinearAlgebra" + +[[deps.CompilerSupportLibraries_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" +version = "1.1.1+0" + +[[deps.DataStructures]] +deps = ["Compat", "InteractiveUtils", "OrderedCollections"] +git-tree-sha1 = "1d0a14036acb104d9e89698bd408f63ab58cdc82" +uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" +version = "0.18.20" + +[[deps.Dates]] +deps = ["Printf"] +uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" + +[[deps.Distributed]] +deps = ["Random", "Serialization", "Sockets"] +uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b" + +[[deps.Downloads]] +deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"] +uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" +version = "1.6.0" + +[[deps.Expat_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "1c6317308b9dc757616f0b5cb379db10494443a7" +uuid = "2e619515-83b5-522b-bb60-26c02a35a201" +version = "2.6.2+0" + +[[deps.FileWatching]] +uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee" + +[[deps.Fontconfig_jll]] +deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Zlib_jll"] +git-tree-sha1 = "db16beca600632c95fc8aca29890d83788dd8b23" +uuid = "a3f928ae-7b40-5064-980b-68af3947d34b" +version = "2.13.96+0" + +[[deps.FreeType2_jll]] +deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "5c1d8ae0efc6c2e7b1fc502cbe25def8f661b7bc" +uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7" +version = "2.13.2+0" + +[[deps.Gettext_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "XML2_jll"] +git-tree-sha1 = "9b02998aba7bf074d14de89f9d37ca24a1a0b046" +uuid = "78b55507-aeef-58d4-861c-77aaff3498b1" +version = "0.21.0+0" + +[[deps.Glib_jll]] +deps = ["Artifacts", "Gettext_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE2_jll", "Zlib_jll"] +git-tree-sha1 = "7c82e6a6cd34e9d935e9aa4051b66c6ff3af59ba" +uuid = "7746bdde-850d-59dc-9ae8-88ece973131d" +version = "2.80.2+0" + +[[deps.Graphite2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "344bf40dcab1073aca04aa0df4fb092f920e4011" +uuid = "3b182d85-2403-5c21-9c21-1e1f0cc25472" +version = "1.3.14+0" + +[[deps.Graphs]] +deps = ["ArnoldiMethod", "Compat", "DataStructures", "Distributed", "Inflate", "LinearAlgebra", "Random", "SharedArrays", "SimpleTraits", "SparseArrays", "Statistics"] +git-tree-sha1 = "4f2b57488ac7ee16124396de4f2bbdd51b2602ad" +uuid = "86223c79-3864-5bf0-83f7-82e725a168b6" +version = "1.11.0" + +[[deps.HarfBuzz_ICU_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "6ccbc4fdf65c8197738c2d68cc55b74b19c97ac2" +uuid = "655565e8-fb53-5cb3-b0cd-aec1ca0647ea" +version = "2.8.1+0" + +[[deps.HarfBuzz_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "129acf094d168394e80ee1dc4bc06ec835e510a3" +uuid = "2e76f6c2-a576-52d4-95c1-20adfe4de566" +version = "2.8.1+1" + +[[deps.HerbConstraints]] +deps = ["DataStructures", "HerbCore", "HerbGrammar", "MLStyle"] +git-tree-sha1 = "2e54da1d19119847b242d1ceda212b180cca36a9" +uuid = "1fa96474-3206-4513-b4fa-23913f296dfc" +version = "0.2.2" + +[[deps.HerbCore]] +git-tree-sha1 = "923877c2715b8166d7ba9f9be2136d70eed87725" +uuid = "2b23ba43-8213-43cb-b5ea-38c12b45bd45" +version = "0.3.0" + +[[deps.HerbGrammar]] +deps = ["AbstractTrees", "DataStructures", "HerbCore", "Serialization", "TreeView"] +git-tree-sha1 = "b4cbf9712dbb3ab281ff4ed517d3cd6bc12f3078" +uuid = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" +version = "0.3.0" + +[[deps.ICU_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "20b6765a3016e1fca0c9c93c80d50061b94218b7" +uuid = "a51ab1cf-af8e-5615-a023-bc2c838bba6b" +version = "69.1.0+0" + +[[deps.Inflate]] +git-tree-sha1 = "ea8031dea4aff6bd41f1df8f2fdfb25b33626381" +uuid = "d25df0c9-e2be-5dd7-82c8-3ad0b3e990b9" +version = "0.1.4" + +[[deps.InteractiveUtils]] +deps = ["Markdown"] +uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" + +[[deps.JLLWrappers]] +deps = ["Artifacts", "Preferences"] +git-tree-sha1 = "7e5d6779a1e09a36db2a7b6cff50942a0a7d0fca" +uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210" +version = "1.5.0" + +[[deps.JpegTurbo_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "c84a835e1a09b289ffcd2271bf2a337bbdda6637" +uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8" +version = "3.0.3+0" + +[[deps.LERC_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "bf36f528eec6634efc60d7ec062008f171071434" +uuid = "88015f11-f218-50d7-93a8-a6af411a945d" +version = "3.0.0+1" + +[[deps.LLVMOpenMP_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "d986ce2d884d49126836ea94ed5bfb0f12679713" +uuid = "1d63c593-3942-5779-bab2-d838dc0a180e" +version = "15.0.7+0" + +[[deps.LZO_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "70c5da094887fd2cae843b8db33920bac4b6f07d" +uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac" +version = "2.10.2+0" + +[[deps.LaTeXStrings]] +git-tree-sha1 = "50901ebc375ed41dbf8058da26f9de442febbbec" +uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" +version = "1.3.1" + +[[deps.LibCURL]] +deps = ["LibCURL_jll", "MozillaCACerts_jll"] +uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" +version = "0.6.4" + +[[deps.LibCURL_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"] +uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" +version = "8.4.0+0" + +[[deps.LibGit2]] +deps = ["Base64", "LibGit2_jll", "NetworkOptions", "Printf", "SHA"] +uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" + +[[deps.LibGit2_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll"] +uuid = "e37daf67-58a4-590a-8e99-b0245dd2ffc5" +version = "1.6.4+0" + +[[deps.LibSSH2_jll]] +deps = ["Artifacts", "Libdl", "MbedTLS_jll"] +uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" +version = "1.11.0+1" + +[[deps.Libdl]] +uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" + +[[deps.Libffi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "0b4a5d71f3e5200a7dff793393e09dfc2d874290" +uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490" +version = "3.2.2+1" + +[[deps.Libgcrypt_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgpg_error_jll"] +git-tree-sha1 = "9fd170c4bbfd8b935fdc5f8b7aa33532c991a673" +uuid = "d4300ac3-e22c-5743-9152-c294e39db1e4" +version = "1.8.11+0" + +[[deps.Libgpg_error_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "fbb1f2bef882392312feb1ede3615ddc1e9b99ed" +uuid = "7add5ba3-2f88-524e-9cd5-f83b8a55f7b8" +version = "1.49.0+0" + +[[deps.Libiconv_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "f9557a255370125b405568f9767d6d195822a175" +uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531" +version = "1.17.0+0" + +[[deps.Libmount_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "0c4f9c4f1a50d8f35048fa0532dabbadf702f81e" +uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9" +version = "2.40.1+0" + +[[deps.Libtiff_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "LERC_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"] +git-tree-sha1 = "3eb79b0ca5764d4799c06699573fd8f533259713" +uuid = "89763e89-9b03-5906-acba-b20f662cd828" +version = "4.4.0+0" + +[[deps.Libuuid_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "5ee6203157c120d79034c748a2acba45b82b8807" +uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700" +version = "2.40.1+0" + +[[deps.LinearAlgebra]] +deps = ["Libdl", "OpenBLAS_jll", "libblastrampoline_jll"] +uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + +[[deps.LittleCMS_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pkg"] +git-tree-sha1 = "110897e7db2d6836be22c18bffd9422218ee6284" +uuid = "d3a379c0-f9a3-5b72-a4c0-6bf4d2e8af0f" +version = "2.12.0+0" + +[[deps.Logging]] +uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" + +[[deps.MLStyle]] +git-tree-sha1 = "bc38dff0548128765760c79eb7388a4b37fae2c8" +uuid = "d8e11817-5142-5d16-987a-aa16d5891078" +version = "0.4.17" + +[[deps.MacroTools]] +deps = ["Markdown", "Random"] +git-tree-sha1 = "2fa9ee3e63fd3a4f7a9a4f4744a52f4856de82df" +uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" +version = "0.5.13" + +[[deps.Markdown]] +deps = ["Base64"] +uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" + +[[deps.MbedTLS_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" +version = "2.28.2+1" + +[[deps.Mmap]] +uuid = "a63ad114-7e13-5084-954f-fe012c677804" + +[[deps.MozillaCACerts_jll]] +uuid = "14a3606d-f60d-562e-9121-12d972cd8159" +version = "2023.1.10" + +[[deps.NetworkOptions]] +uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" +version = "1.2.0" + +[[deps.OpenBLAS_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] +uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" +version = "0.3.23+4" + +[[deps.OpenJpeg_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libtiff_jll", "LittleCMS_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "76374b6e7f632c130e78100b166e5a48464256f8" +uuid = "643b3616-a352-519d-856d-80112ee9badc" +version = "2.4.0+0" + +[[deps.OpenSSL_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "a12e56c72edee3ce6b96667745e6cbbe5498f200" +uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95" +version = "1.1.23+0" + +[[deps.OrderedCollections]] +git-tree-sha1 = "dfdf5519f235516220579f949664f1bf44e741c5" +uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" +version = "1.6.3" + +[[deps.PCRE2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "efcefdf7-47ab-520b-bdef-62a2eaa19f15" +version = "10.42.0+1" + +[[deps.Pixman_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "LLVMOpenMP_jll", "Libdl"] +git-tree-sha1 = "35621f10a7531bc8fa58f74610b1bfb70a3cfc6b" +uuid = "30392449-352a-5448-841d-b1acce4e97dc" +version = "0.43.4+0" + +[[deps.Pkg]] +deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] +uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" +version = "1.10.0" + +[[deps.Poppler_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "Glib_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "OpenJpeg_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "02148a0cb2532f22c0589ceb75c110e168fb3d1f" +uuid = "9c32591e-4766-534b-9725-b71a8799265b" +version = "21.9.0+0" + +[[deps.PrecompileTools]] +deps = ["Preferences"] +git-tree-sha1 = "5aa36f7049a63a1528fe8f7c3f2113413ffd4e1f" +uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a" +version = "1.2.1" + +[[deps.Preferences]] +deps = ["TOML"] +git-tree-sha1 = "9306f6085165d270f7e3db02af26a400d580f5c6" +uuid = "21216c6a-2e73-6563-6e65-726566657250" +version = "1.4.3" + +[[deps.Printf]] +deps = ["Unicode"] +uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" + +[[deps.REPL]] +deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] +uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" + +[[deps.Random]] +deps = ["SHA"] +uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" + +[[deps.Requires]] +deps = ["UUIDs"] +git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7" +uuid = "ae029012-a4dd-5104-9daa-d747884805df" +version = "1.3.0" + +[[deps.SHA]] +uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" +version = "0.7.0" + +[[deps.Serialization]] +uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" + +[[deps.SharedArrays]] +deps = ["Distributed", "Mmap", "Random", "Serialization"] +uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383" + +[[deps.SimpleTraits]] +deps = ["InteractiveUtils", "MacroTools"] +git-tree-sha1 = "5d7e3f4e11935503d3ecaf7186eac40602e7d231" +uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d" +version = "0.9.4" + +[[deps.Sockets]] +uuid = "6462fe0b-24de-5631-8697-dd941f90decc" + +[[deps.SparseArrays]] +deps = ["Libdl", "LinearAlgebra", "Random", "Serialization", "SuiteSparse_jll"] +uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" +version = "1.10.0" + +[[deps.StaticArrays]] +deps = ["LinearAlgebra", "PrecompileTools", "Random", "StaticArraysCore"] +git-tree-sha1 = "9ae599cd7529cfce7fea36cf00a62cfc56f0f37c" +uuid = "90137ffa-7385-5640-81b9-e52037218182" +version = "1.9.4" + + [deps.StaticArrays.extensions] + StaticArraysChainRulesCoreExt = "ChainRulesCore" + StaticArraysStatisticsExt = "Statistics" + + [deps.StaticArrays.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" + +[[deps.StaticArraysCore]] +git-tree-sha1 = "36b3d696ce6366023a0ea192b4cd442268995a0d" +uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" +version = "1.4.2" + +[[deps.Statistics]] +deps = ["LinearAlgebra", "SparseArrays"] +uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" +version = "1.10.0" + +[[deps.SuiteSparse_jll]] +deps = ["Artifacts", "Libdl", "libblastrampoline_jll"] +uuid = "bea87d4a-7f5b-5778-9afe-8cc45184846c" +version = "7.2.1+1" + +[[deps.TOML]] +deps = ["Dates"] +uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" +version = "1.0.3" + +[[deps.Tar]] +deps = ["ArgTools", "SHA"] +uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" +version = "1.10.0" + +[[deps.Test]] +deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] +uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[[deps.TikzGraphs]] +deps = ["Graphs", "LaTeXStrings", "TikzPictures"] +git-tree-sha1 = "e8f41ed9a2cabf6699d9906c195bab1f773d4ca7" +uuid = "b4f28e30-c73f-5eaf-a395-8a9db949a742" +version = "1.4.0" + +[[deps.TikzPictures]] +deps = ["LaTeXStrings", "Poppler_jll", "Requires", "tectonic_jll"] +git-tree-sha1 = "79e2d29b216ef24a0f4f905532b900dcf529aa06" +uuid = "37f6aa50-8035-52d0-81c2-5a1d08754b2d" +version = "3.5.0" + +[[deps.TreeView]] +deps = ["CommonSubexpressions", "Graphs", "MacroTools", "TikzGraphs"] +git-tree-sha1 = "41ddcefb625f2ab0f4d9f2081c2da1af2ccbbf8b" +uuid = "39424ebd-4cf3-5550-a685-96706a953f40" +version = "0.5.1" + +[[deps.UUIDs]] +deps = ["Random", "SHA"] +uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" + +[[deps.Unicode]] +uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" + +[[deps.XML2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Zlib_jll"] +git-tree-sha1 = "52ff2af32e591541550bd753c0da8b9bc92bb9d9" +uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a" +version = "2.12.7+0" + +[[deps.XSLT_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgcrypt_jll", "Libgpg_error_jll", "Libiconv_jll", "Pkg", "XML2_jll", "Zlib_jll"] +git-tree-sha1 = "91844873c4085240b95e795f692c4cec4d805f8a" +uuid = "aed1982a-8fda-507f-9586-7b0439959a61" +version = "1.1.34+0" + +[[deps.Xorg_libX11_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libxcb_jll", "Xorg_xtrans_jll"] +git-tree-sha1 = "afead5aba5aa507ad5a3bf01f58f82c8d1403495" +uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc" +version = "1.8.6+0" + +[[deps.Xorg_libXau_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "6035850dcc70518ca32f012e46015b9beeda49d8" +uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec" +version = "1.0.11+0" + +[[deps.Xorg_libXdmcp_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "34d526d318358a859d7de23da945578e8e8727b7" +uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05" +version = "1.1.4+0" + +[[deps.Xorg_libXext_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "d2d1a5c49fae4ba39983f63de6afcbea47194e85" +uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3" +version = "1.3.6+0" + +[[deps.Xorg_libXrender_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "47e45cd78224c53109495b3e324df0c37bb61fbe" +uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa" +version = "0.9.11+0" + +[[deps.Xorg_libpthread_stubs_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "8fdda4c692503d44d04a0603d9ac0982054635f9" +uuid = "14d82f49-176c-5ed1-bb49-ad3f5cbd8c74" +version = "0.1.1+0" + +[[deps.Xorg_libxcb_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "XSLT_jll", "Xorg_libXau_jll", "Xorg_libXdmcp_jll", "Xorg_libpthread_stubs_jll"] +git-tree-sha1 = "b4bfde5d5b652e22b9c790ad00af08b6d042b97d" +uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b" +version = "1.15.0+0" + +[[deps.Xorg_xtrans_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e92a1a012a10506618f10b7047e478403a046c77" +uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10" +version = "1.5.0+0" + +[[deps.Zlib_jll]] +deps = ["Libdl"] +uuid = "83775a58-1f1d-513f-b197-d71354ab007a" +version = "1.2.13+1" + +[[deps.Zstd_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e678132f07ddb5bfa46857f0d7620fb9be675d3b" +uuid = "3161d3a3-bdf6-5164-811a-617609db77b4" +version = "1.5.6+0" + +[[deps.libblastrampoline_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" +version = "5.8.0+1" + +[[deps.libpng_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "d7015d2e18a5fd9a4f47de711837e980519781a4" +uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f" +version = "1.6.43+1" + +[[deps.nghttp2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" +version = "1.52.0+1" + +[[deps.p7zip_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" +version = "17.4.0+2" + +[[deps.tectonic_jll]] +deps = ["Artifacts", "Fontconfig_jll", "FreeType2_jll", "Graphite2_jll", "HarfBuzz_ICU_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "OpenSSL_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "54867b00af20c70b52a1f9c00043864d8b926a21" +uuid = "d7dd28d6-a5e6-559c-9131-7eb760cdacc5" +version = "0.13.1+0" +""" + +# ╔═╡ Cell order: +# ╟─93b96839-9676-477b-aebb-e50d733a6719 +# ╟─d863795b-5fea-4b69-8f63-e04352bb970a +# ╠═84549502-876a-4ffc-8069-1ffc17622f9a +# ╟─751e1119-fe59-41dc-8785-a3ed0a9cacb9 +# ╠═1030a96d-b739-4232-820b-f21826512251 +# ╟─0224c914-3763-4c55-844c-e9aca430b377 +# ╠═4619b92e-2f9e-429f-9e9a-77b97be7edd1 +# ╟─f8e82610-b0d0-4763-939d-aadc92336c52 +# ╠═f4d34f0a-6c88-48db-84a0-ed3fb01fb7aa +# ╟─8732f232-3069-4fb1-9b6a-0e24b8724451 +# ╠═b68090b2-0eed-4e4a-acdc-6df71747a46d +# ╟─89e8ab14-7097-4088-937f-6453722da9d0 +# ╠═26b9a050-c4b4-42a8-b1d7-e50fb98a7e8a +# ╟─7ebb2202-8aa1-4404-b1dd-ac1669f9a120 +# ╠═9351c7ab-a39f-4f47-a3ee-e1684e2b3c07 +# ╟─3f44d67b-bd18-481e-bca8-bb4904f6b418 +# ╠═58af16a2-7f23-4d06-8836-fb737788ada0 +# ╟─303894cd-a0c2-461c-a671-7330f75b338e +# ╟─5c103ce3-0cb2-4004-ac44-50613a461153 +# ╠═38422e77-cfd8-4a2c-8cf2-f1b430f64ad1 +# ╠═112d4837-e646-4e24-84c8-771b9855aebf +# ╟─e8d4890b-db1c-4b74-8a6d-4388597cead8 +# ╠═842f6873-1f7f-4785-9be5-336dee828ad8 +# ╟─c894f9c1-d608-4a11-9146-ea4b83b40325 +# ╠═c4c058fb-28e6-4aa4-8609-5d0e9d29f42e +# ╠═d17660c0-fe52-4b71-9f7e-c77533801e5b +# ╟─681ac0f0-b207-48d1-9524-e2bb93f79717 +# ╠═df689c72-deeb-4d94-8c10-702c3621bcba +# ╟─5374925a-46ef-49bc-b3b1-14a5afc5f523 +# ╠═d7f632bb-c3a5-4196-b4ce-9fc80ed24034 +# ╟─44aa6111-2374-4261-8616-73925c320b34 +# ╠═f381e96a-a8c6-4ebd-adf2-e6da20ca63de +# ╠═5e0cb560-2a27-4372-945b-4be44fca1fbc +# ╟─eb3bfde8-62c0-4497-b9cd-faba6ec9cf8d +# ╠═112ce5fc-560d-438d-b58f-d52991a1651f +# ╟─327fcddb-9bdc-4ae8-8908-86a0b0619639 +# ╠═2b23504e-8e2c-47c5-b9b8-07b7b0063d2e +# ╟─4fb800af-7ac7-45b0-a3a0-ce153c7eb464 +# ╟─85724067-ff56-40e2-99e5-e26698a4a444 +# ╠═19ef2475-1ebd-4b73-af6b-d7079182f27a +# ╟─7f78d745-8b16-4308-9191-4dbe8bd11da9 +# ╟─1f0ef848-d9b0-414d-9e1a-82beb6633c02 +# ╠═9426a96a-43e3-458d-933e-44b36966db57 +# ╠═5ac3dfc3-54c0-438a-a3b9-d2d24309f532 +# ╟─fc5bd55a-3936-4b93-8f62-b2f708b83a4e +# ╠═32f1afed-5ccf-4378-8f1d-14e293ff2c1a +# ╠═c21e2aa6-55ba-4068-b127-d7b53c17eb45 +# ╠═407f8c03-a0ec-4b23-ba2c-22de526becc8 +# ╠═ad351cf6-8f2b-4cd0-b207-1ecb53eb2e38 +# ╟─00000000-0000-0000-0000-000000000001 +# ╟─00000000-0000-0000-0000-000000000002 From d463fc626ed2b85ef8c1334ea23666c79233e7aa Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Wed, 28 Aug 2024 07:31:45 +0200 Subject: [PATCH 06/13] Replace deprecated cfgrammar. --- docs/src/tutorials/getting_started_with_constraints.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/tutorials/getting_started_with_constraints.jl b/docs/src/tutorials/getting_started_with_constraints.jl index b359985..5791d13 100644 --- a/docs/src/tutorials/getting_started_with_constraints.jl +++ b/docs/src/tutorials/getting_started_with_constraints.jl @@ -29,7 +29,7 @@ We will also redefine the simple arithmetic grammar from the previous tutorial. """ # ╔═╡ aacedef3-08d6-4bc0-a2c3-dd9a9aac340d -grammar = @cfgrammar begin +grammar = @csgrammar begin Int = 1 Int = x Int = - Int From 11faf5a086eecd7da03d7e369ba9f702f1004791 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Thu, 19 Sep 2024 09:12:00 +0200 Subject: [PATCH 07/13] Fix broken code for parameters section of tutorial. --- docs/src/get_started.md | 6 +- docs/src/tutorials/advanced_search.jl | 1307 +++++++++++++++++ .../tutorials/getting_started_with_herb.jl | 6 +- .../tutorials/working_with_interpreters.jl | 64 +- 4 files changed, 1348 insertions(+), 35 deletions(-) create mode 100644 docs/src/tutorials/advanced_search.jl diff --git a/docs/src/get_started.md b/docs/src/get_started.md index cdbbaa8..3dabafd 100644 --- a/docs/src/get_started.md +++ b/docs/src/get_started.md @@ -10,13 +10,13 @@ using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret To define a program synthesis problem, we need a grammar and specification. -First, a grammar can be constructed using the `@cfgrammar` macro included in `HerbGrammar`. Alternatively, we can use the `@csgrammar` macro, which will be shown later on in the tutorial when we use constraints. +First, a grammar can be constructed using the `@csgrammar` macro included in `HerbGrammar`. Here, we describe a simple integer arithmetic example, that can add and multiply an input variable `x` or the integers `1,2`, using ```julia -g = @cfgrammar begin +g = @csgrammar begin Number = |(1:2) Number = x Number = Number + Number @@ -62,7 +62,7 @@ using HerbSearch, HerbSpecification, HerbInterpret, HerbGrammar # define our very simple context-free grammar # Can add and multiply an input variable x or the integers 1,2. -g = @cfgrammar begin +g = @csgrammar begin Number = |(1:2) Number = x Number = Number + Number diff --git a/docs/src/tutorials/advanced_search.jl b/docs/src/tutorials/advanced_search.jl new file mode 100644 index 0000000..ae9ad66 --- /dev/null +++ b/docs/src/tutorials/advanced_search.jl @@ -0,0 +1,1307 @@ +### A Pluto.jl notebook ### +# v0.19.43 + +using Markdown +using InteractiveUtils + +# ╔═╡ a93d954d-0f09-4b6d-a3a5-62bfe39681e2 +using PlutoUI + +# ╔═╡ c4441fa4-09ec-4b81-9681-b13b93a9c9c0 +using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret, HerbConstraints + + +# ╔═╡ dddca175-3d88-45ce-90da-575c0ba38175 +md""" +# Advanced Search Procedures in Herb.jl + +[A more verbose getting started with Herb.jl]() described the concept of a program space and showed how to search it with Herb.jl, using a simple search with a BFS iterator. +This tutorial takes a closer look at advanced search procedures. + +More specifically, you will learn about + +- **Parameters** that can be specified and their effect on the search procedure. +- **Search methods** that can be employed to find a solution program to a program synthesis problem, including basic search (BFS and DFS), stochastic search and genetic search methods. +- **Other functionality** of the module `HerbSearch.jl` + (TODO: why are they in this tutorial?) +""" + +# ╔═╡ 6ab37bbc-73e2-4d9a-a8b2-e715a0b61c8f +TableOfContents() + +# ╔═╡ 67931820-0f43-41e1-898e-5b5bd55e30d1 +md""" +We start with a simple grammar:. +""" + +# ╔═╡ e41c61a4-0b2c-46da-8f7b-fe6dc529c544 +g_1 = @csgrammar begin + Number = |(1:2) + Number = x + Number = Number + Number + Number = Number * Number +end + +# ╔═╡ 9ad0a92a-10d5-458a-8f05-9011c8553609 +md""" +Let's use the simple program `2x+1` as our problem and generate some input-output examples for the problem specification. +""" + +# ╔═╡ 65317911-bc92-4b84-9744-ed784adcab4a +problem_1 = Problem([IOExample(Dict(:x => x), 2x+1) for x ∈ 1:5]) + +# ╔═╡ e9c6bc00-21f5-4a99-8bec-63cf2156c233 +md""" +## Parameters + +Search procedures typically have some hyperparameters that can be cofigured by the user. + +### `max_depth` + +`max_depth` controls the maximum depth of the program trees we want to explore. + +In the following example, we can see the effect of `max_depth` on the number of allocations considered. +""" + +# ╔═╡ 338f19f1-3a62-4462-b8dc-24424d7644f2 +iterator_1 = BFSIterator(g_1, :Number, max_depth=3) + +# ╔═╡ 542cd47e-74cd-4b6f-acc9-bf524222e583 +iterator_2 = BFSIterator(g_1, :Number, max_depth=6) + +# ╔═╡ a6fb2e91-b73a-4032-930f-d884abd539e2 +begin + println("Solution for max_depth = 3:") + solution_1 = @time synth(problem_1, iterator_1) + println(solution_1) + println("----------------------") + println("Solution for max_depth = 6:") + solution_2 = @time synth(problem_1, iterator_2) + println(solution_2) +end + +# ╔═╡ 4b49f206-970e-47d2-af65-336ba65b1019 +md""" +TODO: Explain @time +""" + +# ╔═╡ 35405357-179b-4e77-9bdc-edf5a550b36d +md""" +### `max_enumerations` +Another parameter to use is `max_enumerations`, which describes the maximum number of programs that can be tested at evaluation. +Let's see how many enumerations are necessary to solve our simple problem. +""" + +# ╔═╡ 3954dd49-07a2-4ec2-91b4-9c9596d5c264 +begin + for i in range(1, 50) + println(i, " enumerations") + iterator = BFSIterator(g_1, :Number, max_depth=i) + solution = @time synth(problem_1, iterator) + println(solution) + end +end + +# ╔═╡ 9892e91b-9115-4520-9637-f8d7c8905825 +md""" +TODO: numbers seem to have changed. Not 24 anymore. How reproducible is this anyway? +What does allocations mean? + +We see that only when `i >= 24`, there is a result, after that, increasing `i` does not have any effect on the number of allocations. + +A final parameter we consider here is `allow_evaluation_errors`, which is `false` by default. When this is set to `true`, the program will still run even when an exception is thrown during evaluation. To see the effect of this, we create a new grammar. We can also retrieve the error together with the solution from the search method. +""" + +# ╔═╡ a4e58bbc-7c14-4fce-b35d-688b56e0eb61 +md""" +### `allow_evaluation_errors` + +TODO: What do we mean with 'program will still run'? + +A final parameter we consider here is `allow_evaluation_errors`, which is `false` by default. When set to `true`, the program will still run even when an exception is thrown during evaluation. + +We will use a new example to see the effect of `allow_evaluation_errors`. After defining a new grammar and problem, ... + + +We can also retrieve the error together with the solution from the search method. +""" + +# ╔═╡ 9fb40ceb-8d41-491b-8941-20a8b240eb82 +begin + g_2 = @cfgrammar begin + Number = 1 + List = [] + Index = List[Number] + end +end + +# ╔═╡ 94e0d676-a9c7-4291-8696-15301e541c30 +problem_2 = Problem([IOExample(Dict(), x) for x ∈ 1:5]) + +# ╔═╡ a4a7daed-f89b-44ad-8787-9199c05bf046 +iterator_3 = BFSIterator(g_2, :Number, max_depth=2) + +# ╔═╡ 4821fd3a-ff2d-4991-99ad-76608d11b1da +solution_3 = @time synth(problem_2, iterator_3, allow_evaluation_errors=true) + +# ╔═╡ 8d91b2e3-30b5-4ea2-bd3f-3055bb6d1d5a +# solution = search(g_2, problem_2, :Index, max_depth=2, allow_evaluation_errors=true) + +# ╔═╡ 52332fa2-7ea7-4226-9460-e0bbc905c619 +println("solution: ", solution_3) + +# ╔═╡ c26fea48-f812-4f24-bb8c-680e14a55df7 +md""" +There is another search method called `search_best` which returns both the solution and the possible error. The method returns the best program found so far. In this case, we can also see the error (`typemax(Int)`): +""" + +# ╔═╡ 57f5face-d9d5-441a-8e0e-6ef6319fc178 +solution, error = search_best(g, problem, :Index, max_depth=2, allow_evaluation_errors=true) +println("solution: ", solution) +println("error: ", error) + +# ╔═╡ 9b4b21e0-dc6a-43ae-a511-79988ee99001 +md""" +## Search methods + +We now show examples of using different search procedures, which are initialized by passing different enumerators to the search function. + +### Breadth-First Search + +The breadth-first search will first enumerate all possible programs at the same depth before considering programs with a depth of one more. A tree of the grammar is returned with programs ordered in increasing sizes. We can first `collect` the programs that have a `max-depth` of 2 and a `max_size` of infinite (integer maximum value), where the starting symbol is of type `Real`. This function uses a default heuristic 'left-most first', such that the left-most child in the tree is always explored first. +""" + +# ╔═╡ 3af650d9-19c6-4351-920d-d2361091f628 +g1 = @cfgrammar begin + Real = 1 | 2 + Real = Real * Real +end +programs = collect(get_bfs_enumerator(g1, 2, typemax(Int), :Real)) + +# ╔═╡ d3ff497e-d2c2-4df6-8e4c-cdca70fd0677 +md""" +We can test that this function returns all and only the correct functions. +""" + +# ╔═╡ da7f326c-f0d5-4837-ac9a-5bcad604566e +answer_programs = [ + RuleNode(1), + RuleNode(2), + RuleNode(3, [RuleNode(1), RuleNode(1)]), + RuleNode(3, [RuleNode(1), RuleNode(2)]), + RuleNode(3, [RuleNode(2), RuleNode(1)]), + RuleNode(3, [RuleNode(2), RuleNode(2)]) +] + +println(all(p ∈ programs for p ∈ answer_programs)) + +# ╔═╡ 0020b79a-6352-4e2d-93f6-2a1d7b03ae2c +md""" +### Depth-First Search + +In depth-first search, we first explore a certain branch of the search tree till the `max_depth` or a correct program is reached before we consider the next branch. +""" + +# ╔═╡ 789150a8-862c-48c3-88b8-710b81ab34cf +g1 = @cfgrammar begin +Real = 1 | 2 +Real = Real * Real +end +programs = collect(get_dfs_enumerator(g1, 2, typemax(Int), :Real)) +println(programs) + +# ╔═╡ 243165be-a9d2-484d-8046-811a2b0ba139 +md""" +`get_dfs_enumerator` also has a default left-most heuristic and we consider what the difference is in output. +""" + +# ╔═╡ 3d01c6f1-80a6-4904-97e2-775170e97bbf +g1 = @cfgrammar begin + Real = 1 | 2 + Real = Real * Real +end +programs = collect(get_dfs_enumerator(g1, 2, typemax(Int), :Real, heuristic_rightmost)) +println(programs) + +# ╔═╡ 168c71bf-ce5b-4ab3-b29a-5996981c42a5 +md""" +## Stochastic search +We now introduce a few stochastic search algorithms, for which we first create a simple grammar and a helper function for problems. +""" + +# ╔═╡ a4b522cf-78f0-4d44-88c8-82dd0cdbf952 +grammar = @csgrammar begin + X = |(1:5) + X = X * X + X = X + X + X = X - X + X = x +end + +# ╔═╡ f313edb9-8fd9-4d78-88cd-89226f5c769d +function create_problem(f, range=20) + examples = [IOExample(Dict(:x => x), f(x)) for x ∈ 1:range] + return Problem(examples), examples +end + +# ╔═╡ 0da9053a-959b-471e-8918-662ec63da71c +md""" +### Metropolis-Hastings + +One of the stochastic search methods that is implemented is Metropolis-Hastings (MH), which samples from a distribution of programs based on the grammar. For more information on MH, see for example [this webpage](https://stephens999.github.io/fiveMinuteStats/MH_intro.html). + +The example below uses a simple arithmetic example. You can try running this code block multiple times, which will give different programs, as the search is stochastic. +""" + +# ╔═╡ 0a30fb40-cd45-4661-a501-ae8e45f1e07e +e = x -> x * x + 4 +problem, examples = create_problem(e) +enumerator = get_mh_enumerator(examples, mean_squared_error) +program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3) + +# ╔═╡ 700270ea-90bd-474b-91d9-0e5ed329776a +md""" +### Very Large Scale Neighbourhood Search + +The second implemented stochastic search method is VLSN, which searches for a local optimum in the neighbourhood. For more information, see [this article](https://backend.orbit.dtu.dk/ws/portalfiles/portal/5293785/Pisinger.pdf). + +Given the same grammar as before, we can try it with some simple examples. +""" + +# ╔═╡ 8731f312-bfcf-4f6c-86fa-60014dc146d6 +e = x -> 10 +max_depth = 2 +problem, examples = create_problem(e) +enumerator = get_vlsn_enumerator(examples, mean_squared_error, max_depth) +program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth) + + +# ╔═╡ 46ca65d1-d876-4abc-a562-8d266bad195f +e = x -> x +max_depth = 1 +problem, examples = create_problem(e) +enumerator = get_vlsn_enumerator(examples, mean_squared_error, max_depth) +program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth) + +# ╔═╡ 599194a8-3f47-4917-9143-a5fe0d43029f +md""" +### Simulated Annealing + +The third stochastic search method is called simulated annealing. This is another hill-climbing method to find local optima. For more information, see [this page](https://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/anneal.html). + +We try the example from earlier, but now we can additionally define the `initial_temperature` of the algorithm, which is 1 by default. Change the value below to see the effect. +""" + +# ╔═╡ cb5935ed-d89b-4e25-9243-d201daf18e78 +e = x -> x * x + 4 +initial_temperature = 1 +problem, examples = create_problem(e) +enumerator = get_sa_enumerator(examples, mean_squared_error, initial_temperature) +program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3) + +# ╔═╡ d0c3742e-23e5-4ca1-9e83-b6d1e8a7cded +e = x -> x * x + 4 +initial_temperature = 2 +problem, examples = create_problem(e) +enumerator = get_sa_enumerator(examples, mean_squared_error, initial_temperature) +program, cost = @time search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3) + +# ╔═╡ 5df0ba53-b528-4baf-9980-cafe5d73f9dd +md""" +### Genetic Search + +Genetic search is a type of evolutionary algorithm, which will simulate the process of natural selection and return the 'fittest' program of the population. For more information, see [here](https://www.geeksforgeeks.org/genetic-algorithms/). + +We show the example of finding a lambda function. Try varying the parameters of the genetic search to see what happens. +""" + +# ╔═╡ a434645b-d592-4162-a8b4-b4b04cea30a9 +e = x -> 3 * x * x + (x + 2) +problem, examples = create_problem(e) +enumerator = get_genetic_enumerator(examples, + initial_population_size = 10, + mutation_probability = 0.8, + maximum_initial_population_depth = 3, +) +program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=nothing, max_time=20) + +# ╔═╡ 38cd9032-27c0-4179-a536-ce59a42ff16a +md""" +## Other functionality + +Finally, we showcase two other functionalities of HerbSearch, sampling and heuristics. + +### Sampling +Sampling is implemented for the different stochastic search methods. + +We consider here a simple grammar, which gives different programs for different search depths. +""" + +# ╔═╡ f8415d48-a51d-4845-8425-fd61ed79c06e +grammar = @cfgrammar begin + A = B | C | F + F = G + C = D + D = E +end + +# A->B (depth 1) or A->F->G (depth 2) or A->C->D->E (depth 3) + +# For depth ≤ 1 the only option is A->B +expression = rand(RuleNode, grammar, :A, 1) +@assert rulenode2expr(expression, grammar) in [:B,:C,:F] + +# For depth ≤ 2 the two options are A->B (depth 1) and A->B->G| A->C->G | A->F->G (depth 2) +expression = rand(RuleNode, grammar, :A, 2) +@assert rulenode2expr(expression,grammar) in [:B,:C,:F,:G] + +# ╔═╡ 7f88bf4f-d82c-4e5a-a9eb-93870954c79e +md""" +### Heuristics +""" + +# ╔═╡ 53598f8f-9973-4cad-af0c-280f5531bb21 +md""" +# More interesting domains & Use of constraints +In the following examples, we introduce some larger grammars and show that Herb can still efficiently find the correct program. +""" + +# ╔═╡ bc971069-08c4-493c-a917-8092493d3233 +#Expects to return a program equivalent to 1 + (1 - x) = 2 - x + +g₁ = @csgrammar begin + Element = |(1 : 3) # 1 - 3 + Element = Element + Element # 4 + Element = 1 - Element # 5 + Element = x # 6 +end + +addconstraint!(g₁, ComesAfter(6, [5])) + +examples = [ + IOExample(Dict(:x => 0), 2), + IOExample(Dict(:x => 1), 1), + IOExample(Dict(:x => 2), 0) +] +problem = Problem(examples) +solution = search(g₁, problem, :Element, max_depth=3) + +@assert test_with_input(SymbolTable(g₁), solution, Dict(:x => -2)) == 4 + +# ╔═╡ 61a60c9c-36cf-4e86-b697-748b3524d3b4 +# Expects to return a program equivalent to 4 + x * (x + 3 + 3) = x^2 + 6x + 4 + +g₂ = @csgrammar begin + Element = Element + Element + Element # 1 + Element = Element + Element * Element # 2 + Element = x # 3 + Element = |(3 : 5) # 4 +end + +# Restrict .. + x * x +addconstraint!(g₂, Forbidden(MatchNode(2, [MatchVar(:x), MatchNode(3), MatchNode(3)]))) +# Restrict 4 and 5 in lower level +addconstraint!(g₂, ForbiddenPath([2, 1, 5])) +addconstraint!(g₂, ForbiddenPath([2, 1, 6])) + +examples = [ + IOExample(Dict(:x => 1), 11) + IOExample(Dict(:x => 2), 20) + IOExample(Dict(:x => -1), -1) +] +problem = Problem(examples) +solution = search(g₂, problem, :Element) + +@assert test_with_input(SymbolTable(g₂), solution, Dict(:x => 0)) == 4 + +# ╔═╡ a5b17c81-b667-4b1c-ab15-ddf1a162683b +# Expects to return a program equivalent to (1 - (((1 - x) - 1) - 1)) - 1 = x + 1 + +g₃ = @csgrammar begin + Element = |(1 : 20) # 1 - 20 + Element = Element - 1 # 21 + Element = 1 - Element # 22 + Element = x # 23 +end + +addconstraint!(g₃, ComesAfter(23, [22, 21])) +addconstraint!(g₃, ComesAfter(22, [21])) + +examples = [ + IOExample(Dict(:x => 1), 2) + IOExample(Dict(:x => 10), 11) +] +problem = Problem(examples) +solution = search(g₃, problem, :Element) + +@assert test_with_input(SymbolTable(g₃), solution, Dict(:x => 0)) == 1 +@assert test_with_input(SymbolTable(g₃), solution, Dict(:x => 100)) == 101 + +# ╔═╡ e41feac7-6de6-4223-8a21-341b85da52c0 +# Expects to return a program equivalent to 18 + 4x + +g₄ = @csgrammar begin + Element = |(0 : 20) # 1 - 20 + Element = Element + Element + Element # 21 + Element = Element + Element * Element # 22 + Element = x # 23 +end + +# Enforce ordering on + + +addconstraint!(g₄, Ordered( + MatchNode(21, [MatchVar(:x), MatchVar(:y), MatchVar(:z)]), + [:x, :y, :z] +)) + +examples = [ + IOExample(Dict(:x => 1), 22), + IOExample(Dict(:x => 0), 18), + IOExample(Dict(:x => -1), 14) +] +problem = Problem(examples) +solution = search(g₄, problem, :Element) + +@assert test_with_input(SymbolTable(g₄), solution, Dict(:x => 100)) == 418 + +# ╔═╡ 1c4db74a-4caa-4ce5-815f-b631365c5129 +# Expects to return a program equivalent to (x == 2) ? 1 : (x + 2) + +g₅ = @csgrammar begin + Element = Number # 1 + Element = Bool # 2 + + Number = |(1 : 3) # 3-5 + + Number = Number + Number # 6 + Bool = Number ≡ Number # 7 + Number = x # 8 + + Number = Bool ? Number : Number # 9 + Bool = Bool ? Bool : Bool # 10 +end + +# Forbid ? = ? +addconstraint!(g₅, Forbidden(MatchNode(7, [MatchVar(:x), MatchVar(:x)]))) +# Order = +addconstraint!(g₅, Ordered(MatchNode(7, [MatchVar(:x), MatchVar(:y)]), [:x, :y])) +# Order + +addconstraint!(g₅, Ordered(MatchNode(6, [MatchVar(:x), MatchVar(:y)]), [:x, :y])) + +examples = [ + IOExample(Dict(:x => 0), 2) + IOExample(Dict(:x => 1), 3) + IOExample(Dict(:x => 2), 1) +] +problem = Problem(examples) +solution = search(g₅, problem, :Element) + +@assert test_with_input(SymbolTable(g₅), solution, Dict(:x => 3)) == 5 + +# ╔═╡ 00000000-0000-0000-0000-000000000001 +PLUTO_PROJECT_TOML_CONTENTS = """ +[deps] +HerbConstraints = "1fa96474-3206-4513-b4fa-23913f296dfc" +HerbGrammar = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" +HerbInterpret = "5bbddadd-02c5-4713-84b8-97364418cca7" +HerbSearch = "3008d8e8-f9aa-438a-92ed-26e9c7b4829f" +HerbSpecification = "6d54aada-062f-46d8-85cf-a1ceaf058a06" +PlutoUI = "7f904dfe-b85e-4ff6-b463-dae2292396a8" + +[compat] +HerbConstraints = "~0.2.2" +HerbGrammar = "~0.3.0" +HerbInterpret = "~0.1.3" +HerbSearch = "~0.3.0" +HerbSpecification = "~0.1.0" +PlutoUI = "~0.7.59" +""" + +# ╔═╡ 00000000-0000-0000-0000-000000000002 +PLUTO_MANIFEST_TOML_CONTENTS = """ +# This file is machine-generated - editing it directly is not advised + +julia_version = "1.10.4" +manifest_format = "2.0" +project_hash = "5ee7afaa57cf163e03a42d572d1cb2cb022598e5" + +[[deps.AbstractPlutoDingetjes]] +deps = ["Pkg"] +git-tree-sha1 = "6e1d2a35f2f90a4bc7c2ed98079b2ba09c35b83a" +uuid = "6e696c72-6542-2067-7265-42206c756150" +version = "1.3.2" + +[[deps.AbstractTrees]] +git-tree-sha1 = "2d9c9a55f9c93e8887ad391fbae72f8ef55e1177" +uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" +version = "0.4.5" + +[[deps.ArgTools]] +uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" +version = "1.1.1" + +[[deps.ArnoldiMethod]] +deps = ["LinearAlgebra", "Random", "StaticArrays"] +git-tree-sha1 = "d57bd3762d308bded22c3b82d033bff85f6195c6" +uuid = "ec485272-7323-5ecc-a04f-4719b315124d" +version = "0.4.0" + +[[deps.Artifacts]] +uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" + +[[deps.Base64]] +uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" + +[[deps.Bzip2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "9e2a6b69137e6969bab0152632dcb3bc108c8bdd" +uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0" +version = "1.0.8+1" + +[[deps.Cairo_jll]] +deps = ["Artifacts", "Bzip2_jll", "CompilerSupportLibraries_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "a2f1c8c668c8e3cb4cca4e57a8efdb09067bb3fd" +uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a" +version = "1.18.0+2" + +[[deps.ColorTypes]] +deps = ["FixedPointNumbers", "Random"] +git-tree-sha1 = "b10d0b65641d57b8b4d5e234446582de5047050d" +uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f" +version = "0.11.5" + +[[deps.CommonSubexpressions]] +deps = ["MacroTools", "Test"] +git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7" +uuid = "bbf7d656-a473-5ed7-a52c-81e309532950" +version = "0.3.0" + +[[deps.Compat]] +deps = ["TOML", "UUIDs"] +git-tree-sha1 = "b1c55339b7c6c350ee89f2c1604299660525b248" +uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" +version = "4.15.0" +weakdeps = ["Dates", "LinearAlgebra"] + + [deps.Compat.extensions] + CompatLinearAlgebraExt = "LinearAlgebra" + +[[deps.CompilerSupportLibraries_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" +version = "1.1.1+0" + +[[deps.DataAPI]] +git-tree-sha1 = "abe83f3a2f1b857aac70ef8b269080af17764bbe" +uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" +version = "1.16.0" + +[[deps.DataStructures]] +deps = ["Compat", "InteractiveUtils", "OrderedCollections"] +git-tree-sha1 = "1d0a14036acb104d9e89698bd408f63ab58cdc82" +uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" +version = "0.18.20" + +[[deps.Dates]] +deps = ["Printf"] +uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" + +[[deps.Distributed]] +deps = ["Random", "Serialization", "Sockets"] +uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b" + +[[deps.DocStringExtensions]] +deps = ["LibGit2"] +git-tree-sha1 = "2fb1e02f2b635d0845df5d7c167fec4dd739b00d" +uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae" +version = "0.9.3" + +[[deps.Downloads]] +deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"] +uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" +version = "1.6.0" + +[[deps.Expat_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "1c6317308b9dc757616f0b5cb379db10494443a7" +uuid = "2e619515-83b5-522b-bb60-26c02a35a201" +version = "2.6.2+0" + +[[deps.FileWatching]] +uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee" + +[[deps.FixedPointNumbers]] +deps = ["Statistics"] +git-tree-sha1 = "05882d6995ae5c12bb5f36dd2ed3f61c98cbb172" +uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93" +version = "0.8.5" + +[[deps.Fontconfig_jll]] +deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Zlib_jll"] +git-tree-sha1 = "db16beca600632c95fc8aca29890d83788dd8b23" +uuid = "a3f928ae-7b40-5064-980b-68af3947d34b" +version = "2.13.96+0" + +[[deps.FreeType2_jll]] +deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "5c1d8ae0efc6c2e7b1fc502cbe25def8f661b7bc" +uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7" +version = "2.13.2+0" + +[[deps.Gettext_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "XML2_jll"] +git-tree-sha1 = "9b02998aba7bf074d14de89f9d37ca24a1a0b046" +uuid = "78b55507-aeef-58d4-861c-77aaff3498b1" +version = "0.21.0+0" + +[[deps.Glib_jll]] +deps = ["Artifacts", "Gettext_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE2_jll", "Zlib_jll"] +git-tree-sha1 = "7c82e6a6cd34e9d935e9aa4051b66c6ff3af59ba" +uuid = "7746bdde-850d-59dc-9ae8-88ece973131d" +version = "2.80.2+0" + +[[deps.Graphite2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "344bf40dcab1073aca04aa0df4fb092f920e4011" +uuid = "3b182d85-2403-5c21-9c21-1e1f0cc25472" +version = "1.3.14+0" + +[[deps.Graphs]] +deps = ["ArnoldiMethod", "Compat", "DataStructures", "Distributed", "Inflate", "LinearAlgebra", "Random", "SharedArrays", "SimpleTraits", "SparseArrays", "Statistics"] +git-tree-sha1 = "ebd18c326fa6cee1efb7da9a3b45cf69da2ed4d9" +uuid = "86223c79-3864-5bf0-83f7-82e725a168b6" +version = "1.11.2" + +[[deps.HarfBuzz_ICU_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "6ccbc4fdf65c8197738c2d68cc55b74b19c97ac2" +uuid = "655565e8-fb53-5cb3-b0cd-aec1ca0647ea" +version = "2.8.1+0" + +[[deps.HarfBuzz_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "129acf094d168394e80ee1dc4bc06ec835e510a3" +uuid = "2e76f6c2-a576-52d4-95c1-20adfe4de566" +version = "2.8.1+1" + +[[deps.HerbConstraints]] +deps = ["DataStructures", "HerbCore", "HerbGrammar", "MLStyle"] +git-tree-sha1 = "2e54da1d19119847b242d1ceda212b180cca36a9" +uuid = "1fa96474-3206-4513-b4fa-23913f296dfc" +version = "0.2.2" + +[[deps.HerbCore]] +git-tree-sha1 = "923877c2715b8166d7ba9f9be2136d70eed87725" +uuid = "2b23ba43-8213-43cb-b5ea-38c12b45bd45" +version = "0.3.0" + +[[deps.HerbGrammar]] +deps = ["AbstractTrees", "DataStructures", "HerbCore", "Serialization", "TreeView"] +git-tree-sha1 = "b4cbf9712dbb3ab281ff4ed517d3cd6bc12f3078" +uuid = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" +version = "0.3.0" + +[[deps.HerbInterpret]] +deps = ["HerbCore", "HerbGrammar", "HerbSpecification"] +git-tree-sha1 = "9e19b4ee5f29eb8bb9b1049524728b38e878eca2" +uuid = "5bbddadd-02c5-4713-84b8-97364418cca7" +version = "0.1.3" + +[[deps.HerbSearch]] +deps = ["DataStructures", "HerbConstraints", "HerbCore", "HerbGrammar", "HerbInterpret", "HerbSpecification", "Logging", "MLStyle", "Random", "StatsBase"] +git-tree-sha1 = "472e3f427c148f334dde3837b0bb1549897ed00a" +uuid = "3008d8e8-f9aa-438a-92ed-26e9c7b4829f" +version = "0.3.0" + +[[deps.HerbSpecification]] +git-tree-sha1 = "5385b81e40c3cd62aeea591319896148036863c9" +uuid = "6d54aada-062f-46d8-85cf-a1ceaf058a06" +version = "0.1.0" + +[[deps.Hyperscript]] +deps = ["Test"] +git-tree-sha1 = "179267cfa5e712760cd43dcae385d7ea90cc25a4" +uuid = "47d2ed2b-36de-50cf-bf87-49c2cf4b8b91" +version = "0.0.5" + +[[deps.HypertextLiteral]] +deps = ["Tricks"] +git-tree-sha1 = "7134810b1afce04bbc1045ca1985fbe81ce17653" +uuid = "ac1192a8-f4b3-4bfe-ba22-af5b92cd3ab2" +version = "0.9.5" + +[[deps.ICU_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "20b6765a3016e1fca0c9c93c80d50061b94218b7" +uuid = "a51ab1cf-af8e-5615-a023-bc2c838bba6b" +version = "69.1.0+0" + +[[deps.IOCapture]] +deps = ["Logging", "Random"] +git-tree-sha1 = "8b72179abc660bfab5e28472e019392b97d0985c" +uuid = "b5f81e59-6552-4d32-b1f0-c071b021bf89" +version = "0.2.4" + +[[deps.Inflate]] +git-tree-sha1 = "d1b1b796e47d94588b3757fe84fbf65a5ec4a80d" +uuid = "d25df0c9-e2be-5dd7-82c8-3ad0b3e990b9" +version = "0.1.5" + +[[deps.InteractiveUtils]] +deps = ["Markdown"] +uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" + +[[deps.IrrationalConstants]] +git-tree-sha1 = "630b497eafcc20001bba38a4651b327dcfc491d2" +uuid = "92d709cd-6900-40b7-9082-c6be49f344b6" +version = "0.2.2" + +[[deps.JLLWrappers]] +deps = ["Artifacts", "Preferences"] +git-tree-sha1 = "7e5d6779a1e09a36db2a7b6cff50942a0a7d0fca" +uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210" +version = "1.5.0" + +[[deps.JSON]] +deps = ["Dates", "Mmap", "Parsers", "Unicode"] +git-tree-sha1 = "31e996f0a15c7b280ba9f76636b3ff9e2ae58c9a" +uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6" +version = "0.21.4" + +[[deps.JpegTurbo_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "c84a835e1a09b289ffcd2271bf2a337bbdda6637" +uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8" +version = "3.0.3+0" + +[[deps.LERC_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "bf36f528eec6634efc60d7ec062008f171071434" +uuid = "88015f11-f218-50d7-93a8-a6af411a945d" +version = "3.0.0+1" + +[[deps.LLVMOpenMP_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "d986ce2d884d49126836ea94ed5bfb0f12679713" +uuid = "1d63c593-3942-5779-bab2-d838dc0a180e" +version = "15.0.7+0" + +[[deps.LZO_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "70c5da094887fd2cae843b8db33920bac4b6f07d" +uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac" +version = "2.10.2+0" + +[[deps.LaTeXStrings]] +git-tree-sha1 = "50901ebc375ed41dbf8058da26f9de442febbbec" +uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" +version = "1.3.1" + +[[deps.LibCURL]] +deps = ["LibCURL_jll", "MozillaCACerts_jll"] +uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" +version = "0.6.4" + +[[deps.LibCURL_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"] +uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" +version = "8.4.0+0" + +[[deps.LibGit2]] +deps = ["Base64", "LibGit2_jll", "NetworkOptions", "Printf", "SHA"] +uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" + +[[deps.LibGit2_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll"] +uuid = "e37daf67-58a4-590a-8e99-b0245dd2ffc5" +version = "1.6.4+0" + +[[deps.LibSSH2_jll]] +deps = ["Artifacts", "Libdl", "MbedTLS_jll"] +uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" +version = "1.11.0+1" + +[[deps.Libdl]] +uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" + +[[deps.Libffi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "0b4a5d71f3e5200a7dff793393e09dfc2d874290" +uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490" +version = "3.2.2+1" + +[[deps.Libgcrypt_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgpg_error_jll"] +git-tree-sha1 = "9fd170c4bbfd8b935fdc5f8b7aa33532c991a673" +uuid = "d4300ac3-e22c-5743-9152-c294e39db1e4" +version = "1.8.11+0" + +[[deps.Libgpg_error_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "fbb1f2bef882392312feb1ede3615ddc1e9b99ed" +uuid = "7add5ba3-2f88-524e-9cd5-f83b8a55f7b8" +version = "1.49.0+0" + +[[deps.Libiconv_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "f9557a255370125b405568f9767d6d195822a175" +uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531" +version = "1.17.0+0" + +[[deps.Libmount_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "0c4f9c4f1a50d8f35048fa0532dabbadf702f81e" +uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9" +version = "2.40.1+0" + +[[deps.Libtiff_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "LERC_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"] +git-tree-sha1 = "3eb79b0ca5764d4799c06699573fd8f533259713" +uuid = "89763e89-9b03-5906-acba-b20f662cd828" +version = "4.4.0+0" + +[[deps.Libuuid_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "5ee6203157c120d79034c748a2acba45b82b8807" +uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700" +version = "2.40.1+0" + +[[deps.LinearAlgebra]] +deps = ["Libdl", "OpenBLAS_jll", "libblastrampoline_jll"] +uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + +[[deps.LittleCMS_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pkg"] +git-tree-sha1 = "110897e7db2d6836be22c18bffd9422218ee6284" +uuid = "d3a379c0-f9a3-5b72-a4c0-6bf4d2e8af0f" +version = "2.12.0+0" + +[[deps.LogExpFunctions]] +deps = ["DocStringExtensions", "IrrationalConstants", "LinearAlgebra"] +git-tree-sha1 = "18144f3e9cbe9b15b070288eef858f71b291ce37" +uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688" +version = "0.3.27" + + [deps.LogExpFunctions.extensions] + LogExpFunctionsChainRulesCoreExt = "ChainRulesCore" + LogExpFunctionsChangesOfVariablesExt = "ChangesOfVariables" + LogExpFunctionsInverseFunctionsExt = "InverseFunctions" + + [deps.LogExpFunctions.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + ChangesOfVariables = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0" + InverseFunctions = "3587e190-3f89-42d0-90ee-14403ec27112" + +[[deps.Logging]] +uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" + +[[deps.MIMEs]] +git-tree-sha1 = "65f28ad4b594aebe22157d6fac869786a255b7eb" +uuid = "6c6e2e6c-3030-632d-7369-2d6c69616d65" +version = "0.1.4" + +[[deps.MLStyle]] +git-tree-sha1 = "bc38dff0548128765760c79eb7388a4b37fae2c8" +uuid = "d8e11817-5142-5d16-987a-aa16d5891078" +version = "0.4.17" + +[[deps.MacroTools]] +deps = ["Markdown", "Random"] +git-tree-sha1 = "2fa9ee3e63fd3a4f7a9a4f4744a52f4856de82df" +uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" +version = "0.5.13" + +[[deps.Markdown]] +deps = ["Base64"] +uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" + +[[deps.MbedTLS_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" +version = "2.28.2+1" + +[[deps.Missings]] +deps = ["DataAPI"] +git-tree-sha1 = "ec4f7fbeab05d7747bdf98eb74d130a2a2ed298d" +uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28" +version = "1.2.0" + +[[deps.Mmap]] +uuid = "a63ad114-7e13-5084-954f-fe012c677804" + +[[deps.MozillaCACerts_jll]] +uuid = "14a3606d-f60d-562e-9121-12d972cd8159" +version = "2023.1.10" + +[[deps.NetworkOptions]] +uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" +version = "1.2.0" + +[[deps.OpenBLAS_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] +uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" +version = "0.3.23+4" + +[[deps.OpenJpeg_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libtiff_jll", "LittleCMS_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "76374b6e7f632c130e78100b166e5a48464256f8" +uuid = "643b3616-a352-519d-856d-80112ee9badc" +version = "2.4.0+0" + +[[deps.OpenSSL_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "a12e56c72edee3ce6b96667745e6cbbe5498f200" +uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95" +version = "1.1.23+0" + +[[deps.OrderedCollections]] +git-tree-sha1 = "dfdf5519f235516220579f949664f1bf44e741c5" +uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" +version = "1.6.3" + +[[deps.PCRE2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "efcefdf7-47ab-520b-bdef-62a2eaa19f15" +version = "10.42.0+1" + +[[deps.Parsers]] +deps = ["Dates", "PrecompileTools", "UUIDs"] +git-tree-sha1 = "8489905bcdbcfac64d1daa51ca07c0d8f0283821" +uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0" +version = "2.8.1" + +[[deps.Pixman_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "LLVMOpenMP_jll", "Libdl"] +git-tree-sha1 = "35621f10a7531bc8fa58f74610b1bfb70a3cfc6b" +uuid = "30392449-352a-5448-841d-b1acce4e97dc" +version = "0.43.4+0" + +[[deps.Pkg]] +deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] +uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" +version = "1.10.0" + +[[deps.PlutoUI]] +deps = ["AbstractPlutoDingetjes", "Base64", "ColorTypes", "Dates", "FixedPointNumbers", "Hyperscript", "HypertextLiteral", "IOCapture", "InteractiveUtils", "JSON", "Logging", "MIMEs", "Markdown", "Random", "Reexport", "URIs", "UUIDs"] +git-tree-sha1 = "ab55ee1510ad2af0ff674dbcced5e94921f867a9" +uuid = "7f904dfe-b85e-4ff6-b463-dae2292396a8" +version = "0.7.59" + +[[deps.Poppler_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "Glib_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "OpenJpeg_jll", "Pkg", "libpng_jll"] +git-tree-sha1 = "02148a0cb2532f22c0589ceb75c110e168fb3d1f" +uuid = "9c32591e-4766-534b-9725-b71a8799265b" +version = "21.9.0+0" + +[[deps.PrecompileTools]] +deps = ["Preferences"] +git-tree-sha1 = "5aa36f7049a63a1528fe8f7c3f2113413ffd4e1f" +uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a" +version = "1.2.1" + +[[deps.Preferences]] +deps = ["TOML"] +git-tree-sha1 = "9306f6085165d270f7e3db02af26a400d580f5c6" +uuid = "21216c6a-2e73-6563-6e65-726566657250" +version = "1.4.3" + +[[deps.Printf]] +deps = ["Unicode"] +uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" + +[[deps.REPL]] +deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] +uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" + +[[deps.Random]] +deps = ["SHA"] +uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" + +[[deps.Reexport]] +git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b" +uuid = "189a3867-3050-52da-a836-e630ba90ab69" +version = "1.2.2" + +[[deps.Requires]] +deps = ["UUIDs"] +git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7" +uuid = "ae029012-a4dd-5104-9daa-d747884805df" +version = "1.3.0" + +[[deps.SHA]] +uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" +version = "0.7.0" + +[[deps.Serialization]] +uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" + +[[deps.SharedArrays]] +deps = ["Distributed", "Mmap", "Random", "Serialization"] +uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383" + +[[deps.SimpleTraits]] +deps = ["InteractiveUtils", "MacroTools"] +git-tree-sha1 = "5d7e3f4e11935503d3ecaf7186eac40602e7d231" +uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d" +version = "0.9.4" + +[[deps.Sockets]] +uuid = "6462fe0b-24de-5631-8697-dd941f90decc" + +[[deps.SortingAlgorithms]] +deps = ["DataStructures"] +git-tree-sha1 = "66e0a8e672a0bdfca2c3f5937efb8538b9ddc085" +uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c" +version = "1.2.1" + +[[deps.SparseArrays]] +deps = ["Libdl", "LinearAlgebra", "Random", "Serialization", "SuiteSparse_jll"] +uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" +version = "1.10.0" + +[[deps.StaticArrays]] +deps = ["LinearAlgebra", "PrecompileTools", "Random", "StaticArraysCore"] +git-tree-sha1 = "eeafab08ae20c62c44c8399ccb9354a04b80db50" +uuid = "90137ffa-7385-5640-81b9-e52037218182" +version = "1.9.7" + + [deps.StaticArrays.extensions] + StaticArraysChainRulesCoreExt = "ChainRulesCore" + StaticArraysStatisticsExt = "Statistics" + + [deps.StaticArrays.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" + +[[deps.StaticArraysCore]] +git-tree-sha1 = "192954ef1208c7019899fbf8049e717f92959682" +uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" +version = "1.4.3" + +[[deps.Statistics]] +deps = ["LinearAlgebra", "SparseArrays"] +uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" +version = "1.10.0" + +[[deps.StatsAPI]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "1ff449ad350c9c4cbc756624d6f8a8c3ef56d3ed" +uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0" +version = "1.7.0" + +[[deps.StatsBase]] +deps = ["DataAPI", "DataStructures", "LinearAlgebra", "LogExpFunctions", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"] +git-tree-sha1 = "5cf7606d6cef84b543b483848d4ae08ad9832b21" +uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91" +version = "0.34.3" + +[[deps.SuiteSparse_jll]] +deps = ["Artifacts", "Libdl", "libblastrampoline_jll"] +uuid = "bea87d4a-7f5b-5778-9afe-8cc45184846c" +version = "7.2.1+1" + +[[deps.TOML]] +deps = ["Dates"] +uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" +version = "1.0.3" + +[[deps.Tar]] +deps = ["ArgTools", "SHA"] +uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" +version = "1.10.0" + +[[deps.Test]] +deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] +uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[[deps.TikzGraphs]] +deps = ["Graphs", "LaTeXStrings", "TikzPictures"] +git-tree-sha1 = "e8f41ed9a2cabf6699d9906c195bab1f773d4ca7" +uuid = "b4f28e30-c73f-5eaf-a395-8a9db949a742" +version = "1.4.0" + +[[deps.TikzPictures]] +deps = ["LaTeXStrings", "Poppler_jll", "Requires", "tectonic_jll"] +git-tree-sha1 = "79e2d29b216ef24a0f4f905532b900dcf529aa06" +uuid = "37f6aa50-8035-52d0-81c2-5a1d08754b2d" +version = "3.5.0" + +[[deps.TreeView]] +deps = ["CommonSubexpressions", "Graphs", "MacroTools", "TikzGraphs"] +git-tree-sha1 = "41ddcefb625f2ab0f4d9f2081c2da1af2ccbbf8b" +uuid = "39424ebd-4cf3-5550-a685-96706a953f40" +version = "0.5.1" + +[[deps.Tricks]] +git-tree-sha1 = "eae1bb484cd63b36999ee58be2de6c178105112f" +uuid = "410a4b4d-49e4-4fbc-ab6d-cb71b17b3775" +version = "0.1.8" + +[[deps.URIs]] +git-tree-sha1 = "67db6cc7b3821e19ebe75791a9dd19c9b1188f2b" +uuid = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4" +version = "1.5.1" + +[[deps.UUIDs]] +deps = ["Random", "SHA"] +uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" + +[[deps.Unicode]] +uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" + +[[deps.XML2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Zlib_jll"] +git-tree-sha1 = "d9717ce3518dc68a99e6b96300813760d887a01d" +uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a" +version = "2.13.1+0" + +[[deps.XSLT_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgcrypt_jll", "Libgpg_error_jll", "Libiconv_jll", "XML2_jll", "Zlib_jll"] +git-tree-sha1 = "a54ee957f4c86b526460a720dbc882fa5edcbefc" +uuid = "aed1982a-8fda-507f-9586-7b0439959a61" +version = "1.1.41+0" + +[[deps.Xorg_libX11_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libxcb_jll", "Xorg_xtrans_jll"] +git-tree-sha1 = "afead5aba5aa507ad5a3bf01f58f82c8d1403495" +uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc" +version = "1.8.6+0" + +[[deps.Xorg_libXau_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "6035850dcc70518ca32f012e46015b9beeda49d8" +uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec" +version = "1.0.11+0" + +[[deps.Xorg_libXdmcp_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "34d526d318358a859d7de23da945578e8e8727b7" +uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05" +version = "1.1.4+0" + +[[deps.Xorg_libXext_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "d2d1a5c49fae4ba39983f63de6afcbea47194e85" +uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3" +version = "1.3.6+0" + +[[deps.Xorg_libXrender_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "47e45cd78224c53109495b3e324df0c37bb61fbe" +uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa" +version = "0.9.11+0" + +[[deps.Xorg_libpthread_stubs_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "8fdda4c692503d44d04a0603d9ac0982054635f9" +uuid = "14d82f49-176c-5ed1-bb49-ad3f5cbd8c74" +version = "0.1.1+0" + +[[deps.Xorg_libxcb_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "XSLT_jll", "Xorg_libXau_jll", "Xorg_libXdmcp_jll", "Xorg_libpthread_stubs_jll"] +git-tree-sha1 = "bcd466676fef0878338c61e655629fa7bbc69d8e" +uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b" +version = "1.17.0+0" + +[[deps.Xorg_xtrans_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e92a1a012a10506618f10b7047e478403a046c77" +uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10" +version = "1.5.0+0" + +[[deps.Zlib_jll]] +deps = ["Libdl"] +uuid = "83775a58-1f1d-513f-b197-d71354ab007a" +version = "1.2.13+1" + +[[deps.Zstd_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e678132f07ddb5bfa46857f0d7620fb9be675d3b" +uuid = "3161d3a3-bdf6-5164-811a-617609db77b4" +version = "1.5.6+0" + +[[deps.libblastrampoline_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" +version = "5.8.0+1" + +[[deps.libpng_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "d7015d2e18a5fd9a4f47de711837e980519781a4" +uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f" +version = "1.6.43+1" + +[[deps.nghttp2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" +version = "1.52.0+1" + +[[deps.p7zip_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" +version = "17.4.0+2" + +[[deps.tectonic_jll]] +deps = ["Artifacts", "Fontconfig_jll", "FreeType2_jll", "Graphite2_jll", "HarfBuzz_ICU_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "OpenSSL_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "54867b00af20c70b52a1f9c00043864d8b926a21" +uuid = "d7dd28d6-a5e6-559c-9131-7eb760cdacc5" +version = "0.13.1+0" +""" + +# ╔═╡ Cell order: +# ╟─dddca175-3d88-45ce-90da-575c0ba38175 +# ╠═a93d954d-0f09-4b6d-a3a5-62bfe39681e2 +# ╠═6ab37bbc-73e2-4d9a-a8b2-e715a0b61c8f +# ╠═c4441fa4-09ec-4b81-9681-b13b93a9c9c0 +# ╟─67931820-0f43-41e1-898e-5b5bd55e30d1 +# ╠═e41c61a4-0b2c-46da-8f7b-fe6dc529c544 +# ╟─9ad0a92a-10d5-458a-8f05-9011c8553609 +# ╠═65317911-bc92-4b84-9744-ed784adcab4a +# ╟─e9c6bc00-21f5-4a99-8bec-63cf2156c233 +# ╠═338f19f1-3a62-4462-b8dc-24424d7644f2 +# ╠═542cd47e-74cd-4b6f-acc9-bf524222e583 +# ╠═a6fb2e91-b73a-4032-930f-d884abd539e2 +# ╠═4b49f206-970e-47d2-af65-336ba65b1019 +# ╟─35405357-179b-4e77-9bdc-edf5a550b36d +# ╠═3954dd49-07a2-4ec2-91b4-9c9596d5c264 +# ╠═9892e91b-9115-4520-9637-f8d7c8905825 +# ╠═a4e58bbc-7c14-4fce-b35d-688b56e0eb61 +# ╠═9fb40ceb-8d41-491b-8941-20a8b240eb82 +# ╠═94e0d676-a9c7-4291-8696-15301e541c30 +# ╠═a4a7daed-f89b-44ad-8787-9199c05bf046 +# ╠═4821fd3a-ff2d-4991-99ad-76608d11b1da +# ╠═8d91b2e3-30b5-4ea2-bd3f-3055bb6d1d5a +# ╠═52332fa2-7ea7-4226-9460-e0bbc905c619 +# ╟─c26fea48-f812-4f24-bb8c-680e14a55df7 +# ╠═57f5face-d9d5-441a-8e0e-6ef6319fc178 +# ╟─9b4b21e0-dc6a-43ae-a511-79988ee99001 +# ╠═3af650d9-19c6-4351-920d-d2361091f628 +# ╟─d3ff497e-d2c2-4df6-8e4c-cdca70fd0677 +# ╠═da7f326c-f0d5-4837-ac9a-5bcad604566e +# ╟─0020b79a-6352-4e2d-93f6-2a1d7b03ae2c +# ╠═789150a8-862c-48c3-88b8-710b81ab34cf +# ╟─243165be-a9d2-484d-8046-811a2b0ba139 +# ╠═3d01c6f1-80a6-4904-97e2-775170e97bbf +# ╟─168c71bf-ce5b-4ab3-b29a-5996981c42a5 +# ╠═a4b522cf-78f0-4d44-88c8-82dd0cdbf952 +# ╠═f313edb9-8fd9-4d78-88cd-89226f5c769d +# ╟─0da9053a-959b-471e-8918-662ec63da71c +# ╠═0a30fb40-cd45-4661-a501-ae8e45f1e07e +# ╟─700270ea-90bd-474b-91d9-0e5ed329776a +# ╠═8731f312-bfcf-4f6c-86fa-60014dc146d6 +# ╠═46ca65d1-d876-4abc-a562-8d266bad195f +# ╟─599194a8-3f47-4917-9143-a5fe0d43029f +# ╠═cb5935ed-d89b-4e25-9243-d201daf18e78 +# ╠═d0c3742e-23e5-4ca1-9e83-b6d1e8a7cded +# ╟─5df0ba53-b528-4baf-9980-cafe5d73f9dd +# ╠═a434645b-d592-4162-a8b4-b4b04cea30a9 +# ╟─38cd9032-27c0-4179-a536-ce59a42ff16a +# ╠═f8415d48-a51d-4845-8425-fd61ed79c06e +# ╟─7f88bf4f-d82c-4e5a-a9eb-93870954c79e +# ╟─53598f8f-9973-4cad-af0c-280f5531bb21 +# ╠═bc971069-08c4-493c-a917-8092493d3233 +# ╠═61a60c9c-36cf-4e86-b697-748b3524d3b4 +# ╠═a5b17c81-b667-4b1c-ab15-ddf1a162683b +# ╠═e41feac7-6de6-4223-8a21-341b85da52c0 +# ╠═1c4db74a-4caa-4ce5-815f-b631365c5129 +# ╟─00000000-0000-0000-0000-000000000001 +# ╟─00000000-0000-0000-0000-000000000002 diff --git a/docs/src/tutorials/getting_started_with_herb.jl b/docs/src/tutorials/getting_started_with_herb.jl index 83adbd8..854c27f 100644 --- a/docs/src/tutorials/getting_started_with_herb.jl +++ b/docs/src/tutorials/getting_started_with_herb.jl @@ -25,11 +25,11 @@ First, we start with the setup. We need to access to all the function in the Her md""" ### Defining the program space -Next, we start by creating a grammar. We define a context-free grammar (cfg) as a [`HerbGrammar.ContextSpecificGrammar`](@ref) without any constraints. A cfg is just a simple set of production rules for defining combinations of terminal symbols (in our case real numbers). +Next, we start by creating a grammar. We define a context-free grammar as a [`HerbGrammar.ContextSpecificGrammar`](@ref) without any constraints. A context-free grammar is just a simple set of production rules for defining combinations of terminal symbols (in our case real numbers). Contrary, we could define a context-sensitive grammar, when the production rules only hold in a certain context. However, for more information on this, please see our tutorial on [defining grammars](defining_grammars.md). -For now, we specify a simple grammar for dealing with integers and explain all the rules individually: +For now, we specify a simple grammar (using the `@csgrammar` macro) for dealing with integers and explain all the rules individually: 1. First, we specify our interval `[0:9]` on real numbers and also constrain them to be integer. 2. Then, we can also use the variable `x` to hold an integer. @@ -41,7 +41,7 @@ If you run this cell, you can see all the rules rolled out. """ # ╔═╡ 763b378b-66f9-481e-a3da-ca37825eb255 -g = HerbGrammar.@cfgrammar begin +g = HerbGrammar.@csgrammar begin Real = |(0:9) Real = x Real = Real + Real diff --git a/docs/src/tutorials/working_with_interpreters.jl b/docs/src/tutorials/working_with_interpreters.jl index 66767a4..88f46ee 100644 --- a/docs/src/tutorials/working_with_interpreters.jl +++ b/docs/src/tutorials/working_with_interpreters.jl @@ -6,8 +6,8 @@ using InteractiveUtils # ╔═╡ e0a7076c-9345-40ef-a26e-99e8bad31463 begin - using HerbGrammar - using HerbInterpret + using HerbGrammar + using HerbInterpret end # ╔═╡ 55719688-3940-11ef-1f29-f51dea064ff3 @@ -19,12 +19,12 @@ For example, assume the following grammar. " # ╔═╡ 39eaa982-ba88-49b9-ad52-076a169d0439 -g = @cfgrammar begin - Number = |(1:2) - Number = x - Number = Number + Number - Number = Number * Number - end +g = @csgrammar begin + Number = |(1:2) + Number = x + Number = Number + Number + Number = Number * Number +end # ╔═╡ 2478d5a4-a11e-42aa-87dd-d97a3fa5d378 md" @@ -32,7 +32,7 @@ Let's construct a program `x+3`, which would correspond to the following `RuleNo " # ╔═╡ 1e15898e-568c-4211-ba00-27de61806aeb -myprog = RuleNode(4,[RuleNode(3),RuleNode(1)]) +myprog = RuleNode(4, [RuleNode(3), RuleNode(1)]) # ╔═╡ d43a2094-b215-4d6c-b6d8-8d32fe8898d6 md" @@ -87,7 +87,7 @@ grammar_robots = @csgrammar begin end # ╔═╡ aff77be9-365f-4672-bbd4-07f23528e32e - md" +md" This grammar specifies a simple sequential program with instructions for the robot. A couple of example programs: - `moveRight(); moveLeft(); drop()` - WHILE(notAtTop(), moveUp()) @@ -161,7 +161,7 @@ HerbInterpret = "~0.1.3" PLUTO_MANIFEST_TOML_CONTENTS = """ # This file is machine-generated - editing it directly is not advised -julia_version = "1.9.1" +julia_version = "1.10.4" manifest_format = "2.0" project_hash = "3d463631c2622b04eb5c06a35c60329ce68a7044" @@ -217,7 +217,7 @@ weakdeps = ["Dates", "LinearAlgebra"] [[deps.CompilerSupportLibraries_jll]] deps = ["Artifacts", "Libdl"] uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" -version = "1.0.2+0" +version = "1.1.1+0" [[deps.DataStructures]] deps = ["Compat", "InteractiveUtils", "OrderedCollections"] @@ -370,21 +370,26 @@ version = "1.3.1" [[deps.LibCURL]] deps = ["LibCURL_jll", "MozillaCACerts_jll"] uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" -version = "0.6.3" +version = "0.6.4" [[deps.LibCURL_jll]] deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"] uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" -version = "7.84.0+0" +version = "8.4.0+0" [[deps.LibGit2]] -deps = ["Base64", "NetworkOptions", "Printf", "SHA"] +deps = ["Base64", "LibGit2_jll", "NetworkOptions", "Printf", "SHA"] uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" +[[deps.LibGit2_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll"] +uuid = "e37daf67-58a4-590a-8e99-b0245dd2ffc5" +version = "1.6.4+0" + [[deps.LibSSH2_jll]] deps = ["Artifacts", "Libdl", "MbedTLS_jll"] uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" -version = "1.10.2+0" +version = "1.11.0+1" [[deps.Libdl]] uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" @@ -457,14 +462,14 @@ uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" [[deps.MbedTLS_jll]] deps = ["Artifacts", "Libdl"] uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" -version = "2.28.2+0" +version = "2.28.2+1" [[deps.Mmap]] uuid = "a63ad114-7e13-5084-954f-fe012c677804" [[deps.MozillaCACerts_jll]] uuid = "14a3606d-f60d-562e-9121-12d972cd8159" -version = "2022.10.11" +version = "2023.1.10" [[deps.NetworkOptions]] uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" @@ -473,7 +478,7 @@ version = "1.2.0" [[deps.OpenBLAS_jll]] deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" -version = "0.3.21+4" +version = "0.3.23+4" [[deps.OpenJpeg_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Libtiff_jll", "LittleCMS_jll", "Pkg", "libpng_jll"] @@ -495,7 +500,7 @@ version = "1.6.3" [[deps.PCRE2_jll]] deps = ["Artifacts", "Libdl"] uuid = "efcefdf7-47ab-520b-bdef-62a2eaa19f15" -version = "10.42.0+0" +version = "10.42.0+1" [[deps.Pixman_jll]] deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "LLVMOpenMP_jll", "Libdl"] @@ -506,7 +511,7 @@ version = "0.43.4+0" [[deps.Pkg]] deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" -version = "1.9.0" +version = "1.10.0" [[deps.Poppler_jll]] deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "Glib_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "OpenJpeg_jll", "Pkg", "libpng_jll"] @@ -535,7 +540,7 @@ deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" [[deps.Random]] -deps = ["SHA", "Serialization"] +deps = ["SHA"] uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" [[deps.Requires]] @@ -567,6 +572,7 @@ uuid = "6462fe0b-24de-5631-8697-dd941f90decc" [[deps.SparseArrays]] deps = ["Libdl", "LinearAlgebra", "Random", "Serialization", "SuiteSparse_jll"] uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" +version = "1.10.0" [[deps.StaticArrays]] deps = ["LinearAlgebra", "PrecompileTools", "Random", "StaticArraysCore"] @@ -590,12 +596,12 @@ version = "1.4.3" [[deps.Statistics]] deps = ["LinearAlgebra", "SparseArrays"] uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" -version = "1.9.0" +version = "1.10.0" [[deps.SuiteSparse_jll]] -deps = ["Artifacts", "Libdl", "Pkg", "libblastrampoline_jll"] +deps = ["Artifacts", "Libdl", "libblastrampoline_jll"] uuid = "bea87d4a-7f5b-5778-9afe-8cc45184846c" -version = "5.10.1+6" +version = "7.2.1+1" [[deps.TOML]] deps = ["Dates"] @@ -699,7 +705,7 @@ version = "1.5.0+0" [[deps.Zlib_jll]] deps = ["Libdl"] uuid = "83775a58-1f1d-513f-b197-d71354ab007a" -version = "1.2.13+0" +version = "1.2.13+1" [[deps.Zstd_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl"] @@ -710,7 +716,7 @@ version = "1.5.6+0" [[deps.libblastrampoline_jll]] deps = ["Artifacts", "Libdl"] uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" -version = "5.8.0+0" +version = "5.8.0+1" [[deps.libpng_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Zlib_jll"] @@ -721,12 +727,12 @@ version = "1.6.43+1" [[deps.nghttp2_jll]] deps = ["Artifacts", "Libdl"] uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" -version = "1.48.0+0" +version = "1.52.0+1" [[deps.p7zip_jll]] deps = ["Artifacts", "Libdl"] uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" -version = "17.4.0+0" +version = "17.4.0+2" [[deps.tectonic_jll]] deps = ["Artifacts", "Fontconfig_jll", "FreeType2_jll", "Graphite2_jll", "HarfBuzz_ICU_jll", "HarfBuzz_jll", "ICU_jll", "JLLWrappers", "Libdl", "OpenSSL_jll", "Zlib_jll", "libpng_jll"] From 6b3f8e02aded6e6944af451b90805bed5823e0e2 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Fri, 20 Sep 2024 14:57:11 +0200 Subject: [PATCH 08/13] Improve tutorials. --- docs/Project.toml | 5 ++ .../src/tutorials/abstract_syntax_trees.ipynb | 63 +++++++++++-------- .../tutorials/getting_started_with_herb.jl | 16 ++--- 3 files changed, 50 insertions(+), 34 deletions(-) diff --git a/docs/Project.toml b/docs/Project.toml index 06eab32..f91b5e6 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -4,10 +4,15 @@ DocumenterTools = "35a29f4d-8980-5a13-9543-d66fff28ecb8" Herb = "c09c6b7f-4f63-49de-90d9-97a3563c0f4a" HerbConstraints = "1fa96474-3206-4513-b4fa-23913f296dfc" HerbCore = "2b23ba43-8213-43cb-b5ea-38c12b45bd45" +HerbData = "495a3ad3-8034-41b3-a087-aacf2fd71098" HerbGrammar = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" HerbInterpret = "5bbddadd-02c5-4713-84b8-97364418cca7" HerbSearch = "3008d8e8-f9aa-438a-92ed-26e9c7b4829f" HerbSpecification = "6d54aada-062f-46d8-85cf-a1ceaf058a06" +Pluto = "c3e4b0f8-55cb-11ea-2926-15256bba5781" +PlutoSliderServer = "2fc8631c-6f24-4c5b-bca7-cbb509c42db4" +PlutoStaticHTML = "359b1769-a58e-495b-9770-312e911026ad" +PlutoUI = "7f904dfe-b85e-4ff6-b463-dae2292396a8" PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0" [compat] diff --git a/docs/src/tutorials/abstract_syntax_trees.ipynb b/docs/src/tutorials/abstract_syntax_trees.ipynb index 7220eaf..53be308 100644 --- a/docs/src/tutorials/abstract_syntax_trees.ipynb +++ b/docs/src/tutorials/abstract_syntax_trees.ipynb @@ -54,7 +54,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 1, "id": "69d543a7-3067-4376-b1bb-1a4f9217b786", "metadata": {}, "outputs": [ @@ -76,8 +76,9 @@ "13: Number = Number * Number\n" ] }, + "execution_count": 1, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -134,7 +135,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 2, "id": "9c61d359-1f38-410e-b654-5d36bc9a1d4e", "metadata": {}, "outputs": [ @@ -144,8 +145,9 @@ "13{6,12{11,4}}" ] }, + "execution_count": 2, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -162,7 +164,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 3, "id": "7c933e57-03f8-41ac-a464-70f95b9505c0", "metadata": {}, "outputs": [ @@ -181,7 +183,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 4, "id": "32fbb714-25aa-40d5-9c0b-c20ba8999d0c", "metadata": {}, "outputs": [ @@ -191,8 +193,9 @@ "65" ] }, + "execution_count": 4, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -213,7 +216,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 5, "id": "1f41bec4-d081-48bd-9fee-c30021d4a882", "metadata": {}, "outputs": [ @@ -223,8 +226,9 @@ "fizzbuzz (generic function with 1 method)" ] }, + "execution_count": 5, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -257,7 +261,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 6, "id": "7b0ea30c-e415-4d7e-ba2f-a1590cd4c8fe", "metadata": {}, "outputs": [ @@ -283,8 +287,9 @@ "13: Bool = Bool && Bool\n" ] }, + "execution_count": 6, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -384,7 +389,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 7, "id": "9c897ebc-4a79-45be-b349-c376bffa03df", "metadata": {}, "outputs": [ @@ -394,8 +399,9 @@ "12{13{11{10{1,4}2}11{10{1,3}2}}9{7}12{11{10{1,3}2}9{5}12{11{10{1,4}2}9{6}9{8{1}}}}}" ] }, + "execution_count": 7, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -462,7 +468,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 8, "id": "5a4ba50a-ccda-4e20-9301-051ee839fdf5", "metadata": {}, "outputs": [ @@ -493,7 +499,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 9, "id": "9e0b4ec1-eab1-49b9-be2d-c923098d1850", "metadata": {}, "outputs": [ @@ -507,8 +513,9 @@ " \"22\"" ] }, + "execution_count": 9, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -537,7 +544,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 10, "id": "65bfb9cc-bbdc-46e5-8ebf-9c15c6424339", "metadata": {}, "outputs": [ @@ -547,8 +554,9 @@ "6," ] }, + "execution_count": 10, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -568,7 +576,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 11, "id": "742f7ee8-c088-4331-b17c-dc5a2078ca6b", "metadata": {}, "outputs": [ @@ -599,7 +607,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 12, "id": "59d13ab2-7415-4463-89bf-e09ff55a74d8", "metadata": {}, "outputs": [ @@ -613,8 +621,9 @@ " \"22\"" ] }, + "execution_count": 12, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -633,7 +642,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 13, "id": "74d04a45-1697-4bcd-a5fd-adaa4ef455ff", "metadata": {}, "outputs": [ @@ -643,8 +652,9 @@ "6," ] }, + "execution_count": 13, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ @@ -666,7 +676,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 14, "id": "6f965fe6-f5b1-4059-911c-362f4c1731d6", "metadata": {}, "outputs": [ @@ -697,7 +707,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 15, "id": "41395fec-9053-423b-b0fd-7089bb89c2d0", "metadata": {}, "outputs": [ @@ -711,8 +721,9 @@ " \"22\"" ] }, + "execution_count": 15, "metadata": {}, - "output_type": "display_data" + "output_type": "execute_result" } ], "source": [ diff --git a/docs/src/tutorials/getting_started_with_herb.jl b/docs/src/tutorials/getting_started_with_herb.jl index 854c27f..8e841ae 100644 --- a/docs/src/tutorials/getting_started_with_herb.jl +++ b/docs/src/tutorials/getting_started_with_herb.jl @@ -69,7 +69,7 @@ In the cell below we automatically generate some examples for `x` assigning valu # ╔═╡ 8bf48b7a-0ff5-4015-81d3-ed2eeeceff1c # Create input-output examples -examples = [HerbSpecification.IOExample(Dict(:x => x), 3x+5) for x ∈ 1:5] +examples = [HerbSpecification.IOExample(Dict(:x => x), 3x + 5) for x ∈ 1:5] # ╔═╡ 2baa7f33-c86d-40e2-9253-720ec19e4c43 md""" @@ -126,10 +126,10 @@ Search is done by passing the grammar, the problem and the starting point like b # ╔═╡ cdab3f55-37e4-4aee-bae1-14d3475cbdcd begin - problem_2 = HerbSpecification.Problem("example2", [HerbSpecification.IOExample(Dict(:x => x), 168) for x ∈ 1:5]) - iterator_2 = HerbSearch.BFSIterator(g, :Real, max_depth=4, max_size=30) - expr_2 = HerbSearch.synth(problem_2, iterator_2) - print(expr_2) + problem_2 = HerbSpecification.Problem("example2", [HerbSpecification.IOExample(Dict(:x => x), 168) for x ∈ 1:5]) + iterator_2 = HerbSearch.BFSIterator(g, :Real, max_depth=4, max_size=30) + expr_2 = HerbSearch.synth(problem_2, iterator_2) + print(expr_2) end # ╔═╡ 5ad86beb-eb25-4bae-b0c2-a33d1a38581a @@ -141,9 +141,9 @@ In any case, this concludes our first introduction to the `Herb.jl` program synt # ╔═╡ c06d09a5-138a-4821-8a60-074fa7ec026d begin - problem_3 = HerbSpecification.Problem("example3", [HerbSpecification.IOExample(Dict(:x => x), 167) for x ∈ 1:5]) - expr_3 = HerbSearch.synth(problem_3, iterator_2) - print(expr_3) + problem_3 = HerbSpecification.Problem("example3", [HerbSpecification.IOExample(Dict(:x => x), 167) for x ∈ 1:5]) + expr_3 = HerbSearch.synth(problem_3, iterator_2) + print(expr_3) end # ╔═╡ 00000000-0000-0000-0000-000000000001 From f8b331dc3883a5ab77311aa6017fda9f6ff06c76 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Thu, 26 Sep 2024 18:50:58 +0200 Subject: [PATCH 09/13] Fix rightmost-first heuristic for BFSIterator. Add more explanations. Remove sampling, etc. part at the end. --- docs/src/tutorials/advanced_search.jl | 587 +++++++++++++------------- 1 file changed, 304 insertions(+), 283 deletions(-) diff --git a/docs/src/tutorials/advanced_search.jl b/docs/src/tutorials/advanced_search.jl index ae9ad66..e9fe327 100644 --- a/docs/src/tutorials/advanced_search.jl +++ b/docs/src/tutorials/advanced_search.jl @@ -11,24 +11,31 @@ using PlutoUI using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret, HerbConstraints +# ╔═╡ aa45e602-6219-44ea-82a0-b97639f6a450 +using HerbCore + # ╔═╡ dddca175-3d88-45ce-90da-575c0ba38175 md""" # Advanced Search Procedures in Herb.jl -[A more verbose getting started with Herb.jl]() described the concept of a program space and showed how to search it with Herb.jl, using a simple search with a BFS iterator. -This tutorial takes a closer look at advanced search procedures. +[A more verbose getting started with Herb.jl]() described the concept of a program space and showed how to search it with Herb.jl, using a simple breadth-first-search (BFS) iterator for the search. +This tutorial takes a closer look at advanced search procedures hat can be employed to find a solution program to a program synthesis problem. More specifically, you will learn about - **Parameters** that can be specified and their effect on the search procedure. -- **Search methods** that can be employed to find a solution program to a program synthesis problem, including basic search (BFS and DFS), stochastic search and genetic search methods. -- **Other functionality** of the module `HerbSearch.jl` - (TODO: why are they in this tutorial?) +- **Deterministic search methods** BFS and DFS. +- **Stochastic search methods**, which introduce randomness to search the program space. We will look at Metropolis-Hastings, Very Large Scale Neighbourhood Search, Simulated Annealing and Genetic Search. """ # ╔═╡ 6ab37bbc-73e2-4d9a-a8b2-e715a0b61c8f TableOfContents() +# ╔═╡ 61cee94c-2481-4268-823b-ca596592b63c +md""" +Let's import all the Herb modules that we will use throughout the tutorial. +""" + # ╔═╡ 67931820-0f43-41e1-898e-5b5bd55e30d1 md""" We start with a simple grammar:. @@ -54,13 +61,13 @@ problem_1 = Problem([IOExample(Dict(:x => x), 2x+1) for x ∈ 1:5]) md""" ## Parameters -Search procedures typically have some hyperparameters that can be cofigured by the user. +Search procedures typically have some hyperparameters that you can configure. ### `max_depth` -`max_depth` controls the maximum depth of the program trees we want to explore. +`max_depth` controls the maximum depth of the program trees that are explored during the search, effectively limiting the size and complexity of the synthesized program. The parameter is configured as part of the iterator. -In the following example, we can see the effect of `max_depth` on the number of allocations considered. +In the following example, we consider two different values for `max_depth`. """ # ╔═╡ 338f19f1-3a62-4462-b8dc-24424d7644f2 @@ -69,6 +76,11 @@ iterator_1 = BFSIterator(g_1, :Number, max_depth=3) # ╔═╡ 542cd47e-74cd-4b6f-acc9-bf524222e583 iterator_2 = BFSIterator(g_1, :Number, max_depth=6) +# ╔═╡ 63e97576-1c34-464d-a106-d59d5fb1ee38 +md""" +To see the effect `max_depth` has on the number of memory allocations made during the program synthesis process, we use the `@time` macro. +""" + # ╔═╡ a6fb2e91-b73a-4032-930f-d884abd539e2 begin println("Solution for max_depth = 3:") @@ -80,16 +92,17 @@ begin println(solution_2) end -# ╔═╡ 4b49f206-970e-47d2-af65-336ba65b1019 +# ╔═╡ 58c1a904-4d87-43f7-bcc3-884a8663c1da md""" -TODO: Explain @time +While increasing `max_depth` allows us to explore more complex and deeper program trees, which may lead to a better solution, it also requires more memory allocation and can increase the execution time. """ # ╔═╡ 35405357-179b-4e77-9bdc-edf5a550b36d md""" ### `max_enumerations` -Another parameter to use is `max_enumerations`, which describes the maximum number of programs that can be tested at evaluation. -Let's see how many enumerations are necessary to solve our simple problem. +`max_enumerations` defines the maximum number of candidate programs that can be evaluated before the search is terminated. + +Let's explore how many enumerations are necessary to solve our simple problem. """ # ╔═╡ 3954dd49-07a2-4ec2-91b4-9c9596d5c264 @@ -104,26 +117,17 @@ end # ╔═╡ 9892e91b-9115-4520-9637-f8d7c8905825 md""" -TODO: numbers seem to have changed. Not 24 anymore. How reproducible is this anyway? -What does allocations mean? - -We see that only when `i >= 24`, there is a result, after that, increasing `i` does not have any effect on the number of allocations. - -A final parameter we consider here is `allow_evaluation_errors`, which is `false` by default. When this is set to `true`, the program will still run even when an exception is thrown during evaluation. To see the effect of this, we create a new grammar. We can also retrieve the error together with the solution from the search method. +At `i = 3`, we observe that an optimal program is found. Increasing the number of enumerations beyond that does not affect the solution or the number of memory allocations. """ # ╔═╡ a4e58bbc-7c14-4fce-b35d-688b56e0eb61 md""" ### `allow_evaluation_errors` -TODO: What do we mean with 'program will still run'? - -A final parameter we consider here is `allow_evaluation_errors`, which is `false` by default. When set to `true`, the program will still run even when an exception is thrown during evaluation. +A final parameter we consider here is `allow_evaluation_errors`, which is `false` by default. When `true`, the search continues even if an exception occurs during the evaluation of a candidate program. This allows the search process to handle faulty candidate programs and explore other ones, instead of throwing an error and terminating prematurely. -We will use a new example to see the effect of `allow_evaluation_errors`. After defining a new grammar and problem, ... - -We can also retrieve the error together with the solution from the search method. +We will use a new example to see the effect of `allow_evaluation_errors`. We begin defining a new simple grammar. We then create some input-output examples to specify the problem we want to solve. This time, we choose a problem that we cannot solve with the provided grammar. """ # ╔═╡ 9fb40ceb-8d41-491b-8941-20a8b240eb82 @@ -139,51 +143,77 @@ end problem_2 = Problem([IOExample(Dict(), x) for x ∈ 1:5]) # ╔═╡ a4a7daed-f89b-44ad-8787-9199c05bf046 -iterator_3 = BFSIterator(g_2, :Number, max_depth=2) +iterator_3 = BFSIterator(g_2, :Index, max_depth=2) # ╔═╡ 4821fd3a-ff2d-4991-99ad-76608d11b1da -solution_3 = @time synth(problem_2, iterator_3, allow_evaluation_errors=true) - -# ╔═╡ 8d91b2e3-30b5-4ea2-bd3f-3055bb6d1d5a -# solution = search(g_2, problem_2, :Index, max_depth=2, allow_evaluation_errors=true) +solution_3 = synth(problem_2, iterator_3) -# ╔═╡ 52332fa2-7ea7-4226-9460-e0bbc905c619 -println("solution: ", solution_3) - -# ╔═╡ c26fea48-f812-4f24-bb8c-680e14a55df7 +# ╔═╡ b2eb08d7-3e53-46c5-84b1-e1fa0e07e291 md""" -There is another search method called `search_best` which returns both the solution and the possible error. The method returns the best program found so far. In this case, we can also see the error (`typemax(Int)`): +As expected, an exception occurs during the synthesis process. Now we try the same again, with `allow_evaluation_errors=true`. """ -# ╔═╡ 57f5face-d9d5-441a-8e0e-6ef6319fc178 -solution, error = search_best(g, problem, :Index, max_depth=2, allow_evaluation_errors=true) -println("solution: ", solution) -println("error: ", error) +# ╔═╡ 606070e1-83a7-4cca-a716-4fa459f78772 +solution_4 = synth(problem_2, iterator_3, allow_evaluation_errors=true) + +# ╔═╡ 52332fa2-7ea7-4226-9460-e0bbc905c619 +println("solution: ", solution_4) + +# ╔═╡ c262116e-138e-4133-a032-d2f50bfbf5bd +md""""This time we find a solution, although a suboptimal one.""" # ╔═╡ 9b4b21e0-dc6a-43ae-a511-79988ee99001 md""" ## Search methods -We now show examples of using different search procedures, which are initialized by passing different enumerators to the search function. +Herb.jl provides already implemented, ready-to-use search methods. The core building block of the search is the program iterator, which represents a walk through the program space. All program iterators share the top-level abstract type `ProgramIterator`. For more information on iterators and how to customize them, see [this tutorial](https://herb-ai.github.io/Herb.jl/dev/tutorials/TopDown/). +""" + +# ╔═╡ 115c02c9-ae0c-4623-a61d-831fc6ad55a2 +md""" +### Deterministic search: BFS and DFS -### Breadth-First Search +First, we explore two fundamental deterministic top-down search algorithms: **breadth-first search (BFS)** and **depth-first search (DFS)**. Both algorithms are implemented using the abstract type `TopDownIterator`, which can be customized through the functions priority_function, derivation_heuristic, and hole_heuristic. -The breadth-first search will first enumerate all possible programs at the same depth before considering programs with a depth of one more. A tree of the grammar is returned with programs ordered in increasing sizes. We can first `collect` the programs that have a `max-depth` of 2 and a `max_size` of infinite (integer maximum value), where the starting symbol is of type `Real`. This function uses a default heuristic 'left-most first', such that the left-most child in the tree is always explored first. +#### Breadth-First Search + +The `BFSIterator` enumerates all possible programs at a given depth before progressing to the next level, ensuring that trees are explored in increasing order of size. This guarantees that smaller programs are evaluated first, and larger, more complex ones are considered only after all smaller ones have been processed. + +To explore `BFSIterator`, we define another very simple grammar. """ # ╔═╡ 3af650d9-19c6-4351-920d-d2361091f628 -g1 = @cfgrammar begin - Real = 1 | 2 - Real = Real * Real +begin g3 = @cfgrammar begin + Real = 1 | 2 + Real = Real * Real + end end -programs = collect(get_bfs_enumerator(g1, 2, typemax(Int), :Real)) + +# ╔═╡ 4cb08dba-aea5-4c31-998c-844d1fce8c81 +md""" +Next, we define a `BFSIterator` with a `max_depth` of 2 and a `max_size` of infinite (which we approximate with the maximum value of `Int`), and a starting symbol of type `Real`. By default, `BFSIterator` uses the heuristic 'left-most first', i.e., the left-most child in the tree is always explored first. +""" + +# ╔═╡ f2521a57-267e-4b49-9179-4e9c2e6bdec7 +iteratorbfs = BFSIterator(g3, :Real, max_depth=2, max_size=typemax(Int)) + +# ╔═╡ bf038215-1ecf-4e1c-a9be-e133e4497293 +md""" +To see all possible solution programs the iterator explores, we use `collect`. It returs a list of the programs, ordered by increasing size and depth. +""" + +# ╔═╡ 6aec7358-225a-4764-9a36-da86234b6cf8 +programsbfs = collect(iteratorbfs) + +# ╔═╡ 54ecf6b9-3341-49e0-92e9-71190e06d61b +println(programsbfs) # ╔═╡ d3ff497e-d2c2-4df6-8e4c-cdca70fd0677 md""" -We can test that this function returns all and only the correct functions. +Let's verify that the iterator returns the programs we expect (keep in mind we use a leftmost-first heuristic). """ -# ╔═╡ da7f326c-f0d5-4837-ac9a-5bcad604566e +# ╔═╡ 07b54acf-0c0d-40ac-ae18-fb26094b4aca answer_programs = [ RuleNode(1), RuleNode(2), @@ -193,40 +223,101 @@ answer_programs = [ RuleNode(3, [RuleNode(2), RuleNode(2)]) ] -println(all(p ∈ programs for p ∈ answer_programs)) +# ╔═╡ 9efb01cf-b190-4e3e-aa19-11499ba46489 +println(all(p ∈ programsbfs for p ∈ answer_programs)) # ╔═╡ 0020b79a-6352-4e2d-93f6-2a1d7b03ae2c md""" -### Depth-First Search +#### Depth-First Search + +The `DFSIterator` looks at one branch of the search tree at a time, exploring it unitl a correct program is found or the specified `max_depth` is reached. Only then, the next branch is considered. -In depth-first search, we first explore a certain branch of the search tree till the `max_depth` or a correct program is reached before we consider the next branch. +Let's `collect` the the trees/programs (TODO), using the same grammar but this time `DFSIterator`. """ # ╔═╡ 789150a8-862c-48c3-88b8-710b81ab34cf -g1 = @cfgrammar begin -Real = 1 | 2 -Real = Real * Real +begin + g1 = @cfgrammar begin + Real = 1 | 2 + Real = Real * Real + end + end -programs = collect(get_dfs_enumerator(g1, 2, typemax(Int), :Real)) -println(programs) + +# ╔═╡ db5be2c3-0b36-40b4-bf14-20e2c7063ad7 +iteratordfs = DFSIterator(g3, :Real, max_depth=2, max_size=typemax(Int)) + +# ╔═╡ 4048ff37-e7d1-44ee-bfa3-aa058b6f53b6 +programsdfs = collect(iteratordfs) + +# ╔═╡ 658c55ac-88c8-4657-a8eb-c9c9b91d0ded +println(programsdfs) # ╔═╡ 243165be-a9d2-484d-8046-811a2b0ba139 md""" -`get_dfs_enumerator` also has a default left-most heuristic and we consider what the difference is in output. +`DFSIterator` also uses by default a **leftmost-first** heuristic. If we want to use a **rightmost-first** heuristic instead, we can create our own iterator `DFSIteratorRightmost`. Also see the tutorial [Top Down Iterator](https://herb-ai.github.io/Herb.jl/dev/tutorials/TopDown/) for how to build iterators is Herb.jl. + + + + +TODO: How to use rightmost heuristic? +""" + +# ╔═╡ 3f18989a-ae25-4a66-9930-4459407af695 +# to modify the heuristics, we create a new type +# abstract type DFSIteratorRightmost <: TopDownIterator end + +# ╔═╡ 4b97602a-5226-429f-86ea-8ecac3c807fa +@programiterator DFSIteratorRightmost() <: TopDownIterator + +# ╔═╡ ed198b79-1b95-4531-b148-c1037cfdacf4 +md""" +By default, `priority_function` for a `TopDownIterator` is that of a BFS iterator. Hence, we need to overwrite it: +""" + +# ╔═╡ 3c7f6e75-ea70-4ccd-8ad6-5876d50a587d +function priority_function( + ::DFSIteratorRightmost, + ::AbstractGrammar, + ::AbstractRuleNode, + parent_value::Union{Real, Tuple{Vararg{Real}}}, + isrequeued::Bool +) + if isrequeued + return parent_value; + end + return parent_value - 1; +end + +# ╔═╡ d62918e4-4113-45bd-81ea-d17d007b83f5 +# Alternative +priority_function(::DFSIteratorRightmost) = priority_function(DFSIterator()) + +# ╔═╡ 7480d1e4-e417-4d87-80b7-513a098da70e +md""" +Next, we need to overwrite the `hole_heuristic` to rightmost-first. """ -# ╔═╡ 3d01c6f1-80a6-4904-97e2-775170e97bbf -g1 = @cfgrammar begin - Real = 1 | 2 - Real = Real * Real +# ╔═╡ 7e2af72d-b71c-4234-9bca-cb9a90732a91 +function hole_heuristic(::DFSIteratorRightmost, node::AbstractRuleNode, max_depth::Int)::Union{ExpandFailureReason, HoleReference} + return heuristic_rightmost(node, max_depth); end -programs = collect(get_dfs_enumerator(g1, 2, typemax(Int), :Real, heuristic_rightmost)) -println(programs) + +# ╔═╡ 00d05a7e-ca79-4d6b-828d-b24ef1cb80a2 +iteratordfs_rightmost = DFSIteratorRightmost(g3, :Real, max_depth=2, max_size=typemax(Int)) + +# ╔═╡ c70eb1a1-7d6b-413f-bd85-77e1b8c30b94 +println(collect(iteratordfs_rightmost)) # ╔═╡ 168c71bf-ce5b-4ab3-b29a-5996981c42a5 md""" ## Stochastic search -We now introduce a few stochastic search algorithms, for which we first create a simple grammar and a helper function for problems. + +Whereas deterministic search methods explore the search space in a predictable way, stochastic ones use randomness. + +We will look at several stochastic search algorithms, usinga a new example. + +We start with a simple grammar and a helper function to create the input-output examples for the problem we want to solve. """ # ╔═╡ a4b522cf-78f0-4d44-88c8-82dd0cdbf952 @@ -248,259 +339,154 @@ end md""" ### Metropolis-Hastings -One of the stochastic search methods that is implemented is Metropolis-Hastings (MH), which samples from a distribution of programs based on the grammar. For more information on MH, see for example [this webpage](https://stephens999.github.io/fiveMinuteStats/MH_intro.html). +Metropolis-Hastings (MH) is a method to produce samples from a distribution that may otherwise be difficult to sample. In the context of program synthesis, we want samples from a distribution of programs, based on the grammar. +For more information on MH, see for example [this webpage](https://stephens999.github.io/fiveMinuteStats/MH_intro.html). -The example below uses a simple arithmetic example. You can try running this code block multiple times, which will give different programs, as the search is stochastic. +The example below uses a simple arithmetic example. Try run the code block (TODO) multiple times. You will get different programs every time, as the search is stochastic. """ +# ╔═╡ 4df84c71-99b5-487f-9db5-c048c0c74151 +e_mh = x -> x * x + 4 + +# ╔═╡ afe1872c-6641-4fa0-a53e-50c6b4a712ee +problem_mh, examples_mh = create_problem(e_mh) + +# ╔═╡ 969db94c-d583-40d1-a058-097f8117c2e9 +cost_function = mean_squared_error + + # ╔═╡ 0a30fb40-cd45-4661-a501-ae8e45f1e07e -e = x -> x * x + 4 -problem, examples = create_problem(e) -enumerator = get_mh_enumerator(examples, mean_squared_error) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3) +begin + iteratormh = MHSearchIterator(grammar, :X, examples_mh, cost_function, max_depth=3) #TODO: What should max_depth be? + programmh = synth(problem_mh, iteratormh) +end + +# ╔═╡ 82f7ffa5-2bdc-4153-85fa-d7aca063da12 +programmh # ╔═╡ 700270ea-90bd-474b-91d9-0e5ed329776a md""" ### Very Large Scale Neighbourhood Search -The second implemented stochastic search method is VLSN, which searches for a local optimum in the neighbourhood. For more information, see [this article](https://backend.orbit.dtu.dk/ws/portalfiles/portal/5293785/Pisinger.pdf). +The second stochastic search method we consider is Very Large Scale Neighbourhood Search (VLSN). In each iteration, the neighbourhood of the current candidate program is searched for the local optimum to find a better candidate. -Given the same grammar as before, we can try it with some simple examples. -""" +TODO: more? -# ╔═╡ 8731f312-bfcf-4f6c-86fa-60014dc146d6 -e = x -> 10 -max_depth = 2 -problem, examples = create_problem(e) -enumerator = get_vlsn_enumerator(examples, mean_squared_error, max_depth) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth) +For more information, see [this article](https://backend.orbit.dtu.dk/ws/portalfiles/portal/5293785/Pisinger.pdf). -# ╔═╡ 46ca65d1-d876-4abc-a562-8d266bad195f -e = x -> x -max_depth = 1 -problem, examples = create_problem(e) -enumerator = get_vlsn_enumerator(examples, mean_squared_error, max_depth) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth) +Given the same grammar as before, we can try it with some simple examples. -# ╔═╡ 599194a8-3f47-4917-9143-a5fe0d43029f -md""" -### Simulated Annealing -The third stochastic search method is called simulated annealing. This is another hill-climbing method to find local optima. For more information, see [this page](https://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/anneal.html). -We try the example from earlier, but now we can additionally define the `initial_temperature` of the algorithm, which is 1 by default. Change the value below to see the effect. -""" +Using +large neighborhoods makes it possible to find better candidate solutions in each it- +eration and hence traverse a more promising search path -# ╔═╡ cb5935ed-d89b-4e25-9243-d201daf18e78 -e = x -> x * x + 4 -initial_temperature = 1 -problem, examples = create_problem(e) -enumerator = get_sa_enumerator(examples, mean_squared_error, initial_temperature) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3) +#### Example I +""" -# ╔═╡ d0c3742e-23e5-4ca1-9e83-b6d1e8a7cded -e = x -> x * x + 4 -initial_temperature = 2 -problem, examples = create_problem(e) -enumerator = get_sa_enumerator(examples, mean_squared_error, initial_temperature) -program, cost = @time search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3) +# ╔═╡ e6e5c63b-34e8-40d6-bc12-bd31f40b4b16 +e_vlsn = x -> 10 -# ╔═╡ 5df0ba53-b528-4baf-9980-cafe5d73f9dd -md""" -### Genetic Search +# ╔═╡ 2397f65f-e6b4-4f11-bf66-83440c58b688 +problem_vlsn, examples_vlsn = create_problem(e_vlsn) -Genetic search is a type of evolutionary algorithm, which will simulate the process of natural selection and return the 'fittest' program of the population. For more information, see [here](https://www.geeksforgeeks.org/genetic-algorithms/). +# ╔═╡ 7c738d7b-bf05-40c7-b3b7-1512fbae7299 +iteratorvlsn = VLSNSearchIterator(grammar, :X, examples_vlsn, cost_function, max_depth=2) -We show the example of finding a lambda function. Try varying the parameters of the genetic search to see what happens. -""" +# ╔═╡ 33af905e-e8ca-425d-9805-eb02bec7c26b +programvlsn = synth(problem_vlsn, iteratorvlsn) +# program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth) -# ╔═╡ a434645b-d592-4162-a8b4-b4b04cea30a9 -e = x -> 3 * x * x + (x + 2) -problem, examples = create_problem(e) -enumerator = get_genetic_enumerator(examples, - initial_population_size = 10, - mutation_probability = 0.8, - maximum_initial_population_depth = 3, -) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=nothing, max_time=20) - -# ╔═╡ 38cd9032-27c0-4179-a536-ce59a42ff16a +# ╔═╡ 4208cd17-6c2a-4480-a535-5a7358e0ab25 md""" -## Other functionality - -Finally, we showcase two other functionalities of HerbSearch, sampling and heuristics. - -### Sampling -Sampling is implemented for the different stochastic search methods. +#### Example 2 -We consider here a simple grammar, which gives different programs for different search depths. +TODO: What do we want to show with this example? """ -# ╔═╡ f8415d48-a51d-4845-8425-fd61ed79c06e -grammar = @cfgrammar begin - A = B | C | F - F = G - C = D - D = E -end +# ╔═╡ bea28b36-6613-4895-98f9-27dfd9e57b09 +e_vlsn2 = x -> x -# A->B (depth 1) or A->F->G (depth 2) or A->C->D->E (depth 3) +# ╔═╡ aa95cb5e-926d-4119-8d08-353f37a59039 +problem_vlsn2, examples_vlsn2 = create_problem(e_vlsn2) -# For depth ≤ 1 the only option is A->B -expression = rand(RuleNode, grammar, :A, 1) -@assert rulenode2expr(expression, grammar) in [:B,:C,:F] +# ╔═╡ 285043ef-c295-400f-91c5-f3c6c69ac2bf +iterator_vlsn2 = VLSNSearchIterator(grammar, :X, examples_vlsn2, cost_function, max_depth=1) -# For depth ≤ 2 the two options are A->B (depth 1) and A->B->G| A->C->G | A->F->G (depth 2) -expression = rand(RuleNode, grammar, :A, 2) -@assert rulenode2expr(expression,grammar) in [:B,:C,:F,:G] +# ╔═╡ 36f0e0cf-c871-42c9-956e-054767cbf693 +program_vlsn2 = synth(problem_vlsn2, iterator_vlsn2) -# ╔═╡ 7f88bf4f-d82c-4e5a-a9eb-93870954c79e +# ╔═╡ 599194a8-3f47-4917-9143-a5fe0d43029f md""" -### Heuristics +### Simulated Annealing + +Simualted Annealing (SA) explores smaller, incremental changes in the candidate program per iteration and refines the solution over time. It is a modified hill-climbing algorithm that, instead of picking the best move (program modification?), picks a random move. If the move improves the solution (candidate program), it is always accepted. Ocassionally, a move towards a worse solution is accepted to escape loca optima. However, this strategy follows a cooling (annealing) schedule, i.e., it starts exploring widely (at high temperature) and gradually becomes more selective (at low temperture), accepting worse solutions less often. + +For more information, see [this page](https://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/anneal.html). """ -# ╔═╡ 53598f8f-9973-4cad-af0c-280f5531bb21 +# ╔═╡ dd6aee87-cd96-4be1-b8fb-03fffee5ea43 md""" -# More interesting domains & Use of constraints -In the following examples, we introduce some larger grammars and show that Herb can still efficiently find the correct program. +We use the same example as for MH. SA additionally has the option to specify the `initial_temperature` for the annealing (default `initial_temperature=1`). Let's see what effect changing the temperature from 1 to 2 has on the solution program. """ -# ╔═╡ bc971069-08c4-493c-a917-8092493d3233 -#Expects to return a program equivalent to 1 + (1 - x) = 2 - x - -g₁ = @csgrammar begin - Element = |(1 : 3) # 1 - 3 - Element = Element + Element # 4 - Element = 1 - Element # 5 - Element = x # 6 -end - -addconstraint!(g₁, ComesAfter(6, [5])) +# ╔═╡ f6a3ed3f-bcf9-4f22-a86e-33255d88e0b2 +e_sa = x -> x * x + 4 -examples = [ - IOExample(Dict(:x => 0), 2), - IOExample(Dict(:x => 1), 1), - IOExample(Dict(:x => 2), 0) -] -problem = Problem(examples) -solution = search(g₁, problem, :Element, max_depth=3) +# ╔═╡ e25d115f-7549-4606-b96c-9ef700810f7b +problem_sa, examples_sa = create_problem(e_sa) -@assert test_with_input(SymbolTable(g₁), solution, Dict(:x => -2)) == 4 +# ╔═╡ 94f2bd5e-e11e-42e7-9a3e-3c9d5ae43cd4 +initial_temperature1 = 1 -# ╔═╡ 61a60c9c-36cf-4e86-b697-748b3524d3b4 -# Expects to return a program equivalent to 4 + x * (x + 3 + 3) = x^2 + 6x + 4 +# ╔═╡ eb851d7b-803e-45f6-ad10-fa0bde78826a +iterator_sa1 = SASearchIterator(grammar, :X, examples_sa, cost_function, max_depth=3, initial_temperature = initial_temperature1) -g₂ = @csgrammar begin - Element = Element + Element + Element # 1 - Element = Element + Element * Element # 2 - Element = x # 3 - Element = |(3 : 5) # 4 -end +# ╔═╡ 73304e3f-05bf-4f0c-9acd-fc8afa87b460 +program_sa1 = synth(problem_sa, iterator_sa1) -# Restrict .. + x * x -addconstraint!(g₂, Forbidden(MatchNode(2, [MatchVar(:x), MatchNode(3), MatchNode(3)]))) -# Restrict 4 and 5 in lower level -addconstraint!(g₂, ForbiddenPath([2, 1, 5])) -addconstraint!(g₂, ForbiddenPath([2, 1, 6])) +# ╔═╡ 07f11eb1-6b45-441a-a481-57628bad23ae +initial_temperature2 = 2 -examples = [ - IOExample(Dict(:x => 1), 11) - IOExample(Dict(:x => 2), 20) - IOExample(Dict(:x => -1), -1) -] -problem = Problem(examples) -solution = search(g₂, problem, :Element) +# ╔═╡ 4ff69f0a-6626-4593-b361-a2387eecc731 +iterator_sa2 = SASearchIterator(grammar, :X, examples_sa, cost_function, max_depth=3, initial_temperature = initial_temperature2) -@assert test_with_input(SymbolTable(g₂), solution, Dict(:x => 0)) == 4 +# ╔═╡ 0f7f228c-4f95-4f1a-9ca2-2d03384a00c0 -# ╔═╡ a5b17c81-b667-4b1c-ab15-ddf1a162683b -# Expects to return a program equivalent to (1 - (((1 - x) - 1) - 1)) - 1 = x + 1 -g₃ = @csgrammar begin - Element = |(1 : 20) # 1 - 20 - Element = Element - 1 # 21 - Element = 1 - Element # 22 - Element = x # 23 -end +# ╔═╡ 5df0ba53-b528-4baf-9980-cafe5d73f9dd +md""" +### Genetic Search -addconstraint!(g₃, ComesAfter(23, [22, 21])) -addconstraint!(g₃, ComesAfter(22, [21])) +Genetic search is a type of evolutionary algorithm, which simulates the process of natural selection. It evolves a population of candidate programs through operations like mutation, crossover (recombination), and selection. Then, the fitness of each program is assessed (i.e., how well it satisfies the given specifications). Only the 'fittest' programs are selected for the next generation, thus gradually refining the population of candidate programs. -examples = [ - IOExample(Dict(:x => 1), 2) - IOExample(Dict(:x => 10), 11) -] -problem = Problem(examples) -solution = search(g₃, problem, :Element) +For more information, see [here](https://www.geeksforgeeks.org/genetic-algorithms/). -@assert test_with_input(SymbolTable(g₃), solution, Dict(:x => 0)) == 1 -@assert test_with_input(SymbolTable(g₃), solution, Dict(:x => 100)) == 101 +We show the example of finding a lambda function. Try varying the parameters of the genetic search to see what happens. +""" -# ╔═╡ e41feac7-6de6-4223-8a21-341b85da52c0 -# Expects to return a program equivalent to 18 + 4x +# ╔═╡ 99ea1c20-ca2c-4d77-bc3b-06814db1d666 +e_gs = x -> 3 * x * x + (x + 2) -g₄ = @csgrammar begin - Element = |(0 : 20) # 1 - 20 - Element = Element + Element + Element # 21 - Element = Element + Element * Element # 22 - Element = x # 23 -end +# ╔═╡ d991edb9-2291-42a7-97ff-58c456515505 +problem_gs, examples_gs = create_problem(e_gs) -# Enforce ordering on + + -addconstraint!(g₄, Ordered( - MatchNode(21, [MatchVar(:x), MatchVar(:y), MatchVar(:z)]), - [:x, :y, :z] -)) +# ╔═╡ d17c2a79-3c14-4c50-a4aa-465f3da011a2 -examples = [ - IOExample(Dict(:x => 1), 22), - IOExample(Dict(:x => 0), 18), - IOExample(Dict(:x => -1), 14) -] -problem = Problem(examples) -solution = search(g₄, problem, :Element) - -@assert test_with_input(SymbolTable(g₄), solution, Dict(:x => 100)) == 418 - -# ╔═╡ 1c4db74a-4caa-4ce5-815f-b631365c5129 -# Expects to return a program equivalent to (x == 2) ? 1 : (x + 2) - -g₅ = @csgrammar begin - Element = Number # 1 - Element = Bool # 2 - - Number = |(1 : 3) # 3-5 - - Number = Number + Number # 6 - Bool = Number ≡ Number # 7 - Number = x # 8 - - Number = Bool ? Number : Number # 9 - Bool = Bool ? Bool : Bool # 10 -end -# Forbid ? = ? -addconstraint!(g₅, Forbidden(MatchNode(7, [MatchVar(:x), MatchVar(:x)]))) -# Order = -addconstraint!(g₅, Ordered(MatchNode(7, [MatchVar(:x), MatchVar(:y)]), [:x, :y])) -# Order + -addconstraint!(g₅, Ordered(MatchNode(6, [MatchVar(:x), MatchVar(:y)]), [:x, :y])) - -examples = [ - IOExample(Dict(:x => 0), 2) - IOExample(Dict(:x => 1), 3) - IOExample(Dict(:x => 2), 1) -] -problem = Problem(examples) -solution = search(g₅, problem, :Element) +# ╔═╡ 069591a3-b89b-4fc6-afba-2145e32852b7 +iterator_gs = GeneticSearchIterator(grammar, :X, examples_gs, population_size = 10, mutation_probability = 0.8, maximum_initial_population_depth = 3) -@assert test_with_input(SymbolTable(g₅), solution, Dict(:x => 3)) == 5 +# ╔═╡ 5bef5754-d81b-4160-8ed6-396d02853d9a +program_gs, error_gs = synth(problem_gs, iterator_gs) # ╔═╡ 00000000-0000-0000-0000-000000000001 PLUTO_PROJECT_TOML_CONTENTS = """ [deps] HerbConstraints = "1fa96474-3206-4513-b4fa-23913f296dfc" +HerbCore = "2b23ba43-8213-43cb-b5ea-38c12b45bd45" HerbGrammar = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" HerbInterpret = "5bbddadd-02c5-4713-84b8-97364418cca7" HerbSearch = "3008d8e8-f9aa-438a-92ed-26e9c7b4829f" @@ -509,6 +495,7 @@ PlutoUI = "7f904dfe-b85e-4ff6-b463-dae2292396a8" [compat] HerbConstraints = "~0.2.2" +HerbCore = "~0.3.0" HerbGrammar = "~0.3.0" HerbInterpret = "~0.1.3" HerbSearch = "~0.3.0" @@ -522,7 +509,7 @@ PLUTO_MANIFEST_TOML_CONTENTS = """ julia_version = "1.10.4" manifest_format = "2.0" -project_hash = "5ee7afaa57cf163e03a42d572d1cb2cb022598e5" +project_hash = "e8df13c29bfd5a0060b6c72e678efa3df418b0d7" [[deps.AbstractPlutoDingetjes]] deps = ["Pkg"] @@ -1251,6 +1238,7 @@ version = "0.13.1+0" # ╟─dddca175-3d88-45ce-90da-575c0ba38175 # ╠═a93d954d-0f09-4b6d-a3a5-62bfe39681e2 # ╠═6ab37bbc-73e2-4d9a-a8b2-e715a0b61c8f +# ╟─61cee94c-2481-4268-823b-ca596592b63c # ╠═c4441fa4-09ec-4b81-9681-b13b93a9c9c0 # ╟─67931820-0f43-41e1-898e-5b5bd55e30d1 # ╠═e41c61a4-0b2c-46da-8f7b-fe6dc529c544 @@ -1259,49 +1247,82 @@ version = "0.13.1+0" # ╟─e9c6bc00-21f5-4a99-8bec-63cf2156c233 # ╠═338f19f1-3a62-4462-b8dc-24424d7644f2 # ╠═542cd47e-74cd-4b6f-acc9-bf524222e583 +# ╟─63e97576-1c34-464d-a106-d59d5fb1ee38 # ╠═a6fb2e91-b73a-4032-930f-d884abd539e2 -# ╠═4b49f206-970e-47d2-af65-336ba65b1019 +# ╟─58c1a904-4d87-43f7-bcc3-884a8663c1da # ╟─35405357-179b-4e77-9bdc-edf5a550b36d # ╠═3954dd49-07a2-4ec2-91b4-9c9596d5c264 -# ╠═9892e91b-9115-4520-9637-f8d7c8905825 -# ╠═a4e58bbc-7c14-4fce-b35d-688b56e0eb61 +# ╟─9892e91b-9115-4520-9637-f8d7c8905825 +# ╟─a4e58bbc-7c14-4fce-b35d-688b56e0eb61 # ╠═9fb40ceb-8d41-491b-8941-20a8b240eb82 # ╠═94e0d676-a9c7-4291-8696-15301e541c30 # ╠═a4a7daed-f89b-44ad-8787-9199c05bf046 # ╠═4821fd3a-ff2d-4991-99ad-76608d11b1da -# ╠═8d91b2e3-30b5-4ea2-bd3f-3055bb6d1d5a +# ╟─b2eb08d7-3e53-46c5-84b1-e1fa0e07e291 +# ╠═606070e1-83a7-4cca-a716-4fa459f78772 # ╠═52332fa2-7ea7-4226-9460-e0bbc905c619 -# ╟─c26fea48-f812-4f24-bb8c-680e14a55df7 -# ╠═57f5face-d9d5-441a-8e0e-6ef6319fc178 +# ╟─c262116e-138e-4133-a032-d2f50bfbf5bd # ╟─9b4b21e0-dc6a-43ae-a511-79988ee99001 +# ╟─115c02c9-ae0c-4623-a61d-831fc6ad55a2 # ╠═3af650d9-19c6-4351-920d-d2361091f628 +# ╟─4cb08dba-aea5-4c31-998c-844d1fce8c81 +# ╠═f2521a57-267e-4b49-9179-4e9c2e6bdec7 +# ╟─bf038215-1ecf-4e1c-a9be-e133e4497293 +# ╠═6aec7358-225a-4764-9a36-da86234b6cf8 +# ╠═54ecf6b9-3341-49e0-92e9-71190e06d61b # ╟─d3ff497e-d2c2-4df6-8e4c-cdca70fd0677 -# ╠═da7f326c-f0d5-4837-ac9a-5bcad604566e -# ╟─0020b79a-6352-4e2d-93f6-2a1d7b03ae2c +# ╠═07b54acf-0c0d-40ac-ae18-fb26094b4aca +# ╠═9efb01cf-b190-4e3e-aa19-11499ba46489 +# ╠═0020b79a-6352-4e2d-93f6-2a1d7b03ae2c # ╠═789150a8-862c-48c3-88b8-710b81ab34cf -# ╟─243165be-a9d2-484d-8046-811a2b0ba139 -# ╠═3d01c6f1-80a6-4904-97e2-775170e97bbf -# ╟─168c71bf-ce5b-4ab3-b29a-5996981c42a5 +# ╠═db5be2c3-0b36-40b4-bf14-20e2c7063ad7 +# ╠═4048ff37-e7d1-44ee-bfa3-aa058b6f53b6 +# ╠═658c55ac-88c8-4657-a8eb-c9c9b91d0ded +# ╠═243165be-a9d2-484d-8046-811a2b0ba139 +# ╠═aa45e602-6219-44ea-82a0-b97639f6a450 +# ╠═3f18989a-ae25-4a66-9930-4459407af695 +# ╠═4b97602a-5226-429f-86ea-8ecac3c807fa +# ╟─ed198b79-1b95-4531-b148-c1037cfdacf4 +# ╠═3c7f6e75-ea70-4ccd-8ad6-5876d50a587d +# ╠═d62918e4-4113-45bd-81ea-d17d007b83f5 +# ╟─7480d1e4-e417-4d87-80b7-513a098da70e +# ╠═7e2af72d-b71c-4234-9bca-cb9a90732a91 +# ╠═00d05a7e-ca79-4d6b-828d-b24ef1cb80a2 +# ╠═c70eb1a1-7d6b-413f-bd85-77e1b8c30b94 +# ╠═168c71bf-ce5b-4ab3-b29a-5996981c42a5 # ╠═a4b522cf-78f0-4d44-88c8-82dd0cdbf952 # ╠═f313edb9-8fd9-4d78-88cd-89226f5c769d # ╟─0da9053a-959b-471e-8918-662ec63da71c +# ╠═4df84c71-99b5-487f-9db5-c048c0c74151 +# ╠═afe1872c-6641-4fa0-a53e-50c6b4a712ee +# ╠═969db94c-d583-40d1-a058-097f8117c2e9 # ╠═0a30fb40-cd45-4661-a501-ae8e45f1e07e +# ╠═82f7ffa5-2bdc-4153-85fa-d7aca063da12 # ╟─700270ea-90bd-474b-91d9-0e5ed329776a -# ╠═8731f312-bfcf-4f6c-86fa-60014dc146d6 -# ╠═46ca65d1-d876-4abc-a562-8d266bad195f +# ╠═e6e5c63b-34e8-40d6-bc12-bd31f40b4b16 +# ╠═2397f65f-e6b4-4f11-bf66-83440c58b688 +# ╠═7c738d7b-bf05-40c7-b3b7-1512fbae7299 +# ╠═33af905e-e8ca-425d-9805-eb02bec7c26b +# ╟─4208cd17-6c2a-4480-a535-5a7358e0ab25 +# ╠═bea28b36-6613-4895-98f9-27dfd9e57b09 +# ╠═aa95cb5e-926d-4119-8d08-353f37a59039 +# ╠═285043ef-c295-400f-91c5-f3c6c69ac2bf +# ╠═36f0e0cf-c871-42c9-956e-054767cbf693 # ╟─599194a8-3f47-4917-9143-a5fe0d43029f -# ╠═cb5935ed-d89b-4e25-9243-d201daf18e78 -# ╠═d0c3742e-23e5-4ca1-9e83-b6d1e8a7cded +# ╠═dd6aee87-cd96-4be1-b8fb-03fffee5ea43 +# ╠═f6a3ed3f-bcf9-4f22-a86e-33255d88e0b2 +# ╠═e25d115f-7549-4606-b96c-9ef700810f7b +# ╠═94f2bd5e-e11e-42e7-9a3e-3c9d5ae43cd4 +# ╠═eb851d7b-803e-45f6-ad10-fa0bde78826a +# ╠═73304e3f-05bf-4f0c-9acd-fc8afa87b460 +# ╠═07f11eb1-6b45-441a-a481-57628bad23ae +# ╠═4ff69f0a-6626-4593-b361-a2387eecc731 +# ╠═0f7f228c-4f95-4f1a-9ca2-2d03384a00c0 # ╟─5df0ba53-b528-4baf-9980-cafe5d73f9dd -# ╠═a434645b-d592-4162-a8b4-b4b04cea30a9 -# ╟─38cd9032-27c0-4179-a536-ce59a42ff16a -# ╠═f8415d48-a51d-4845-8425-fd61ed79c06e -# ╟─7f88bf4f-d82c-4e5a-a9eb-93870954c79e -# ╟─53598f8f-9973-4cad-af0c-280f5531bb21 -# ╠═bc971069-08c4-493c-a917-8092493d3233 -# ╠═61a60c9c-36cf-4e86-b697-748b3524d3b4 -# ╠═a5b17c81-b667-4b1c-ab15-ddf1a162683b -# ╠═e41feac7-6de6-4223-8a21-341b85da52c0 -# ╠═1c4db74a-4caa-4ce5-815f-b631365c5129 +# ╠═99ea1c20-ca2c-4d77-bc3b-06814db1d666 +# ╠═d991edb9-2291-42a7-97ff-58c456515505 +# ╠═d17c2a79-3c14-4c50-a4aa-465f3da011a2 +# ╠═069591a3-b89b-4fc6-afba-2145e32852b7 +# ╠═5bef5754-d81b-4160-8ed6-396d02853d9a # ╟─00000000-0000-0000-0000-000000000001 # ╟─00000000-0000-0000-0000-000000000002 From 661270e523b8e572311b995e593205272cf99215 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Mon, 30 Sep 2024 11:59:44 +0200 Subject: [PATCH 10/13] Better way to use rightmost-heuristic. Improvements on tutorial text. --- docs/src/tutorials/advanced_search.jl | 193 ++++++++++++-------------- 1 file changed, 86 insertions(+), 107 deletions(-) diff --git a/docs/src/tutorials/advanced_search.jl b/docs/src/tutorials/advanced_search.jl index e9fe327..3c42a59 100644 --- a/docs/src/tutorials/advanced_search.jl +++ b/docs/src/tutorials/advanced_search.jl @@ -8,12 +8,9 @@ using InteractiveUtils using PlutoUI # ╔═╡ c4441fa4-09ec-4b81-9681-b13b93a9c9c0 -using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret, HerbConstraints +using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret, HerbConstraints, HerbCore -# ╔═╡ aa45e602-6219-44ea-82a0-b97639f6a450 -using HerbCore - # ╔═╡ dddca175-3d88-45ce-90da-575c0ba38175 md""" # Advanced Search Procedures in Herb.jl @@ -183,7 +180,7 @@ To explore `BFSIterator`, we define another very simple grammar. """ # ╔═╡ 3af650d9-19c6-4351-920d-d2361091f628 -begin g3 = @cfgrammar begin +begin g_3 = @cfgrammar begin Real = 1 | 2 Real = Real * Real end @@ -195,7 +192,7 @@ Next, we define a `BFSIterator` with a `max_depth` of 2 and a `max_size` of infi """ # ╔═╡ f2521a57-267e-4b49-9179-4e9c2e6bdec7 -iteratorbfs = BFSIterator(g3, :Real, max_depth=2, max_size=typemax(Int)) +iterator_bfs = BFSIterator(g_3, :Real, max_depth=2, max_size=typemax(Int)) # ╔═╡ bf038215-1ecf-4e1c-a9be-e133e4497293 md""" @@ -203,10 +200,10 @@ To see all possible solution programs the iterator explores, we use `collect`. I """ # ╔═╡ 6aec7358-225a-4764-9a36-da86234b6cf8 -programsbfs = collect(iteratorbfs) +programs_bfs = collect(iterator_bfs) # ╔═╡ 54ecf6b9-3341-49e0-92e9-71190e06d61b -println(programsbfs) +println(programs_bfs) # ╔═╡ d3ff497e-d2c2-4df6-8e4c-cdca70fd0677 md""" @@ -224,78 +221,45 @@ answer_programs = [ ] # ╔═╡ 9efb01cf-b190-4e3e-aa19-11499ba46489 -println(all(p ∈ programsbfs for p ∈ answer_programs)) +println(all(p ∈ programs_bfs for p ∈ answer_programs)) # ╔═╡ 0020b79a-6352-4e2d-93f6-2a1d7b03ae2c md""" #### Depth-First Search -The `DFSIterator` looks at one branch of the search tree at a time, exploring it unitl a correct program is found or the specified `max_depth` is reached. Only then, the next branch is considered. +The `DFSIterator` explores one branch of the search tree at a time, fully traversing it unitl a correct program is found or the specified `max_depth` is reached. Only after completing the current branch, it proceeds to the next branch. -Let's `collect` the the trees/programs (TODO), using the same grammar but this time `DFSIterator`. +As before, we `collect` the candidate programs using the same grammar, but a `DFSIterator`. """ -# ╔═╡ 789150a8-862c-48c3-88b8-710b81ab34cf -begin - g1 = @cfgrammar begin - Real = 1 | 2 - Real = Real * Real - end - -end - # ╔═╡ db5be2c3-0b36-40b4-bf14-20e2c7063ad7 -iteratordfs = DFSIterator(g3, :Real, max_depth=2, max_size=typemax(Int)) +iterator_dfs = DFSIterator(g_3, :Real, max_depth=2, max_size=typemax(Int)) # ╔═╡ 4048ff37-e7d1-44ee-bfa3-aa058b6f53b6 -programsdfs = collect(iteratordfs) +programs_dfs = collect(iterator_dfs) # ╔═╡ 658c55ac-88c8-4657-a8eb-c9c9b91d0ded -println(programsdfs) +println(programs_dfs) # ╔═╡ 243165be-a9d2-484d-8046-811a2b0ba139 md""" -`DFSIterator` also uses by default a **leftmost-first** heuristic. If we want to use a **rightmost-first** heuristic instead, we can create our own iterator `DFSIteratorRightmost`. Also see the tutorial [Top Down Iterator](https://herb-ai.github.io/Herb.jl/dev/tutorials/TopDown/) for how to build iterators is Herb.jl. - - - - -TODO: How to use rightmost heuristic? +`DFSIterator` also uses by default a **leftmost-first** heuristic. If we want to use a **rightmost-first** heuristic instead, we can create our own iterator `DFSIteratorRightmost` as a sub-type of `TopDownIterator`, using the `@programiterator` macro. Then we implement the functions `priority_function` and `hole_heuristic`. Also see the tutorial [Top Down Iterator](https://herb-ai.github.io/Herb.jl/dev/tutorials/TopDown/) for how to build iterators is Herb.jl. """ -# ╔═╡ 3f18989a-ae25-4a66-9930-4459407af695 -# to modify the heuristics, we create a new type -# abstract type DFSIteratorRightmost <: TopDownIterator end - # ╔═╡ 4b97602a-5226-429f-86ea-8ecac3c807fa @programiterator DFSIteratorRightmost() <: TopDownIterator # ╔═╡ ed198b79-1b95-4531-b148-c1037cfdacf4 md""" -By default, `priority_function` for a `TopDownIterator` is that of a BFS iterator. Hence, we need to overwrite it: +By default, `priority_function` for a `TopDownIterator` is that of a BFS iterator. Hence, we need to provide a new implementation. For this, we can make use of the existing `priority_function` of `DFSIterator`. """ -# ╔═╡ 3c7f6e75-ea70-4ccd-8ad6-5876d50a587d -function priority_function( - ::DFSIteratorRightmost, - ::AbstractGrammar, - ::AbstractRuleNode, - parent_value::Union{Real, Tuple{Vararg{Real}}}, - isrequeued::Bool -) - if isrequeued - return parent_value; - end - return parent_value - 1; -end - # ╔═╡ d62918e4-4113-45bd-81ea-d17d007b83f5 -# Alternative priority_function(::DFSIteratorRightmost) = priority_function(DFSIterator()) # ╔═╡ 7480d1e4-e417-4d87-80b7-513a098da70e md""" -Next, we need to overwrite the `hole_heuristic` to rightmost-first. +Next, we need to implement the `hole_heuristic` to be rightmost-first. """ # ╔═╡ 7e2af72d-b71c-4234-9bca-cb9a90732a91 @@ -304,24 +268,41 @@ function hole_heuristic(::DFSIteratorRightmost, node::AbstractRuleNode, max_dept end # ╔═╡ 00d05a7e-ca79-4d6b-828d-b24ef1cb80a2 -iteratordfs_rightmost = DFSIteratorRightmost(g3, :Real, max_depth=2, max_size=typemax(Int)) +iteratordfs_rightmost = DFSIteratorRightmost(g_3, :Real, max_depth=2, max_size=typemax(Int)) + +# ╔═╡ e0e8042d-ae41-4046-ab4f-5954a0d1cfb7 +programs_dfs_rightmost = collect(iteratordfs_rightmost) # ╔═╡ c70eb1a1-7d6b-413f-bd85-77e1b8c30b94 -println(collect(iteratordfs_rightmost)) +println(programs_dfs_rightmost) + +# ╔═╡ 02010940-df9f-4847-b0be-0bc9c6bb2ad4 +md""" +We observe that the order of programs has changed. We can also test if both DFS iterators return the same programs: +""" + +# ╔═╡ f5edcb4d-da72-4eeb-b55e-0ace1697133a +Set(programs_dfs)==Set(programs_dfs_rightmost) # ╔═╡ 168c71bf-ce5b-4ab3-b29a-5996981c42a5 md""" ## Stochastic search -Whereas deterministic search methods explore the search space in a predictable way, stochastic ones use randomness. +While deterministic search methods explore the search space in a predictable way, stochastic ones introduce randomness to allow for more flexibility. + +In this section, we will look at the stochastic search algorithms: Metropolis-Hastings (MH), Very Large Scale Neighbourhood Search (VLSNS), and Simulated Annealing (SA). In Herb.jl, all of these search methodsthe share a common supertype `StochasticSearchIterator`, which defines the following fields + - `examples` + - `cost_function` + - `initial_temperature` + - `evaluation_function`. -We will look at several stochastic search algorithms, usinga a new example. +They are customized by overriding the functions `neighbourhood`, `propose`, `accept` and `temperature` as required. We start with a simple grammar and a helper function to create the input-output examples for the problem we want to solve. """ # ╔═╡ a4b522cf-78f0-4d44-88c8-82dd0cdbf952 -grammar = @csgrammar begin +g_4 = @csgrammar begin X = |(1:5) X = X * X X = X + X @@ -335,14 +316,23 @@ function create_problem(f, range=20) return Problem(examples), examples end +# ╔═╡ 8e75ec35-94dc-4292-ab36-83731b3b9318 +md""" +Throughout the stochastic search examples, we will use mean-squared-error as cost function. The cost function helps to guide the search by evaluating how well a candidate program solves the given task. This is used to decide whether a proposed program should be accepted or rejected. +""" + +# ╔═╡ eebb9554-b657-4be3-aecf-c0869a2b38fa +cost_function = mean_squared_error + # ╔═╡ 0da9053a-959b-471e-8918-662ec63da71c md""" ### Metropolis-Hastings -Metropolis-Hastings (MH) is a method to produce samples from a distribution that may otherwise be difficult to sample. In the context of program synthesis, we want samples from a distribution of programs, based on the grammar. +Metropolis-Hastings (MH) is a method to produce samples from a distribution that may otherwise be difficult to sample. In the context of program synthesis, we sample from a distribution of programs defined by the grammar. + For more information on MH, see for example [this webpage](https://stephens999.github.io/fiveMinuteStats/MH_intro.html). -The example below uses a simple arithmetic example. Try run the code block (TODO) multiple times. You will get different programs every time, as the search is stochastic. +To illustrate MH, we use a simple arithmetic example. """ # ╔═╡ 4df84c71-99b5-487f-9db5-c048c0c74151 @@ -351,14 +341,16 @@ e_mh = x -> x * x + 4 # ╔═╡ afe1872c-6641-4fa0-a53e-50c6b4a712ee problem_mh, examples_mh = create_problem(e_mh) -# ╔═╡ 969db94c-d583-40d1-a058-097f8117c2e9 -cost_function = mean_squared_error - +# ╔═╡ 69e91ae9-8475-47dd-826e-8c229faa11e8 +md""" +Run the following code block to define the iterator and perform the program synthesis multiple times. Since the search process is stochastic, you will likely see different solution programs with each run. +""" # ╔═╡ 0a30fb40-cd45-4661-a501-ae8e45f1e07e begin - iteratormh = MHSearchIterator(grammar, :X, examples_mh, cost_function, max_depth=3) #TODO: What should max_depth be? + iteratormh = MHSearchIterator(g_4, :X, examples_mh, cost_function, max_depth=3) programmh = synth(problem_mh, iteratormh) + println("Sollution using MH: ", programmh) end # ╔═╡ 82f7ffa5-2bdc-4153-85fa-d7aca063da12 @@ -368,43 +360,29 @@ programmh md""" ### Very Large Scale Neighbourhood Search -The second stochastic search method we consider is Very Large Scale Neighbourhood Search (VLSN). In each iteration, the neighbourhood of the current candidate program is searched for the local optimum to find a better candidate. - -TODO: more? +TODO: what do we want to show with the examples? +The second stochastic search method we consider is Very Large Scale Neighbourhood Search (VLSN). In each iteration, the algorithm searches the neighbourhood of the current candidate program for a local optimum, aiming to find a better candidate solution. For more information, see [this article](https://backend.orbit.dtu.dk/ws/portalfiles/portal/5293785/Pisinger.pdf). Given the same grammar as before, we can try it with some simple examples. - - - -Using -large neighborhoods makes it possible to find better candidate solutions in each it- -eration and hence traverse a more promising search path - -#### Example I """ # ╔═╡ e6e5c63b-34e8-40d6-bc12-bd31f40b4b16 e_vlsn = x -> 10 # ╔═╡ 2397f65f-e6b4-4f11-bf66-83440c58b688 -problem_vlsn, examples_vlsn = create_problem(e_vlsn) +problem_vlsn1, examples_vlsn1 = create_problem(e_vlsn) # ╔═╡ 7c738d7b-bf05-40c7-b3b7-1512fbae7299 -iteratorvlsn = VLSNSearchIterator(grammar, :X, examples_vlsn, cost_function, max_depth=2) +iterator_vlsn1 = VLSNSearchIterator(g_4, :X, examples_vlsn1, cost_function, max_depth=2) # ╔═╡ 33af905e-e8ca-425d-9805-eb02bec7c26b -programvlsn = synth(problem_vlsn, iteratorvlsn) -# program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth) +program_vlsn1 = synth(problem_vlsn1, iterator_vlsn1) -# ╔═╡ 4208cd17-6c2a-4480-a535-5a7358e0ab25 -md""" -#### Example 2 - -TODO: What do we want to show with this example? -""" +# ╔═╡ 7cbaa721-2bab-4a27-8c18-83f3f81bb336 +println("Solution: ", program_vlsn1) # ╔═╡ bea28b36-6613-4895-98f9-27dfd9e57b09 e_vlsn2 = x -> x @@ -413,16 +391,21 @@ e_vlsn2 = x -> x problem_vlsn2, examples_vlsn2 = create_problem(e_vlsn2) # ╔═╡ 285043ef-c295-400f-91c5-f3c6c69ac2bf -iterator_vlsn2 = VLSNSearchIterator(grammar, :X, examples_vlsn2, cost_function, max_depth=1) +iterator_vlsn2 = VLSNSearchIterator(g_4, :X, examples_vlsn2, cost_function, max_depth=1) # ╔═╡ 36f0e0cf-c871-42c9-956e-054767cbf693 program_vlsn2 = synth(problem_vlsn2, iterator_vlsn2) +# ╔═╡ bff8026e-49ee-4200-ae86-7687767389a5 +println("Solution: ", program_vlsn2) + # ╔═╡ 599194a8-3f47-4917-9143-a5fe0d43029f md""" ### Simulated Annealing -Simualted Annealing (SA) explores smaller, incremental changes in the candidate program per iteration and refines the solution over time. It is a modified hill-climbing algorithm that, instead of picking the best move (program modification?), picks a random move. If the move improves the solution (candidate program), it is always accepted. Ocassionally, a move towards a worse solution is accepted to escape loca optima. However, this strategy follows a cooling (annealing) schedule, i.e., it starts exploring widely (at high temperature) and gradually becomes more selective (at low temperture), accepting worse solutions less often. +Simulated Annealing (SA) explores smaller, incremental changes to the candidate program in each iteration, gradually refining the solution. It is a variation of the hill-climbing algorithm: Instead of always selecting the best move, SA picks a random move. If the move improves the solution (i.e., the candidate program), it is accepted. + +Occasionally, SA will accept a move that worsens the solution. This allows the algorithm to escape local optima and explore more of the solution space. However, this strategy follows a cooling (annealing) schedule: at the beginning (high temperature), the algorithm explores more broadly and is more likely to accept worse solutions. As the temperature decreases, it becomes more selective, accepting worse solutions less often. For more information, see [this page](https://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/anneal.html). """ @@ -432,17 +415,14 @@ md""" We use the same example as for MH. SA additionally has the option to specify the `initial_temperature` for the annealing (default `initial_temperature=1`). Let's see what effect changing the temperature from 1 to 2 has on the solution program. """ -# ╔═╡ f6a3ed3f-bcf9-4f22-a86e-33255d88e0b2 -e_sa = x -> x * x + 4 - # ╔═╡ e25d115f-7549-4606-b96c-9ef700810f7b -problem_sa, examples_sa = create_problem(e_sa) +problem_sa, examples_sa = create_problem(e_mh) # ╔═╡ 94f2bd5e-e11e-42e7-9a3e-3c9d5ae43cd4 initial_temperature1 = 1 # ╔═╡ eb851d7b-803e-45f6-ad10-fa0bde78826a -iterator_sa1 = SASearchIterator(grammar, :X, examples_sa, cost_function, max_depth=3, initial_temperature = initial_temperature1) +iterator_sa1 = SASearchIterator(g_4, :X, examples_sa, cost_function, max_depth=3, initial_temperature = initial_temperature1) # ╔═╡ 73304e3f-05bf-4f0c-9acd-fc8afa87b460 program_sa1 = synth(problem_sa, iterator_sa1) @@ -451,14 +431,16 @@ program_sa1 = synth(problem_sa, iterator_sa1) initial_temperature2 = 2 # ╔═╡ 4ff69f0a-6626-4593-b361-a2387eecc731 -iterator_sa2 = SASearchIterator(grammar, :X, examples_sa, cost_function, max_depth=3, initial_temperature = initial_temperature2) +iterator_sa2 = SASearchIterator(g_4, :X, examples_sa, cost_function, max_depth=3, initial_temperature = initial_temperature2) # ╔═╡ 0f7f228c-4f95-4f1a-9ca2-2d03384a00c0 # ╔═╡ 5df0ba53-b528-4baf-9980-cafe5d73f9dd md""" -### Genetic Search +## Genetic Search + +TODO: talk a bit about the iterator. Genetic search is a type of evolutionary algorithm, which simulates the process of natural selection. It evolves a population of candidate programs through operations like mutation, crossover (recombination), and selection. Then, the fitness of each program is assessed (i.e., how well it satisfies the given specifications). Only the 'fittest' programs are selected for the next generation, thus gradually refining the population of candidate programs. @@ -473,11 +455,8 @@ e_gs = x -> 3 * x * x + (x + 2) # ╔═╡ d991edb9-2291-42a7-97ff-58c456515505 problem_gs, examples_gs = create_problem(e_gs) -# ╔═╡ d17c2a79-3c14-4c50-a4aa-465f3da011a2 - - # ╔═╡ 069591a3-b89b-4fc6-afba-2145e32852b7 -iterator_gs = GeneticSearchIterator(grammar, :X, examples_gs, population_size = 10, mutation_probability = 0.8, maximum_initial_population_depth = 3) +iterator_gs = GeneticSearchIterator(g_4, :X, examples_gs, population_size = 10, mutation_probability = 0.8, maximum_initial_population_depth = 3) # ╔═╡ 5bef5754-d81b-4160-8ed6-396d02853d9a program_gs, error_gs = synth(problem_gs, iterator_gs) @@ -1273,29 +1252,30 @@ version = "0.13.1+0" # ╟─d3ff497e-d2c2-4df6-8e4c-cdca70fd0677 # ╠═07b54acf-0c0d-40ac-ae18-fb26094b4aca # ╠═9efb01cf-b190-4e3e-aa19-11499ba46489 -# ╠═0020b79a-6352-4e2d-93f6-2a1d7b03ae2c -# ╠═789150a8-862c-48c3-88b8-710b81ab34cf +# ╟─0020b79a-6352-4e2d-93f6-2a1d7b03ae2c # ╠═db5be2c3-0b36-40b4-bf14-20e2c7063ad7 # ╠═4048ff37-e7d1-44ee-bfa3-aa058b6f53b6 # ╠═658c55ac-88c8-4657-a8eb-c9c9b91d0ded -# ╠═243165be-a9d2-484d-8046-811a2b0ba139 -# ╠═aa45e602-6219-44ea-82a0-b97639f6a450 -# ╠═3f18989a-ae25-4a66-9930-4459407af695 +# ╟─243165be-a9d2-484d-8046-811a2b0ba139 # ╠═4b97602a-5226-429f-86ea-8ecac3c807fa # ╟─ed198b79-1b95-4531-b148-c1037cfdacf4 -# ╠═3c7f6e75-ea70-4ccd-8ad6-5876d50a587d # ╠═d62918e4-4113-45bd-81ea-d17d007b83f5 # ╟─7480d1e4-e417-4d87-80b7-513a098da70e # ╠═7e2af72d-b71c-4234-9bca-cb9a90732a91 # ╠═00d05a7e-ca79-4d6b-828d-b24ef1cb80a2 +# ╠═e0e8042d-ae41-4046-ab4f-5954a0d1cfb7 # ╠═c70eb1a1-7d6b-413f-bd85-77e1b8c30b94 -# ╠═168c71bf-ce5b-4ab3-b29a-5996981c42a5 +# ╟─02010940-df9f-4847-b0be-0bc9c6bb2ad4 +# ╠═f5edcb4d-da72-4eeb-b55e-0ace1697133a +# ╟─168c71bf-ce5b-4ab3-b29a-5996981c42a5 # ╠═a4b522cf-78f0-4d44-88c8-82dd0cdbf952 # ╠═f313edb9-8fd9-4d78-88cd-89226f5c769d +# ╟─8e75ec35-94dc-4292-ab36-83731b3b9318 +# ╠═eebb9554-b657-4be3-aecf-c0869a2b38fa # ╟─0da9053a-959b-471e-8918-662ec63da71c # ╠═4df84c71-99b5-487f-9db5-c048c0c74151 # ╠═afe1872c-6641-4fa0-a53e-50c6b4a712ee -# ╠═969db94c-d583-40d1-a058-097f8117c2e9 +# ╟─69e91ae9-8475-47dd-826e-8c229faa11e8 # ╠═0a30fb40-cd45-4661-a501-ae8e45f1e07e # ╠═82f7ffa5-2bdc-4153-85fa-d7aca063da12 # ╟─700270ea-90bd-474b-91d9-0e5ed329776a @@ -1303,14 +1283,14 @@ version = "0.13.1+0" # ╠═2397f65f-e6b4-4f11-bf66-83440c58b688 # ╠═7c738d7b-bf05-40c7-b3b7-1512fbae7299 # ╠═33af905e-e8ca-425d-9805-eb02bec7c26b -# ╟─4208cd17-6c2a-4480-a535-5a7358e0ab25 +# ╠═7cbaa721-2bab-4a27-8c18-83f3f81bb336 # ╠═bea28b36-6613-4895-98f9-27dfd9e57b09 # ╠═aa95cb5e-926d-4119-8d08-353f37a59039 # ╠═285043ef-c295-400f-91c5-f3c6c69ac2bf # ╠═36f0e0cf-c871-42c9-956e-054767cbf693 +# ╠═bff8026e-49ee-4200-ae86-7687767389a5 # ╟─599194a8-3f47-4917-9143-a5fe0d43029f -# ╠═dd6aee87-cd96-4be1-b8fb-03fffee5ea43 -# ╠═f6a3ed3f-bcf9-4f22-a86e-33255d88e0b2 +# ╟─dd6aee87-cd96-4be1-b8fb-03fffee5ea43 # ╠═e25d115f-7549-4606-b96c-9ef700810f7b # ╠═94f2bd5e-e11e-42e7-9a3e-3c9d5ae43cd4 # ╠═eb851d7b-803e-45f6-ad10-fa0bde78826a @@ -1321,7 +1301,6 @@ version = "0.13.1+0" # ╟─5df0ba53-b528-4baf-9980-cafe5d73f9dd # ╠═99ea1c20-ca2c-4d77-bc3b-06814db1d666 # ╠═d991edb9-2291-42a7-97ff-58c456515505 -# ╠═d17c2a79-3c14-4c50-a4aa-465f3da011a2 # ╠═069591a3-b89b-4fc6-afba-2145e32852b7 # ╠═5bef5754-d81b-4160-8ed6-396d02853d9a # ╟─00000000-0000-0000-0000-000000000001 From 4e9ebd3ab955df97fe160c33c43c20fc64a094fc Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Mon, 30 Sep 2024 12:32:24 +0200 Subject: [PATCH 11/13] Deletes md/notbook files of tutorials that are replaced by Pluto notebooks. Update htmls for docs. --- docs/src/tutorials/abstract_syntax_trees.html | 18 + .../src/tutorials/abstract_syntax_trees.ipynb | 750 ------------------ docs/src/tutorials/abstract_syntax_trees.md | 435 ---------- docs/src/tutorials/advanced_search.ipynb | 616 -------------- docs/src/tutorials/advanced_search.md | 403 ---------- docs/src/tutorials/defining_grammars.html | 17 + docs/src/tutorials/defining_grammars.ipynb | 531 ------------- docs/src/tutorials/defining_grammars.md | 590 -------------- .../getting_started_with_constraints.html | 4 +- .../getting_started_with_constraints.ipynb | 481 ----------- .../getting_started_with_constraints.md | 738 ----------------- .../tutorials/getting_started_with_herb.html | 4 +- .../tutorials/getting_started_with_herb.ipynb | 346 -------- .../tutorials/getting_started_with_herb.md | 180 ----- .../tutorials/working_with_interpreters.html | 4 +- .../tutorials/working_with_interpreters.ipynb | 271 ------- .../tutorials/working_with_interpreters.md | 141 ---- 17 files changed, 41 insertions(+), 5488 deletions(-) create mode 100644 docs/src/tutorials/abstract_syntax_trees.html delete mode 100644 docs/src/tutorials/abstract_syntax_trees.ipynb delete mode 100644 docs/src/tutorials/abstract_syntax_trees.md delete mode 100644 docs/src/tutorials/advanced_search.ipynb delete mode 100644 docs/src/tutorials/advanced_search.md create mode 100644 docs/src/tutorials/defining_grammars.html delete mode 100644 docs/src/tutorials/defining_grammars.ipynb delete mode 100644 docs/src/tutorials/defining_grammars.md delete mode 100644 docs/src/tutorials/getting_started_with_constraints.ipynb delete mode 100644 docs/src/tutorials/getting_started_with_constraints.md delete mode 100644 docs/src/tutorials/getting_started_with_herb.ipynb delete mode 100644 docs/src/tutorials/getting_started_with_herb.md delete mode 100644 docs/src/tutorials/working_with_interpreters.ipynb delete mode 100644 docs/src/tutorials/working_with_interpreters.md diff --git a/docs/src/tutorials/abstract_syntax_trees.html b/docs/src/tutorials/abstract_syntax_trees.html new file mode 100644 index 0000000..f002fb3 --- /dev/null +++ b/docs/src/tutorials/abstract_syntax_trees.html @@ -0,0 +1,18 @@ + + + + + + + + +
\ No newline at end of file diff --git a/docs/src/tutorials/abstract_syntax_trees.ipynb b/docs/src/tutorials/abstract_syntax_trees.ipynb deleted file mode 100644 index 53be308..0000000 --- a/docs/src/tutorials/abstract_syntax_trees.ipynb +++ /dev/null @@ -1,750 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "adb6401d-97cf-4926-86d1-d93cd888cced", - "metadata": {}, - "source": [ - "# Herb tutorial: Abstract syntax trees" - ] - }, - { - "cell_type": "markdown", - "id": "6aab945c-b81f-43f3-bcb3-36521ed3a50e", - "metadata": {}, - "source": [ - "In this tutorial, you will learn\n", - " \n", - "- How to represent a computer program as an abstract syntax tree in Herb.\n", - "- How to replace parts of the tree to modify the program." - ] - }, - { - "cell_type": "markdown", - "id": "56460a31-bfdd-4c33-8a2b-a0db1430477e", - "metadata": {}, - "source": [ - "## Abstract syntax trees\n", - "\n", - "The syntactic structure of a computer program can be represented in a hierarchical tree structure, a so-called _Abstract Syntax Tree (AST)_. The syntax of a programming language is typically defined using a formal grammar, a set of rules on how valid programs can be constructed. ASTs are derived from the grammar, but are abstractions in the sense that they omit details such as parenthesis, semicolons, etc. and only retain what's necessary to capture the program structure. \n", - "\n", - "In the context of program synthesis, ASTs are often used to define the space of all possible programs which is searched to find one that satisfies the given specifications. During the search process, different ASTs, each corresponding to a different program, are generated and evaluated until a suitable one is found.\n", - "\n", - "Each _node_ of the AST represents a construct in the program (e.g., a variable, an operator, a statement, or a function) and this construct corresponds to a rule in the formal grammar. \n", - "An _edge_ describes the relationship between constructs, and the tree structure captures the nesting of constructs. " - ] - }, - { - "cell_type": "markdown", - "id": "3f28908e-adcf-47ff-b369-556bc6e2dae4", - "metadata": {}, - "source": [ - "## A simple example program\n", - "\n", - "We first consider the simple program 5*(x+3). We will define a grammar that is sufficient to represent this program and use it to construct a AST for our program." - ] - }, - { - "cell_type": "markdown", - "id": "4d6961c8-2fd5-4cd9-b4a1-166d46500aaa", - "metadata": {}, - "source": [ - "### Define the grammar" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "69d543a7-3067-4376-b1bb-1a4f9217b786", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "1: Number = 0\n", - "2: Number = 1\n", - "3: Number = 2\n", - "4: Number = 3\n", - "5: Number = 4\n", - "6: Number = 5\n", - "7: Number = 6\n", - "8: Number = 7\n", - "9: Number = 8\n", - "10: Number = 9\n", - "11: Number = x\n", - "12: Number = Number + Number\n", - "13: Number = Number * Number\n" - ] - }, - "execution_count": 1, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "using HerbCore, HerbGrammar, HerbInterpret\n", - "\n", - "grammar = @csgrammar begin\n", - " Number = |(0:9)\n", - " Number = x\n", - " Number = Number + Number\n", - " Number = Number * Number\n", - "end\n" - ] - }, - { - "cell_type": "markdown", - "id": "8de36f71-5051-4d1b-b725-c2a94fbe9119", - "metadata": {}, - "source": [ - "### Construct the syntax tree" - ] - }, - { - "cell_type": "markdown", - "id": "33b9e13e-f444-496f-8931-56a87887ae9c", - "metadata": {}, - "source": [ - "The AST of this program is shown in the diagram below. The number in each node refers to the index of the corresponding rule in our grammar. " - ] - }, - { - "cell_type": "markdown", - "id": "5578f49b-741b-4f4a-a660-ce5bb4673ea1", - "metadata": {}, - "source": [ - "```mermaid\n", - " flowchart \n", - " id1((13)) ---\n", - " id2((6))\n", - " id1 --- id3((12))\n", - " id4((11))\n", - " id5((4))\n", - " id3 --- id4\n", - " id3 --- id5\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "a41f36e1-ce66-4303-8ad1-d74aeba72a70", - "metadata": {}, - "source": [ - "In `Herb.jl`, the `HerbCore.RuleNode` is used to represent both an individual node, but also entire ASTs or sub-trees. This is achieved by nesting instances of `RuleNode`. A `RuleNode` can be instantiated by providing the index of the grammar rule that the node represents and a vector of child nodes. " - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "9c61d359-1f38-410e-b654-5d36bc9a1d4e", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "13{6,12{11,4}}" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "syntaxtree = RuleNode(13, [RuleNode(6), RuleNode(12, [RuleNode(11), RuleNode(4)])])" - ] - }, - { - "cell_type": "markdown", - "id": "da78a5d9-038e-4d97-ab1a-32efc830c1e8", - "metadata": {}, - "source": [ - "We can confirm that our AST is correct by displaying it in a more human-readable way, using `HerbGrammar.rulenode2expr` and by testing it on a few input examples using `HerbInterpret.execute_on_input`." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "7c933e57-03f8-41ac-a464-70f95b9505c0", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "5 * (x + 3)\n" - ] - } - ], - "source": [ - "program = rulenode2expr(syntaxtree, grammar)\n", - "println(program)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "32fbb714-25aa-40d5-9c0b-c20ba8999d0c", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "65" - ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# test solution on inputs\n", - "output = execute_on_input(grammar, syntaxtree, Dict(:x => 10))" - ] - }, - { - "cell_type": "markdown", - "id": "b724f97d-ae36-455c-94b8-10632beffe6e", - "metadata": {}, - "source": [ - "## Another example: FizzBuzz\n", - "\n", - "Let's look at a more interesting example. \n", - "The program `fizbuzz()` is based on the popular _FizzBuzz_ problem. Given an integer number, the program simply returns a `String` of that number, but replace numbers divisible by 3 with `\"Fizz\"`, numbers divisible by 5 with `\"Buzz\"`, and number divisible by both 3 and 5 with `\"FizzBuzz\"`." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "1f41bec4-d081-48bd-9fee-c30021d4a882", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "fizzbuzz (generic function with 1 method)" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "function fizzbuzz(x)\n", - " if x % 5 == 0 && x % 3 == 0\n", - " return \"FizzBuzz\"\n", - " else\n", - " if x % 3 == 0\n", - " return \"Fizz\"\n", - " else\n", - " if x % 5 == 0\n", - " return \"Buzz\"\n", - " else\n", - " return string(x)\n", - " end\n", - " end\n", - " end\n", - "end" - ] - }, - { - "cell_type": "markdown", - "id": "bc5a248d-87e7-4eda-9a3a-80620da45f9c", - "metadata": {}, - "source": [ - "### Define the grammar\n", - "\n", - "Let's define a grammar with all the rules that we need." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "7b0ea30c-e415-4d7e-ba2f-a1590cd4c8fe", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "1: Int = input1\n", - "2: Int = 0\n", - "3: Int = 3\n", - "4: Int = 5\n", - "5: String = Fizz\n", - "6: String = Buzz\n", - "7: String = FizzBuzz\n", - "8: String = string(Int)\n", - "9: Return = String\n", - "10: Int = Int % Int\n", - "11: Bool = Int == Int\n", - "12: Int = if Bool\n", - " Int\n", - "else\n", - " Int\n", - "end\n", - "13: Bool = Bool && Bool\n" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "grammar_fizzbuzz = @csgrammar begin\n", - " Int = input1\n", - " Int = 0 | 3 | 5\n", - " String = \"Fizz\" | \"Buzz\" | \"FizzBuzz\"\n", - " String = string(Int)\n", - " Return = String\n", - " Int = Int % Int\n", - " Bool = Int == Int\n", - " Int = Bool ? Int : Int\n", - " Bool = Bool && Bool\n", - "end" - ] - }, - { - "cell_type": "markdown", - "id": "f0c06383-ce23-446e-8b9f-b323abacfbcb", - "metadata": {}, - "source": [ - "### Construct the syntax tree" - ] - }, - { - "cell_type": "markdown", - "id": "57fc9e37-d243-49c4-9fc9-7669a3cc7e47", - "metadata": {}, - "source": [ - "Given the grammar, the AST of `fizzbuzz()` looks like this:" - ] - }, - { - "cell_type": "markdown", - "id": "219585f5-a4b2-473a-ad98-b3ceb45e37c0", - "metadata": {}, - "source": [ - "```mermaid\n", - "flowchart \n", - " id1((12)) --- id21((13))\n", - " id1--- id22((9))\n", - " id1--- id23((12))\n", - "\n", - " id21 --- id31((11))\n", - " id21 --- id32((11))\n", - "\n", - " id31 --- id41((10))\n", - " id31 --- id42((2))\n", - "\n", - " id41 --- id51((1))\n", - " id41 --- id52((4))\n", - "\n", - " id32 --- id43((10)) \n", - " id32 --- id44((2))\n", - "\n", - " id43 --- id53((1))\n", - " id43 --- id54((3))\n", - "\n", - " id22 --- id33((7))\n", - " id23 --- id34((11))\n", - "\n", - " id34 --- id45((10))\n", - " id34 --- id46((2))\n", - "\n", - " id45 --- id55((1))\n", - " id45 --- id56((3))\n", - "\n", - " id23 --- id35((9))\n", - " id35 --- id47((5))\n", - "\n", - " id23 --- id36((12))\n", - " id36 --- id48((11))\n", - " id48 --- id57((10))\n", - " id57 --- id61((1))\n", - " id57 --- id62((4))\n", - " id48 --- id58((2))\n", - "\n", - " id36 --- id49((9))\n", - " id49 --- id59((6))\n", - "\n", - " id36 --- id410((9))\n", - " id410 --- id510((8))\n", - " id510 --- id63((1))\n", - " \n", - " \n", - " \n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "ca814e00-95e9-4b18-971d-ca82335bcb98", - "metadata": {}, - "source": [ - "As before, we use nest instanced of `RuleNode` to implement the AST." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "9c897ebc-4a79-45be-b349-c376bffa03df", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "12{13{11{10{1,4}2}11{10{1,3}2}}9{7}12{11{10{1,3}2}9{5}12{11{10{1,4}2}9{6}9{8{1}}}}}" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "fizzbuzz_syntaxtree = RuleNode(12, [\n", - " RuleNode(13, [\n", - " RuleNode(11, [\n", - " RuleNode(10, [\n", - " RuleNode(1),\n", - " RuleNode(4)\n", - " ]),\n", - " RuleNode(2)\n", - " ]),\n", - " RuleNode(11, [\n", - " RuleNode(10, [\n", - " RuleNode(1),\n", - " RuleNode(3)\n", - " ]),\n", - " RuleNode(2)\n", - " ])\n", - " ]),\n", - " RuleNode(9, [\n", - " RuleNode(7)\n", - " \n", - " ]),\n", - " RuleNode(12, [\n", - " RuleNode(11, [\n", - " RuleNode(10, [\n", - " RuleNode(1),\n", - " RuleNode(3),\n", - " ]),\n", - " RuleNode(2)\n", - " ]),\n", - " RuleNode(9, [\n", - " RuleNode(5)\n", - " ]),\n", - " RuleNode(12, [\n", - " RuleNode(11, [\n", - " RuleNode(10, [\n", - " RuleNode(1),\n", - " RuleNode(4)\n", - " ]),\n", - " RuleNode(2)\n", - " ]),\n", - " RuleNode(9, [\n", - " RuleNode(6)\n", - " ]),\n", - " RuleNode(9, [\n", - " RuleNode(8, [\n", - " RuleNode(1)\n", - " ])\n", - " ])\n", - " ])\n", - " ]) \n", - " ])" - ] - }, - { - "cell_type": "markdown", - "id": "ac0f888f-0376-48b2-bc68-cbbde67f650f", - "metadata": {}, - "source": [ - "And we check our syntax tree is correct:" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "5a4ba50a-ccda-4e20-9301-051ee839fdf5", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "if input1 % 5 == 0 && input1 % 3 == 0\n", - " \"FizzBuzz\"\n", - "else\n", - " if input1 % 3 == 0\n", - " \"Fizz\"\n", - " else\n", - " if input1 % 5 == 0\n", - " \"Buzz\"\n", - " else\n", - " string(input1)\n", - " end\n", - " end\n", - "end\n" - ] - } - ], - "source": [ - "program = rulenode2expr(fizzbuzz_syntaxtree, grammar_fizzbuzz)\n", - "println(program)" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "9e0b4ec1-eab1-49b9-be2d-c923098d1850", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "4-element Vector{Any}:\n", - " \"Fizz\"\n", - " \"Buzz\"\n", - " \"FizzBuzz\"\n", - " \"22\"" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# test solution on inputs\n", - "input = [Dict(:input1 => 3), Dict(:input1 => 5), Dict(:input1 =>15), Dict(:input1 => 22)]\n", - "output = execute_on_input(grammar_fizzbuzz, fizzbuzz_syntaxtree, input)" - ] - }, - { - "cell_type": "markdown", - "id": "2cd4b5a5-f715-49e7-a3e2-dbc638af8330", - "metadata": {}, - "source": [ - "### Modify the AST/program\n", - "\n", - "There are several ways to modify an AST and hence, a program. You can\n", - "\n", - "- directly replace a node with `HerbCore.swap_node()`\n", - "- insert a rule node with `insert!`\n", - "\n", - "Let's modify our example such that if the input number is divisible by 3, the program returns \"Buzz\" instead of \"Fizz\". \n", - "We use `swap_node()` to replace the node of the AST that corresponds to rule 5 in the grammar (`String = Fizz`) with rule 6 (`String = Buzz`). To do so, `swap_node()` needs the tree that contains the node we want to modify, the new node we want to replace the node with, and the path to that node.\n", - "\n", - "Note that `swap_node()` modifies the tree, hence we make a deep copy of it first." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "65bfb9cc-bbdc-46e5-8ebf-9c15c6424339", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "6," - ] - }, - "execution_count": 10, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "modified_fizzbuzz_syntaxtree = deepcopy(fizzbuzz_syntaxtree)\n", - "newnode = RuleNode(6)\n", - "path = [3, 2, 1]\n", - "swap_node(modified_fizzbuzz_syntaxtree, newnode, path)" - ] - }, - { - "cell_type": "markdown", - "id": "9bc85789-00de-4f54-aa80-1e2ab789750e", - "metadata": {}, - "source": [ - "Let's confirm that we modified the AST, and hence the program, correctly:" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "742f7ee8-c088-4331-b17c-dc5a2078ca6b", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "if input1 % 5 == 0 && input1 % 3 == 0\n", - " \"FizzBuzz\"\n", - "else\n", - " if input1 % 3 == 0\n", - " \"Buzz\"\n", - " else\n", - " if input1 % 5 == 0\n", - " \"Buzz\"\n", - " else\n", - " string(input1)\n", - " end\n", - " end\n", - "end\n" - ] - } - ], - "source": [ - "program = rulenode2expr(modified_fizzbuzz_syntaxtree, grammar_fizzbuzz)\n", - "println(program)" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "59d13ab2-7415-4463-89bf-e09ff55a74d8", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "4-element Vector{Any}:\n", - " \"Buzz\"\n", - " \"Buzz\"\n", - " \"FizzBuzz\"\n", - " \"22\"" - ] - }, - "execution_count": 12, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# test solution on same inputs as before\n", - "output = execute_on_input(grammar_fizzbuzz, modified_fizzbuzz_syntaxtree, input)" - ] - }, - { - "cell_type": "markdown", - "id": "550839d5-3219-4097-b015-538d84a4416f", - "metadata": {}, - "source": [ - "An alternative way to modify the AST is by using `insert!()`. This requires to provide the location of the node that we want to as `NodeLoc`. `NodeLoc` points to a node in the tree and consists of the parent and the child index of the node.\n", - "Again, we make a deep copy of the original AST first." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "74d04a45-1697-4bcd-a5fd-adaa4ef455ff", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "6," - ] - }, - "execution_count": 13, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "modified_fizzbuzz_syntaxtree = deepcopy(fizzbuzz_syntaxtree)\n", - "# get the node we want to modify and instantiate a NodeLoc from it.\n", - "node = get_node_at_location(modified_fizzbuzz_syntaxtree, [3, 2, 1])\n", - "nodeloc = NodeLoc(node, 0)\n", - "# replace the node\n", - "insert!(node, nodeloc, newnode)" - ] - }, - { - "cell_type": "markdown", - "id": "9f0bf320-92bd-4e64-b8ed-6ea841a91cc0", - "metadata": {}, - "source": [ - "Again, we check that we modified the program as intended:" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "6f965fe6-f5b1-4059-911c-362f4c1731d6", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "if input1 % 5 == 0 && input1 % 3 == 0\n", - " \"FizzBuzz\"\n", - "else\n", - " if input1 % 3 == 0\n", - " \"Buzz\"\n", - " else\n", - " if input1 % 5 == 0\n", - " \"Buzz\"\n", - " else\n", - " string(input1)\n", - " end\n", - " end\n", - "end\n" - ] - } - ], - "source": [ - "program = rulenode2expr(modified_fizzbuzz_syntaxtree, grammar_fizzbuzz)\n", - "println(program)" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "41395fec-9053-423b-b0fd-7089bb89c2d0", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "4-element Vector{Any}:\n", - " \"Buzz\"\n", - " \"Buzz\"\n", - " \"FizzBuzz\"\n", - " \"22\"" - ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# test on same inputs as before\n", - "output = execute_on_input(grammar_fizzbuzz, modified_fizzbuzz_syntaxtree, input)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Julia 1.10.3", - "language": "julia", - "name": "julia-1.10" - }, - "language_info": { - "file_extension": ".jl", - "mimetype": "application/julia", - "name": "julia", - "version": "1.10.3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/docs/src/tutorials/abstract_syntax_trees.md b/docs/src/tutorials/abstract_syntax_trees.md deleted file mode 100644 index 85b1e31..0000000 --- a/docs/src/tutorials/abstract_syntax_trees.md +++ /dev/null @@ -1,435 +0,0 @@ -# Herb tutorial: Abstract syntax trees - -In this tutorial, you will learn - -- How to represent a computer program as an abstract syntax tree in Herb. -- How to replace parts of the tree to modify the program. - -## Abstract syntax trees - -The syntactic structure of a computer program can be represented in a hierarchical tree structure, a so-called _Abstract Syntax Tree (AST)_. The syntax of a programming language is typically defined using a formal grammar, a set of rules on how valid programs can be constructed. ASTs are derived from the grammar, but are abstractions in the sense that they omit details such as parenthesis, semicolons, etc. and only retain what's necessary to capture the program structure. - -In the context of program synthesis, ASTs are often used to define the space of all possible programs which is searched to find one that satisfies the given specifications. During the search process, different ASTs, each corresponding to a different program, are generated and evaluated until a suitable one is found. - -Each _node_ of the AST represents a construct in the program (e.g., a variable, an operator, a statement, or a function) and this construct corresponds to a rule in the formal grammar. -An _edge_ describes the relationship between constructs, and the tree structure captures the nesting of constructs. - -## A simple example program - -We first consider the simple program 5*(x+3). We will define a grammar that is sufficient to represent this program and use it to construct a AST for our program. - -### Define the grammar - - -```julia -using HerbCore, HerbGrammar, HerbInterpret - -grammar = @csgrammar begin - Number = |(0:9) - Number = x - Number = Number + Number - Number = Number * Number -end - -``` - - - 1: Number = 0 - 2: Number = 1 - 3: Number = 2 - 4: Number = 3 - 5: Number = 4 - 6: Number = 5 - 7: Number = 6 - 8: Number = 7 - 9: Number = 8 - 10: Number = 9 - 11: Number = x - 12: Number = Number + Number - 13: Number = Number * Number - - - -### Construct the syntax tree - -The AST of this program is shown in the diagram below. The number in each node refers to the index of the corresponding rule in our grammar. - -```mermaid - flowchart - id1((13)) --- - id2((6)) - id1 --- id3((12)) - id4((11)) - id5((4)) - id3 --- id4 - id3 --- id5 -``` - -In `Herb.jl`, the `HerbCore.RuleNode` is used to represent both an individual node, but also entire ASTs or sub-trees. This is achieved by nesting instances of `RuleNode`. A `RuleNode` can be instantiated by providing the index of the grammar rule that the node represents and a vector of child nodes. - - -```julia -syntaxtree = RuleNode(13, [RuleNode(6), RuleNode(12, [RuleNode(11), RuleNode(4)])]) -``` - - - 13{6,12{11,4}} - - -We can confirm that our AST is correct by displaying it in a more human-readable way, using `HerbGrammar.rulenode2expr` and by testing it on a few input examples using `HerbInterpret.execute_on_input`. - - -```julia -program = rulenode2expr(syntaxtree, grammar) -println(program) -``` - - 5 * (x + 3) - - - -```julia -# test solution on inputs -output = execute_on_input(grammar, syntaxtree, Dict(:x => 10)) -``` - - - 65 - - -## Another example: FizzBuzz - -Let's look at a more interesting example. -The program `fizbuzz()` is based on the popular _FizzBuzz_ problem. Given an integer number, the program simply returns a `String` of that number, but replace numbers divisible by 3 with `"Fizz"`, numbers divisible by 5 with `"Buzz"`, and number divisible by both 3 and 5 with `"FizzBuzz"`. - - -```julia -function fizzbuzz(x) - if x % 5 == 0 && x % 3 == 0 - return "FizzBuzz" - else - if x % 3 == 0 - return "Fizz" - else - if x % 5 == 0 - return "Buzz" - else - return string(x) - end - end - end -end -``` - - - fizzbuzz (generic function with 1 method) - - -### Define the grammar - -Let's define a grammar with all the rules that we need. - - -```julia -grammar_fizzbuzz = @csgrammar begin - Int = input1 - Int = 0 | 3 | 5 - String = "Fizz" | "Buzz" | "FizzBuzz" - String = string(Int) - Return = String - Int = Int % Int - Bool = Int == Int - Int = Bool ? Int : Int - Bool = Bool && Bool -end -``` - - - 1: Int = input1 - 2: Int = 0 - 3: Int = 3 - 4: Int = 5 - 5: String = Fizz - 6: String = Buzz - 7: String = FizzBuzz - 8: String = string(Int) - 9: Return = String - 10: Int = Int % Int - 11: Bool = Int == Int - 12: Int = if Bool - Int - else - Int - end - 13: Bool = Bool && Bool - - - -### Construct the syntax tree - -Given the grammar, the AST of `fizzbuzz()` looks like this: - -```mermaid -flowchart - id1((12)) --- id21((13)) - id1--- id22((9)) - id1--- id23((12)) - - id21 --- id31((11)) - id21 --- id32((11)) - - id31 --- id41((10)) - id31 --- id42((2)) - - id41 --- id51((1)) - id41 --- id52((4)) - - id32 --- id43((10)) - id32 --- id44((2)) - - id43 --- id53((1)) - id43 --- id54((3)) - - id22 --- id33((7)) - id23 --- id34((11)) - - id34 --- id45((10)) - id34 --- id46((2)) - - id45 --- id55((1)) - id45 --- id56((3)) - - id23 --- id35((9)) - id35 --- id47((5)) - - id23 --- id36((12)) - id36 --- id48((11)) - id48 --- id57((10)) - id57 --- id61((1)) - id57 --- id62((4)) - id48 --- id58((2)) - - id36 --- id49((9)) - id49 --- id59((6)) - - id36 --- id410((9)) - id410 --- id510((8)) - id510 --- id63((1)) - - - -``` - -As before, we use nest instanced of `RuleNode` to implement the AST. - - -```julia -fizzbuzz_syntaxtree = RuleNode(12, [ - RuleNode(13, [ - RuleNode(11, [ - RuleNode(10, [ - RuleNode(1), - RuleNode(4) - ]), - RuleNode(2) - ]), - RuleNode(11, [ - RuleNode(10, [ - RuleNode(1), - RuleNode(3) - ]), - RuleNode(2) - ]) - ]), - RuleNode(9, [ - RuleNode(7) - - ]), - RuleNode(12, [ - RuleNode(11, [ - RuleNode(10, [ - RuleNode(1), - RuleNode(3), - ]), - RuleNode(2) - ]), - RuleNode(9, [ - RuleNode(5) - ]), - RuleNode(12, [ - RuleNode(11, [ - RuleNode(10, [ - RuleNode(1), - RuleNode(4) - ]), - RuleNode(2) - ]), - RuleNode(9, [ - RuleNode(6) - ]), - RuleNode(9, [ - RuleNode(8, [ - RuleNode(1) - ]) - ]) - ]) - ]) - ]) -``` - - - 12{13{11{10{1,4}2}11{10{1,3}2}}9{7}12{11{10{1,3}2}9{5}12{11{10{1,4}2}9{6}9{8{1}}}}} - - -And we check our syntax tree is correct: - - -```julia -program = rulenode2expr(fizzbuzz_syntaxtree, grammar_fizzbuzz) -println(program) -``` - - if input1 % 5 == 0 && input1 % 3 == 0 - "FizzBuzz" - else - if input1 % 3 == 0 - "Fizz" - else - if input1 % 5 == 0 - "Buzz" - else - string(input1) - end - end - end - - - -```julia -# test solution on inputs -input = [Dict(:input1 => 3), Dict(:input1 => 5), Dict(:input1 =>15), Dict(:input1 => 22)] -output = execute_on_input(grammar_fizzbuzz, fizzbuzz_syntaxtree, input) -``` - - - 4-element Vector{Any}: - "Fizz" - "Buzz" - "FizzBuzz" - "22" - - -### Modify the AST/program - -There are several ways to modify an AST and hence, a program. You can - -- directly replace a node with `HerbCore.swap_node()` -- insert a rule node with `insert!` - -Let's modify our example such that if the input number is divisible by 3, the program returns "Buzz" instead of "Fizz". -We use `swap_node()` to replace the node of the AST that corresponds to rule 5 in the grammar (`String = Fizz`) with rule 6 (`String = Buzz`). To do so, `swap_node()` needs the tree that contains the node we want to modify, the new node we want to replace the node with, and the path to that node. - -Note that `swap_node()` modifies the tree, hence we make a deep copy of it first. - - -```julia -modified_fizzbuzz_syntaxtree = deepcopy(fizzbuzz_syntaxtree) -newnode = RuleNode(6) -path = [3, 2, 1] -swap_node(modified_fizzbuzz_syntaxtree, newnode, path) -``` - - - 6, - - -Let's confirm that we modified the AST, and hence the program, correctly: - - -```julia -program = rulenode2expr(modified_fizzbuzz_syntaxtree, grammar_fizzbuzz) -println(program) -``` - - if input1 % 5 == 0 && input1 % 3 == 0 - "FizzBuzz" - else - if input1 % 3 == 0 - "Buzz" - else - if input1 % 5 == 0 - "Buzz" - else - string(input1) - end - end - end - - - -```julia -# test solution on same inputs as before -output = execute_on_input(grammar_fizzbuzz, modified_fizzbuzz_syntaxtree, input) -``` - - - 4-element Vector{Any}: - "Buzz" - "Buzz" - "FizzBuzz" - "22" - - -An alternative way to modify the AST is by using `insert!()`. This requires to provide the location of the node that we want to as `NodeLoc`. `NodeLoc` points to a node in the tree and consists of the parent and the child index of the node. -Again, we make a deep copy of the original AST first. - - -```julia -modified_fizzbuzz_syntaxtree = deepcopy(fizzbuzz_syntaxtree) -# get the node we want to modify and instantiate a NodeLoc from it. -node = get_node_at_location(modified_fizzbuzz_syntaxtree, [3, 2, 1]) -nodeloc = NodeLoc(node, 0) -# replace the node -insert!(node, nodeloc, newnode) -``` - - - 6, - - -Again, we check that we modified the program as intended: - - -```julia -program = rulenode2expr(modified_fizzbuzz_syntaxtree, grammar_fizzbuzz) -println(program) -``` - - if input1 % 5 == 0 && input1 % 3 == 0 - "FizzBuzz" - else - if input1 % 3 == 0 - "Buzz" - else - if input1 % 5 == 0 - "Buzz" - else - string(input1) - end - end - end - - - -```julia -# test on same inputs as before -output = execute_on_input(grammar_fizzbuzz, modified_fizzbuzz_syntaxtree, input) -``` - - - 4-element Vector{Any}: - "Buzz" - "Buzz" - "FizzBuzz" - "22" - diff --git a/docs/src/tutorials/advanced_search.ipynb b/docs/src/tutorials/advanced_search.ipynb deleted file mode 100644 index dc4c248..0000000 --- a/docs/src/tutorials/advanced_search.ipynb +++ /dev/null @@ -1,616 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Advanced Search Procedures in Herb.jl\n", - "\n", - "In this tutorial, we show how to use the search procedure using more advanced methods." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "using Pkg\n", - "Pkg.add([\"HerbGrammar\", \"HerbSpecification\", \"HerbSearch\", \"HerbInterpret\", \"HerbConstraints\"])\n", - "using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret, HerbConstraints" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We start with a simple grammar and a simple problem." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g = @cfgrammar begin\n", - " Number = |(1:2)\n", - " Number = x\n", - " Number = Number + Number\n", - " Number = Number * Number\n", - "end\n", - "\n", - "problem = Problem([IOExample(Dict(:x => x), 2x+1) for x ∈ 1:5])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Parameters\n", - "\n", - "We can use a search strategy, where we can specify different parameters. For example, by setting the `max_depth`, we limit the depth of the search. In the next example, we can see the effect of the depth on the number of allocations considered. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "solution = @time search(g, problem, :Number, max_depth=3)\n", - "println(solution)\n", - "\n", - "solution = @time search(g, problem, :Number, max_depth=6)\n", - "println(solution)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Another parameter to use is `max_enumerations`, which limits the number of programs that can be tested at evaluation. We can see the number of enumerations necessary to solve a simple problem in the next example." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "for i in range(1, 50)\n", - " println(i, \" enumerations\")\n", - " solution = @time search(g, problem, :Number, max_enumerations=i)\n", - " println(solution)\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We see that only when `i >= 24`, there is a result, after that, increasing `i` does not have any effect on the number of allocations. \n", - "\n", - "A final parameter we consider here is `allow_evaluation_errors`, which is `false` by default. When this is set to `true`, the program will still run even when an exception is thrown during evaluation. To see the effect of this, we create a new grammar. We can also retrieve the error together with the solution from the search method." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g = @cfgrammar begin\n", - " Number = 1\n", - " List = []\n", - " Index = List[Number]\n", - "end\n", - "\n", - "problem = Problem([IOExample(Dict(), x) for x ∈ 1:5])\n", - "solution = search(g, problem, :Index, max_depth=2, allow_evaluation_errors=true)\n", - "println(\"solution: \", solution)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There is another search method called `search_best` which returns both the solution and the possible error. The method returns the best program found so far. In this case, we can also see the error (`typemax(Int)`):" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "solution, error = search_best(g, problem, :Index, max_depth=2, allow_evaluation_errors=true)\n", - "println(\"solution: \", solution)\n", - "println(\"error: \", error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Search methods\n", - "\n", - "We now show examples of using different search procedures, which are initialized by passing different enumerators to the search function.\n", - "\n", - "### Breadth-First Search\n", - "\n", - "The breadth-first search will first enumerate all possible programs at the same depth before considering programs with a depth of one more. A tree of the grammar is returned with programs ordered in increasing sizes. We can first `collect` the programs that have a `max-depth` of 2 and a `max_size` of infinite (integer maximum value), where the starting symbol is of type `Real`. This function uses a default heuristic 'left-most first', such that the left-most child in the tree is always explored first." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g1 = @cfgrammar begin\n", - " Real = 1 | 2\n", - " Real = Real * Real\n", - "end\n", - "programs = collect(get_bfs_enumerator(g1, 2, typemax(Int), :Real))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can test that this function returns all and only the correct functions. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "answer_programs = [\n", - " RuleNode(1),\n", - " RuleNode(2),\n", - " RuleNode(3, [RuleNode(1), RuleNode(1)]),\n", - " RuleNode(3, [RuleNode(1), RuleNode(2)]),\n", - " RuleNode(3, [RuleNode(2), RuleNode(1)]),\n", - " RuleNode(3, [RuleNode(2), RuleNode(2)])\n", - "]\n", - "\n", - "println(all(p ∈ programs for p ∈ answer_programs))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Depth-First Search\n", - "\n", - "In depth-first search, we first explore a certain branch of the search tree till the `max_depth` or a correct program is reached before we consider the next branch. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g1 = @cfgrammar begin\n", - "Real = 1 | 2\n", - "Real = Real * Real\n", - "end\n", - "programs = collect(get_dfs_enumerator(g1, 2, typemax(Int), :Real))\n", - "println(programs)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "`get_dfs_enumerator` also has a default left-most heuristic and we consider what the difference is in output. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g1 = @cfgrammar begin\n", - " Real = 1 | 2\n", - " Real = Real * Real\n", - "end\n", - "programs = collect(get_dfs_enumerator(g1, 2, typemax(Int), :Real, heuristic_rightmost))\n", - "println(programs)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Stochastic search\n", - "We now introduce a few stochastic search algorithms, for which we first create a simple grammar and a helper function for problems." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "grammar = @csgrammar begin\n", - " X = |(1:5)\n", - " X = X * X\n", - " X = X + X\n", - " X = X - X\n", - " X = x\n", - "end" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "function create_problem(f, range=20)\n", - " examples = [IOExample(Dict(:x => x), f(x)) for x ∈ 1:range]\n", - " return Problem(examples), examples\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Metropolis-Hastings\n", - "\n", - "One of the stochastic search methods that is implemented is Metropolis-Hastings (MH), which samples from a distribution of programs based on the grammar. For more information on MH, see for example [this webpage](https://stephens999.github.io/fiveMinuteStats/MH_intro.html).\n", - "\n", - "The example below uses a simple arithmetic example. You can try running this code block multiple times, which will give different programs, as the search is stochastic. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "e = x -> x * x + 4\n", - "problem, examples = create_problem(e)\n", - "enumerator = get_mh_enumerator(examples, mean_squared_error)\n", - "program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Very Large Scale Neighbourhood Search \n", - "\n", - "The second implemented stochastic search method is VLSN, which searches for a local optimum in the neighbourhood. For more information, see [this article](https://backend.orbit.dtu.dk/ws/portalfiles/portal/5293785/Pisinger.pdf).\n", - "\n", - "Given the same grammar as before, we can try it with some simple examples." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "e = x -> 10\n", - "max_depth = 2\n", - "problem, examples = create_problem(e)\n", - "enumerator = get_vlsn_enumerator(examples, mean_squared_error, max_depth)\n", - "program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth)\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "e = x -> x\n", - "max_depth = 1\n", - "problem, examples = create_problem(e)\n", - "enumerator = get_vlsn_enumerator(examples, mean_squared_error, max_depth)\n", - "program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Simulated Annealing\n", - "\n", - "The third stochastic search method is called simulated annealing. This is another hill-climbing method to find local optima. For more information, see [this page](https://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/anneal.html).\n", - "\n", - "We try the example from earlier, but now we can additionally define the `initial_temperature` of the algorithm, which is 1 by default. Change the value below to see the effect." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "e = x -> x * x + 4\n", - "initial_temperature = 1\n", - "problem, examples = create_problem(e)\n", - "enumerator = get_sa_enumerator(examples, mean_squared_error, initial_temperature)\n", - "program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3) " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "e = x -> x * x + 4\n", - "initial_temperature = 2\n", - "problem, examples = create_problem(e)\n", - "enumerator = get_sa_enumerator(examples, mean_squared_error, initial_temperature)\n", - "program, cost = @time search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Genetic Search\n", - "\n", - "Genetic search is a type of evolutionary algorithm, which will simulate the process of natural selection and return the 'fittest' program of the population. For more information, see [here](https://www.geeksforgeeks.org/genetic-algorithms/).\n", - "\n", - "We show the example of finding a lambda function. Try varying the parameters of the genetic search to see what happens." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "e = x -> 3 * x * x + (x + 2)\n", - "problem, examples = create_problem(e)\n", - "enumerator = get_genetic_enumerator(examples, \n", - " initial_population_size = 10,\n", - " mutation_probability = 0.8,\n", - " maximum_initial_population_depth = 3,\n", - ")\n", - "program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=nothing, max_time=20) " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Other functionality\n", - "\n", - "Finally, we showcase two other functionalities of HerbSearch, sampling and heuristics.\n", - "\n", - "### Sampling\n", - "Sampling is implemented for the different stochastic search methods.\n", - "\n", - "We consider here a simple grammar, which gives different programs for different search depths." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "grammar = @cfgrammar begin \n", - " A = B | C | F\n", - " F = G\n", - " C = D\n", - " D = E\n", - "end\n", - "\n", - "# A->B (depth 1) or A->F->G (depth 2) or A->C->D->E (depth 3)\n", - "\n", - "# For depth ≤ 1 the only option is A->B\n", - "expression = rand(RuleNode, grammar, :A, 1)\n", - "@assert rulenode2expr(expression, grammar) in [:B,:C,:F]\n", - "\n", - "# For depth ≤ 2 the two options are A->B (depth 1) and A->B->G| A->C->G | A->F->G (depth 2)\n", - "expression = rand(RuleNode, grammar, :A, 2)\n", - "@assert rulenode2expr(expression,grammar) in [:B,:C,:F,:G]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Heuristics" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# More interesting domains & Use of constraints\n", - "In the following examples, we introduce some larger grammars and show that Herb can still efficiently find the correct program." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Expects to return a program equivalent to 1 + (1 - x) = 2 - x\n", - "\n", - "g₁ = @csgrammar begin\n", - " Element = |(1 : 3) # 1 - 3\n", - " Element = Element + Element # 4\n", - " Element = 1 - Element # 5\n", - " Element = x # 6\n", - "end\n", - "\n", - "addconstraint!(g₁, ComesAfter(6, [5]))\n", - "\n", - "examples = [\n", - " IOExample(Dict(:x => 0), 2),\n", - " IOExample(Dict(:x => 1), 1),\n", - " IOExample(Dict(:x => 2), 0)\n", - "]\n", - "problem = Problem(examples)\n", - "solution = search(g₁, problem, :Element, max_depth=3)\n", - "\n", - "@assert test_with_input(SymbolTable(g₁), solution, Dict(:x => -2)) == 4" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Expects to return a program equivalent to 4 + x * (x + 3 + 3) = x^2 + 6x + 4\n", - "\n", - "g₂ = @csgrammar begin\n", - " Element = Element + Element + Element # 1\n", - " Element = Element + Element * Element # 2\n", - " Element = x # 3\n", - " Element = |(3 : 5) # 4\n", - "end\n", - "\n", - "# Restrict .. + x * x\n", - "addconstraint!(g₂, Forbidden(MatchNode(2, [MatchVar(:x), MatchNode(3), MatchNode(3)])))\n", - "# Restrict 4 and 5 in lower level\n", - "addconstraint!(g₂, ForbiddenPath([2, 1, 5]))\n", - "addconstraint!(g₂, ForbiddenPath([2, 1, 6]))\n", - "\n", - "examples = [\n", - " IOExample(Dict(:x => 1), 11)\n", - " IOExample(Dict(:x => 2), 20)\n", - " IOExample(Dict(:x => -1), -1)\n", - "]\n", - "problem = Problem(examples)\n", - "solution = search(g₂, problem, :Element)\n", - "\n", - "@assert test_with_input(SymbolTable(g₂), solution, Dict(:x => 0)) == 4" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Expects to return a program equivalent to (1 - (((1 - x) - 1) - 1)) - 1 = x + 1\n", - "\n", - "g₃ = @csgrammar begin\n", - " Element = |(1 : 20) # 1 - 20\n", - " Element = Element - 1 # 21\n", - " Element = 1 - Element # 22\n", - " Element = x # 23\n", - "end\n", - "\n", - "addconstraint!(g₃, ComesAfter(23, [22, 21]))\n", - "addconstraint!(g₃, ComesAfter(22, [21]))\n", - "\n", - "examples = [\n", - " IOExample(Dict(:x => 1), 2)\n", - " IOExample(Dict(:x => 10), 11)\n", - "]\n", - "problem = Problem(examples)\n", - "solution = search(g₃, problem, :Element)\n", - "\n", - "@assert test_with_input(SymbolTable(g₃), solution, Dict(:x => 0)) == 1\n", - "@assert test_with_input(SymbolTable(g₃), solution, Dict(:x => 100)) == 101" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Expects to return a program equivalent to 18 + 4x\n", - "\n", - "g₄ = @csgrammar begin\n", - " Element = |(0 : 20) # 1 - 20\n", - " Element = Element + Element + Element # 21\n", - " Element = Element + Element * Element # 22\n", - " Element = x # 23\n", - "end\n", - "\n", - "# Enforce ordering on + +\n", - "addconstraint!(g₄, Ordered(\n", - " MatchNode(21, [MatchVar(:x), MatchVar(:y), MatchVar(:z)]),\n", - " [:x, :y, :z]\n", - "))\n", - "\n", - "examples = [\n", - " IOExample(Dict(:x => 1), 22),\n", - " IOExample(Dict(:x => 0), 18),\n", - " IOExample(Dict(:x => -1), 14)\n", - "]\n", - "problem = Problem(examples)\n", - "solution = search(g₄, problem, :Element)\n", - "\n", - "@assert test_with_input(SymbolTable(g₄), solution, Dict(:x => 100)) == 418" - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "metadata": {}, - "outputs": [], - "source": [ - "# Expects to return a program equivalent to (x == 2) ? 1 : (x + 2)\n", - "\n", - "g₅ = @csgrammar begin\n", - " Element = Number # 1\n", - " Element = Bool # 2\n", - "\n", - " Number = |(1 : 3) # 3-5\n", - " \n", - " Number = Number + Number # 6\n", - " Bool = Number ≡ Number # 7\n", - " Number = x # 8\n", - " \n", - " Number = Bool ? Number : Number # 9\n", - " Bool = Bool ? Bool : Bool # 10\n", - "end\n", - "\n", - "# Forbid ? = ?\n", - "addconstraint!(g₅, Forbidden(MatchNode(7, [MatchVar(:x), MatchVar(:x)])))\n", - "# Order =\n", - "addconstraint!(g₅, Ordered(MatchNode(7, [MatchVar(:x), MatchVar(:y)]), [:x, :y]))\n", - "# Order +\n", - "addconstraint!(g₅, Ordered(MatchNode(6, [MatchVar(:x), MatchVar(:y)]), [:x, :y]))\n", - "\n", - "examples = [\n", - " IOExample(Dict(:x => 0), 2)\n", - " IOExample(Dict(:x => 1), 3)\n", - " IOExample(Dict(:x => 2), 1)\n", - "]\n", - "problem = Problem(examples)\n", - "solution = search(g₅, problem, :Element)\n", - "\n", - "@assert test_with_input(SymbolTable(g₅), solution, Dict(:x => 3)) == 5" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Julia 1.9.0", - "language": "julia", - "name": "julia-1.9" - }, - "language_info": { - "file_extension": ".jl", - "mimetype": "application/julia", - "name": "julia", - "version": "1.9.0" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/docs/src/tutorials/advanced_search.md b/docs/src/tutorials/advanced_search.md deleted file mode 100644 index bb10fa1..0000000 --- a/docs/src/tutorials/advanced_search.md +++ /dev/null @@ -1,403 +0,0 @@ -# Getting started - -You can either paste this code into the Julia REPL or run it using the `advanced_search.ipynb` notebook. Alternatively you can copy the code into a file like `get_started.jl`, followed by running `julia get_started.jl`. - -To start, we import the necessary packages. - -```julia -using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret -``` - -We start with the same simple grammar from the main file [get_started.md](../get_started.md). - -```julia -g = @cfgrammar begin - Number = |(1:2) - Number = x - Number = Number + Number - Number = Number * Number -end -``` - -We use a simple problem. - -```julia - problem = Problem([IOExample(Dict(:x => x), 2x+1) for x ∈ 1:5]) -``` - -## Parameters - -We can use a search strategy, where we can specify different parameters. For example, by setting the `max_depth`, we limit the depth of the search. In the next example, we can see the effect of the depth on the number of allocations considered. - -```julia -solution = @time search(g, problem, :Number, max_depth=3) ->>> 0.003284 seconds (50.08 k allocations: 2.504 MiB) -println(solution) ->>> (x + 1) + x - -solution = @time search(g, problem, :Number, max_depth=6) ->>> 0.005696 seconds (115.28 k allocations: 5.910 MiB) -println(solution) ->>> (1 + x) + x -``` - -Another parameter to use is `max_enumerations`, which limits the number of programs that can be tested at evaluation. We can see the number of enumerations necessary to solve a simple problem in the next example. - -```julia -for i in range(1, 50) - println(i, " enumerations") - solution = @time search(g, problem, :Number, max_enumerations=i) - println(solution) -end - ->>> .... ->>> 23 enums: nothing ->>> 0.010223 seconds (117.01 k allocations: 5.935 MiB, 44.23% gc time) ->>> 24 enums: (1 + x) + x ->>> 0.005305 seconds (117.01 k allocations: 5.935 MiB) ->>> 25 enums: (1 + x) + x ->>> 0.005381 seconds (117.01 k allocations: 5.935 MiB) ->>> ... -``` - -We see that only when `i >= 24`, there is a result. After that, increasing `i` does not have any effect on the number of allocations. - -A final parameter we consider here is `allow_evaluation_errors`, which is `false` by default. When this is set to `true`, the program will still run even when an exception is thrown during evaluation. To see the effect of this, we create a new grammar. We can also retrieve the error together with the solution from the search method. - -```julia -g = @cfgrammar begin - Number = 1 - List = [] - Index = List[Number] -end - -problem = Problem([IOExample(Dict(), x) for x ∈ 1:5]) -solution = search(g, problem, :Index, max_depth=2, allow_evaluation_errors=true) -println(solution) ->>> nothing -``` - -There is also another search method called `search_best` which returns both the solution and the possible error. The method returns the best program found so far. In this case, we can also see the error: - -```julia -solution, error = search_best(g, problem, :Index, max_depth=2, allow_evaluation_errors=true) -println(solution) ->>> nothing -println(error) ->>> 9223372036854775807 # or: typemax(Int) -``` - -## Search methods - -We now show examples of using different search procedures. - -### Breadth-First Search - -The breadth-first search will first enumerate all possible programs at the same depth before considering programs with a depth of one more. A tree of the grammar is returned with programs ordered in increasing sizes. We can first `collect` the programs that have a `max-depth` of 2 and a `max_size` of infinite (integer maximum value), where the starting symbol is of type `Real`. This function uses a default heuristic 'left-most first', such that the left-most child in the tree is always explored first. - -```julia -g1 = @cfgrammar begin - Real = 1 | 2 - Real = Real * Real -end -programs = collect(get_bfs_enumerator(g1, 2, typemax(Int), :Real)) -``` - -We can test that this function returns all and only the correct functions. - -```julia -answer_programs = [ - RuleNode(1), - RuleNode(2), - RuleNode(3, [RuleNode(1), RuleNode(1)]), - RuleNode(3, [RuleNode(1), RuleNode(2)]), - RuleNode(3, [RuleNode(2), RuleNode(1)]), - RuleNode(3, [RuleNode(2), RuleNode(2)]) -] - -println(all(p ∈ programs for p ∈ answer_programs)) ->>> true -``` -### Depth-First Search - -In depth-first search, we first explore a certain branch of the search tree till the `max_depth` or a correct program is reached before we consider the next branch. - -```julia -g1 = @cfgrammar begin -Real = 1 | 2 -Real = Real * Real -end -programs = collect(get_dfs_enumerator(g1, 2, typemax(Int), :Real)) -println(programs) ->>> RuleNode[1,, 3{1,1}, 3{1,2}, 3{2,1}, 3{2,2}, 2,] -``` - -`get_dfs_enumerator` also has a default left-most heuristic and we consider what the difference is in output. - - -```julia -g1 = @cfgrammar begin - Real = 1 | 2 - Real = Real * Real -end -programs = collect(get_dfs_enumerator(g1, 2, typemax(Int), :Real, heuristic_rightmost)) -println(programs) ->>> RuleNode[1,, 3{1,1}, 3{2,1}, 3{1,2}, 3{2,2}, 2,] -``` - -## Stochastic search - -For the examples below, we use this grammar and helper function. -```julia -grammar = @csgrammar begin - X = |(1:5) - X = X * X - X = X + X - X = X - X - X = x -end -function create_problem(f, range=20) - examples = [IOExample(Dict(:x => x), f(x)) for x ∈ 1:range] - return Problem(examples), examples -end -``` - -### Metropolis-Hastings - -One of the stochastic search methods that is implemented is Metropolis-Hastings (MH), which samples from a distribution of programs based on the grammar. For more information on MH, see for example [this webpage](https://stephens999.github.io/fiveMinuteStats/MH_intro.html). - - - -The below example uses a simple arithmetic example. As the search method is stochastic, different programs may be returned, as shown below. - -```julia -e = Meta.parse("x -> x * x + 4") -problem, examples = create_problem(eval(e)) -enumerator = get_mh_enumerator(examples, mean_squared_error) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3) - ->>> (:(x * x - (1 - 5)), 0) ->>> (:(4 + x * x), 0) ->>> (:(x * x + 4), 0) -``` - -### Very Large Scale Neighbourhood Search - -The second implemented stochastic search method is VLSN, which search for a local optimum in the neighbourhood. For more information, see [this article](https://backend.orbit.dtu.dk/ws/portalfiles/portal/5293785/Pisinger.pdf). - -Given the same grammar as before, we can try with some simple examples. - -```julia -e = Meta.parse("x -> 10") -max_depth = 2 -problem, examples = create_problem(eval(e)) -enumerator = get_vlsn_enumerator(examples, mean_squared_error, max_depth) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth) ->>> (:(5 + 5), 0) -``` - -```julia -e = Meta.parse("x -> x") -max_depth = 1 -problem, examples = create_problem(eval(e)) -enumerator = get_vlsn_enumerator(examples, mean_squared_error, max_depth) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=max_depth) ->>> (:x, 0) -``` - -### Simulated Annealing - -The third stochastic search method is called simulated annealing. This is another hill-climbing method to find local optima. For more information, see [this page](https://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/anneal.html). - -We try the example from earlier, but now we can additionally define the `initial_temperature` of the algorithm, which is 1 by default. Two possible answers to the program are given as well. - -```julia -e = Meta.parse("x -> x * x + 4") -initial_temperature = 2 -problem, examples = create_problem(eval(e)) -enumerator = get_sa_enumerator(examples, mean_squared_error, initial_temperature) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=3) ->>> (:(4 + x * x), 0) ->>> (:(x * x + (5 - 1)), 0) -``` - -### Genetic Search - -Genetic search is a type of evolutionary algorithm, which will simulate the process of natural selection and return the 'fittest' program of the population. For more information, see [here](https://www.geeksforgeeks.org/genetic-algorithms/). - -We show the example of finding a lambda function. - -```julia -e = x -> 3 * x * x + (x + 2) -problem, examples = create_problem(e) -enumerator = get_genetic_enumerator(examples, - initial_population_size = 10, - mutation_probability = 0.8, - maximum_initial_population_depth = 3, -) -program, cost = search_best(grammar, problem, :X, enumerator=enumerator, error_function=mse_error_function, max_depth=nothing, max_time=20) ->>> (:(((((x - 5) + x) + x * x) + 1) + (((((2 + x * x) + 3) + x * x) + 1) - ((x - x) + x))), 0) ->>> (:(x * 1 + (x * ((x + x) + x * 1) + (1 + 1) * 1)), 0) ->>> (:((((x + x) + x) + 2) * x + ((x - x) + (2 - x))), 0) -``` - -## Other functionality - -Finally, we showcase two other functionalities of HerbSearch, sampling and heuristics. - -### Sampling -Sampling is implemented for the different stochastic search methods. - -We consider here a simple grammar, which gives different programs for different search depths. - -```julia -grammar = @cfgrammar begin - A = B | C | F - F = G - C = D - D = E -end - -# A->B (depth 1) or A->F->G (depth 2) or A->C->D->E (depth 3) - -# For depth ≤ 1 the only option is A->B -expression = rand(RuleNode, grammar, :A, 1) -@assert rulenode2expr(expression, grammar) in [:B,:C,:F] - -# For depth ≤ 2 the two options are A->B (depth 1) and A->B->G| A->C->G | A->F->G (depth 2) -expression = rand(RuleNode, grammar, :A, 2) -@assert rulenode2expr(expression,grammar) in [:B,:C,:F,:G] -``` - -### Heuristics - - -# Examples with larger domains and constraints -Here, we showcase a few examples using a more complicated grammar and a few types of constraints -```julia -# Expects to return a program equivalent to 1 + (1 - x) = 2 - x - -g₁ = @csgrammar begin - Element = |(1 : 3) # 1 - 3 - Element = Element + Element # 4 - Element = 1 - Element # 5 - Element = x # 6 -end - -addconstraint!(g₁, ComesAfter(6, [5])) - -examples = [ - IOExample(Dict(:x => 0), 2), - IOExample(Dict(:x => 1), 1), - IOExample(Dict(:x => 2), 0) -] -problem = Problem(examples) -solution = search(g₁, problem, :Element, max_depth=3) - -@assert test_with_input(SymbolTable(g₁), solution, Dict(:x => -2)) == 4 - -# Expects to return a program equivalent to 4 + x * (x + 3 + 3) = x^2 + 6x + 4 - -g₂ = @csgrammar begin - Element = Element + Element + Element # 1 - Element = Element + Element * Element # 2 - Element = x # 3 - Element = |(3 : 5) # 4 -end - -# Restrict .. + x * x -addconstraint!(g₂, Forbidden(MatchNode(2, [MatchVar(:x), MatchNode(3), MatchNode(3)]))) -# Restrict 4 and 5 in lower level -addconstraint!(g₂, ForbiddenPath([2, 1, 5])) -addconstraint!(g₂, ForbiddenPath([2, 1, 6])) - -examples = [ - IOExample(Dict(:x => 1), 11) - IOExample(Dict(:x => 2), 20) - IOExample(Dict(:x => -1), -1) -] -problem = Problem(examples) -solution = search(g₂, problem, :Element) - -@assert test_with_input(SymbolTable(g₂), solution, Dict(:x => 0)) == 4 - -# Expects to return a program equivalent to (1 - (((1 - x) - 1) - 1)) - 1 = x + 1 - -g₃ = @csgrammar begin - Element = |(1 : 20) # 1 - 20 - Element = Element - 1 # 21 - Element = 1 - Element # 22 - Element = x # 23 -end - -addconstraint!(g₃, ComesAfter(23, [22, 21])) -addconstraint!(g₃, ComesAfter(22, [21])) - -examples = [ - IOExample(Dict(:x => 1), 2) - IOExample(Dict(:x => 10), 11) -] -problem = Problem(examples) -solution = search(g₃, problem, :Element) - -@assert test_with_input(SymbolTable(g₃), solution, Dict(:x => 0)) == 1 -@assert test_with_input(SymbolTable(g₃), solution, Dict(:x => 100)) == 101 - -# Expects to return a program equivalent to 18 + 4x - -g₄ = @csgrammar begin - Element = |(0 : 20) # 1 - 20 - Element = Element + Element + Element # 21 - Element = Element + Element * Element # 22 - Element = x # 23 -end - -# Enforce ordering on + + -addconstraint!(g₄, Ordered( - MatchNode(21, [MatchVar(:x), MatchVar(:y), MatchVar(:z)]), - [:x, :y, :z] -)) - -examples = [ - IOExample(Dict(:x => 1), 22), - IOExample(Dict(:x => 0), 18), - IOExample(Dict(:x => -1), 14) -] -problem = Problem(examples) -solution = search(g₄, problem, :Element) - -@assert test_with_input(SymbolTable(g₄), solution, Dict(:x => 100)) == 418 - -# Expects to return a program equivalent to (x == 2) ? 1 : (x + 2) - -g₅ = @csgrammar begin - Element = Number # 1 - Element = Bool # 2 - - Number = |(1 : 3) # 3-5 - - Number = Number + Number # 6 - Bool = Number ≡ Number # 7 - Number = x # 8 - - Number = Bool ? Number : Number # 9 - Bool = Bool ? Bool : Bool # 10 -end - -# Forbid ? = ? -addconstraint!(g₅, Forbidden(MatchNode(7, [MatchVar(:x), MatchVar(:x)]))) -# Order = -addconstraint!(g₅, Ordered(MatchNode(7, [MatchVar(:x), MatchVar(:y)]), [:x, :y])) -# Order + -addconstraint!(g₅, Ordered(MatchNode(6, [MatchVar(:x), MatchVar(:y)]), [:x, :y])) - -examples = [ - IOExample(Dict(:x => 0), 2) - IOExample(Dict(:x => 1), 3) - IOExample(Dict(:x => 2), 1) -] -problem = Problem(examples) -solution = search(g₅, problem, :Element) - -@assert test_with_input(SymbolTable(g₅), solution, Dict(:x => 3)) == 5 -``` \ No newline at end of file diff --git a/docs/src/tutorials/defining_grammars.html b/docs/src/tutorials/defining_grammars.html new file mode 100644 index 0000000..63bbd5c --- /dev/null +++ b/docs/src/tutorials/defining_grammars.html @@ -0,0 +1,17 @@ + + + + + + + +
\ No newline at end of file diff --git a/docs/src/tutorials/defining_grammars.ipynb b/docs/src/tutorials/defining_grammars.ipynb deleted file mode 100644 index dec24cb..0000000 --- a/docs/src/tutorials/defining_grammars.ipynb +++ /dev/null @@ -1,531 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Defining Grammars in Herb.jl using HerbGrammar\n", - "\n", - "The program space in Herb.jl is defined using a grammar. \n", - "This notebook demonstrates how such a grammar can be created. \n", - "There are multiple kinds of grammars, but they can all be defined in a very similar way." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Setup\n", - "First, we import the necessary Herb packages." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "using HerbGrammar, HerbConstraints" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Creating a simple grammar\n", - "\n", - "This cell contains a very simple arithmetic grammar. \n", - "The grammar is defined using the `@csgrammar` macro. \n", - "This macro converts the grammar definition in the form of a Julia expression into Herb's internal grammar representation. \n", - "Macro's are executed during compilation.\n", - "If you want to load a grammar during execution, have a look at the `HerbGrammar.expr2csgrammar` function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g₁ = HerbGrammar.@csgrammar begin\n", - " Int = 1\n", - " Int = 2\n", - " Int = 3\n", - " Int = Int * Int\n", - " Int = Int + Int\n", - "end" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Defining every integer one-by-one can be quite tedious. Therefore, it is also possible to use the following syntax that makes use of a Julia iterator:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g₂ = HerbGrammar.@csgrammar begin\n", - " Int = |(0:9)\n", - " Int = Int * Int\n", - " Int = Int + Int\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can do the same with lists:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g₃ = HerbGrammar.@csgrammar begin\n", - " Int = |([0, 2, 4, 6, 8])\n", - " Int = Int * Int\n", - " Int = Int + Int\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Variables can also be added to the grammar by just using the variable name:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g₄ = HerbGrammar.@csgrammar begin\n", - " Int = |(0:9)\n", - " Int = Int * Int\n", - " Int = Int + Int\n", - " Int = x\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Grammars can also work with functions. \n", - "After all, `+` and `*` are just infix operators for Julia's identically-named functions.\n", - "You can use functions that are provided by Julia, or functions that you wrote yourself:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "f(a) = a + 1\n", - "\n", - "g₅ = HerbGrammar.@csgrammar begin\n", - " Int = |(0:9)\n", - " Int = Int * Int\n", - " Int = Int + Int\n", - " Int = f(Int)\n", - " Int = x\n", - "end" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Similarly, we can also define the operator times (x) manually." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "×(a, b) = a * b\n", - "\n", - "g₆ = HerbGrammar.@csgrammar begin\n", - " Int = |(0:9)\n", - " Int = a\n", - " Int = Int + Int\n", - " Int = Int × Int\n", - "end" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Working with grammars\n", - "\n", - "If you want to implement something using these grammars, it is useful to know about the functions that you can use to manipulate grammars and extract information. \n", - "This section is not complete, but it aims to give an overview of the most important functions. \n", - "\n", - "It is recommended to also read up on [Julia metaprogramming](https://docs.julialang.org/en/v1/manual/metaprogramming/) if you are not already familiar with the concept.\n", - "\n", - "One of the most important things about grammars is that each rule has an index associated with it:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g₇ = HerbGrammar.@csgrammar begin\n", - " Int = |(0:9)\n", - " Int = Int + Int\n", - " Int = Int * Int\n", - " Int = x\n", - "end\n", - "\n", - "collect(enumerate(g₇.rules))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can use this index to extract information from the grammar." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### isterminal\n", - "\n", - "`isterminal` returns `true` if a rule is terminal, i.e. it cannot be expanded. For example, rule 1 is terminal, but rule 11 is not, since it contains the non-terminal symbol `:Int`. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.isterminal(g₇, 1)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.isterminal(g₇, 11)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### return_type\n", - "\n", - "This function is rather obvious; it returns the non-terminal symbol that corresponds to a certain rule. The return type for all rules in our grammar is `:Int`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.return_type(g₇, 11)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### child_types\n", - "\n", - "`child_types` returns the types of the nonterminal children of a rule in a vector.\n", - "If you just want to know how many children a rule has, and not necessarily which types they have, you can use `nchildren`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.child_types(g₇, 11)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.nchildren(g₇, 11)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### nonterminals\n", - "\n", - "The `nonterminals` function can be used to obtain a list of all nonterminals in the grammar." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.nonterminals(g₇)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Adding rules\n", - "\n", - "It is also possible to add rules to a grammar during execution. This can be done using the `add_rule!` function. The exclamatin mark is a Julia convention and is appended to name if a function modifies its arguments (in our example the grammar).\n", - "\n", - "A rule can be provided in the same syntax as is used in the grammar definition.\n", - "The rule should be of the `Expr` type, which is a built-in type for representing expressions. \n", - "An easy way of creating `Expr` values in Julia is to encapsulate it in brackets and use a colon as prefix:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.add_rule!(g₇, :(Int = Int - Int))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Removing rules\n", - "\n", - "It is also possible to remove rules in Herb.jl, however, this is a bit more involved. \n", - "As said before, rules have an index associated with them. \n", - "The internal representation of programs that are defined by the grammar makes use of those indices for efficiency.\n", - "Blindly removing a rule would shift the indices of other rules, and this could mean that existing programs get a different meaning or become invalid. \n", - "\n", - "Therefore, there are two functions for removing rules:\n", - "\n", - "- `remove_rule!` removes a rule from the grammar, but fills its place with a placeholder. Therefore, the indices stay the same, and only programs that use the removed rule become invalid.\n", - "- `cleanup_removed_rules!` removes all placeholders and shifts the indices of the other rules.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.remove_rule!(g₇, 11)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.cleanup_removed_rules!(g₇)" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Context-sensitive grammars\n", - "\n", - "Context-sensitive grammars introduce additional constraints compared to context-free grammars (like the simple grammar examples above).\n", - "As before, we use the `@csgrammar` macro:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g₈ = HerbGrammar.@csgrammar begin\n", - " Int = |(0:9)\n", - " Int = Int + Int\n", - " Int = Int * Int\n", - " Int = x\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Constraints can be added using the `addconstraint!` function, which takes a context-sensitive grammar and a constraint and adds the constraint to the grammar.\n", - "\n", - "For example, we can add a `` constraint to enforce that the input symbol `x` (rule 13) appears at least once in the program, to avoid programs that are just a constant. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.addconstraint!(g₈, Contains(13))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There is a dedicated tutorial for constraints in Herb.jl and how to work with them." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Probabilistic grammars\n", - "\n", - "Herb.jl also supports probabilistic grammars. \n", - "These grammars allow the user to assign a probability to each rule in the grammar.\n", - "A probabilistic grammar can be defined in a very similar way to a standard grammar, but has some slightly different syntax:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g₉ = HerbGrammar.@pcsgrammar begin\n", - " 0.4 : Int = |(0:9)\n", - " 0.2 : Int = Int + Int\n", - " 0.1 : Int = Int * Int\n", - " 0.3 : Int = x\n", - "end\n", - "\n", - "for r ∈ 1:length(g₃.rules)\n", - " p = HerbGrammar.probability(g₈, r)\n", - "\n", - " println(\"$p : $r\")\n", - "end" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The numbers before each rule represent the probability assigned to that rule.\n", - "The total probability for each return type should add up to 1.0.\n", - "If this isn't the case, Herb.jl will normalize the probabilities.\n", - "\n", - "If a single line in the grammar definition represents multiple rules, such as `0.4 : Int = |(0:9)`, the probability will be evenly divided over all these rules." - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## File writing\n", - "\n", - "### Saving & loading context-free grammars\n", - "\n", - "If you want to store a grammar on the disk, you can use the `store_csg`, `read_csg` and functions to store and read grammars respectively. \n", - "The `store_csg` grammar can also be used to store probabilistic grammars. To read probabilistic grammars, use `read_pcsg`.\n", - "The stored grammar files can also be opened using a text editor to be modified, as long as the contents of the file doesn't violate the syntax for defining grammars." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.store_csg(g₇, \"demo.txt\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.read_csg(\"demo.txt\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Saving & loading context-sensitive grammars\n", - "\n", - "Saving and loading context-sensitive grammars is very similar to how it is done with context-free grammars.\n", - "The only difference is that an additional file is created for the constraints. \n", - "The file that contains the grammars can be edited and can also be read using the reader for context-free grammars.\n", - "The file that contains the constraints cannot be edited." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "HerbGrammar.store_csg( g₈, \"demo.grammar\", \"demo.constraints\")\n", - "g₈, g₈.constraints" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "g₉ = HerbGrammar.read_csg(\"demo.grammar\", \"demo.constraints\")\n", - "g₉, g₉.constraints" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Julia 1.10.3", - "language": "julia", - "name": "julia-1.10" - }, - "language_info": { - "file_extension": ".jl", - "mimetype": "application/julia", - "name": "julia", - "version": "1.10.3" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/docs/src/tutorials/defining_grammars.md b/docs/src/tutorials/defining_grammars.md deleted file mode 100644 index 6223b12..0000000 --- a/docs/src/tutorials/defining_grammars.md +++ /dev/null @@ -1,590 +0,0 @@ -# Defining Grammars in Herb.jl using HerbGrammar - -The program space in Herb.jl is defined using a grammar. -This notebook demonstrates how such a grammar can be created. -There are multiple kinds of grammars, but they can all be defined in a very similar way. - -### Setup -First, we import the necessary Herb packages. - - -```julia -using HerbGrammar, HerbConstraints -``` - -### Creating a simple grammar - -This cell contains a very simple arithmetic grammar. -The grammar is defined using the `@cfgrammar` macro. -This macro converts the grammar definition in the form of a Julia expression into Herb's internal grammar representation. -Macro's are executed during compilation. -If you want to load a grammar during execution, have a look at the `HerbGrammar.expr2cfgrammar` function. - - -```julia -g₁ = HerbGrammar.@cfgrammar begin - Int = 1 - Int = 2 - Int = 3 - Int = Int * Int - Int = Int + Int -end -``` - - - 1: Int = 1 - 2: Int = 2 - 3: Int = 3 - 4: Int = Int * Int - 5: Int = Int + Int - - - -Defining every integer one-by-one can be quite tedious. Therefore, it is also possible to use the following syntax that makes use of a Julia iterator: - - -```julia -g₂ = HerbGrammar.@cfgrammar begin - Int = |(0:9) - Int = Int * Int - Int = Int + Int -end -``` - - - 1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = Int * Int - 12: Int = Int + Int - - - -You can do the same with lists: - - -```julia -g₃ = HerbGrammar.@cfgrammar begin - Int = |([0, 2, 4, 6, 8]) - Int = Int * Int - Int = Int + Int -end -``` - - - 1: Int = 0 - 2: Int = 2 - 3: Int = 4 - 4: Int = 6 - 5: Int = 8 - 6: Int = Int * Int - 7: Int = Int + Int - - - -Variables can also be added to the grammar by just using the variable name: - - -```julia -g₄ = HerbGrammar.@cfgrammar begin - Int = |(0:9) - Int = Int * Int - Int = Int + Int - Int = x -end -``` - - - 1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = Int * Int - 12: Int = Int + Int - 13: Int = x - - - -Grammars can also work with functions. -After all, `+` and `*` are just infix operators for Julia's identically-named functions. -You can use functions that are provided by Julia, or functions that you wrote yourself: - - -```julia -f(a) = a + 1 - -g₅ = HerbGrammar.@cfgrammar begin - Int = |(0:9) - Int = Int * Int - Int = Int + Int - Int = f(Int) - Int = x -end -``` - - - 1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = Int * Int - 12: Int = Int + Int - 13: Int = f(Int) - 14: Int = x - - - -Similarly, we can also define the operator times (x) manually. - - -```julia -×(a, b) = a * b - -g₆ = HerbGrammar.@cfgrammar begin - Int = |(0:9) - Int = a - Int = Int + Int - Int = Int × Int -end -``` - - - 1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = a - 12: Int = Int + Int - 13: Int = Int × Int - - - -### Working with grammars - -If you want to implement something using these grammars, it is useful to know about the functions that you can use to manipulate grammars and extract information. -This section is not necessarily complete, but it aims to give an overview of the most important functions. - -It is recommended to also read up on [Julia metaprogramming](https://docs.julialang.org/en/v1/manual/metaprogramming/) if you are not already familiar with that. - -One of the most important things about grammars is that each rule has an index associated with it: - - -```julia -g₇ = HerbGrammar.@cfgrammar begin - Int = |(0:9) - Int = Int + Int - Int = Int * Int - Int = x -end - -collect(enumerate(g₇.rules)) -``` - - - 13-element Vector{Tuple{Int64, Any}}: - (1, 0) - (2, 1) - (3, 2) - (4, 3) - (5, 4) - (6, 5) - (7, 6) - (8, 7) - (9, 8) - (10, 9) - (11, :(Int + Int)) - (12, :(Int * Int)) - (13, :x) - - -We can use this index to extract information from the grammar. - -### isterminal - -`isterminal` returns `true` if a rule is terminal, i.e. it cannot be expanded. For example, rule 1 is terminal, but rule 11 is not, since it contains the non-terminal symbol `:Int`. - - -```julia -HerbGrammar.isterminal(g₇, 1) -``` - - - true - - - -```julia -HerbGrammar.isterminal(g₇, 11) -``` - - - false - - -### return_type - -This function is rather obvious; it returns the non-terminal symbol that corresponds to a certain rule. The return type for all rules in our grammar is `:Int`. - - -```julia -HerbGrammar.return_type(g₇, 11) -``` - - - :Int - - -### child_types - -`child_types` returns the types of the nonterminal children of a rule in a vector. -If you just want to know how many children a rule has, and not necessarily which types they have, you can use `nchildren` - - -```julia -HerbGrammar.child_types(g₇, 11) -``` - - - 2-element Vector{Symbol}: - :Int - :Int - - - -```julia -HerbGrammar.nchildren(g₇, 11) -``` - - - 2 - - -### nonterminals - -The `nonterminals` function can be used to obtain a list of all nonterminals in the grammar. - - -```julia -HerbGrammar.nonterminals(g₇) -``` - - - 1-element Vector{Symbol}: - :Int - - -### Adding rules - -It is also possible to add rules to a grammar during execution. This can be done using the `add_rule!` function. -As with most functions in Julia that end with an exclamation mark, this function modifies its argument (the grammar). - -A rule can be provided in the same syntax as is used in the grammar definition. -The rule should be of the `Expr` type, which is a built-in type for representing expressions. -An easy way of creating `Expr` values in Julia is to encapsulate it in brackets and use a colon as prefix: - - -```julia -HerbGrammar.add_rule!(g₇, :(Int = Int - Int)) -``` - - - 1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = Int + Int - 12: Int = Int * Int - 13: Int = x - 14: Int = Int - Int - - - -### Removing rules - -It is also possible to remove rules in Herb.jl, however, this is a bit more involved. -As said before, rules have an index associated with them. -The internal representation of programs that are defined by the grammar makes use of those indices for efficiency. -Blindly removing a rule would shift the indices of other rules, and this could mean that existing programs get a different meaning or become invalid. - -Therefore, there are two functions for removing rules: - -- `remove_rule!` removes a rule from the grammar, but fills its place with a placeholder. Therefore, the indices stay the same, and only programs that use the removed rule become invalid. -- `cleanup_removed_rules!` removes all placeholders and shifts the indices of the other rules. - - - -```julia -HerbGrammar.remove_rule!(g₇, 11) -``` - - - 1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: nothing = nothing - 12: Int = Int * Int - 13: Int = x - 14: Int = Int - Int - - - - -```julia -HerbGrammar.cleanup_removed_rules!(g₇) -``` - - - 1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = Int * Int - 12: Int = x - 13: Int = Int - Int - - - -## Context-sensitive grammars - -Context-sensitive grammars allow additional constraints to be added with respect to context-free grammars. -The syntax for defining a context-sensitive grammar is identical to defining a context-sensitive grammar: - - -```julia -g₈ = HerbGrammar.@csgrammar begin - Int = |(0:9) - Int = Int + Int - Int = Int * Int - Int = x -end -``` - - - 1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = Int + Int - 12: Int = Int * Int - 13: Int = x - - - -Constraints can be added using the `addconstraint!` function, which takes a context-sensitive grammar and a constraint and adds the constraint to the grammar. -Currently, Herb.jl only has propagators constraints. -These constraints each have a corresponding `propagate` function that removes all options that violate that constraint from the domain. -At the moment, there are three propagator constraints: - -- `ComesAfter(rule, predecessors)`: It is only possible to use rule `rule` when `predecessors` are in its path to the root. -- `Forbidden(sequence)`: Forbids the derivation specified as a path in an expression tree. -- `Ordered(order)`: Rules have to be used in the specified order. That is, rule at index K can only be used if rules at indices `[1...K-1]` are used in the left subtree of the current expression. - -Below, an example is given of a context-sensitive grammar with a `ComesAfter` constraint: - - -```julia -HerbGrammar.addconstraint!(g₈, HerbConstraints.ComesAfter(1, [9])) -``` - - - 1-element Vector{HerbCore.Constraint}: - ComesAfter(1, [9]) - - -### Probabilistic grammars - -Herb.jl also supports probabilistic grammars. -These grammars allow the user to assign a probability to each rule in the grammar. -A probabilistic grammar can be defined in a very similar way to a standard grammar, but has some slightly different syntax: - - -```julia -g₉ = HerbGrammar.@pcfgrammar begin - 0.4 : Int = |(0:9) - 0.2 : Int = Int + Int - 0.1 : Int = Int * Int - 0.3 : Int = x -end - -for r ∈ 1:length(g₃.rules) - p = HerbGrammar.probability(g₈, r) - - println("$p : $r") -end -``` - - 0.07692307692307693 : 1 - 0.07692307692307693 : 2 - 0.07692307692307693 : 3 - 0.07692307692307693 : 4 - 0.07692307692307693 : 5 - 0.07692307692307693 : 6 - 0.07692307692307693 : 7 - - - ┌ Warning: Requesting probability in a non-probabilistic grammar. - │ Uniform distribution is assumed. - └ @ HerbGrammar /Users/issahanou/.julia/packages/HerbGrammar/x0E9w/src/grammar_base.jl:155 - ┌ Warning: Requesting probability in a non-probabilistic grammar. - │ Uniform distribution is assumed. - └ @ HerbGrammar /Users/issahanou/.julia/packages/HerbGrammar/x0E9w/src/grammar_base.jl:155 - ┌ Warning: Requesting probability in a non-probabilistic grammar. - │ Uniform distribution is assumed. - └ @ HerbGrammar /Users/issahanou/.julia/packages/HerbGrammar/x0E9w/src/grammar_base.jl:155 - ┌ Warning: Requesting probability in a non-probabilistic grammar. - │ Uniform distribution is assumed. - └ @ HerbGrammar /Users/issahanou/.julia/packages/HerbGrammar/x0E9w/src/grammar_base.jl:155 - ┌ Warning: Requesting probability in a non-probabilistic grammar. - │ Uniform distribution is assumed. - └ @ HerbGrammar /Users/issahanou/.julia/packages/HerbGrammar/x0E9w/src/grammar_base.jl:155 - ┌ Warning: Requesting probability in a non-probabilistic grammar. - │ Uniform distribution is assumed. - └ @ HerbGrammar /Users/issahanou/.julia/packages/HerbGrammar/x0E9w/src/grammar_base.jl:155 - ┌ Warning: Requesting probability in a non-probabilistic grammar. - │ Uniform distribution is assumed. - └ @ HerbGrammar /Users/issahanou/.julia/packages/HerbGrammar/x0E9w/src/grammar_base.jl:155 - - -The numbers before each rule represent the probability assigned to that rule. -The total probability for each return type should add up to 1.0. -If this isn't the case, Herb.jl will normalize the probabilities. - -If a single line in the grammar definition represents multiple rules, such as `0.4 : Int = |(0:9)`, the probability will be evenly divided over all these rules. - -## File writing - -### Saving & loading context-free grammars - -If you want to store a grammar on the disk, you can use the `store_cfg`, `read_cfg` and functions to store and read grammars respectively. -The `store_cfg` grammar can also be used to store probabilistic grammars. Reading probabilistic grammars can be done using `read_pcfg`. -The stored grammar files can also be opened using a text editor to be modified, as long as the contents of the file doesn't violate the syntax for defining grammars. - - -```julia -HerbGrammar.store_cfg("demo.txt", g₇) -``` - - -```julia -HerbGrammar.read_cfg("demo.txt") -``` - - - 1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = Int * Int - 12: Int = x - 13: Int = Int - Int - - - -### Saving & loading context-sensitive grammars - -Saving and loading context-sensitive grammars is very similar to how it is done with context-free grammars. -The only difference is that an additional file is created for the constraints. -The file that contains the grammars can be edited and can also be read using the reader for context-free grammars. -The file that contains the constraints cannot be edited. - - -```julia -HerbGrammar.store_csg("demo.grammar", "demo.constraints", g₈) -g₈, g₈.constraints -``` - - - (1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = Int + Int - 12: Int = Int * Int - 13: Int = x - , HerbCore.Constraint[ComesAfter(1, [9])]) - - - -```julia -g₉ = HerbGrammar.read_csg("demo.grammar", "demo.constraints") -g₉, g₉.constraints -``` - - - (1: Int = 0 - 2: Int = 1 - 3: Int = 2 - 4: Int = 3 - 5: Int = 4 - 6: Int = 5 - 7: Int = 6 - 8: Int = 7 - 9: Int = 8 - 10: Int = 9 - 11: Int = Int + Int - 12: Int = Int * Int - 13: Int = x - , HerbCore.Constraint[ComesAfter(1, [9])]) - diff --git a/docs/src/tutorials/getting_started_with_constraints.html b/docs/src/tutorials/getting_started_with_constraints.html index a16e59f..67e1c71 100644 --- a/docs/src/tutorials/getting_started_with_constraints.html +++ b/docs/src/tutorials/getting_started_with_constraints.html @@ -3,11 +3,11 @@ diff --git a/docs/src/tutorials/getting_started_with_constraints.ipynb b/docs/src/tutorials/getting_started_with_constraints.ipynb deleted file mode 100644 index 2bb63b3..0000000 --- a/docs/src/tutorials/getting_started_with_constraints.ipynb +++ /dev/null @@ -1,481 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Getting started with HerbConstraints\n", - "\n", - "When enumerating programs using a grammar, we will encounter many redundant programs. For example, `x`, `-(-x)` and `1 * x` are syntactically different programs, but they have the same semantics. Grammar constraints aim to speed up synthesis by eliminating such redundant programs and thereby reducing the size of the program space." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Setup\n", - "\n", - "For this tutorial, we need to import the following modules of the Herb.jl framework:\n", - "\n", - "* `HerbCore` for the necessary data strucutes, like `Hole`s and `RuleNode`s\n", - "* `HerbGrammar` to define the grammar\n", - "* `HerbConstraints` to define the constraints\n", - "* `HerbSearch` to execute a constrained enumeration\n", - "\n", - "We will also redefine the simple arithmetic grammar from the previous tutorial." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "using HerbCore, HerbGrammar, HerbConstraints, HerbSearch\n", - "\n", - "grammar = @cfgrammar begin\n", - " Int = 1\n", - " Int = x\n", - " Int = - Int\n", - " Int = Int + Int\n", - " Int = Int * Int\n", - "end" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Working with constraints\n", - "\n", - "To show the effects of constraints, we will first enumerate all programs without constraints (up to a maximum size of 3 AST nodes).\n", - "\n", - "(To make sure the grammar doesn't have any constraints, we can clear the constraints using `clearconstraints!`. This is not needed at this point, but could come in handy if your REPL holds a reference to a constrained version of the grammar)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "clearconstraints!(grammar)\n", - "iter = BFSIterator(grammar, :Int, max_size=3)\n", - "\n", - "for program ∈ iter\n", - " println(rulenode2expr(program, grammar))\n", - "end\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Upon inspection, we can already see some redundant programs, like `1 * 1` and `-(-1)`. To eliminate these redundant programs, we will set up some constraints that prevent these patterns from appearing. Then we will create another iteratator to enumerate all programs that satisfy the defined grammar constraints.\n", - "\n", - "To make the forbidden pattern constraint general, we will use a special type of rulenode: `VarNode(:A)`. This node matches with any subtree and can be used to forbid multiple forbidden patterns using a single constraint. For example, `Forbidden(RuleNode(minus, [RuleNode(minus, [VarNode(:A)])])))` forbids:\n", - "\n", - "* `-(-1)`\n", - "* `-(-X)`\n", - "* `-(-(1 + 1))`\n", - "* `1 + -(-(1 + 1))`\n", - "* etc" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "one = 1\n", - "x = 2\n", - "minus = 3\n", - "plus = 4\n", - "times = 5\n", - "\n", - "addconstraint!(grammar, Forbidden(RuleNode(times, [RuleNode(one), VarNode(:A)]))) # forbid 1*A\n", - "addconstraint!(grammar, Forbidden(RuleNode(minus, [RuleNode(minus, [VarNode(:A)])]))) # forbid -(-A)\n", - "\n", - "iter = BFSIterator(grammar, :Int, max_size=3)\n", - "\n", - "for program ∈ iter\n", - " println(rulenode2expr(program, grammar))\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Forbidden Constraint\n", - "\n", - "The `Forbidden` constraint forbids any subtree in the program that matches a given template tree. Such a template tree can consist of 3 node types:\n", - "* `RuleNode(1)`. Matches exactly the given rule.\n", - "* `DomainRuleNode(BitVector((0, 0, 0, 1, 1)), children)`. Matches any rule in its bitvector domain. In this case, rule 4 and 5.\n", - "* `VarNode(:A)`. Matches any subtree. If another VarNode of the same name is used, the subtrees have to be the same." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#this constraint forbids A+A and A*A\n", - "constraint = Forbidden(DomainRuleNode(BitVector((0, 0, 0, 1, 1)), [VarNode(:A), VarNode(:A)]))\n", - "\n", - "# Without this constraint, we encounter 154 programs\n", - "clearconstraints!(grammar)\n", - "iter = BFSIterator(grammar, :Int, max_size=5)\n", - "println(length(iter))\n", - "\n", - "# With this constraint, we encounter 106 programs\n", - "clearconstraints!(grammar)\n", - "addconstraint!(grammar, constraint)\n", - "iter = BFSIterator(grammar, :Int, max_size=5)\n", - "println(length(iter))\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Contains Constraint\n", - "\n", - "The `Contains` constraint enforces that a given rule appears in the program tree at least once. \n", - "\n", - "In the arithmetic grammar, this constraint can be used to ensure the input symbol `x` is used in the program. Otherwise, the program is just a constant." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "clearconstraints!(grammar)\n", - "addconstraint!(grammar, Contains(2)) #rule 2 should be used in the program\n", - "iter = BFSIterator(grammar, :Int, max_size=3)\n", - "\n", - "for program ∈ iter\n", - " println(rulenode2expr(program, grammar))\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Contains Subtree Constraint\n", - "\n", - "Similarly to the `Contains` constraint, the `ContainsSubtree` can be used to enforce a given template tree is used in the program at least once." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "clearconstraints!(grammar)\n", - "addconstraint!(grammar, ContainsSubtree(RuleNode(times, [RuleNode(x), RuleNode(x)]))) #x*x should be in the program tree\n", - "iter = BFSIterator(grammar, :Int, max_size=4)\n", - "\n", - "for program ∈ iter\n", - " println(rulenode2expr(program, grammar))\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Ordered Constraint\n", - "\n", - "The `Ordered` constraint enforces an `<=` ordering on a provided list of variables. With this constraint, we can break symmetries based on commutativity. For example, `1+x` and `x+1` are semantically equivalent. By imposing an `Ordered` constraint, we can eliminate one of the symmetric variants.\n", - "\n", - "To define an `Ordered` constraint, we need to provide it with a template tree including at least two differently named `VarNode`s. And additionally, an ordering of the variables in the tree.\n", - "\n", - "In the upcoming example we will set up a template tree representing `a+b` and `a*b`.\n", - "Then, we will impose an ordering `a<=b` on all the subtrees that match the template.\n", - "\n", - "The result is that our iterator skips the redundant programs `x+1` and `x*1`, as they are already represented by `1+x` and `1*x`.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "clearconstraints!(grammar)\n", - "\n", - "template_tree = DomainRuleNode(BitVector((0, 0, 0, 1, 1)), [VarNode(:a), VarNode(:b)])\n", - "order = [:a, :b]\n", - "\n", - "addconstraint!(grammar, Ordered(template_tree, order))\n", - "iter = BFSIterator(grammar, :Int, max_size=3)\n", - "\n", - "for program ∈ iter\n", - " println(rulenode2expr(program, grammar))\n", - "end\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Forbidden Sequence Constraint\n", - "\n", - "The `ForbiddenSequence` constraints forbids a given sequence of rule nodes in a vertical path of the tree. \n", - "\n", - "An optional second argument, `ignore_if`, can be used to overrule the constraint in case any of the rules on the `ignore_if` list are present. \n", - "\n", - "Below we will define the constraint `ForbiddenSequence([plus, one], ignore_if=[times])`. It forbids an `1` after an `+` unless an `*` disrupts the sequence.\n", - "\n", - "This constraint will **forbid** the following programs:\n", - "\n", - "* x + 1\n", - "* x + -1\n", - "* x + -(-1)\n", - "* x + (x + 1)\n", - "* x * (x + 1)\n", - "\n", - "But it will **allow** the following program (as * disrupts the sequence):\n", - "\n", - "* x + (x * 1)\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "constraint = ForbiddenSequence([plus, one], ignore_if=[times])\n", - "addconstraint!(grammar, constraint)\n", - "iter = BFSIterator(grammar, :Int, max_size=3)\n", - "\n", - "for program ∈ iter\n", - " println(rulenode2expr(program, grammar))\n", - "end\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Custom Constraint\n", - "\n", - "To implement a new constraint, we need to define two structs: an `AbstractGrammarConstraint` and an `AbstractLocalConstraint`.\n", - "\n", - "A **grammar constraint** is a high-level constraint on the grammar itself and does not refer to a location in the tree. For example, the `Forbidden` constraint is responsible for forbidding a template tree everywhere in the tree. To divide the work of constraint propagation, the grammar constraint will post several local constraints that are responsible for propagating the constraint at each particular location.\n", - "\n", - "A **local constraint** is a rooted version of a grammar constraint. Each local constraint holds a `path` field that points to a location in the tree where this constraint applies." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Suppose we want to implement a simple custom constraint that forbids a given `rule` twice in a row. \n", - "\n", - "Each time a new AST node is added to a tree, the `on_new_node` function is called to notify that an unseen node has been added to the tree at path `path`. Our grammar constraint has the opportunity to react to this event. In this example, we will post a new local constraint at the new location using the `post!` function.\n", - "\n", - "(Don't worry about the `HerbConstraints.` prefixes. Normally, constraints are defined within the HerbConstraints repository, so there is no need to specify the namespace)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "\"\"\"\n", - "Forbids the consecutive application of the specified rule.\n", - "For example, CustomConstraint(4) forbids the tree 4(1, 4(1, 1)) as it applies rule 4 twice in a row.\n", - "\"\"\"\n", - "struct ForbidConsecutive <: AbstractGrammarConstraint\n", - " rule::Int\n", - "end\n", - "\n", - "\"\"\"\n", - "Post a local constraint on each new node that appears in the tree\n", - "\"\"\"\n", - "function HerbConstraints.on_new_node(solver::Solver, constraint::ForbidConsecutive, path::Vector{Int})\n", - " HerbConstraints.post!(solver, LocalForbidConsecutive(path, constraint.rule))\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we will define our local constraint. This constraint is responsible for propagating the constraint at a given path. The `propagate!` method can use several solver functions to manipulate the tree. The following **tree manipulations** can be used to remove rules from the domain of a hole at a given path:\n", - "\n", - "* `remove!(solver::Solver, path::Vector{Int}, rule_index::Int)`\n", - "* `remove!(solver::Solver, path::Vector{Int}, rules::Vector{Int})`\n", - "* `remove_all_but!(solver::Solver, path::Vector{Int}, new_domain::BitVector)`\n", - "* `remove_above!(solver::Solver, path::Vector{Int}, rule_index::Int)`\n", - "* `remove_below!(solver::Solver, path::Vector{Int}, rule_index::Int)`\n", - "* `make_equal!(solver::Solver, node1::AbstractRuleNode, node2::AbstractRuleNode)` (a high level manipulation that requires `node1` and `node2` to be in the tree)\n", - "\n", - "In addition to tree manipulations, the following solver functions can be used to communicate new information to the solver:\n", - "\n", - "* `set_infeasible!(solver)`. If a propagator detects an inconsistency, the solver should be notified and cancel any other scheduled propagators.\n", - "* `deactivate!(solver, constraint)`. If a constraint is satisfied, it should deactivate itself to prevent re-propagation.\n", - "* `post!(solver, constraint)` A constraint is allowed to post new local constraints. This might be helpful if a constraint can be reduced to a smaller constraint.\n", - "\n", - "The solver manages all constraints and the program tree we propagate on. Applying tree manipulations might cause a chain reaction of other propagators, so the shape of the tree might update as we propagate. The get the latest information about the tree, we should use the following getter functions:\n", - "\n", - "* `get_tree(solver)` returns the root node of the current (partial) program tree\n", - "* `isfeasible(solver)` returns the a flag indicating if the solver is not violating any (other) constriants.\n", - "* `get_path(solver, node)` returns the path at which the node is located.\n", - "* `get_node_at_location(solver, path)` returns the node that is currently at the given path (be aware that this instance might be replaced by manipulations).\n", - "* `get_hole_at_location(solver, path)` same as get node at location, but asserts the node is a hole (domain size >= 2).\n", - "\n", - "To get information about a node, we can use the following getter functions:\n", - "\n", - "* `isfilled(node)`. Returns true if the node is a `RuleNode` or has domain size 1.\n", - "* `get_rule(node)`. Get the rule of a filled node.\n", - "* `get_children(node)`. Get the children of a node.\n", - "* `node.domain[rule]`. Given the node is a hole, return true if `rule` is in the domain.\n", - "\n", - "Finally, another useful function for propagators is `pattern_match(node1, node2)`. This function compares two trees and returns a `PatternMatchResult` that indicates if the nodes match, and potentially indicate which holes need to be filled to complete the match.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "\"\"\"\n", - "Forbids the consecutive application of the specified rule at path `path`.\n", - "\"\"\"\n", - "struct LocalForbidConsecutive <: AbstractLocalConstraint\n", - " path::Vector{Int}\n", - " rule::Int\n", - "end\n", - "\n", - "\"\"\"\n", - "Propagates the constraints by preventing a consecutive application of the specified rule.\n", - "\"\"\"\n", - "function HerbConstraints.propagate!(solver::Solver, constraint::LocalForbidConsecutive)\n", - " node = get_node_at_location(solver, constraint.path)\n", - " if isfilled(node)\n", - " if get_rule(node) == constraint.rule\n", - " #the specified rule is used, make sure the rule will not be used by any of the children\n", - " for (i, child) ∈ enumerate(get_children(node))\n", - " if isfilled(child)\n", - " if get_rule(child) == constraint.rule\n", - " #the specified rule was used twice in a row, which is violating the constraint\n", - " set_infeasible!(solver)\n", - " return\n", - " end\n", - " elseif child.domain[constraint.rule]\n", - " child_path = push!(copy(constraint.path), i)\n", - " remove!(solver, child_path, constraint.rule) # remove the rule from the domain of the child\n", - " end\n", - " end\n", - " end\n", - " elseif node.domain[constraint.rule]\n", - " #our node is a hole with the specified rule in its domain\n", - " #we will now check if any of the children already uses the specified rule\n", - " softfail = false\n", - " for (i, child) ∈ enumerate(get_children(node))\n", - " if isfilled(child)\n", - " if get_rule(child) == constraint.rule\n", - " #the child holds the specified rule, so the parent cannot have this rule\n", - " remove!(solver, constraint.path, constraint.rule)\n", - " end\n", - " elseif child.domain[constraint.rule]\n", - " #the child is a hole and contains the specified node. since there are 2 holes involved, we will softfail.\n", - " softfail = true\n", - " end\n", - " end\n", - " if softfail\n", - " #we cannot deactivate the constraint, because it needs to be repropagated\n", - " return\n", - " end\n", - " end\n", - "\n", - " #the constraint is satisfied and can be deactivated\n", - " HerbConstraints.deactivate!(solver, constraint)\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Posting a local constraint will trigger the initial propagation. To re-propagate, the constraint needs to be rescheduled for propagation.\n", - "\n", - "Whenever the tree is manipulated, we will make a `shouldschedule` check to see if our constraint needs to be rescheduled for propagation based on the manipulation.\n", - "\n", - "In our case, we want to repropagate if either:\n", - "* a tree manipulation occured at the `constraint.path`\n", - "* a tree manipulation occured at the child of the `constraint.path`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "\n", - "\"\"\"\n", - "Gets called whenever an tree manipulation occurs at the given `path`.\n", - "Returns true iff the `constraint` should be rescheduled for propagation.\n", - "\"\"\"\n", - "function HerbConstraints.shouldschedule(solver::Solver, constraint::LocalForbidConsecutive, path::Vector{Int})::Bool\n", - " return (path == constraint.path) || (path == constraint.path[1:end-1])\n", - "end\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "With all the components implemented, we can do a constrained enumeration using our new `ForbidConsecutive` constraint." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "clearconstraints!(grammar)\n", - "\n", - "addconstraint!(grammar, ForbidConsecutive(minus))\n", - "addconstraint!(grammar, ForbidConsecutive(plus))\n", - "addconstraint!(grammar, ForbidConsecutive(times))\n", - "\n", - "iter = BFSIterator(grammar, :Int, max_size=6)\n", - "\n", - "for program ∈ iter\n", - " println(rulenode2expr(program, grammar))\n", - "end" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Julia 1.9.4", - "language": "julia", - "name": "julia-1.9" - }, - "language_info": { - "file_extension": ".jl", - "mimetype": "application/julia", - "name": "julia", - "version": "1.9.4" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/docs/src/tutorials/getting_started_with_constraints.md b/docs/src/tutorials/getting_started_with_constraints.md deleted file mode 100644 index 7c95e90..0000000 --- a/docs/src/tutorials/getting_started_with_constraints.md +++ /dev/null @@ -1,738 +0,0 @@ -# Getting started with HerbConstraints - -When enumerating programs using a grammar, we will encounter many redundant programs. For example, `x`, `-(-x)` and `1 * x` are syntactically different programs, but they have the same semantics. Grammar constraints aim to speed up synthesis by eliminating such redundant programs and thereby reducing the size of the program space. - -### Setup - -For this tutorial, we need to import the following modules of the Herb.jl framework: - -* `HerbCore` for the necessary data strucutes, like `Hole`s and `RuleNode`s -* `HerbGrammar` to define the grammar -* `HerbConstraints` to define the constraints -* `HerbSearch` to execute a constrained enumeration - -We will also redefine the simple arithmetic grammar from the previous tutorial. - - -```julia -using HerbCore, HerbGrammar, HerbConstraints, HerbSearch - -grammar = @cfgrammar begin - Int = 1 - Int = x - Int = - Int - Int = Int + Int - Int = Int * Int -end -``` - - - 1: Int = 1 - 2: Int = x - 3: Int = -Int - 4: Int = Int + Int - 5: Int = Int * Int - - - -### Working with constraints - -To show the effects of constraints, we will first enumerate all programs without constraints (up to a maximum size of 3 AST nodes). - -(To make sure the grammar doesn't have any constraints, we can clear the constraints using `clearconstraints!`. This is not needed at this point, but could come in handy if your REPL holds a reference to a constrained version of the grammar) - - -```julia -clearconstraints!(grammar) -iter = BFSIterator(grammar, :Int, max_size=3) - -for program ∈ iter - println(rulenode2expr(program, grammar)) -end - -``` - - 1 - x - -1 - -x - 1 * 1 - -(-1) - 1x - -(-x) - x * x - x * 1 - x + 1 - x + x - 1 + x - 1 + 1 - - -Upon inspection, we can already see some redundant programs, like `1 * 1` and `-(-1)`. To eliminate these redundant programs, we will set up some constraints that prevent these patterns from appearing. Then we will create another iteratator to enumerate all programs that satisfy the defined grammar constraints. - -To make the forbidden pattern constraint general, we will use a special type of rulenode: `VarNode(:A)`. This node matches with any subtree and can be used to forbid multiple forbidden patterns using a single constraint. For example, `Forbidden(RuleNode(minus, [RuleNode(minus, [VarNode(:A)])])))` forbids: - -* `-(-1)` -* `-(-X)` -* `-(-(1 + 1))` -* `1 + -(-(1 + 1))` -* etc - - -```julia -one = 1 -x = 2 -minus = 3 -plus = 4 -times = 5 - -addconstraint!(grammar, Forbidden(RuleNode(times, [RuleNode(one), VarNode(:A)]))) # forbid 1*A -addconstraint!(grammar, Forbidden(RuleNode(minus, [RuleNode(minus, [VarNode(:A)])]))) # forbid -(-A) - -iter = BFSIterator(grammar, :Int, max_size=3) - -for program ∈ iter - println(rulenode2expr(program, grammar)) -end -``` - - 1 - x - -1 - -x - x * 1 - x * x - x + x - x + 1 - 1 + 1 - 1 + x - - -### Forbidden Constraint - -The `Forbidden` constraint forbids any subtree in the program that matches a given template tree. Such a template tree can consist of 3 node types: -* `RuleNode(1)`. Matches exactly the given rule. -* `DomainRuleNode(BitVector((0, 0, 0, 1, 1)), children)`. Matches any rule in its bitvector domain. In this case, rule 4 and 5. -* `VarNode(:A)`. Matches any subtree. If another VarNode of the same name is used, the subtrees have to be the same. - - -```julia -#this constraint forbids A+A and A*A -constraint = Forbidden(DomainRuleNode(BitVector((0, 0, 0, 1, 1)), [VarNode(:A), VarNode(:A)])) - -# Without this constraint, we encounter 154 programs -clearconstraints!(grammar) -iter = BFSIterator(grammar, :Int, max_size=5) -println(length(iter)) - -# With this constraint, we encounter 106 programs -clearconstraints!(grammar) -addconstraint!(grammar, constraint) -iter = BFSIterator(grammar, :Int, max_size=5) -println(length(iter)) - -``` - - 154 - 106 - - -### Contains Constraint - -The `Contains` constraint enforces that a given rule appears in the program tree at least once. - -In the arithmetic grammar, this constraint can be used to ensure the input symbol `x` is used in the program. Otherwise, the program is just a constant. - - -```julia -clearconstraints!(grammar) -addconstraint!(grammar, Contains(2)) #rule 2 should be used in the program -iter = BFSIterator(grammar, :Int, max_size=3) - -for program ∈ iter - println(rulenode2expr(program, grammar)) -end -``` - - x - -x - -(-x) - 1x - x * x - x * 1 - x + 1 - x + x - 1 + x - - -### Contains Subtree Constraint - -Similarly to the `Contains` constraint, the `ContainsSubtree` can be used to enforce a given template tree is used in the program at least once. - - -```julia -clearconstraints!(grammar) -addconstraint!(grammar, ContainsSubtree(RuleNode(times, [RuleNode(x), RuleNode(x)]))) #x*x should be in the program tree -iter = BFSIterator(grammar, :Int, max_size=4) - -for program ∈ iter - println(rulenode2expr(program, grammar)) -end -``` - - x * x - -(x * x) - - -### Ordered Constraint - -The `Ordered` constraint enforces an `<=` ordering on a provided list of variables. With this constraint, we can break symmetries based on commutativity. For example, `1+x` and `x+1` are semantically equivalent. By imposing an `Ordered` constraint, we can eliminate one of the symmetric variants. - -To define an `Ordered` constraint, we need to provide it with a template tree including at least two differently named `VarNode`s. And additionally, an ordering of the variables in the tree. - -In the upcoming example we will set up a template tree representing `a+b` and `a*b`. -Then, we will impose an ordering `a<=b` on all the subtrees that match the template. - -The result is that our iterator skips the redundant programs `x+1` and `x*1`, as they are already represented by `1+x` and `1*x`. - - - -```julia -clearconstraints!(grammar) - -template_tree = DomainRuleNode(BitVector((0, 0, 0, 1, 1)), [VarNode(:a), VarNode(:b)]) -order = [:a, :b] - -addconstraint!(grammar, Ordered(template_tree, order)) -iter = BFSIterator(grammar, :Int, max_size=3) - -for program ∈ iter - println(rulenode2expr(program, grammar)) -end - -``` - - 1 - x - -1 - -x - 1 * 1 - -(-1) - 1x - -(-x) - x * x - x + x - 1 + x - 1 + 1 - - -### Forbidden Sequence Constraint - -The `ForbiddenSequence` constraints forbids a given sequence of rule nodes in a vertical path of the tree. - -An optional second argument, `ignore_if`, can be used to overrule the constraint in case any of the rules on the `ignore_if` list are present. - -Below we will define the constraint `ForbiddenSequence([plus, one], ignore_if=[times])`. It forbids an `1` after an `+` unless an `*` disrupts the sequence. - -This constraint will **forbid** the following programs: - -* x + 1 -* x + -1 -* x + -(-1) -* x + (x + 1) -* x * (x + 1) - -But it will **allow** the following program (as * disrupts the sequence): - -* x + (x * 1) - - - -```julia -constraint = ForbiddenSequence([plus, one], ignore_if=[times]) -addconstraint!(grammar, constraint) -iter = BFSIterator(grammar, :Int, max_size=3) - -for program ∈ iter - println(rulenode2expr(program, grammar)) -end - -``` - - 1 - x - -1 - -x - 1 * 1 - -(-1) - 1x - -(-x) - x * x - x + x - - -### Custom Constraint - -To implement a new constraint, we need to define two structs: an `AbstractGrammarConstraint` and an `AbstractLocalConstraint`. - -A **grammar constraint** is a high-level constraint on the grammar itself and does not refer to a location in the tree. For example, the `Forbidden` constraint is responsible for forbidding a template tree everywhere in the tree. To divide the work of constraint propagation, the grammar constraint will post several local constraints that are responsible for propagating the constraint at each particular location. - -A **local constraint** is a rooted version of a grammar constraint. Each local constraint holds a `path` field that points to a location in the tree where this constraint applies. - -Suppose we want to implement a simple custom constraint that forbids a given `rule` twice in a row. - -Each time a new AST node is added to a tree, the `on_new_node` function is called to notify that an unseen node has been added to the tree at path `path`. Our grammar constraint has the opportunity to react to this event. In this example, we will post a new local constraint at the new location using the `post!` function. - -(Don't worry about the `HerbConstraints.` prefixes. Normally, constraints are defined within the HerbConstraints repository, so there is no need to specify the namespace) - - -```julia -""" -Forbids the consecutive application of the specified rule. -For example, CustomConstraint(4) forbids the tree 4(1, 4(1, 1)) as it applies rule 4 twice in a row. -""" -struct ForbidConsecutive <: AbstractGrammarConstraint - rule::Int -end - -""" -Post a local constraint on each new node that appears in the tree -""" -function HerbConstraints.on_new_node(solver::Solver, constraint::ForbidConsecutive, path::Vector{Int}) - HerbConstraints.post!(solver, LocalForbidConsecutive(path, constraint.rule)) -end -``` - - - HerbConstraints.on_new_node - - -Next, we will define our local constraint. This constraint is responsible for propagating the constraint at a given path. The `propagate!` method can use several solver functions to manipulate the tree. The following **tree manipulations** can be used to remove rules from the domain of a hole at a given path: - -* `remove!(solver::Solver, path::Vector{Int}, rule_index::Int)` -* `remove!(solver::Solver, path::Vector{Int}, rules::Vector{Int})` -* `remove_all_but!(solver::Solver, path::Vector{Int}, new_domain::BitVector)` -* `remove_above!(solver::Solver, path::Vector{Int}, rule_index::Int)` -* `remove_below!(solver::Solver, path::Vector{Int}, rule_index::Int)` -* `make_equal!(solver::Solver, node1::AbstractRuleNode, node2::AbstractRuleNode)` (a high level manipulation that requires `node1` and `node2` to be in the tree) - -In addition to tree manipulations, the following solver functions can be used to communicate new information to the solver: - -* `set_infeasible!(solver)`. If a propagator detects an inconsistency, the solver should be notified and cancel any other scheduled propagators. -* `deactivate!(solver, constraint)`. If a constraint is satisfied, it should deactivate itself to prevent re-propagation. -* `post!(solver, constraint)` A constraint is allowed to post new local constraints. This might be helpful if a constraint can be reduced to a smaller constraint. - -The solver manages all constraints and the program tree we propagate on. Applying tree manipulations might cause a chain reaction of other propagators, so the shape of the tree might update as we propagate. The get the latest information about the tree, we should use the following getter functions: - -* `get_tree(solver)` returns the root node of the current (partial) program tree -* `isfeasible(solver)` returns the a flag indicating if the solver is not violating any (other) constriants. -* `get_path(solver, node)` returns the path at which the node is located. -* `get_node_at_location(solver, path)` returns the node that is currently at the given path (be aware that this instance might be replaced by manipulations). -* `get_hole_at_location(solver, path)` same as get node at location, but asserts the node is a hole (domain size >= 2). - -To get information about a node, we can use the following getter functions: - -* `isfilled(node)`. Returns true if the node is a `RuleNode` or has domain size 1. -* `get_rule(node)`. Get the rule of a filled node. -* `get_children(node)`. Get the children of a node. -* `node.domain[rule]`. Given the node is a hole, return true if `rule` is in the domain. - -Finally, another useful function for propagators is `pattern_match(node1, node2)`. This function compares two trees and returns a `PatternMatchResult` that indicates if the nodes match, and potentially indicate which holes need to be filled to complete the match. - - - -```julia -""" -Forbids the consecutive application of the specified rule at path `path`. -""" -struct LocalForbidConsecutive <: AbstractLocalConstraint - path::Vector{Int} - rule::Int -end - -""" -Propagates the constraints by preventing a consecutive application of the specified rule. -""" -function HerbConstraints.propagate!(solver::Solver, constraint::LocalForbidConsecutive) - node = get_node_at_location(solver, constraint.path) - if isfilled(node) - if get_rule(node) == constraint.rule - #the specified rule is used, make sure the rule will not be used by any of the children - for (i, child) ∈ enumerate(get_children(node)) - if isfilled(child) - if get_rule(child) == constraint.rule - #the specified rule was used twice in a row, which is violating the constraint - set_infeasible!(solver) - return - end - elseif child.domain[constraint.rule] - child_path = push!(copy(constraint.path), i) - remove!(solver, child_path, constraint.rule) # remove the rule from the domain of the child - end - end - end - elseif node.domain[constraint.rule] - #our node is a hole with the specified rule in its domain - #we will now check if any of the children already uses the specified rule - softfail = false - for (i, child) ∈ enumerate(get_children(node)) - if isfilled(child) - if get_rule(child) == constraint.rule - #the child holds the specified rule, so the parent cannot have this rule - remove!(solver, constraint.path, constraint.rule) - end - elseif child.domain[constraint.rule] - #the child is a hole and contains the specified node. since there are 2 holes involved, we will softfail. - softfail = true - end - end - if softfail - #we cannot deactivate the constraint, because it needs to be repropagated - return - end - end - - #the constraint is satisfied and can be deactivated - HerbConstraints.deactivate!(solver, constraint) -end -``` - - - HerbConstraints.propagate! - - -Posting a local constraint will trigger the initial propagation. To re-propagate, the constraint needs to be rescheduled for propagation. - -Whenever the tree is manipulated, we will make a `shouldschedule` check to see if our constraint needs to be rescheduled for propagation based on the manipulation. - -In our case, we want to repropagate if either: -* a tree manipulation occured at the `constraint.path` -* a tree manipulation occured at the child of the `constraint.path` - - -```julia - -""" -Gets called whenever an tree manipulation occurs at the given `path`. -Returns true iff the `constraint` should be rescheduled for propagation. -""" -function HerbConstraints.shouldschedule(solver::Solver, constraint::LocalForbidConsecutive, path::Vector{Int})::Bool - return (path == constraint.path) || (path == constraint.path[1:end-1]) -end - -``` - - - HerbConstraints.shouldschedule - - -With all the components implemented, we can do a constrained enumeration using our new `ForbidConsecutive` constraint. - - -```julia -clearconstraints!(grammar) - -addconstraint!(grammar, ForbidConsecutive(minus)) -addconstraint!(grammar, ForbidConsecutive(plus)) -addconstraint!(grammar, ForbidConsecutive(times)) - -iter = BFSIterator(grammar, :Int, max_size=6) - -for program ∈ iter - println(rulenode2expr(program, grammar)) -end -``` - - 1 - x - -1 - -x - 1 * 1 - 1x - x * x - x * 1 - x + 1 - x + x - 1 + x - 1 + 1 - 1 * -1 - 1 * -x - x * -x - x * -1 - x + -1 - x + -x - 1 + -x - -(1 * 1) - 1 + -1 - -(1x) - -(x * x) - -(x * 1) - -((x + 1)) - -((x + x)) - -1 * 1 - -((1 + x)) - -1 * x - -((1 + 1)) - -x * x - -x * 1 - -x + 1 - -x + x - -1 + x - -1 + 1 - (1 + 1) * 1 - -(1 * -1) - (1 + 1) * x - -(1 * -x) - (1 + x) * x - -(x * -x) - (1 + x) * 1 - -(x * -1) - (x + x) * 1 - -((x + -1)) - (x + x) * x - -((x + -x)) - (x + 1) * x - -((1 + -x)) - (x + 1) * 1 - -((1 + -1)) - x * 1 + 1 - x * 1 + x - x * x + x - x * x + 1 - 1x + 1 - 1x + x - -(-1 * 1) - 1 * 1 + x - -(-1 * x) - 1 * 1 + 1 - -(-x * x) - -(-x * 1) - -((-x + 1)) - 1 * (1 + 1) - -((-x + x)) - 1 * (1 + x) - -((-1 + x)) - 1 * (x + x) - -((-1 + 1)) - 1 * (x + 1) - x * (x + 1) - x * (x + x) - x * (1 + x) - x * (1 + 1) - -1 * -1 - x + 1 * 1 - -1 * -x - x + 1x - -x * -x - x + x * x - -x * -1 - x + x * 1 - -x + -1 - 1 + x * 1 - -x + -x - 1 + x * x - -1 + -x - 1 + 1x - -1 + -1 - 1 + 1 * 1 - -1 * (1 + 1) - 1 * (1 + -1) - -1 * (1 + x) - 1 * (1 + -x) - -1 * (x + x) - 1 * (x + -x) - -1 * (x + 1) - 1 * (x + -1) - -x * (x + 1) - x * (x + -1) - -x * (x + x) - x * (x + -x) - -x * (1 + x) - x * (1 + -x) - -x * (1 + 1) - x * (1 + -1) - -x + 1 * 1 - x + 1 * -1 - -x + 1x - x + 1 * -x - -x + x * x - x + x * -x - -x + x * 1 - x + x * -1 - -1 + x * 1 - 1 + x * -1 - -1 + x * x - 1 + x * -x - -1 + 1x - 1 + 1 * -x - -1 + 1 * 1 - 1 + 1 * -1 - 1 * -(1 * 1) - (1 + -1) * 1 - 1 * -(1x) - (1 + -1) * x - 1 * -(x * x) - (1 + -x) * x - 1 * -(x * 1) - (1 + -x) * 1 - 1 * -((x + 1)) - (x + -x) * 1 - 1 * -((x + x)) - (x + -x) * x - 1 * -((1 + x)) - (x + -1) * x - 1 * -((1 + 1)) - (x + -1) * 1 - x * -((1 + 1)) - x * -1 + 1 - x * -((1 + x)) - x * -1 + x - x * -((x + x)) - x * -x + x - x * -((x + 1)) - x * -x + 1 - x * -(x * 1) - 1 * -x + 1 - x * -(x * x) - 1 * -x + x - x * -(1x) - 1 * -1 + x - x * -(1 * 1) - 1 * -1 + 1 - x + -(1 * 1) - x + -(1x) - 1 * (-1 + 1) - x + -(x * x) - 1 * (-1 + x) - x + -(x * 1) - 1 * (-x + x) - x + -((x + 1)) - 1 * (-x + 1) - x + -((x + x)) - x * (-x + 1) - x + -((1 + x)) - x * (-x + x) - x + -((1 + 1)) - x * (-1 + x) - 1 + -((1 + 1)) - x * (-1 + 1) - 1 + -((1 + x)) - x + -1 * 1 - 1 + -((x + x)) - x + -1 * x - 1 + -((x + 1)) - x + -x * x - 1 + -(x * 1) - x + -x * 1 - 1 + -(x * x) - 1 + -x * 1 - 1 + -(1x) - 1 + -x * x - 1 + -(1 * 1) - 1 + -1 * x - 1 + -1 * 1 - -(1 * 1) * 1 - -(1 * 1) * x - -((1 + 1) * 1) - -(1x) * x - -((1 + 1) * x) - -(1x) * 1 - -((1 + x) * x) - -(x * x) * 1 - -((1 + x) * 1) - -(x * x) * x - -((x + x) * 1) - -(x * 1) * x - -((x + x) * x) - -(x * 1) * 1 - -((x + 1) * x) - -((x + 1)) * 1 - -((x + 1) * 1) - -((x + 1)) * x - -((x * 1 + 1)) - -((x + x)) * x - -((x * 1 + x)) - -((x + x)) * 1 - -((x * x + x)) - -((1 + x)) * 1 - -((x * x + 1)) - -((1 + x)) * x - -((1x + 1)) - -((1 + 1)) * x - -((1x + x)) - -((1 + 1)) * 1 - -((1 * 1 + x)) - -((1 + 1)) + 1 - -((1 * 1 + 1)) - -((1 + 1)) + x - -((1 + x)) + x - -(-1 * -1) - -((1 + x)) + 1 - -(-1 * -x) - -((x + x)) + 1 - -(-x * -x) - -((x + x)) + x - -(-x * -1) - -((x + 1)) + x - -((-x + -1)) - -((x + 1)) + 1 - -((-x + -x)) - -(x * 1) + 1 - -((-1 + -x)) - -(x * 1) + x - -((-1 + -1)) - -(x * x) + x - -(x * x) + 1 - (-1 + 1) * 1 - -(1x) + 1 - (-1 + 1) * x - -(1x) + x - (-1 + x) * x - -(1 * 1) + x - (-1 + x) * 1 - -(1 * 1) + 1 - (-x + x) * 1 - (-x + x) * x - (1 + 1) * -1 - (-x + 1) * x - (1 + 1) * -x - (-x + 1) * 1 - (1 + x) * -x - -x * 1 + 1 - (1 + x) * -1 - -x * 1 + x - (x + x) * -1 - -x * x + x - (x + x) * -x - -x * x + 1 - (x + 1) * -x - -1 * x + 1 - (x + 1) * -1 - -1 * x + x - x * 1 + -1 - -1 * 1 + x - x * 1 + -x - -1 * 1 + 1 - x * x + -x - x * x + -1 - -(1 * (1 + 1)) - 1x + -1 - -(1 * (1 + x)) - 1x + -x - -(1 * (x + x)) - 1 * 1 + -x - -(1 * (x + 1)) - 1 * 1 + -1 - -(x * (x + 1)) - -(x * (x + x)) - -(x * (1 + x)) - -(x * (1 + 1)) - -((x + 1 * 1)) - -((x + 1x)) - -((x + x * x)) - -((x + x * 1)) - -((1 + x * 1)) - -((1 + x * x)) - -((1 + 1x)) - -((1 + 1 * 1)) - diff --git a/docs/src/tutorials/getting_started_with_herb.html b/docs/src/tutorials/getting_started_with_herb.html index 990d2a6..c21184d 100644 --- a/docs/src/tutorials/getting_started_with_herb.html +++ b/docs/src/tutorials/getting_started_with_herb.html @@ -3,11 +3,11 @@ diff --git a/docs/src/tutorials/getting_started_with_herb.ipynb b/docs/src/tutorials/getting_started_with_herb.ipynb deleted file mode 100644 index f9690e0..0000000 --- a/docs/src/tutorials/getting_started_with_herb.ipynb +++ /dev/null @@ -1,346 +0,0 @@ -{ - "cells": [ - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Search\n", - "\n", - "This notebook describes how you can search a program space as defined by a grammar.\n", - "Specifically, we will look at example-based search, where the goal is to find a program that is able to transform the inputs of every example to the corresponding output." - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Setup\n", - "First, we start with the setup. We need to access to all the function in the Herb.jl framework." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret, HerbConstraints" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Defining the program space\n", - "\n", - "Next, we start by creating a grammar. We define a context-free grammar (cfg) as a [`HerbGrammar.ContextSpecificGrammar`](@ref) without any constraints. A cfg is just a simple set of production rules for defining combinations of terminal symbols (in our case real numbers). \n", - "\n", - "Contrary, we could define a context-sensitive grammar, when the production rules only hold in a certain context. However, for more information on this, please see our tutorial on [defining grammars](defining_grammars.md).\n", - "\n", - "For now, we specify a simple grammar for dealing with integers and explain all the rules individually:\n", - "\n", - "1. First, we specify our interval `[0:9]` on real numbers and also constrain them to be integer.\n", - "2. Then, we can also use the variable `x` to hold an integer.\n", - "3. The third rule determines we can add two integers.\n", - "4. The fourth rule determines we can subtract an integer from another.\n", - "5. Finally, we also allow the multiplication of two integers.\n", - "\n", - "If you run this cell, you can see all the rules rolled out." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "1: Real = 0\n", - "2: Real = 1\n", - "3: Real = 2\n", - "4: Real = 3\n", - "5: Real = 4\n", - "6: Real = 5\n", - "7: Real = 6\n", - "8: Real = 7\n", - "9: Real = 8\n", - "10: Real = 9\n", - "11: Real = x\n", - "12: Real = Real + Real\n", - "13: Real = Real - Real\n", - "14: Real = Real * Real\n" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "g = HerbGrammar.@cfgrammar begin\n", - " Real = |(0:9)\n", - " Real = x\n", - " Real = Real + Real\n", - " Real = Real - Real\n", - " Real = Real * Real\n", - "end" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Defining the problem" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As mentioned before, we are looking at example-based search. \n", - "This means that the problem is defined by a set of input-output examples. \n", - "A single example hence consists of an input and an output.\n", - "The input is defined as a dictionary, with a value assigned to each variable in the grammar.\n", - "It is important to write the variable name as a `Symbol` instead of a string.\n", - "A `Symbol` in Julia is written with a colon prefix, i.e. `:x`. \n", - "The output of the input-output example is just a single value for this specific grammar, but could possibly relate to e.g. arrays of values, too.\n", - "\n", - "In the cell below we automatically generate some examples for `x` assigning values `1-5`." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "5-element Vector{IOExample}:\n", - " IOExample(Dict{Symbol, Any}(:x => 1), 8)\n", - " IOExample(Dict{Symbol, Any}(:x => 2), 11)\n", - " IOExample(Dict{Symbol, Any}(:x => 3), 14)\n", - " IOExample(Dict{Symbol, Any}(:x => 4), 17)\n", - " IOExample(Dict{Symbol, Any}(:x => 5), 20)" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# Create input-output examples\n", - "examples = [HerbSpecification.IOExample(Dict(:x => x), 3x+5) for x ∈ 1:5]" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now that we have some input-output examples, we can define the problem. \n", - "Next to the examples, a problem also contains a name meant to link to the file path, which can be used to keep track of current examples. \n", - "For now, this is irrelevant, and you can give the program any name you like." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "Problem{Vector{IOExample}}(\"example\", IOExample[IOExample(Dict{Symbol, Any}(:x => 1), 8), IOExample(Dict{Symbol, Any}(:x => 2), 11), IOExample(Dict{Symbol, Any}(:x => 3), 14), IOExample(Dict{Symbol, Any}(:x => 4), 17), IOExample(Dict{Symbol, Any}(:x => 5), 20)])" - ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "problem = HerbSpecification.Problem(\"example\", examples)" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Searching\n", - "\n", - "Now that we have defined the search space and the goal of the search, we can start the search. \n", - "\n", - "Of course, our problem is underdefined as there might be multiple programs that satisfy our examples. \n", - "Let us consider the case where we also have a ternary if-then-else operator and standard boolean operators in our grammar: we could synthesize the program `x ≤ 5 ? 3x+5 : 0`. \n", - "This program satisfies all our examples, but we don't expect it to generalize very well.\n", - "\n", - "To search through a program space, we first need to define a [`HerbSearch.ProgramIterator`](@ref), which can be instantiated with different iterators, for now we use a simple [`HerbSearch.BFSIterator`](@ref). For more advanced search methods check out our tutorial on [advanced search](.advanced_search.md). For more information about iterators, check out our tutorial on [working with interpreters](.working_with_interpreters.md). \n", - "\n", - "In general, we assume that a smaller program is more general than a larger program. \n", - "Therefore, we search for the smallest program in our grammar that satisfies our examples. \n", - "This can be done using a breadth-first search over the program/search space.\n", - "\n", - "This search is very basic; it makes use of an enumeration technique, where we enumerate programs one-by-one until we find a program that matches our examples. The search procedure has a built-in default evaluator to verify the candidate programs with the given input. The search procedure also has a built-in search procedure using breadth-first search. \n", - "\n", - "So, we only need to give our grammar and the problem to our search procedure, along with a starting `Symbol`, in our case a `Real`. " - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "BFSIterator(GenericSolver(1: Real = 0\n", - "2: Real = 1\n", - "3: Real = 2\n", - "4: Real = 3\n", - "5: Real = 4\n", - "6: Real = 5\n", - "7: Real = 6\n", - "8: Real = 7\n", - "9: Real = 8\n", - "10: Real = 9\n", - "11: Real = x\n", - "12: Real = Real + Real\n", - "13: Real = Real - Real\n", - "14: Real = Real * Real\n", - ", SolverState(hole[Bool[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]],, Set{AbstractLocalConstraint}(), true), DataStructures.PriorityQueue{AbstractLocalConstraint, Int64, Base.Order.ForwardOrdering}(), nothing, true, false, 9223372036854775807, 9223372036854775807))" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "iterator = BFSIterator(g, :Real)" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(12{14{4,11}6}, optimal_program)" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "synth(problem, iterator)" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As you can see, the search procedure found the correct program!" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Defining the search procedure\n", - "\n", - "In the previous case, we used the built-ins of the search procedure. However, we can also give a custom enumerator to the search procedure and define a few more values.\n", - "\n", - "We first define a new problem to test with, we are looking for the programs that can compute the value `167`. We immediately pass the examples to the problem and then set up the new search.\n", - "\n", - "Search is done by passing the grammar, the problem and the starting point like before. We now also specify the enumeration function to be used, and now we use depth-first search. Then, we give the maximum depth of the programs we want to search for `(3)`, the maximum number of nodes in the Abstract Syntax Tree that exists during search `(10)`, and the maximum time in seconds allowed for the search." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "(14{14{4,9}8}, optimal_program)" - ] - } - ], - "source": [ - "problem = HerbSpecification.Problem(\"example2\", [HerbSpecification.IOExample(Dict(:x => x), 168) for x ∈ 1:5])\n", - "iterator = HerbSearch.BFSIterator(g, :Real, max_depth=4, max_size=30)\n", - "expr = HerbSearch.synth(problem, iterator)\n", - "print(expr)" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We see that our synthesizer can find a program to construct the value `168`, though a fun experiment would be trying to get the value `167`, what do you think would happen? You can try below, using the same iterator.\n", - "\n", - "In any case, this concludes our first introduction to the `Herb.jl` program synthesis framework. You can see more examples in this repository, or explore yourself. Enjoy!" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33m\u001b[1m┌ \u001b[22m\u001b[39m\u001b[33m\u001b[1mWarning: \u001b[22m\u001b[39mUniformSolver is iterating over more than 1000000 solutions...\n", - "\u001b[33m\u001b[1m└ \u001b[22m\u001b[39m\u001b[90m@ HerbSearch ~/.julia/packages/HerbSearch/PIeQW/src/uniform_iterator.jl:96\u001b[39m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "(12{13{6,1}14{14{3,10}10}}, optimal_program)" - ] - } - ], - "source": [ - "problem = HerbSpecification.Problem(\"example3\", [HerbSpecification.IOExample(Dict(:x => x), 167) for x ∈ 1:5])\n", - "expr = HerbSearch.synth(problem, iterator)\n", - "print(expr)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Julia 1.10.3", - "language": "julia", - "name": "julia-1.10" - }, - "language_info": { - "file_extension": ".jl", - "mimetype": "application/julia", - "name": "julia", - "version": "1.10.3" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/docs/src/tutorials/getting_started_with_herb.md b/docs/src/tutorials/getting_started_with_herb.md deleted file mode 100644 index 3467ee9..0000000 --- a/docs/src/tutorials/getting_started_with_herb.md +++ /dev/null @@ -1,180 +0,0 @@ -# Search - -This notebook describes how you can search a program space as defined by a grammar. -Specifically, we will look at example-based search, where the goal is to find a program that is able to transform the inputs of every example to the corresponding output. - -### Setup -First, we start with the setup. We need to access to all the function in the Herb.jl framework. - - -```julia -using HerbGrammar, HerbSpecification, HerbSearch, HerbInterpret, HerbConstraints -``` - -### Defining the program space - -Next, we start by creating a grammar. We define a context-free grammar (cfg) as a [`HerbGrammar.ContextSpecificGrammar`](@ref) without any constraints. A cfg is just a simple set of production rules for defining combinations of terminal symbols (in our case real numbers). - -Contrary, we could define a context-sensitive grammar, when the production rules only hold in a certain context. However, for more information on this, please see our tutorial on [defining grammars](defining_grammars.md). - -For now, we specify a simple grammar for dealing with integers and explain all the rules individually: - -1. First, we specify our interval `[0:9]` on real numbers and also constrain them to be integer. -2. Then, we can also use the variable `x` to hold an integer. -3. The third rule determines we can add two integers. -4. The fourth rule determines we can subtract an integer from another. -5. Finally, we also allow the multiplication of two integers. - -If you run this cell, you can see all the rules rolled out. - - -```julia -g = HerbGrammar.@cfgrammar begin - Real = |(0:9) - Real = x - Real = Real + Real - Real = Real - Real - Real = Real * Real -end -``` - - - 1: Real = 0 - 2: Real = 1 - 3: Real = 2 - 4: Real = 3 - 5: Real = 4 - 6: Real = 5 - 7: Real = 6 - 8: Real = 7 - 9: Real = 8 - 10: Real = 9 - 11: Real = x - 12: Real = Real + Real - 13: Real = Real - Real - 14: Real = Real * Real - - - -### Defining the problem - -As mentioned before, we are looking at example-based search. -This means that the problem is defined by a set of input-output examples. -A single example hence consists of an input and an output. -The input is defined as a dictionary, with a value assigned to each variable in the grammar. -It is important to write the variable name as a `Symbol` instead of a string. -A `Symbol` in Julia is written with a colon prefix, i.e. `:x`. -The output of the input-output example is just a single value for this specific grammar, but could possibly relate to e.g. arrays of values, too. - -In the cell below we automatically generate some examples for `x` assigning values `1-5`. - - -```julia -# Create input-output examples -examples = [HerbSpecification.IOExample(Dict(:x => x), 3x+5) for x ∈ 1:5] -``` - - - 5-element Vector{IOExample}: - IOExample(Dict{Symbol, Any}(:x => 1), 8) - IOExample(Dict{Symbol, Any}(:x => 2), 11) - IOExample(Dict{Symbol, Any}(:x => 3), 14) - IOExample(Dict{Symbol, Any}(:x => 4), 17) - IOExample(Dict{Symbol, Any}(:x => 5), 20) - - -Now that we have some input-output examples, we can define the problem. -Next to the examples, a problem also contains a name meant to link to the file path, which can be used to keep track of current examples. -For now, this is irrelevant, and you can give the program any name you like. - - -```julia -problem = HerbSpecification.Problem("example", examples) -``` - - - Problem{Vector{IOExample}}("example", IOExample[IOExample(Dict{Symbol, Any}(:x => 1), 8), IOExample(Dict{Symbol, Any}(:x => 2), 11), IOExample(Dict{Symbol, Any}(:x => 3), 14), IOExample(Dict{Symbol, Any}(:x => 4), 17), IOExample(Dict{Symbol, Any}(:x => 5), 20)]) - - -### Searching - -Now that we have defined the search space and the goal of the search, we can start the search. - -Of course, our problem is underdefined as there might be multiple programs that satisfy our examples. -Let us consider the case where we also have a ternary if-then-else operator and standard boolean operators in our grammar: we could synthesize the program `x ≤ 5 ? 3x+5 : 0`. -This program satisfies all our examples, but we don't expect it to generalize very well. - -To search through a program space, we first need to define a [`HerbSearch.ProgramIterator`](@ref), which can be instantiated with different iterators, for now we use a simple [`HerbSearch.BFSIterator`](@ref). For more advanced search methods check out our tutorial on [advanced search](.advanced_search.md). For more information about iterators, check out our tutorial on [working with interpreters](.working_with_interpreters.md). - -In general, we assume that a smaller program is more general than a larger program. -Therefore, we search for the smallest program in our grammar that satisfies our examples. -This can be done using a breadth-first search over the program/search space. - -This search is very basic; it makes use of an enumeration technique, where we enumerate programs one-by-one until we find a program that matches our examples. The search procedure has a built-in default evaluator to verify the candidate programs with the given input. The search procedure also has a built-in search procedure using breadth-first search. - -So, we only need to give our grammar and the problem to our search procedure, along with a starting `Symbol`, in our case a `Real`. - - -```julia -iterator = BFSIterator(g, :Real) -``` - - - BFSIterator(1: Real = 0 - 2: Real = 1 - 3: Real = 2 - 4: Real = 3 - 5: Real = 4 - 6: Real = 5 - 7: Real = 6 - 8: Real = 7 - 9: Real = 8 - 10: Real = 9 - 11: Real = x - 12: Real = Real + Real - 13: Real = Real - Real - 14: Real = Real * Real - , :Real, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807) - - - -```julia -synth(problem, iterator) -``` - - - (12{14{11,4}6}, optimal_program) - - -As you can see, the search procedure found the correct program! - -### Defining the search procedure - -In the previous case, we used the built-ins of the search procedure. However, we can also give a custom enumerator to the search procedure and define a few more values. - -We first define a new problem to test with, we are looking for the programs that can compute the value `167`. We immediately pass the examples to the problem and then set up the new search. - -Search is done by passing the grammar, the problem and the starting point like before. We now also specify the enumeration function to be used, and now we use depth-first search. Then, we give the maximum depth of the programs we want to search for `(3)`, the maximum number of nodes in the Abstract Syntax Tree that exists during search `(10)`, and the maximum time in seconds allowed for the search. - - -```julia -problem = HerbSpecification.Problem("example2", [HerbSpecification.IOExample(Dict(:x => x), 168) for x ∈ 1:5]) -iterator = HerbSearch.BFSIterator(g, :Real, max_depth=4, max_size=30, max_time=180) -expr = HerbSearch.synth(problem, iterator) -print(expr) -``` - - (14{7,14{5,8}}, optimal_program) - -We see that our synthesizer can find a program to construct the value `168`, though a fun experiment would be trying to get the value `167`, what do you think would happen? You can try below, using the same iterator. - -In any case, this concludes our first introduction to the `Herb.jl` program synthesis framework. You can see more examples in this repository, or explore yourself. Enjoy! - - -```julia -problem = HerbSpecification.Problem("example3", [HerbSpecification.IOExample(Dict(:x => x), 167) for x ∈ 1:5]) -expr = HerbSearch.synth(problem, iterator) -print(expr) -``` - - (12{14{7,14{10,4}}6}, optimal_program) diff --git a/docs/src/tutorials/working_with_interpreters.html b/docs/src/tutorials/working_with_interpreters.html index e4a9971..f9ae880 100644 --- a/docs/src/tutorials/working_with_interpreters.html +++ b/docs/src/tutorials/working_with_interpreters.html @@ -3,11 +3,11 @@ diff --git a/docs/src/tutorials/working_with_interpreters.ipynb b/docs/src/tutorials/working_with_interpreters.ipynb deleted file mode 100644 index ce05015..0000000 --- a/docs/src/tutorials/working_with_interpreters.ipynb +++ /dev/null @@ -1,271 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Using the Julia interpreter\n", - "\n", - "To know how good a candidate program is, program synthesisers execute them. The easiest way to execute a program is to rely on Julia itself. To leverage the Julia interpreter, you only have to ensure that your programs are valid Julia expressions. \n", - "\n", - "First, let's import the necessary packages" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "using HerbGrammar, HerbInterpret" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "Now, assume the following grammar." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "1: Number = 1\n", - "2: Number = 2\n", - "3: Number = x\n", - "4: Number = Number + Number\n", - "5: Number = Number * Number\n" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "g = @cfgrammar begin\n", - " Number = |(1:2)\n", - " Number = x\n", - " Number = Number + Number\n", - " Number = Number * Number\n", - " end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's construct a program `x+3`, which would correspond to the following `RuleNode` representation" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "4{3,1}" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "myprog = RuleNode(4,[RuleNode(3),RuleNode(1)])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To run this program, we have to convert it into a Julia expression, which we can do in the following way:" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - ":(x + 1)" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "myprog_julia = rulenode2expr(myprog, g)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now we have a valid Julia expression, but we are still missing one key ingredient: we have to inform the interpreter about the special symbols. In our case, these are `:x` and `:+`. To do so, we need to create a symbol table, which is nothing more than a dictionary mapping symbols to their values:" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "Dict{Symbol, Any} with 2 entries:\n", - " :+ => +\n", - " :x => 2" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "symboltable = Dict{Symbol,Any}(:x => 2, :+ => +)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now we can execute our program through the defaul interpreter available in `HerbInterpret`:\n" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "3" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "interpret(symboltable, myprog_julia)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And that's it!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Defining a custom interpreter\n", - "\n", - "A disadvantage of the default Julia interpreter is that it needs to traverse abstract syntax tree twice -- once to convert it into a Julia expression, and the second time to execute that expression. Program execution is regularly the most consuming part of the entire pipeline and, by eliminating one of these steps, we can cut the runtime in half.\n", - "\n", - "We can define an interpreter that works directly over `RuleNode`s. \n", - "Consider the scenario in which we want to write programs for robot navigation: imagine a 2D world in which the robot can move around and pick up a ball. The programs we could write direct the robot to go up, down, left, and right. For convenience, the programming language also offers conditionals and loops:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "grammar_robots = @csgrammar begin\n", - " Start = Sequence #1\n", - "\n", - " Sequence = Operation #2\n", - " Sequence = (Operation; Sequence) #3\n", - " Operation = Transformation #4\n", - " Operation = ControlStatement #5\n", - "\n", - " Transformation = moveRight() | moveDown() | moveLeft() | moveUp() | drop() | grab() #6\n", - " ControlStatement = IF(Condition, Sequence, Sequence) #12\n", - " ControlStatement = WHILE(Condition, Sequence) #13\n", - "\n", - " Condition = atTop() | atBottom() | atLeft() | atRight() | notAtTop() | notAtBottom() | notAtLeft() | notAtRight() #14\n", - "end" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This grammar specifies a simple sequential program with instructions for the robot. A couple of example programs:\n", - " - `moveRight(); moveLeft(); drop()`\n", - " - WHILE(notAtTop(), moveUp())\n", - "\n", - "The idea behind this programming language is that the program specifies a set of transformations over a state of the robot world. Thus, a program can only be executed over a particular state. In this case, the state represents the size of the 2D world, the current position of a robot, the current position of a ball, and whether the robot is currently holding a ball. The execution of a particular instruction acts as a state transformation: each instruction takes a state as an input, transforms it, and passes it to the subsequent instruction. For example, execution of the program `moveRight(); moveLeft(); drop()` would proceed as:\n", - " 1. take an input state, \n", - " 2. pass it to the `moveRight()` instruction,\n", - " 3. pass the output of `moveRight()` to `moveLeft()` instructions,\n", - " 4. pass the output of `moveLeft()` to `drop()`,\n", - " 5. return the output of `drop()`.\n", - "\n", - "\n", - " The following is only one possible way to implement a custom interpreter, but it demonstrates a general template that can always be followed.\n", - "\n", - " We want to implement the following function, which would take in a program in the form of a `RuleNode`, a grammar, and a starting state, and return the state obtained after executing the program:\n", - "\n", - " interpret(prog::AbstractRuleNode, grammar::ContextSensitiveGrammar, state::RobotState)::RobotState\n", - "\n", - " As `RuleNode`s only store indices of derivation rules from the grammar, not the functions themselves, we will first pull the function call associated with every derivation rule. In Julia, this is indicated by the top-level symbol of the rules. For example, the top-level symbol for the derivation rule 6 is `:moveRight`; for rule 12, that is `:IF`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The remaining functions follow a similar idea. (You can see the full implementation of this interpreter [here](https://github.com/Herb-AI/HerbBenchmarks.jl/blob/new-robots/src/data/Robots_2020/robots_primitives.jl))." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "PLUTO_PROJECT_TOML_CONTENTS = \"\"\"\n", - "[deps]\n", - "HerbGrammar = \"4ef9e186-2fe5-4b24-8de7-9f7291f24af7\"\n", - "HerbInterpret = \"5bbddadd-02c5-4713-84b8-97364418cca7\"\n", - "\n", - "[compat]\n", - "HerbGrammar = \"~0.3.0\"\n", - "HerbInterpret = \"~0.1.3\"\n", - "\"\"\"" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Julia 1.9.4", - "language": "julia", - "name": "julia-1.9" - }, - "language_info": { - "file_extension": ".jl", - "mimetype": "application/julia", - "name": "julia", - "version": "1.9.4" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/docs/src/tutorials/working_with_interpreters.md b/docs/src/tutorials/working_with_interpreters.md deleted file mode 100644 index d1cc62e..0000000 --- a/docs/src/tutorials/working_with_interpreters.md +++ /dev/null @@ -1,141 +0,0 @@ -# Using the Julia interpreter - -To know how good a candidate program is, program synthesisers execute them. The easiest way to execute a program is to rely on Julia itself. To leverage the Julia interpreter, you only have to ensure that your programs are valid Julia expressions. - -First, let's import the necessary packages - - -```julia -using HerbGrammar, HerbInterpret -``` - - -Now, assume the following grammar. - - -```julia -g = @cfgrammar begin - Number = |(1:2) - Number = x - Number = Number + Number - Number = Number * Number - end -``` - - - 1: Number = 1 - 2: Number = 2 - 3: Number = x - 4: Number = Number + Number - 5: Number = Number * Number - - - -Let's construct a program `x+3`, which would correspond to the following `RuleNode` representation - - -```julia -myprog = RuleNode(4,[RuleNode(3),RuleNode(1)]) -``` - - - 4{3,1} - - -To run this program, we have to convert it into a Julia expression, which we can do in the following way: - - -```julia -myprog_julia = rulenode2expr(myprog, g) -``` - - - :(x + 1) - - -Now we have a valid Julia expression, but we are still missing one key ingredient: we have to inform the interpreter about the special symbols. In our case, these are `:x` and `:+`. To do so, we need to create a symbol table, which is nothing more than a dictionary mapping symbols to their values: - - -```julia -symboltable = Dict{Symbol,Any}(:x => 2, :+ => +) -``` - - - Dict{Symbol, Any} with 2 entries: - :+ => + - :x => 2 - - -Now we can execute our program through the defaul interpreter available in `HerbInterpret`: - - - -```julia -interpret(symboltable, myprog_julia) -``` - - - 3 - - -And that's it! - -# Defining a custom interpreter - -A disadvantage of the default Julia interpreter is that it needs to traverse abstract syntax tree twice -- once to convert it into a Julia expression, and the second time to execute that expression. Program execution is regularly the most consuming part of the entire pipeline and, by eliminating one of these steps, we can cut the runtime in half. - -We can define an interpreter that works directly over `RuleNode`s. -Consider the scenario in which we want to write programs for robot navigation: imagine a 2D world in which the robot can move around and pick up a ball. The programs we could write direct the robot to go up, down, left, and right. For convenience, the programming language also offers conditionals and loops: - - -```julia -grammar_robots = @csgrammar begin - Start = Sequence #1 - - Sequence = Operation #2 - Sequence = (Operation; Sequence) #3 - Operation = Transformation #4 - Operation = ControlStatement #5 - - Transformation = moveRight() | moveDown() | moveLeft() | moveUp() | drop() | grab() #6 - ControlStatement = IF(Condition, Sequence, Sequence) #12 - ControlStatement = WHILE(Condition, Sequence) #13 - - Condition = atTop() | atBottom() | atLeft() | atRight() | notAtTop() | notAtBottom() | notAtLeft() | notAtRight() #14 -end -``` - -This grammar specifies a simple sequential program with instructions for the robot. A couple of example programs: - - `moveRight(); moveLeft(); drop()` - - WHILE(notAtTop(), moveUp()) - -The idea behind this programming language is that the program specifies a set of transformations over a state of the robot world. Thus, a program can only be executed over a particular state. In this case, the state represents the size of the 2D world, the current position of a robot, the current position of a ball, and whether the robot is currently holding a ball. The execution of a particular instruction acts as a state transformation: each instruction takes a state as an input, transforms it, and passes it to the subsequent instruction. For example, execution of the program `moveRight(); moveLeft(); drop()` would proceed as: - 1. take an input state, - 2. pass it to the `moveRight()` instruction, - 3. pass the output of `moveRight()` to `moveLeft()` instructions, - 4. pass the output of `moveLeft()` to `drop()`, - 5. return the output of `drop()`. - - - The following is only one possible way to implement a custom interpreter, but it demonstrates a general template that can always be followed. - - We want to implement the following function, which would take in a program in the form of a `RuleNode`, a grammar, and a starting state, and return the state obtained after executing the program: - - interpret(prog::AbstractRuleNode, grammar::ContextSensitiveGrammar, state::RobotState)::RobotState - - As `RuleNode`s only store indices of derivation rules from the grammar, not the functions themselves, we will first pull the function call associated with every derivation rule. In Julia, this is indicated by the top-level symbol of the rules. For example, the top-level symbol for the derivation rule 6 is `:moveRight`; for rule 12, that is `:IF`. - -The remaining functions follow a similar idea. (You can see the full implementation of this interpreter [here](https://github.com/Herb-AI/HerbBenchmarks.jl/blob/new-robots/src/data/Robots_2020/robots_primitives.jl)). - - -```julia -PLUTO_PROJECT_TOML_CONTENTS = """ -[deps] -HerbGrammar = "4ef9e186-2fe5-4b24-8de7-9f7291f24af7" -HerbInterpret = "5bbddadd-02c5-4713-84b8-97364418cca7" - -[compat] -HerbGrammar = "~0.3.0" -HerbInterpret = "~0.1.3" -""" -``` From 8609bb194db4c0a85381176811a56c87ae104850 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Mon, 30 Sep 2024 12:47:35 +0200 Subject: [PATCH 12/13] Change section headings for BFS/DFS. --- docs/src/tutorials/advanced_search.jl | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/src/tutorials/advanced_search.jl b/docs/src/tutorials/advanced_search.jl index 3c42a59..8bb6ee3 100644 --- a/docs/src/tutorials/advanced_search.jl +++ b/docs/src/tutorials/advanced_search.jl @@ -161,18 +161,21 @@ md""""This time we find a solution, although a suboptimal one.""" # ╔═╡ 9b4b21e0-dc6a-43ae-a511-79988ee99001 md""" -## Search methods +## Top-down search Herb.jl provides already implemented, ready-to-use search methods. The core building block of the search is the program iterator, which represents a walk through the program space. All program iterators share the top-level abstract type `ProgramIterator`. For more information on iterators and how to customize them, see [this tutorial](https://herb-ai.github.io/Herb.jl/dev/tutorials/TopDown/). + +First, we explore two fundamental deterministic top-down search algorithms: **breadth-first search (BFS)** and **depth-first search (DFS)**. Both algorithms are implemented using the abstract type `TopDownIterator`, which can be customized through the functions +- `priority_function` +- `derivation_heuristic` +- `hole_heuristic` """ # ╔═╡ 115c02c9-ae0c-4623-a61d-831fc6ad55a2 md""" -### Deterministic search: BFS and DFS - First, we explore two fundamental deterministic top-down search algorithms: **breadth-first search (BFS)** and **depth-first search (DFS)**. Both algorithms are implemented using the abstract type `TopDownIterator`, which can be customized through the functions priority_function, derivation_heuristic, and hole_heuristic. -#### Breadth-First Search +### Breadth-First Search The `BFSIterator` enumerates all possible programs at a given depth before progressing to the next level, ensuring that trees are explored in increasing order of size. This guarantees that smaller programs are evaluated first, and larger, more complex ones are considered only after all smaller ones have been processed. @@ -225,7 +228,7 @@ println(all(p ∈ programs_bfs for p ∈ answer_programs)) # ╔═╡ 0020b79a-6352-4e2d-93f6-2a1d7b03ae2c md""" -#### Depth-First Search +### Depth-First Search The `DFSIterator` explores one branch of the search tree at a time, fully traversing it unitl a correct program is found or the specified `max_depth` is reached. Only after completing the current branch, it proceeds to the next branch. From 27701f8609d273815a815fc41f16c0010f4132e8 Mon Sep 17 00:00:00 2001 From: Pamela Wochner Date: Thu, 3 Oct 2024 09:43:45 +0200 Subject: [PATCH 13/13] Defines priority function for DFS rightmost properly. Removes unnecessary begin...end, and TODO notes. --- docs/src/tutorials/advanced_search.html | 17 ++++++++ docs/src/tutorials/advanced_search.jl | 54 +++++++++++-------------- 2 files changed, 41 insertions(+), 30 deletions(-) create mode 100644 docs/src/tutorials/advanced_search.html diff --git a/docs/src/tutorials/advanced_search.html b/docs/src/tutorials/advanced_search.html new file mode 100644 index 0000000..c43adcc --- /dev/null +++ b/docs/src/tutorials/advanced_search.html @@ -0,0 +1,17 @@ + + + + + + + +
\ No newline at end of file diff --git a/docs/src/tutorials/advanced_search.jl b/docs/src/tutorials/advanced_search.jl index 8bb6ee3..ba70384 100644 --- a/docs/src/tutorials/advanced_search.jl +++ b/docs/src/tutorials/advanced_search.jl @@ -103,13 +103,11 @@ Let's explore how many enumerations are necessary to solve our simple problem. """ # ╔═╡ 3954dd49-07a2-4ec2-91b4-9c9596d5c264 -begin - for i in range(1, 50) +for i in range(1, 50) println(i, " enumerations") iterator = BFSIterator(g_1, :Number, max_depth=i) solution = @time synth(problem_1, iterator) println(solution) - end end # ╔═╡ 9892e91b-9115-4520-9637-f8d7c8905825 @@ -128,12 +126,10 @@ We will use a new example to see the effect of `allow_evaluation_errors`. We beg """ # ╔═╡ 9fb40ceb-8d41-491b-8941-20a8b240eb82 -begin - g_2 = @cfgrammar begin - Number = 1 - List = [] - Index = List[Number] - end +g_2 = @csgrammar begin + Number = 1 + List = [] + Index = List[Number] end # ╔═╡ 94e0d676-a9c7-4291-8696-15301e541c30 @@ -183,10 +179,9 @@ To explore `BFSIterator`, we define another very simple grammar. """ # ╔═╡ 3af650d9-19c6-4351-920d-d2361091f628 -begin g_3 = @cfgrammar begin +g_3 = @csgrammar begin Real = 1 | 2 Real = Real * Real - end end # ╔═╡ 4cb08dba-aea5-4c31-998c-844d1fce8c81 @@ -254,11 +249,22 @@ md""" # ╔═╡ ed198b79-1b95-4531-b148-c1037cfdacf4 md""" -By default, `priority_function` for a `TopDownIterator` is that of a BFS iterator. Hence, we need to provide a new implementation. For this, we can make use of the existing `priority_function` of `DFSIterator`. +By default, `priority_function` for a `TopDownIterator` is that of a BFS iterator. Hence, we need to provide a new implementation. """ -# ╔═╡ d62918e4-4113-45bd-81ea-d17d007b83f5 -priority_function(::DFSIteratorRightmost) = priority_function(DFSIterator()) +# ╔═╡ 75b1abfd-19ed-43f5-ac65-f8ffde76c581 +function priority_function( + ::DFSIteratorRightmost, + ::AbstractGrammar, + ::AbstractRuleNode, + parent_value::Union{Real, Tuple{Vararg{Real}}}, + isrequeued::Bool +) + if isrequeued + return parent_value; + end + return parent_value - 1; +end # ╔═╡ 7480d1e4-e417-4d87-80b7-513a098da70e md""" @@ -351,20 +357,15 @@ Run the following code block to define the iterator and perform the program synt # ╔═╡ 0a30fb40-cd45-4661-a501-ae8e45f1e07e begin - iteratormh = MHSearchIterator(g_4, :X, examples_mh, cost_function, max_depth=3) - programmh = synth(problem_mh, iteratormh) - println("Sollution using MH: ", programmh) + iterator_mh = MHSearchIterator(g_4, :X, examples_mh, cost_function, max_depth=3) + program_mh = synth(problem_mh, iterator_mh) + println("Sollution using MH: ", program_mh) end -# ╔═╡ 82f7ffa5-2bdc-4153-85fa-d7aca063da12 -programmh - # ╔═╡ 700270ea-90bd-474b-91d9-0e5ed329776a md""" ### Very Large Scale Neighbourhood Search -TODO: what do we want to show with the examples? - The second stochastic search method we consider is Very Large Scale Neighbourhood Search (VLSN). In each iteration, the algorithm searches the neighbourhood of the current candidate program for a local optimum, aiming to find a better candidate solution. For more information, see [this article](https://backend.orbit.dtu.dk/ws/portalfiles/portal/5293785/Pisinger.pdf). @@ -436,15 +437,10 @@ initial_temperature2 = 2 # ╔═╡ 4ff69f0a-6626-4593-b361-a2387eecc731 iterator_sa2 = SASearchIterator(g_4, :X, examples_sa, cost_function, max_depth=3, initial_temperature = initial_temperature2) -# ╔═╡ 0f7f228c-4f95-4f1a-9ca2-2d03384a00c0 - - # ╔═╡ 5df0ba53-b528-4baf-9980-cafe5d73f9dd md""" ## Genetic Search -TODO: talk a bit about the iterator. - Genetic search is a type of evolutionary algorithm, which simulates the process of natural selection. It evolves a population of candidate programs through operations like mutation, crossover (recombination), and selection. Then, the fitness of each program is assessed (i.e., how well it satisfies the given specifications). Only the 'fittest' programs are selected for the next generation, thus gradually refining the population of candidate programs. For more information, see [here](https://www.geeksforgeeks.org/genetic-algorithms/). @@ -1262,7 +1258,7 @@ version = "0.13.1+0" # ╟─243165be-a9d2-484d-8046-811a2b0ba139 # ╠═4b97602a-5226-429f-86ea-8ecac3c807fa # ╟─ed198b79-1b95-4531-b148-c1037cfdacf4 -# ╠═d62918e4-4113-45bd-81ea-d17d007b83f5 +# ╠═75b1abfd-19ed-43f5-ac65-f8ffde76c581 # ╟─7480d1e4-e417-4d87-80b7-513a098da70e # ╠═7e2af72d-b71c-4234-9bca-cb9a90732a91 # ╠═00d05a7e-ca79-4d6b-828d-b24ef1cb80a2 @@ -1280,7 +1276,6 @@ version = "0.13.1+0" # ╠═afe1872c-6641-4fa0-a53e-50c6b4a712ee # ╟─69e91ae9-8475-47dd-826e-8c229faa11e8 # ╠═0a30fb40-cd45-4661-a501-ae8e45f1e07e -# ╠═82f7ffa5-2bdc-4153-85fa-d7aca063da12 # ╟─700270ea-90bd-474b-91d9-0e5ed329776a # ╠═e6e5c63b-34e8-40d6-bc12-bd31f40b4b16 # ╠═2397f65f-e6b4-4f11-bf66-83440c58b688 @@ -1300,7 +1295,6 @@ version = "0.13.1+0" # ╠═73304e3f-05bf-4f0c-9acd-fc8afa87b460 # ╠═07f11eb1-6b45-441a-a481-57628bad23ae # ╠═4ff69f0a-6626-4593-b361-a2387eecc731 -# ╠═0f7f228c-4f95-4f1a-9ca2-2d03384a00c0 # ╟─5df0ba53-b528-4baf-9980-cafe5d73f9dd # ╠═99ea1c20-ca2c-4d77-bc3b-06814db1d666 # ╠═d991edb9-2291-42a7-97ff-58c456515505