Create subsection "Nested workflow" in the docs

[agoscinski]

Moves the graph builder, if task and while task examples into this subsection. Also removed the graph_builder.ipynb since it is now generated. Solves issue #195

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.60%. Comparing base (5937b88) to head (3146393).
Report is 84 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #277      +/-   ##
==========================================
+ Coverage   75.75%   80.60%   +4.85%     
==========================================
  Files          70       66       -4     
  Lines        4615     5135     +520     
==========================================
+ Hits         3496     4139     +643     
+ Misses       1119      996     -123     

Flag	Coverage Δ
python-3.11	80.52% <ø> (+4.85%)	⬆️
python-3.12	80.52% <ø> (?)
python-3.9	80.56% <ø> (+4.82%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[agoscinski]

test_pause_task_after_submit seems a bit unstable it failed already two times in different jobs

[superstar54]

Looking at the if and while task again, because we now have new If task and while task, thus there are two ways to implement if

• if task, dose not need nested workgraph
• use graph_builder, thus nested workgraph.

thus it's better to keep if and while separately, and not move to the nested_workgraph.

I suggest at the end of the graph_builder.py, we make a link to show the application in the if and while.

[agoscinski]

I dont think the graph_builder example is really about the dynamic part as it just forwards inputs like any task does it. We could make another example that runs different tasks depending on the inputs. Something like

@task.calcfunction()
def add_one(x):
    return x.value+1

@task.calcfunction()
def modulo_five(x):
    return x % 5

@graph_builder(outputs = [...])
def my_modular(i: orm.Int):
    wg = WorkGraph()
    if i.value < 5:
        task = wg.add_task(add_one)
    else:
        task = wg.add_task(modulo_five)
    # need wg.selector to expose for linking?
    return wg

Then one could say that this does not preserve provenance and directly interlude to the If workgraph

[agoscinski]

Added an example for a dynamic use case of the graph_builder

[superstar54]

Hi @agoscinski , Thanks for the work.

I would suggest not creating a dynamic-workflows section, but moving all three notebooks into the howto, so that user can see the dynamic, if and while immediately.

[superstar54]

This is a duplicate as in the if. I would suggest using a for loop inside the graph_builder so that the number of tasks depends on an input value, making the work graph dynamic.

[agoscinski]

I think we have quite a few for loops in the examples too therefore I don't see duplication as a reason to rather use for loops. I feel this example is more educative because you are changing the type of task, you are in different branch of your program depending on your input which might be more intuitive to be dynamic.

But I can also just add both.

[superstar54]

Then one could say that this does not preserve provenance and directly interlude to the If workgraph

Why ti does not preserve provenance?

[agoscinski]

I would suggest not creating a dynamic-workflows section, but moving all three notebooks into the howto, so that user can see the dynamic, if and while immediately.

Okay, but the howto's starting to get messy and need some structure at some point, but I think for now one still can put everything there

[agoscinski]

Why ti does not preserve provenance?

I mean more that it does not store provenance as transparently as using the If WorkGraph, but looking at the provenance graph, It is not really capturing the if-then-else logic, but just only some additional information about the condition. But at least in the GUI the if-then-else is capture more transparently.

[agoscinski]

I merged now the nested and dynamic example because it seemed strange to see them separate

[superstar54]

Hi @agoscinski , thanks for the update!

The dynamic_graph_builder.py is not used, I think you want to remove it.

[superstar54]

Better to give an overview: user can create nested WorkGraph in two ways:

• Create a Task from the workgraph
• Use graph builder.

Also discuss what's the upside and downside of the two approaches, or later in each section.

[agoscinski]

Have done something similar

[superstar54]

Remove this empty line, otherwise the format will be wrong.

[superstar54]

This belongs to the Create a Task from the workgraph

[agoscinski]

It is at the beginning now

[superstar54]

Actually, one can link the two WorkGraph tasks. But we need to write the code in a different way:

def add_multiply(x=None, y=None, z=None):
    wg = WorkGraph()
    wg.add_task(add, name="add", x=x, y=y)
    wg.add_task(multiply, name="multiply", x=z)
    wg.add_link(wg.tasks["add"].outputs[0], wg.tasks["multiply"].inputs["y"])
    return wg

wg = WorkGraph("nested_workgraph")
add_multiply1 = wg.add_task(add_multiply(x=2, y=3, z=4), name="add_multiply1")
add_multiply2 = wg.add_task(add_multiply(y=5, z=6), name="add_multiply2")
wg.add_link(add_multiply1.outputs["multiply.result"], add_multiply2.inputs["add.x"])
wg.run()

The difference between the above code and a graph_builder task is that in graph_builder, the workgraph is dynamic and only created during execution, while the above workgraph is static, so that we can access the socket directly, e.g., add.x.
Both can used to create nested workgrpah, but graph_builder is a black box, which is the downside. The upside is that it allows dynamic workgraph generation.

[agoscinski]

Okay used that example and explained a bit

[agoscinski]

Hm.. it is not properly working

# define add task
@task.calcfunction()
def add(x, y):
    return x + y


# define multiply task
@task.calcfunction()
def multiply(x, y):
    return x * y


def add_multiply(x=None, y=None, z=None):
    wg = WorkGraph()
    wg.add_task(add, name="add", x=x, y=y)
    wg.add_task(multiply, name="multiply", x=z)
    wg.add_link(wg.tasks["add"].outputs[0], wg.tasks["multiply"].inputs["y"])
    return wg


wg = WorkGraph("nested_workgraph")
# Creating a task from the WorkGraph
add_multiply1 = wg.add_task(add_multiply(x=Int(2), y=Int(3), z=Int(4)))
add_multiply2 = wg.add_task(add_multiply(x=Int(2), y=Int(3)))
# link the output of a task to the input of another task
wg.add_link(add_multiply1.outputs[0], add_multiply2.inputs["multiply.x"])
wg.to_html()

# %%
# Run the workgraph

wg.run()

Gives

Error: Error in task multiply: Cannot convert value of type <class 'aiida.orm.utils.managers.NodeLinksManager'> to AiiDA type.

[agoscinski]

Okay now it works!

[superstar54]

Since you mentioned this part at the beginning, so better to move this section before graph_builder.

[agoscinski]

Is now integrated into the first part

[superstar54]

LGTM, thanks!