Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple Inputs & Outputs #30

Open
nickzoic opened this issue Mar 18, 2024 · 2 comments
Open

Multiple Inputs & Outputs #30

nickzoic opened this issue Mar 18, 2024 · 2 comments

Comments

@nickzoic
Copy link
Collaborator

We've kind of gotten away with having a single input queue and output queue for each node.
However, this has led to some weird behavioural differences:

  • For ProductPlugins, eg: Join, there must be exactly two inputs and they are consumed separately
  • For everything else, there can be multiple inputs and they are consumed indiscriminately

This might not seem like much of a difference but it adds significant weirdness to the pipeline code.
Maybe it'd make sense for nodes to have 0, 1 or 2 input "ports" and have this visible in the UI.

Also, a lot of "filter" operations, eg: selecting records matching a pattern, could use a second output for
"failures". Effectively an "else" output. Sure you can add another complementary filter node but there's always
the risk that you'll miss something. Plus there's no output from writers. So maybe it'd make sense for nodes to have 0, 1 or 2 output "ports" and have this visible in the UI.

@nickzoic
Copy link
Collaborator Author

nickzoic commented Apr 2, 2024

plugins get num_inputs and num_outputs properties in 9350584 and the GUI now prevents adding links where these are 0 but actually supporting multiple ports is a bigger piece of work ...

@nickzoic
Copy link
Collaborator Author

Note that the new(ish) Filter plugin is a step in this direction: currently the "outputs" option allows setting a column to a value, but it could also allow selection of between multiple output "ports".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant