Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: Create workflow_v2 processor #2607

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jem-davies
Copy link

@jem-davies jem-davies commented May 25, 2024

This Draft Pull Request is to request interim feedback on the proposed solution to the issue #2599.

A new Benthos Processor workflow_v2 has been created here.

Some example benthos config files have been included in ./benthos/cmd/benthos.

  • config_old.yaml - runs the example DAG from the workflow processor docs using the workflow processor
  • config.yaml - runs the example DAG from the workflow processor docs using the new workflow_v2 processor

The config_old.yaml will output:

INFO STAGE A FINISHED   
INFO STAGE C FINISHED   
INFO STAGE B FINISHED   
INFO STAGE E FINISHED   
INFO STAGE D FINISHED   
INFO STAGE F FINISHED   

The config.yaml will output:

INFO STAGE A FINISHED   
INFO STAGE C FINISHED   
INFO STAGE E FINISHED   
INFO STAGE F FINISHED   
INFO STAGE B FINISHED   
INFO STAGE D FINISHED   

Notice that E will start / finish before B is started.

I would greatly appreciate some feedback regarding the proposed solution before I spend more time fixing some of the things that I know that I need to do:

  • unit tests
  • writing docs
  • implementing the ability to restart DAG execution at particular places described on the workflow documentation under structured metadata
  • Currently there is no logic to process batches > 1 .

Also I am a quite new to Go - so could well have made some newbie mistakes.

Specific questions that I have are:

  • Should this be a new processor i.e V2 or should the existing workflow be altered?
  • The ability to infer the DAG from the request_map fields of the branch - Is this something that the V2 needs to have?

@jem-davies jem-davies requested a review from Jeffail as a code owner May 25, 2024 14:18
@jem-davies jem-davies changed the title Create workflow_v2 processor Draft: Create workflow_v2 processor May 25, 2024
@jem-davies jem-davies marked this pull request as draft May 25, 2024 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant