Kubeflow Pipelines 2.3.x planning

[rimolive]

Hello, Pipelines community!

The Pipelines WG is planning the future releases of KFP and we'd like to hear from you what are the features you want to see in the coming releases.

Let's use this thread to discuss the most desired features, and don't forget to join Pipelines WG to discuss the roadmap for the next releases.

[tom-pavz]

Hello! Here are a few issues I would love to see get into a sooner release than later for kfp!

1. [feature] display main.log as artifact for each step #10036 (total dealbreaker for using KFP v2 IMO)
2. When the backend will support PipelineTaskFinalStatus? kubeflow#7422 (this issue seems to be have been created in the incorrect GH repo, but I am seeing it in my current 2.1.0 deploy of KFP, so I think it is valid)
3. [bug] {{workflow.status}} and {{workflow.failures}} Behavior in KFP v2 dsl.ExitHandler versus v1 #10917
4. [frontend] Filter pipeline text box shows error when typing anything in it #10241
5. [frontend] first time choosing a pipeline definition is VERY slow #10897

[postera-james]

Agreed! in particular #10036 is an essential feature to replicate before a move to v2 is possible for anyone using any kind of scaling

[HumairAK]

Some more:

• Documentation for s3-prefixes (the feature itself is already merged and will go out on the next 2.3 release)
  • Relevant issue: [feature] remove hard-coded configs in KFP V2 launcher #9689
• WIP: feat(api/sdk/backend): implement PipelineConfig and setting ttl on pipelines. Fixes: #10899 #10942 (implementation is almost complete)
• Upgrade to latest Argo 3.4.17 #10977
  • if we can do argo 3.5 latest that's even better
• [sdk] ExitHandler doesn't compile with task Inputs #10187
• This is a priority in our downstream, and the work on this is almost done, we can forward this implementation to upstream kfp for 2.3: [feature] Abstract the Artifact Storage API via API server #10872
• [feature] Add clear architectural diagrams on how KFP works and fill out the operator guides #10957
• feat(backend): Support loop parallelism #8718 (implementation is basically done)
• [feature] Option to disable Caching in V2 at the KFP, Pipeline, and Run level  #10839
• [feature] Mounting emptyDir option on kfp v2 workflow-task using kfp-kubernetes #10656

[StefanoFioravanzo]

We also need an issue to track V1/V2 feature parity. It's not necessarily related to 2.3.X, but more broadly we need a high-level understanding and summary of what's missing

[HumairAK]

With 2.3 out, it seems we didn't get quite all of these, so those issues unaccounted for should probably be pushed to the 2.4.x planning doc, or milestone if we get one created.

[rimolive]

Agreed. Closing this discussion, but let's work on creating the milestones for 2.4