Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ai_ prometheus metrics produced with inconsistent pipeline tag #3185

Open
pwilczynskiclearcode opened this issue Sep 30, 2024 · 2 comments · May be fixed by #3191
Open

ai_ prometheus metrics produced with inconsistent pipeline tag #3185

pwilczynskiclearcode opened this issue Sep 30, 2024 · 2 comments · May be fixed by #3191
Assignees
Labels
AI Issues and PR related to the AI-video branch. status: triage this issue has not been evaluated yet

Comments

@pwilczynskiclearcode
Copy link
Contributor

pwilczynskiclearcode commented Sep 30, 2024

It's maybe not a bug but an inconvenience resulting in multiple similar tags being produced.

  • pipeline value produced in
    monitor.AIProcessingError(err.Error(), pipeline, ... is equal to text-to-image, image-to-image, upscale, image-to-video, audio-to-text, segment-anything-2 - lowercased with - as a separator.
  • In submitSegmentAnything2 it's produced as segment anything 2 (lower case, space separated).
  • no orchestrators available... errors are produced using capName resulting in Text to image, Image to image, Image to video, Upscale, Audio to text, Segment anything 2 (First letter uppercase, space separated).

And because of that when trying to filter based on pipeline on Grafana we get "duplicates":
Screenshot 2024-09-27 at 15 52 02

We need some unification.

@pwilczynskiclearcode pwilczynskiclearcode added the AI Issues and PR related to the AI-video branch. label Sep 30, 2024
@github-actions github-actions bot added the status: triage this issue has not been evaluated yet label Sep 30, 2024
@ad-astra-video
Copy link
Contributor

ad-astra-video commented Oct 2, 2024

This is a good find! Thank you for bringing this forward!

Could we use the PipelineToCapability function to make them all use the capability name? The PipelineToCapability function was introduced with the maxPricePerCapability addition to parse the pipelines input into the json pricing input to capabilities to try and avoid another a long switch statement to manually assign the capabilities based on the text input (see handleAIRequest as an example).

Open to alternative implementations of the PipelineToCapability function to make it less brittle. Maybe should remove the string parsing and just put in a switch statement in PipelineToCapability to assign the capabilities more directly? Other ideas welcomed here! An example of the brittle nature of this function is I provided incorrect advice to rename the Large Language Model capability description added with LLMs to Large language model in review of that PR. I think it should instead be Llm to correctly find the capability for the llm pipeline in the PipelineToCapability function.

@pwilczynskiclearcode
Copy link
Contributor Author

I already posted a PR trying to transform capability to pipeline name (pipeline names aren't defined globally by any const).
The prometheus tag/label is called pipeline so I wanted to produce it in "pipeline" format, not capability.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AI Issues and PR related to the AI-video branch. status: triage this issue has not been evaluated yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants