Add pipeline support for prompt Embeds [40 LPT]

[rickstaa]

Overview

Currently, applications using the Livepeer pipelines lack a mechanism to prioritize parts of a prompt. This can be achieved using weighted prompts (i.e., prompt embeds). Adding support for weighted prompts in the T2I, I2I, and upscaling pipelines would greatly enhance the user experience. We invite the community to help us implement this important feature 🛠️.

Required Skillset

• Basic proficiency in Python.
• Optional: Experience with the Hugginface diffusers library.
• Optional: Experience with uvicorn and FastAPI libraries.

Bounty Requirements

1. An easy method for users to utilize prompt embeds with the current pipelines, ideally without adding new parameters. The ++/-- syntax should be recognized by Livepeer, creating a prompt embed to be passed through the superresolution pipeline.
2. A report detailing any performance changes (e.g., inference speed) resulting from adding this feature.

Implementation Tips

• Refer to the Diffusers weighted prompts documentation.
• Consult the AI runner development guide for testing your implementation.
• Review the superresolution Hugging Face pipeline, which includes a clear example of implementing this feature.
• After completing your implementation, run the gen_openapi.py file and the make command in the main folder to regenerate the OpenAPI spec.
• Check the developer documentation in the ai-worker repository to see how you can debug your changes (i.e. https://github.com/livepeer/ai-worker/tree/main/dev and https://github.com/livepeer/ai-worker/tree/main/runner/dev).

How to Apply

Warning

Please wait for the issue to be assigned to you before starting work. To prevent duplication of effort, submissions for unassigned issues will not be accepted.

1. Express Your Interest: Comment on the issue to let us know you're interested.
2. Wait for Review: Our team will review expressions of interest within 14 days and select the best candidate.
3. Get Assigned: If selected, we'll assign the GitHub issue to you.
4. Start Working: Begin your work! For help or guidance, join the #🛋│developer-lounge channel on our Discord server.
5. Submit Your Work: Create a pull request in the relevant repository and request a review.
6. Notify Us: Comment on the GitHub issue when your pull request is ready for review.
7. Receive Your Bounty: Once your pull request is approved, we'll arrange the bounty payment.
8. Gain Recognition: Your valuable contributions will be showcased in our project's changelog.

Thank you for your interest in contributing to our project 💛!

[tegridydev]

Interested!

[rickstaa]

Interested!

@tegridydev, thank you for your interest in implementing this bounty. I reviewed your GitHub profile and your experience with Python. I'm happy to announce that you are accepted to work on this bounty and qualify for the 40 LPT reward 🎉. Welcome to the Livepeer open-source community 🚀. I have provided the necessary information in this issue, but if anything is unclear, please don't hesitate to comment below or reach out in the #🛋│developer-lounge channel on our Discord server.

[rickstaa]

@tegridydev as discussed in DM I removed you from the bounty so somebody else can tackle it 👍.

[rickstaa]

@JJassonn69, if you're interested in taking on this bounty, you can apply similar logic as used in the prompt splitting. Please refer to this link for guidance: Prompt Splitting Example.

You can create a function that returns both prompt and prompt_embeds, clearing the prompt and generating embeds as needed. However, feel free to design and implement the solution in any way you see fit.

[nishantkluhera]

Hello! I'm interested in this bounty if it's already not been assigned

[rickstaa]

Hi @nishantkluhera, thank you for your interest in contributing to the Livepeer open-source ecosystem! Unfortunately, this bounty has already been completed by @JJassonn69, and you can view the finished pull request here. It seems we forgot to assign him to the task—apologies for any confusion. We hope to see you tackle another bounty soon! 🚀