[WIP] AI video prototype #2959
[WIP] AI video prototype #2959
Conversation
yondonfu
force-pushed
the
ai-video
branch
5 times, most recently
from
feb3f3e
to
563d199
Compare
|
d32b1b5 temporarily disables Linux arm64 builds because they fail due to an error related to not being able to find zlib during ffmpeg compilation. This error doesn't occur for amd64 builds so I suspect the issue is related to the amd64 -> arm64 cross-compilation process. I noticed that we compile an arm64 specific version of x264 before compiling ffmpeg - perhaps we have to do something similar with zlib? zlib is currently required as a dependency as of 133050d#diff-4ae778054809274731b9da0c6a5a869c0bd214e92f954a5c9c39181748c2f175 which enabled the png decoder and image2 muxer which are used to demux + decode a sequence of PNG files so they can be encoded into an mp4 file. Ideally, we would replace the PNG demux/decode component by passing tensors (that represent frames) outputted by a model directly from GPU memory to NVENC using torchaudio.StreamWriter, but torchaudio.StreamWriter doesn't support RGB -> YUV conversion on the GPU yet - it can still encode a larger, less-streaming friendly (my understanding is yuv420p is preferred for streaming) RGB output, but I didn't jump to implement this yet due to current limitations. Until this replacement happens, zlib would be a required dependency to support the temporary PNG demux + decode component. |
|
@yondonfu Weird - on release go-livepeer right now zlib is dynamically linked. I'll have a look. |
This commit ensures the ai network software is in sync with the main branch.
This commit enables the macos and linux-amd64 builds.
This commit removes some redundant code in the bluild action.
This commit makes a small improvement to the doc comments in the build action. This was done to align the code with https://github.com/livepeer/go-livepeer/pull/3148/files.
This commit released the v0.7.7-ai.3 from the AI subnet software.
#3116) This commit allows Gateways to specify the maximum pricing they are willing to pay for a given capability and model combination. Gateways use a single MaxPrice set at launch with the maxPricePerUnit flag, which serves as the default. Additionally, they can specify a JSON config using the maxPricePerCapability flag to set prices per capability and model ID. The maxPrice per capability can also be adjusted via the setBroadcastConfig endpoint of the CLI webserver.
This commit released the v0.7.7-ai.3 from the AI subnet software.
This commit released the v0.7.8-ai.1 from the AI subnet software.
This commit applies a small code improvement.
This commit removes a redundant command which I introduced in the last commit.
This commit adds support for the new [segment anything 2](https://ai.meta.com/sam2/) pipeline (SAM2) that was added to the AI-worker in [this pull request](livepeer/ai-worker#185). While the new SAM pipeline can also do video segmentation this will be done in a subsequent pull request. Co-authored-by: John | Elite Encoder <[email protected]> Co-authored-by: Peter Schroedl <[email protected]> Co-authored-by: Rick Staa <[email protected]>
This commit released the new AI network software.
This commit updates the ai-worker to the one with the changed worker types.
This reverts commit e01daa2.
This commit ensures that the go-livepeer code uses the new worker classes that were defined in livepeer/ai-worker#191.
This commit updates the AI error handling behavoir so that BadRequest errors are forwarded to the user. Co-authored-by: Rick Staa <[email protected]>
While merging the main branch into the AI branch, the fragile AI selection algorithm broke due to changes in the transcoding selection logic, which the AI algorithm relies on. This commit provides a temporary patch to ensure the selection process continues to function while we work on improving the AI selection algorithm.
Add ochestrator_version tag to ai_request_latency_score and ai_request_errors metrics
This commit adds the necessary changes to support the new LLM pipeline introduced in the [ai-worker](https://github.com/livepeer/ai-worker) version [v0.7.0](https://github.com/livepeer/ai-worker/releases/tag/v0.7.0).
* fix: correct the order of the capability list This commit addresses an issue where the capability list was ordered incorrectly, which was introduced in https://github.com/livepeer/go-livepeer/pull/3114/files. The incorrect ordering caused a break in the SAM2 pipeline, which is now resolved. * Refactor Capabilities to use const values instead of iota * Refactor Capabilities to use const values instead of iota --------- Co-authored-by: Rafał Leszko <[email protected]>
Opening this draft PR to kick off CI build process for the WIP
ai-videobranch. This PR should not be merged as-is and the code on this branch will likely be refactored + cleaned up separately later on.