Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Orchestrator crash, possibly related to pricing #3057

Closed
eliteprox opened this issue May 16, 2024 · 2 comments
Closed

Orchestrator crash, possibly related to pricing #3057

eliteprox opened this issue May 16, 2024 · 2 comments
Assignees
Labels
AI Issues and PR related to the AI-video branch. type: bug Something isn't working

Comments

@eliteprox
Copy link
Contributor

Describe the bug

Runtime error occurred on production orchestrator:

I0516 16:39:30.673213       1 block_watcher.go:454] Polling blocks from=211907963 to=211907982
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2729f64]

goroutine 723 [running]:
math/big.(*Rat).Num(...)
        /usr/local/go/src/math/big/rat.go:409
github.com/livepeer/go-livepeer/core.(*orchestrator).priceInfo(0xc000ea9090, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, 0xdf, ...}, ...)
        /src/core/orchestrator.go:374 +0x204
github.com/livepeer/go-livepeer/core.(*orchestrator).PriceInfo(0xc0009937b8?, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, 0xdf, ...}, ...)
        /src/core/orchestrator.go:287 +0x5a
github.com/livepeer/go-livepeer/server.getPriceInfo({0xb22b40?, 0xc000ea9090?}, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, ...}, ...)
        /src/server/rpc.go:363 +0xe4
github.com/livepeer/go-livepeer/server.orchestratorInfoWithCaps({0xb22b40, 0xc000ea9090}, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, ...}, ...)
        /src/server/rpc.go:374 +0xaa
github.com/livepeer/go-livepeer/server.orchestratorInfo(...)
        /src/server/rpc.go:367
github.com/livepeer/go-livepeer/server.getOrchestrator({0xb22b40, 0xc000ea9090}, 0xc000877440)
        /src/server/rpc.go:341 +0x2b2
github.com/livepeer/go-livepeer/server.(*lphttp).GetOrchestrator(0x5138c0?, {0xc000877440?, 0x18d03c6?}, 0x0?)
        /src/server/rpc.go:185 +0x25
github.com/livepeer/go-livepeer/net._Orchestrator_GetOrchestrator_Handler({0x57ab20?, 0xc000579500}, {0xb10a50, 0xc0007337d0}, 0xc00078a9a0, 0x0)
        /src/net/lp_rpc_grpc.pb.go:109 +0x169
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000e4e1e0, {0xb19d20, 0xc0006666e0}, 0xc000a9c120, 0xc000579590, 0x321ddc0, 0x0)
        /go/pkg/mod/google.golang.org/[email protected]/server.go:1358 +0xe15
google.golang.org/grpc.(*Server).handleStream(0xc000e4e1e0, {0xb19d20, 0xc0006666e0}, 0xc000a9c120, 0x0)
        /go/pkg/mod/google.golang.org/[email protected]/server.go:1735 +0x9e7
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /go/pkg/mod/google.golang.org/[email protected]/server.go:970 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 693
        /go/pkg/mod/google.golang.org/[email protected]/server.go:981 +0x145

Just backlogging this for now, will check more into it shortly

Reproduction steps

Run production AI Orchestrator with following command

docker run -d --restart unless-stopped -v /livepeer/ai/data:/root/.lpData/ -v /var/run/docker.sock:/var/run/docker.sock --network host --gpus all livepeer/go-livepeer:ai-video -network=arbitrum-one-mainnet -orchestrator -transcoder -serviceAddr 76.215.8.207:8936 -v 6 -nvidia "all" -aiWorker -aiModels /root/.lpData/aiModels.json -aiModelsDir /livepeer/ai/data/models -ticketEV 2999999999999 -ethUrl ***** -ethOrchAddr 0x104a7ca059a35fd4def5ecb16600b2caa1fe1361 -ethPassword *** -ethAcctAddr 0x0074780FefF1FD0277FAD6ccdb5a29908df6051F

aiModels.json config:

[
    {
        "pipeline": "text-to-image",
        "model_id": "SG161222/RealVisXL_V4.0_Lightning",
        "price_per_unit": 4768371,
        "warm": true
    }
]

Expected behaviour

Orchestrator should not crash

Severity

Major

Screenshots / Live demo link

No response

OS

None

Running on

None

AI go-livepeer version

docker pull ghcr.io/livepeer/go-livepeer/build:sha-816920a51e883ac1e690fc15b7837e2189aa130b

AI go-livepeer commit hash

816920a

Additional context

No response

@eliteprox eliteprox added AI Issues and PR related to the AI-video branch. type: bug Something isn't working labels May 16, 2024
@eliteprox eliteprox changed the title Orchestrator crash Orchestrator crash, possibly related to pricing May 16, 2024
@eliteprox
Copy link
Contributor Author

Issue seems to be resolved by adding -pricePerPixel into startup flag #3047

@rickstaa
Copy link
Contributor

Fixed in b7a9fb7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AI Issues and PR related to the AI-video branch. type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants