2nd of two inline tasks gets new affinityassistant annotation but no new assistant pod

[jrhunger]

Using simple knative func to build/deploy a function:

$ func create --language python test1
$ func deploy --remote --registry myregistry -p test1
Creating Pipeline resources
Running Pipeline Task: Building function image on the cluster
Running Pipeline Task: Deploying function to the cluster
(hangs here)

Looking further at what was deployed:

$ kubectl get pipeline
NAME                         AGE
test1-pack-upload-pipeline   8m10s

$ kubectl get pipelinerun
NAME                                   SUCCEEDED   REASON    STARTTIME   COMPLETIONTIME
test1-pack-upload-pipeline-run-dc8h8   Unknown     Running   4m30s

$ kubectl get taskrun
NAME                                          SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
test1-pack-upload-pipeline-run-dc8h8-build    True        Succeeded   6m17s       5m17s
test1-pack-upload-pipeline-run-dc8h8-deploy   Unknown     Pending     5m17s

$ kubectl get taskrun test1-pack-upload-pipeline-run-dc8h8-build -o yaml | grep affinity-assistant
    pipeline.tekton.dev/affinity-assistant: affinity-assistant-3a3177cc7e
$ kubectl get taskrun test1-pack-upload-pipeline-run-dc8h8-deploy -o yaml | grep affinity-assistant
    pipeline.tekton.dev/affinity-assistant: affinity-assistant-14c03b2b85

$ kubectl get pods | grep affinity
affinity-assistant-3a3177cc7e-0                    1/1     Running     0          9m27s

$ kubectl -n tekton-pipelines logs deploy/tekton-pipelines-controller | grep 3a3177cc7e
{"severity":"info","timestamp":"2024-05-28T19:25:37.507Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/affinity_assistant.go:148","message":"Created StatefulSet affinity-assistant-3a3177cc7e in namespace default","commit":"d649ded","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"d58391ca-71a8-41dd-a4e7-a53a32a4968a","knative.dev/key":"default/test1-pack-upload-pipeline-run-dc8h8"}
$ kubectl -n tekton-pipelines logs deploy/tekton-pipelines-controller | grep 14c03b2b85
$

Any idea where to look next? Where did the 14c03b2b85 affinity tag come from?

The pipeline has two tasks defined inline (not as separate task objects): build and deploy. Each has workspace "source-workspace" defined. The pipelinerun sets source-workspace to pvc test1-pack-upload-pipeline-pvc. There is only one PVC in the ns, and both the build and deploy pods specify to mount it, as expected.

I did try disabling affinity and this worked once by luck, but then failed the next time i tried it because the deploy pod got scheduled to a different node than the affinity-assistant, which was mounting the PVC it needed.

$ func version
v0.41.0
## deployed tekton-pipelines on friday from latest release:
$kubectl -n tekton-pipelines get pods -o yaml | grep image:
      image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/events:v0.60.0@sha256:514e49ebed13bd2b0114b79f014a6809d5e4460eed741fa8d18f19f978e12693
      image: sha256:4ef2c6d40d442130e6ab0e9b55a038b5c5926af45374231f8e051d02c136d5d1
      image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller:v0.60.0@sha256:f39e414b0c05d8b56035094f0c535048f9424087d1cb6f926816f52fb7d78aab
      image: sha256:f20500cd79e506104550eb0ea1a42e6879063dd025c5248327587258f8863856
      image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/webhook:v0.60.0@sha256:0842aef1322eba5e8a230e749957554dc415f0d88594289de563f9fd8593bd1a
      image: sha256:1e359d649767455608e590348331baf0595cd5fad06549e517c09aa3f9ca8a83