init-dind-externals container slows down pod initialization

[djahandarie]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.9.3

Deployment Method

Other

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Observe any pod created by ARC, especially on a node with lots of I/O going on. Look at kubectl describe pod for the pod. You will see that there is a considerable delay between "Started container init-dind-externals" and the line after it, which is due to the initi-dind-externals container taking time to run.

Describe the bug

On our cluster, we find that init-dind-externals container takes anywhere from 5s to multiple mins to run, depending on the I/O load of the machine it's running on. It seems to do a massive copy of tons of node modules etc. I'm not sure exactly how big of a copy this is but it seems rather large.

Describe the expected behavior

I would expect there is some more efficient way to do this, either in terms of populating the dind image with the necessary files beforehand, or at least reducing the number of files that need to be copied (like node modules...).

Additional Context

N/A

Controller Logs

N/A

Runner Pod Logs

  Normal  Started    3m43s  kubelet                                Started container init-dind-externals
  Normal  Pulled     12s    kubelet                                Container image "us-east4-docker.pkg.dev/kouzoh-github-actions-prod/runner-bases/debian-dind-sidecar:stable" already present on machine

(This is an example of the init-dind-externals container taking 3m31s total to run in order for the pod to initialize)

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.