Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inefficient loading of fixtures #1986

Open
mrgrain opened this issue Oct 8, 2024 · 0 comments
Open

Inefficient loading of fixtures #1986

mrgrain opened this issue Oct 8, 2024 · 0 comments
Labels

Comments

@mrgrain
Copy link
Contributor

mrgrain commented Oct 8, 2024

The change in #1811 replaced parallel fixturization (= infusing a snippet into a fixture to convert it from human readable but non-compiling code to less human friendly but compiling code) with a serial approach to limit file system spikes.

As it turns out the loading of fixtures is highly inefficient and involves multiple FS calls per snippet. However there is a 1:N relationship between fixtures and snippets, i.e. a single fixture can be used for multiple snippets. That means the number of FS calls to fixturize snippets can be limited to the number of fixtures.

While in the worst case this can still be equal to the number of snippets, in practice it is unlikely.

We can address this inefficiency by loading all fixtures in serial (which will have FS calls), and then execute fixturization of the snippets in parallel (now without FS calls).

Additionally we can look into limiting the parallelism to a constant load.

@mrgrain mrgrain added the p2 label Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant