Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there a way in which we can create long living threads? #678

Open
LRagji opened this issue Oct 19, 2024 · 1 comment
Open

Is there a way in which we can create long living threads? #678

LRagji opened this issue Oct 19, 2024 · 1 comment
Labels
enhancement New feature or request good-first-issue Good for newcomers never-stale Issues/PRs that are not stale

Comments

@LRagji
Copy link

LRagji commented Oct 19, 2024

My request may sound strange, but bare with me, currently piscina is designed for worker threads is more functional stateless approach and this may work for 90% use cases, but my use case requires me to hold a instance of class on a thread and keep it long running. The reason for this is cause i hold iterator on the worker thread side and exchange bulk messages for iteration between it and main thread.. can this be done via piscina?

What i am doing is opening sqlite DBs queries on worker threads in iterator mode and doing a k-way merge on main thread.. and i dont want to keep reopening these queries.. any suggestions?

@metcoder95
Copy link
Member

Currently not without an overhead.

Basically Piscina threads are meant to be destroyed if idle. This happens immediately by default, and can be extended accordingly.

Sadly, even passing Inifinity won’t work as we rely on a setTimeout that will take care of verifying if the worker is idle or not and destroy it if so.

One workaround is to have a single long-running that that abstains the worker from being idle, this will require to set you concurrentTaskPerWorker to N + 1 value.

I can see the benefit of having long-running threads, so happy to support a contribution to implement the feature.

@metcoder95 metcoder95 added the enhancement New feature or request label Oct 20, 2024
@metcoder95 metcoder95 added good-first-issue Good for newcomers never-stale Issues/PRs that are not stale labels Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good-first-issue Good for newcomers never-stale Issues/PRs that are not stale
Projects
None yet
Development

No branches or pull requests

2 participants