Strategy for managing upstream churn

[JSKenyon]

This is a high level but persistent problem. Due to the somewhat advanced usage of various dask/dask-ms/numba features in QuartiCal, release versions can easily be broken by upstream changes. The simple solution to this problem is to use a tool like poetry to freeze in known working versions on release. This is more robust to upstream change, but more brittle in the long term as it means installing QC as a dependency/in the same environment as another package may become difficult.

This issue is here to help me gather my thoughts/solicit opinions from users.

[bennahugo]

Yes it is very tricky. Pinning has a lot of advantages in terms of maintaining working releases, but when you move to target a new version of Python it becomes a wholesale whack-an-upstream-regression operation. I do think we need to more strictly adhere to supporting LTS versions (and e.g. the versions of Python they bundle) of Ubuntu though and have a firmer schedule on switchover. I do still prefer to have fixed versions on releases though - for me the pros outweigh the cons. Software don't need to be in installable in the same Python environment - we have stimela to link them together for that reason.

…

On Thu, Jan 25, 2024, 13:47 JSKenyon ***@***.***> wrote: This is a high level but persistent problem. Due to the somewhat advanced usage of various dask/dask-ms/numba features in QuartiCal, release versions can easily be broken by upstream changes. The simple solution to this problem is to use a tool like poetry to freeze in known working versions on release. This is more robust to upstream change, but more brittle in the long term as it means installing QC as a dependency/in the same environment as another package may become difficult. This issue is here to help me gather my thoughts/solicit opinions from users. — Reply to this email directly, view it on GitHub <#317>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6TSGJB766KT3LG5PHTYQJA47AVCNFSM6AAAAABCKMC562VHI2DSMVQWIX3LMV43ASLTON2WKOZSGEYDAMRRGUZTGMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sjperkins]

installing QC as a dependency/in the same environment as another package may become difficult

In general I think users should be discouraged from doing this, even though they might be surprised that they shouldn't!

[JSKenyon]

I think that I am gravitating towards using poetry for end-user installations. The lock file should ensure that each release remains installable regardless of upstream churn. The project will remain pip installable with the caveat that the looser dependencies may lead to breakages. I am tempted to drop support for Python3.8 as it results in very stale versions in the lock file (due to upstream deprecations). I think that this is likely fine as alternative Python versions are available via deadsnakes and QC should typically be installed in a virtual environment.

[JSKenyon]

Once again, I have convinced myself that poetry doesn't really resolve the problem as poetry.lock is not honoured when installing from PyPI. It may be part of the solution though. I think that I am going to adopt the following strategy:

1. Use poetry with only minimum version specifications in pyproject.toml.
2. Use the resulting poetry.lock to establish the most up-to-date versions which can be installed.
3. Use the results of 2 to pin the maximum versions in pyproject.toml on each release.

I believe that the above strategy should produce relatively robust PyPI packages without being too strict. It also keeps things simple for users while simultaneously giving us a way to sync dev environments (if required).