Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preparations to switch to APK package manager #23706

Open
aparcar opened this issue Mar 21, 2024 · 44 comments
Open

Preparations to switch to APK package manager #23706

aparcar opened this issue Mar 21, 2024 · 44 comments

Comments

@aparcar
Copy link
Member

aparcar commented Mar 21, 2024

Hi all, some fellow developers and me worked for some time now on making APK the new package manager for OpenWrt, replacing the unmaintained OPKG fork we've been using for the longest time.

APK is actively developed and used in multiple other distributions, e.g. Alpine Linux 🎉

While there is till some work ahead, I'd like to prepare everyone who maintains a package to verify that the PKG_VERSION follows Semantic Versioning <major>.<minor>.<fixup>[.<sub1>...]. APK uses a deterministic algorithm to compare versions and does not like random strings, except a valid hash prefixed with a ~.

If in doubt, please use the Docker container below to verify the used version is valid:

docker run --rm -it ghcr.io/aparcar/apk-valid-version <VERSION> [<VERSION> ...]

It will print whatever version is not valid, if you get a zero exit code, you're fine.

Please feel free to reach out for assistance and have a look at the core migration of versions.

@aparcar aparcar pinned this issue Mar 21, 2024
@aparcar
Copy link
Member Author

aparcar commented Mar 21, 2024

Below packages that would cause trouble. Please bear in mind that PKG_RELEASE is soonish prefixed with an r and may only contain a number.

Naughty list:

  • ack 3.7.0+perl5.38-r1
  • atlas-probe 0~2.6.3-r1
  • bcm2711-eeprom v.2024.01.05-2712-r2 fixed in master branch
  • bcm2712-eeprom v.2024.01.05-2712-r2 fixed in master branch
  • bcm27xx-eeprom v.2024.01.05-2712-r2 fixed in master branch
  • bluez-tools 20201025.f653217-r2
  • convbin 2.4.3_b34-r1
  • debootstrap 1.0.128+nmu2+deb12u1-r1
  • faad2 0~2.11.1-r1
  • fft-eval 2019-11-27-r4d3b6faee428e3bd9f44ab6a3d70585ec50484a1
  • hfsfsck 540.1.linux3-r2
  • https-dns-proxy 2023-11-19-r1
  • isc-dhcp-client-ipv4 4.4.3-P1-r7
  • isc-dhcp-client-ipv6 4.4.3-P1-r7
  • isc-dhcp-dyndns 4.4.3-P1-r7
  • isc-dhcp-omshell-ipv4 4.4.3-P1-r7
  • isc-dhcp-omshell-ipv6 4.4.3-P1-r7
  • isc-dhcp-relay-ipv4 4.4.3-P1-r7
  • isc-dhcp-relay-ipv6 4.4.3-P1-r7
  • isc-dhcp-server-ipv4 4.4.3-P1-r7
  • isc-dhcp-server-ipv6 4.4.3-P1-r7
  • ksmbd-avahi-service 0~3.5.1-r1 fixed in master branch
  • ksmbd-hotplug 0~3.5.1-r1 fixed in master branch
  • ksmbd-server 0~3.5.1-r1 fixed in master branch
  • libcli 2022.07.06~V1.10.7
  • libdb47 4.7.25.4.NC-r7
  • libdb47xx 4.7.25.4.NC-r7
  • libedit 20230828-3.1-r1
  • libfaad2 0~2.11.1-r1
  • libinih r58-r1
  • libinireader r58-r1
  • libmad 0~0.16.4-r1 fixed in master branch
  • libmbim 0~1.30.0-r1 fixed in master branch
  • libmspack 0.10.1alpha-r1
  • libnet-1.2.x 1.2-rc3-r4
  • libopenssl-afalg_sync 1.2.0-beta.1-r5
  • libow 3.2p4-r2
  • libow-capi 3.2p4-r2
  • libpagekite 0.91.201110C-r2
  • libqmi 0~1.34.0-r1 fixed in master branch
  • libradiotap 2020-06-22-r5
  • libuwifi 2020-03-10-r2
  • libuwsc-mbedtls 3.3.5-r5.1
  • libuwsc-nossl 3.3.5-r5.1
  • libuwsc-openssl 3.3.5-r5.1
  • libuwsc-wolfssl 3.3.5-r5.1
  • libvorbisidec 1.0.3-20180319-r2
  • libx264 2024-01-13-r1
  • lua-curl-v3 0.3.13-snapshot-r2
  • lua-openssl 0~0.8.5-1-r1
  • luajit 2.1.0-beta3-r7
  • luajit2 2.1-20231117-r1
  • luasoap 2014-08-21-raf1e100281cee4b972df10121e37e51d53367a98
  • luv 1.40.0-0-r2
  • mbim-utils 0~1.30.0-r1 fixed in master branch
  • mdio-tools 0~1.3.1-r1
  • mdns-utils IETF104-r5
  • mdnsd IETF104-r5
  • mdnsresponder IETF104-r5
  • mkhfs 540.1.linux3-r2
  • modemmanager 0~1.22.0-r12 fixed in master branch
  • netdiscover 0.3-pre-beta7-r2
  • netifyd 2023.08.10~v4.4.7-r1
  • netjson-monitoring 0~0.1.1-r2
  • node v20.11.1-r1
  • node-npm v20.11.1-r1
  • nsutils 0.2-2022-05-13-d6570bdec8435dfc781b95f6b404dedf965294dd-r1
  • ntfs-3g 2022.5.17-r1-fuseint
  • ntfs-3g-low 2022.5.17-r1-fuseint
  • ntfs-3g-utils 2022.5.17-r1-fuseint
  • ntp-keygen 4.2.8p15-r4
  • ntp-utils 4.2.8p15-r4
  • ntpd 4.2.8p15-r4
  • ntpdate 4.2.8p15-r4
  • oci-runtime-tests 1.0.0-pre20210326-r3
  • oci-runtime-tool 1.0.0-pre20210326-r3
  • openocd 0~v0.12.0-r2
  • openssh-client 9.7p1-r1
  • openssh-client-utils 9.7p1-r1
  • openssh-keygen 9.7p1-r1
  • openssh-moduli 9.7p1-r1
  • openssh-server 9.7p1-r1
  • openssh-server-pam 9.7p1-r1
  • openssh-sftp-avahi-service 9.7p1-r1
  • openssh-sftp-client 9.7p1-r1
  • openssh-sftp-server 9.7p1-r1
  • openwisp-config 0~1.0.1-r2
  • openwisp-monitoring 0~0.1.1-r2
  • owfs 3.2p4-r2
  • owftpd 3.2p4-r2
  • owhttpd 3.2p4-r2
  • owserver 3.2p4-r2
  • owshell 3.2p4-r2
  • pagekitec 0.91.201110C-r2
  • perl-ack 3.7.0+perl5.38-r1
  • perl-authen-sasl 2.16+perl5.38-r2
  • perl-authen-sasl-xs 1.00+perl5.38-r2
  • perl-cgi 4.57+perl5.38-r1
  • perl-compress-bzip2 2.28+perl5.38-r1
  • perl-dbi 1.643+perl5.38-r1
  • perl-device-serialport 1.04+perl5.38-r4
  • perl-device-usb 0.38+perl5.38-r1
  • perl-encode-locale 1.05+perl5.38-r2
  • perl-file-listing 6.04+perl5.38-r2
  • perl-file-next 1.18+perl5.38-r1
  • perl-file-rsyncp 0.74+perl5.38-r1
  • perl-file-sharedir-install 0.13+perl5.38-r1
  • perl-html-form 6.07+perl5.38-r1
  • perl-html-parser 3.75+perl5.38-r1
  • perl-html-tagset 3.20+perl5.38-r4
  • perl-html-tree 3.23+perl5.38-r5
  • perl-http-cookies 6.08+perl5.38-r1
  • perl-http-daemon 6.06+perl5.38-r1
  • perl-http-date 6.05+perl5.38-r2
  • perl-http-message 6.22+perl5.38-r1
  • perl-http-negotiate 6.01+perl5.38-r2
  • perl-http-server-simple 0.52+perl5.38-r1
  • perl-inline 0.86+perl5.38-r1
  • perl-inline-c 0.81+perl5.38-r1
  • perl-io-html 1.001+perl5.38-r2
  • perl-lockfile-simple 0.208+perl5.38-r4
  • perl-lwp-mediatypes 6.04+perl5.38-r1
  • perl-net-cidr-lite 0.21+perl5.38-r1
  • perl-net-dns 1.41+perl5.38-r1
  • perl-net-http 6.19+perl5.38-r1
  • perl-net-telnet 3.04+perl5.38-r4
  • perl-netaddr-ip 4.079+perl5.38-r1
  • perl-parse-recdescent 1.967015+perl5.38-r2
  • perl-parse-yapp 1.21+perl5.38-r1
  • perl-sub-uplevel 0.2800+perl5.38-r1
  • perl-test-harness 3.42+perl5.38-r1
  • perl-test-warn 0.36+perl5.38-r1
  • perl-text-csv_xs 1.52+perl5.38-r1
  • perl-time-moment 0.44+perl5.38-r1
  • perl-try-tiny 0.31+perl5.38-r1
  • perl-uri 1.76+perl5.38-r1
  • perl-www 6.43+perl5.38-r2
  • perl-www-curl 4.17+perl5.38-r7
  • perl-www-mechanize 1.96+perl5.38-r1
  • perl-www-robotrules 6.02+perl5.38-r2
  • perl-xml-parser 2.46+perl5.38-r1
  • pingcheck 2022-03-01-r1
  • pos2kml 2.4.3_b34-r1
  • proto-bonding 2021-04-09-r3
  • pservice 2017-08-29-r3
  • python3-dateutil 2.9.0.post0-r1
  • python3-dateutil-src 2.9.0.post0-r1
  • qauld 2.0.0-beta.15-r1
  • qcsuper 2022-08-11-r1
  • qmi-utils 0~1.34.0-r1 fixed in master branch
  • re2 2023-02-01-r3
  • rnx2rtkp 2.4.3_b34-r1
  • rtkrcv 2.4.3_b34-r1
  • softethervpn-base 4.38-9760-r2
  • softethervpn-bridge 4.38-9760-r2
  • softethervpn-client 4.38-9760-r2
  • softethervpn-server 4.38-9760-r2
  • spamassassin 4.0.0+perl5.38-r1
  • spamc 4.0.0+perl5.38-r1
  • spamc-ssl 4.0.0+perl5.38-r1
  • sqm-scripts-extra 2016-06-08-r1
  • squeezelite-custom 1.9.9-1432-r2
  • squeezelite-dynamic 1.9.9-1432-r2
  • squeezelite-full 1.9.9-1432-r2
  • sslh v1.22c-r1
  • str2str 2.4.3_b34-r1
  • sudo 1.9.15p5-r1
  • svox 1.0+git20130326-r3
  • tinc 1.1pre18-r3
  • tmate-ssh-server 511fd2bd852464e76824279609a34ee93fe148a4-r2
  • tunneldigger-broker 0~v0.4.0-r1
  • tunneldigger-broker-src 0~v0.4.0-r1
  • tvheadend 2023-06-05-r2
  • udpxy 1.0-25.1-r1 fixed in master branch
  • v2ray-geoip 202403210039-1
  • v2ray-geosite 20240316051411-1
  • v2ray-geosite-ir 202403180026-1
  • xupnpd 2018-11-20-r2 fixed in master branch
  • zlog 0~1.2.17-r1

@brada4
Copy link

brada4 commented Mar 21, 2024

isc-dhcp is EOL https://www.isc.org/dhcp/

@BKPepe
Copy link
Member

BKPepe commented Mar 23, 2024

Dear package maintainers,

There is no need to open individual pull requests for your packages, you are maintaining. That will be really time-consuming for you and for us. We will do it here exactly the same as it was done in the main repo via treewide commit.

@stokito
Copy link
Contributor

stokito commented Mar 24, 2024

I didn't know that the OpenWrt is going to switch to APK. Here is more details on this https://forum.openwrt.org/t/rfc-announcing-apkwrt/120096

@pesa1234
Copy link
Contributor

Dear, I've update ksmbd #23750, I'm not maintainer but if I can why don't help? Thanks guys

@stangri
Copy link
Member

stangri commented Mar 25, 2024

@aparcar Just a heads up, most of my packages either:

  • depended on the old style of version/revision to be presented by the binary to be compatible with upstream-maintained products (client to a server or a peer in a mixed network for example)
  • contained dependencies on opkg and were incompatible with upcoming APK in the ways other than version/release number

I believe I have fixed most of those, but I may have to submit further PRs if needed.

Also, the https-dns-proxy should be fixed with the latest PR.

PS. If the PR needs to be made against packages/luci repo for reasons other than APK migration, should I refrain from moving to the new release syntax?

@neheb
Copy link
Contributor

neheb commented Apr 7, 2024

libmad was fixed.

@pesa1234
Copy link
Contributor

pesa1234 commented Apr 7, 2024

ksmbd-* fixed

@neheb
Copy link
Contributor

neheb commented Apr 21, 2024

I noticed mtd-rw has a bad version too. #23977 fixes.

@ErwanMAS
Copy link
Contributor

ErwanMAS commented May 5, 2024

@aparcar

root@debian:~# docker run --rm -it ghcr.io/aparcar/apk-valid-version
Unable to find image 'ghcr.io/aparcar/apk-valid-version:latest' locally
docker: Error response from daemon: Head "https://ghcr.io/v2/aparcar/apk-valid-version/manifests/latest": unauthorized.
See 'docker run --help'.

@stokito
Copy link
Contributor

stokito commented May 19, 2024

The sslh migration (just as a version update) #24192
Its author confirmed that he will now always use a semantic versioning

@ttc0419
Copy link

ttc0419 commented Jun 12, 2024

Quick question, is 24x going to use apk?

@aparcar
Copy link
Member Author

aparcar commented Jun 12, 2024

Quick question, is 24x going to use apk?

No final decision on this.

@stangri
Copy link
Member

stangri commented Jul 9, 2024

Quick question, is 24x going to use apk?

No final decision on this.

If 24.xx is NOT going to use APK by default, would the 24.xx building tools (toolchain, SDK, CI) still prepend r to the PKG_RELEASE?

@aparcar
Copy link
Member Author

aparcar commented Aug 13, 2024

@openwrt/packages-write I kindly ask every maintainer to check the failing package builds here and see if the PKG_VERSION need some adoption: https://buildbot.aparcar.org/faillogs/x86_64/packages/

@hnyman
Copy link
Contributor

hnyman commented Aug 14, 2024

@aparcar
In addition of versioning, there appears to be also dependency trouble.
Some packages depend explicitly on opkg, and building an actual firmware will fail if apk is selected. Apk also finds more conflicts that opkg.

I built yesterday an APK enabled firmware, and had to remove one package (openssh sftp server) due to the versioning reasons, like expected.
But then at image finalization I ran into conflicts with e.g. @dibdot popular 'adblock' which explicitly depends on opkg.
Similarly, some packages depend on uclient-fetch,, which apk finds to be in conflict with GNU wget, unlike opkg does.

@ErwanMAS
Copy link
Contributor

@aparcar

root@debian:~# docker run --rm -it ghcr.io/aparcar/apk-valid-version
Unable to find image 'ghcr.io/aparcar/apk-valid-version:latest' locally
docker: Error response from daemon: Head "https://ghcr.io/v2/aparcar/apk-valid-version/manifests/latest": unauthorized.
See 'docker run --help'.

@aparcar i want to try your docker image and i have this error ^

@dibdot
Copy link
Contributor

dibdot commented Aug 14, 2024

But then at image finalization I ran into conflicts with e.g. @dibdot popular 'adblock' which explicitly depends on opkg.

@hnyman I'll remove the opkg dependency in adblock with the next minor update - thanks for heads up!

Edit: the opkg dependency has been removed in adblock-4.2.2 (see 34db79b)

@Payne-X6
Copy link
Contributor

@aparcar What to do when apk compatible version format opkg compares as an older version than the original? See libedit.

@nxhack
Copy link
Contributor

nxhack commented Aug 27, 2024

node, node-npm: I have sent a pull request.

Update: merged

@hnyman
Copy link
Contributor

hnyman commented Nov 9, 2024

List of still failing packages can be seen from the test/staging apk buildbot faillogs:
https://downloads.staging.openwrt.org/snapshots/faillogs/

only a few architectures have been compiled so far, but seems that e.g. ddns-scripts, nginx, ntfs-3g, many perl packages etc. still fail due to versioning reasons.

@vortexilation
Copy link
Contributor

Does anyone have a clue about perl-file-next or perl related packages?

ERROR: info field 'version' has invalid value: package version is invalid
ERROR: failed to create package: /home/user/works/openwrt/bin/packages/x86_64/packages/perl-file-next-1.18+perl5.40-r1.apk: package version is invalid

Looking at for example perl-file-next Makefile PKG_VERSION, i don't see anything wrong with it :

PKG_NAME:=perl-file-next
PKG_VERSION:=1.18
PKG_RELEASE:=1

What would be the fixes then?.

@aparcar
Copy link
Member Author

aparcar commented Nov 11, 2024

Seems like you attach perl5.40 to the version itself, which doesn't fly. How about something like perl-file-next-1.18.5.40-r1? This is what we did for some kernel packages, first the package version attached by the Kernel. You may not any arbitrary strings to the version anymore.

@1715173329
Copy link
Member

guess we need change

PKG_VERSION:=$(PKG_VERSION)+perl$(PERL_VERSION2)

to PKG_VERSION:=$(PKG_VERSION).$(PERL_VERSION2)

@vortexilation
Copy link
Contributor

vortexilation commented Nov 11, 2024

@aparcar
Thank you, will be helpful for fixing other packages version related things.

@1715173329
Compiled fine now, thanks, I think you should open a PR as using APK is becoming default now.
These will be fixed all perl APK related problems as listed above.

@PussAzuki
Copy link

I found that when compiling nginx-ssl I get an error with the following output

"depends:nginx-ssl-util>=1.5-1 libc libopenssl3 libpthread libpcre2 nginx-ssl-util zlib"

How to fix nginx makefile?

@hnyman
Copy link
Contributor

hnyman commented Nov 12, 2024

How to fix nginx makefile?

EXTRA_DEPENDS:=nginx-ssl-util (>=1.5-1)

By removing the extra version comparison there.
@Ansuel probably thinks about better fix, but that should be enough for your own use.

(Not sure if actually just "1.5-r1" would be enough there, as old "1.5-1" is not accepted by apk)

nxhack added a commit to nxhack/packages that referenced this issue Nov 13, 2024
@Ra2-IFV
Copy link
Contributor

Ra2-IFV commented Nov 13, 2024

I see a lot of package use this:

PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz

Then PKG_VERSION must match upstream version, this breaks apk support if upstream version use other naming syntax. Do we have a solution for this?

@Ra2-IFV
Copy link
Contributor

Ra2-IFV commented Nov 13, 2024

I found that when compiling nginx-ssl I get an error with the following output

"depends:nginx-ssl-util>=1.5-1 libc libopenssl3 libpthread libpcre2 nginx-ssl-util zlib"

How to fix nginx makefile?

#25335

@hnyman
Copy link
Contributor

hnyman commented Nov 13, 2024

PKG_VERSION must match upstream version, this breaks apk support if upstream version use other naming syntax. Do we have a solution for this?

Some examples of solutions, mainly involving extra variables in Makefile to differentiate PKG_VERSION and the name needed for upstream file (and possibly for compilation PKG_BUILD_DIR):

@Ra2-IFV
Copy link
Contributor

Ra2-IFV commented Nov 13, 2024

It works, but it would be better if there is a standard for this

@hnyman
Copy link
Contributor

hnyman commented Nov 13, 2024

If you check the currently failing packages in faillogs, you notice that there is substantial variation to the upstream names. There is no single cure.
https://downloads.openwrt.org/snapshots/faillogs/aarch64_cortex-a53/packages/

nxhack added a commit to nxhack/packages that referenced this issue Nov 13, 2024
neheb pushed a commit that referenced this issue Nov 13, 2024
See: #23706

Signed-off-by: Hirokazu MORIKAWA <[email protected]>
robimarko added a commit to robimarko/packages that referenced this issue Nov 15, 2024
robimarko added a commit that referenced this issue Nov 15, 2024
@hnyman
Copy link
Contributor

hnyman commented Nov 17, 2024

The list of currently failing packages in the packages feed is already below 10 packages.

https://downloads.openwrt.org/snapshots/faillogs/x86_64/packages/

debootstrap/ hfsprogs/ kafs-client/ micropython-lib/ netifyd/ oci-runtime-tools/ python-dateutil/ qcsuper/ sox/

( and also lua-curl-v3 and netdiscover, but there is already a PR for them )

@Ansuel
Copy link
Member

Ansuel commented Nov 17, 2024

@hnyman always dreap to have 0 faillogs there... at least for some targets :(

@Ansuel
Copy link
Member

Ansuel commented Nov 17, 2024

@Ansuel
Copy link
Member

Ansuel commented Nov 17, 2024

/home/ansuel/openwrt-ansuel/staging_dir/host/bin/fakeroot /home/ansuel/openwrt-ansuel/staging_dir/host/bin/apk mkpkg --info "name:pcapsipdump" --info "version:2020.03.03~157-r1" --info "description:pcapsipdump is a tool for dumping SIP sessions (plus RTP traffic, if available) to disk in a fashion similar to "tcpdump -w" (format is exactly the same), but one file per SIP session (even if there are thousands of concurrent SIP sessions)." --info "arch:aarch64_cortex-a53" --info "license:GPL-2.0+" --info "origin:feeds/telephony/net/pcapsipdump" --info "url:http://sourceforge.net/projects/pcapsipdump/" --info "maintainer:" --info "provides:" --script "post-install:/home/ansuel/openwrt-ansuel/build_dir/target-aarch64_cortex-a53_musl/pcapsipdump-2020.03.03~157/apk-aarch64_cortex-a53/pcapsipdump/post-install" --script "pre-deinstall:/home/ansuel/openwrt-ansuel/build_dir/target-aarch64_cortex-a53_musl/pcapsipdump-2020.03.03~157/apk-aarch64_cortex-a53/pcapsipdump/pre-deinstall" --info "depends:libc libstdcpp6 libpcap1" --files "/home/ansuel/openwrt-ansuel/build_dir/target-aarch64_cortex-a53_musl/pcapsipdump-2020.03.03~157/ipkg-aarch64_cortex-a53/pcapsipdump" --output "/home/ansuel/openwrt-ansuel/bin/packages/aarch64_cortex-a53/telephony/pcapsipdump-2020.03.03~157-r1.apk" --sign "/home/ansuel/openwrt-ansuel/private-key.pem"

Ok we need to escape the description...

@hnyman
Copy link
Contributor

hnyman commented Nov 17, 2024

Yeah the description handling shout be fixed.
Not sure if it should be escaping or sanitising...
I am so old-school, that having possible string delimiters (double quotes) in the field sounds error-prone in the long run.

@Ansuel
Copy link
Member

Ansuel commented Nov 17, 2024

@hnyman I sanitized the package description and also pushed a commit that escape them just to handle anything that is not officially supported.

hnyman pushed a commit to hnyman/packages that referenced this issue Nov 17, 2024
See: openwrt#23706

Signed-off-by: Hirokazu MORIKAWA <[email protected]>
(cherry picked from commit de9f4eb)
hnyman pushed a commit to hnyman/packages that referenced this issue Nov 17, 2024
See: openwrt#23706

Signed-off-by: Robert Marko <[email protected]>
(cherry picked from commit 3d6cfc7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests