Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for Windows ARM64 support #3107

Open
dennisameling opened this issue Mar 12, 2021 · 230 comments
Open

Tracking issue for Windows ARM64 support #3107

dennisameling opened this issue Mar 12, 2021 · 230 comments

Comments

@dennisameling
Copy link

dennisameling commented Mar 12, 2021

There's been a discussion about Windows ARM64 support here: #3021

I made a discussion comment there to keep track of the outstanding tasks to get basic ARM64 support to Git for Windows. However, @Alovchin91 informed me via Twitter that it was hard to keep track of the progress that way, requesting a dedicated issue. So here it is 😊

Open tasks for basic ARM64 support:

Thanks to the changes that we did in the last few days, there's now a fully working build of Git for Windows ARM64.

Test build for ARM64 users, built on March 12, 2021: https://github.com/dennisameling/git/releases/tag/v2.31.0-rc2.windows.2

I think we're pretty much ready to publish a beta version of Git for Windows ARM64! @dscho what do you think? Please let me know if I missed something in the list above.

@dscho
Copy link
Member

dscho commented Mar 12, 2021

I'm a bit focused on making sure that the v2.31.0 release goes smoothly, and will be happy to pay more attention to the ARM64 side of things after that release is out (which will probably happen Tuesday or Wednesday this coming week).

@Alovchin91
Copy link

Hi @dennisameling ,

Thanks for your wonderful work! ❤️ Have been using MinGit lately (which is a 💎 by itself) and it works great on my SPX!

One thing I just found:

C:\Projects\rustup>git log --help
fatal: 'C:/MinGit/arm64/share/doc/git-doc': not a documentation directory.

While MinGit doesn't seem to contain git-doc anyway, what caught my attention is that git.exe tries to search for documentation in the arm64 directory, while docs seem to still be in mingw32.

Could you please check on your side, on a complete Git setup? Thanks a lot! 😃

@dscho
Copy link
Member

dscho commented Mar 14, 2021

The documentation is not built as part of the CMake-based build. It will take quite some work to implement that. @Alovchin91 that's your chance to get involved! 😉

@Alovchin91
Copy link

@dscho I would be happy to help, but right now my capacity is extremely limited unfortunately and I have never had any proper experience with Cmake.

But wouldn't it be enough to point Git.exe from arm64 folder to the documentation in mingw32? 🤔

@dscho
Copy link
Member

dscho commented Mar 14, 2021

But wouldn't it be enough to point Git.exe from arm64 folder to the documentation in mingw32? 🤔

Not really, because the build configuration may differ in more than just the CPU architecture, even in ways affecting how/what parts of the documentation are compiled.

@Alovchin91
Copy link

Alovchin91 commented Mar 14, 2021

Hmm, I think right now my concern is mostly about where does arm64 version look for docs in a full installer-based version of Git 🤔

I saw that it's also absent from the "normal" mingw32 MinGit build, so I suppose missing docs in those builds is not arm64 specific issue.

@dscho
Copy link
Member

dscho commented Mar 14, 2021

Right, documentation is excluded from MinGit builds.

But looking for the documentation in the wrong location, just because there is no documentation in the correct location, isn't a solution for the non-MinGit flavors of Git for Windows, either ;-)

@Alovchin91
Copy link

Ah, right, now I get your point 😁 For some reason I thought that the most part of the tools are redirected to mingw32 folder and arm64 only contains wrappers or helpers 😅

@dscho
Copy link
Member

dscho commented Mar 14, 2021

Thanks to @dennisameling tireless work, the Git executables are built for Windows/ARM64. The i686 version of the MINGW tools we're still relying on are things like curl.exe, but we do include all of the i686 Git executables at the moment, too.

@dennisameling
Copy link
Author

Can confirm the help docs are only present in mingw32\share\doc\git-doc on a full installation and not in arm64\share\doc\git-doc.

The help function looks at GIT_HTML_PATH to find the help docs:

git/builtin/help.c

Lines 464 to 482 in 959b2f0

static void get_html_page_path(struct strbuf *page_path, const char *page)
{
struct stat st;
char *to_free = NULL;
if (!html_path)
html_path = to_free = system_path(GIT_HTML_PATH);
/* Check that we have a git documentation directory. */
if (!strstr(html_path, "://")) {
if (stat(mkpath("%s/git.html", html_path), &st)
|| !S_ISREG(st.st_mode))
die("'%s': not a documentation directory.", html_path);
}
strbuf_init(page_path, 0);
strbuf_addf(page_path, "%s/%s.html", html_path, page);
free(to_free);
}

This is defined in the CMake file:

add_compile_definitions(PAGER_ENV="LESS=FRX LV=-c"
GIT_EXEC_PATH="libexec/git-core"
GIT_LOCALE_PATH="share/locale"
GIT_MAN_PATH="share/man"
GIT_INFO_PATH="share/info"
GIT_HTML_PATH="share/doc/git-doc"
DEFAULT_HELP_FORMAT="html"
DEFAULT_GIT_TEMPLATE_DIR="share/git-core/templates"
GIT_VERSION="${PROJECT_VERSION}.GIT"
GIT_USER_AGENT="git/${PROJECT_VERSION}.GIT"
BINDIR="bin"
GIT_BUILT_FROM_COMMIT="")

@dscho would you be okay with a clause in the CMake file for Win ARM64 that sets GIT_HTML_PATH to ../mingw32/share/doc/git-doc for the time being? We can leverage the already available CMAKE_GENERATOR_PLATFORM for that logic.

@dscho
Copy link
Member

dscho commented Mar 19, 2021

would you be okay with a clause in the CMake file for Win ARM64 that sets GIT_HTML_PATH to ../mingw32/share/doc/git-doc for the time being?

I wonder how much work it would be to optionally use asciidoctor or asciidoc, if configured. These lines describe how asciidoc is called: https://github.com/git/git/blob/v2.31.0/Documentation/Makefile#L125-L133 And these describe how asciidoctor would be called instead: https://github.com/git/git/blob/v2.31.0/Documentation/Makefile#L179-L190

@dennisameling
Copy link
Author

I wonder how much work it would be to optionally use asciidoctor or asciidoc, if configured.

Will have a look in the coming days.

Just created a PR for 64-bit ARM64 installer support: git-for-windows/build-extra#333

@dennisameling
Copy link
Author

@dscho now that the 64-bit ARM64 installer PR has been merged, do you think we're ready to release a beta build of Git for Windows ARM64? I'm happy to look into the docs thing with asciidoctor and asciidoc mentioned above, but will only have time for that after May 7 due to work responsibilities and a break 😊 Thanks in advance!

@rimrul
Copy link
Member

rimrul commented Apr 22, 2021

Check if the daily "Git for Windows update check" works correctly on ARM64

It'll probably run daily and check for updates, but when it detects an update, it'll try to download and install the x86 installer.

@dscho
Copy link
Member

dscho commented Apr 23, 2021

do you think we're ready to release a beta build of Git for Windows ARM64?

Yes, I think we're getting to the point where we want to invite testers (I, unfortunately, am not eligible because I lack the hardware).

I'm happy to look into the docs thing with asciidoctor and asciidoc mentioned above

Fantastic.

I do think that it should be possible to add that to the CMakeLists.txt definition (a quick search brought up gdamore/nanomsg@5592124 which might be a good example to follow, although I would try not to have a separate Boolean to enable documentation; Rather, I would make it conditional on a check that tries to find asciidoc or asciidoctor, allowing the user to specify a hard-coded path to either to side-step the discovery).

It'll probably run daily and check for updates, but when it detects an update, it'll try to download and install the x86 installer.

Indeed.

For microsoft/git, I already did something slightly ugly to redirect the git update-git-for-windows command: microsoft#321 (comment). Essentially, it is a hack that edits the git-update-git-for-windows script to target a different update site, via a few sed commands.

I could imagine that we would probably want to improve on that e.g. by enhancing git-update-git-for-windows itself. One option would be to search for a file, say, git-update-git-for-windows.ini next to the script (when called via the PATH, as git update-git-for-windows will do internally, "$0" contains the absolute path to the script, therefore "$0.ini"would refer to that.inifile's path). If found, it would override a couple of things. Something along these lines (applies tobuild-extra`, but reader beware: this is totally untested):

diff --git a/git-extra/git-update-git-for-windows b/git-extra/git-update-git-for-windows
index 7121dd7be..50ff5524a 100755
--- a/git-extra/git-update-git-for-windows
+++ b/git-extra/git-update-git-for-windows
@@ -174,7 +174,22 @@ update_git_for_windows () {
 		;;
 	esac
 
-	latest_tag_url=https://gitforwindows.org/latest-tag.txt
+	if test -f "$0.ini"
+	then
+		fork="$(git config -f "$0.ini" update.fromFork)"
+		test -n "$releases_url" || {
+			echo "Could not find update.fromFork in $0.ini" >&2
+			return 1
+		}
+		releases_url=https://api.github.com/repos/$fork/releases
+		latest_tag_url=$releases_url/latest
+		latest_eval='latest=${latest_tag#*\"tag_name\": \"}; latest=${latest%%\"*}'
+	else
+		releases_url=https://api.github.com/repos/git-for-windows/git/releases
+		latest_tag_url=https://gitforwindows.org/latest-tag.txt
+		latest_eval='latest=${latest_tag#v}'
+	fi
+
 	latest_tag=$(http_get $latest_tag_url) ||
 	case $?,"$proxy" in
 	7,)
@@ -190,7 +205,7 @@ update_git_for_windows () {
 		;;
 	esac
 
-	latest=${latest_tag#v}
+	eval "$latest_eval"
 	# Did we ask about this version already?
 	recently_seen="$(get_recently_seen)"
 	test -n "$quiet" && test "x$recently_seen" = "x$latest" && return
@@ -215,7 +230,6 @@ update_git_for_windows () {
 	esac
 
 	echo "Update $latest is available" >&2
-	releases_url=https://api.github.com/repos/git-for-windows/git/releases
 	releases=$(http_get $releases_url/latest) || return
 	download=$(echo "$releases" |
 		grep '"browser_download_url": "' |

@dscho
Copy link
Member

dscho commented Apr 23, 2021

❓ Check if the daily "Git for Windows update check" works correctly on ARM64

Right, completely forgot to address that part. We do trust uname -m in the git-update-git-for-windows script, but on ARM, we cannot (because our MSYS2 will be an i686 or an x86_64 one).

@rimrul
Copy link
Member

rimrul commented Apr 23, 2021

I could imagine that we would probably want to improve on that e.g. by enhancing git-update-git-for-windows itself. One option would be to search for a file, say, git-update-git-for-windows.ini next to the script (when called via the PATH, as git update-git-for-windows will do internally, "$0" contains the absolute path to the script, therefore "$0.ini"would refer to that.ini` file's path). If found, it would override a couple of things.

I was thinking a lot simpler: We could check $MSYSTEM or $MINGW_MOUNT_POINT to detect we're on an ARM install. Admittedly, I did not think about microsoft/git when I had that idea.

@dscho
Copy link
Member

dscho commented Apr 26, 2021

I don't really trust MSYSTEM and MINGW_MOUNT_POINT at this stage ;-) If git.exe sets MSYSTEM, for example, it will not have the expected value.

@dscho
Copy link
Member

dscho commented Apr 28, 2021

I opened git-for-windows/build-extra#338 to help with this.

@dennisameling
Copy link
Author

Incredibly excited that git-for-windows/build-extra#340 has been merged. I almost don't dare to ask, but do you think we're now ready to release a beta version of Git for Windows ARM64, @dscho (apart from the request for the docs, which I hope to look into in the coming two weeks)? 🤞🏼

Yes, I think we're getting to the point where we want to invite testers (I, unfortunately, am not eligible because I lack the hardware).

I ordered and received a Raspberry Pi 4 last week, then installed Windows on ARM 19043 (21H1) on it. The performance is surprisingly good, but every once in a while there's a little driver-related hiccup that the Windows on Raspberry team is working hard on. If you want, I can send you the TeamViewer ID + password so you can test a bit on your end as well 👍🏼

@Alovchin91
Copy link

I have the Surface Pro X so I can also help with testing 🙂 RDP is also an option.

@dscho
Copy link
Member

dscho commented May 18, 2021

Right now, I would like to focus on Git for Windows v2.32.0, for which -rc0 came out yesterday, and the final version is expected around June 7th/8th.

After that, I'm all game for ARM64 and will take y'all up on the kind offers to let me connect to your hardware.

@Alovchin91
Copy link

Well... I wonder if BUILD next week brings anything interesting to arm64 world anyway 🤷 Maybe an arm64 GitHub Actions pipeline? One could dream...

@rimrul
Copy link
Member

rimrul commented May 19, 2021

@dennisameling Could you add the issue with git --version --build-options that I pointed out in #3083 to the list?

While it isn't a show stopper for the ARM beta it would be nice to have that fixed before we get more ARM bug reports.

@Alovchin91
Copy link

Okay, I hope it's finally time to prepare the first ARM64 beta! 🤗 Please let me know if I can help with anything. I could provide an ARM64 virtual machine on my Surface Pro X for a build pipeline, for example 🙂

@dennisameling
Copy link
Author

dennisameling commented Jul 10, 2021

Just updated the list above with open issues. I recently came across the fact that Visual C++ is needed since we use MSVC for the ARM64 builds. We can either include the Visual C++ Redistributable package in the installer, or statically build using MSVC (if possible). Do you have any preference here @dscho? I guess the latter option would be best as software like GitHub Desktop is depending on MinGit, so a static build would make their lives easier as well.

@rimrul
Copy link
Member

rimrul commented Jul 10, 2021

I don't think UCRT can be statically linked and redistributing vcredist would probably be a GPL violation, so linking the vcredist download page is probably our only option.

@dscho
Copy link
Member

dscho commented Jun 17, 2024

...And now it fails with some other cryptic message:

That makes me believe that the GitHub App is maybe not installed on your fork?

I swear that repo was not made to be forked...

Indeed it wasn't. It was made to support Git for Windows' own processes.

@Alovchin91
Copy link

Alovchin91 commented Jun 17, 2024

That makes me believe that the GitHub App is maybe not installed on your fork?

It fails when trying to do "Mirror Check Run" on git-for-windows/git repo:

https://github.com/git-for-windows/git-for-windows-automation/blob/292b89bd88329ba348c08e906438a0967a2cc1a1/.github/workflows/git-artifacts.yml#L34-L35

Does that mean I also need to fork the git repo? 🤔

Or maybe I can skip this step altogether? All I want to do is essentially build pacman packages from an existing tag.

Okay, scratch all that. I have removed this step altogether and now I can continue.

@Alovchin91
Copy link

I believe there's now an issue with git-sdk-arm64, check out this run: https://github.com/Alovchin91/git-for-windows-automation/actions/runs/9555876276/job/26340005470

error: target not found: mingw-w64-clang-aarch64-asciidoctor

And quite a number of the following lines (though not sure if they are errors):

/usr/bin/makepkg: line 935: /usr/bin/gettext: No such file or directory

@Alovchin91
Copy link

Alovchin91 commented Jun 17, 2024

Using the full setup-git-for-windows-sdk flavor allowed to move on, but now there are thousands of errors like:

module error : failed to open /usr/lib/libxslt-plugins/nwalsh_com_xsl_documentation_1_0.dll

See https://github.com/Alovchin91/git-for-windows-automation/actions/runs/9556010330/job/26340435149, Build mingw-w64-clang-aarch64-git step.

The packages have been successfully built though.

@dennisameling
Copy link
Author

I believe there's now an issue with git-sdk-arm64, check out this run: https://github.com/Alovchin91/git-for-windows-automation/actions/runs/9555876276/job/26340005470

error: target not found: mingw-w64-clang-aarch64-asciidoctor

That should be fixed with this PR: git-for-windows/git-sdk-arm64#19

And quite a number of the following lines (though not sure if they are errors):

/usr/bin/makepkg: line 935: /usr/bin/gettext: No such file or directory

I noticed those too, but they're also in the x86 and x64 flavors, e.g. here: https://github.com/git-for-windows/git-for-windows-automation/actions/runs/9346419477/job/25721230283#step:19:112

Not sure if that's a known issue at all.

@dscho
Copy link
Member

dscho commented Jun 18, 2024

And quite a number of the following lines (though not sure if they are errors):

/usr/bin/makepkg: line 935: /usr/bin/gettext: No such file or directory

I noticed those too, but they're also in the x86 and x64 flavors, e.g. here: https://github.com/git-for-windows/git-for-windows-automation/actions/runs/9346419477/job/25721230283#step:19:112

Not sure if that's a known issue at all.

#4952

@dscho
Copy link
Member

dscho commented Jun 19, 2024

And quite a number of the following lines (though not sure if they are errors):

/usr/bin/makepkg: line 935: /usr/bin/gettext: No such file or directory

I noticed those too, but they're also in the x86 and x64 flavors, e.g. here: https://github.com/git-for-windows/git-for-windows-automation/actions/runs/9346419477/job/25721230283#step:19:112
Not sure if that's a known issue at all.

#4952

I believe that this will be addressed via git-for-windows/build-extra#560.

@tuanle07
Copy link

Any updates on this? Been waiting for git on Arm64 for years. :(

@dscho
Copy link
Member

dscho commented Aug 20, 2024

Any updates on this? Been waiting for git on Arm64 for years. :(

A better use of your time might be to help the effort.

As to updates: there is e.g. git-for-windows/build-extra#574, git-for-windows/build-extra#572, git-for-windows/git-sdk-arm64#21, git-for-windows/build-extra#574 and of course https://inbox.sourceware.org/cygwin/[email protected]/#t.

Git for Windows/ARM64 would certainly benefit from competent help by motivated contributors.

@dennisameling
Copy link
Author

dennisameling commented Sep 9, 2024

Now that the asciidoctor issue is out of the way (thanks @dscho!), I was able to build the arm64 artifacts again. And I have some good news:

@dscho given that the pipelines seem to be a lot more stable now, how do you feel about releasing a beta version of GfW arm64? I think one final blocker would be the discussion on the artifact names. WDYT?

@Alovchin91
Copy link

Alovchin91 commented Sep 9, 2024

Just FYI, I have been using the artefacts from Dennis's runs for more than half a year now (I'm installing it inside MSYS2 proper from the packages archive) and I have never had a single issue with it. The only thing that I had to workaround is Pacman but that's because Pacman itself kept hanging randomly after the 6.0.2-12 patch (I've posted it somewhere in one of the discussions; never tried to upgrade it past 6.0.2-11 afterwards). I work in a humongous mono-repo (although without LFS but I believe Git LFS is a separate thing) so there it is.

@dscho
Copy link
Member

dscho commented Nov 12, 2024

I am still a bit uneasy about the double-fork deadlock, even though there seems to be some movement on about a potential work-around.

The full solution would be a working Cygwin for Windows/ARM64. While this is still hard, also on that front there has been some recent advancements:

In short, while I encourage nobody to hold their breath for it, but instead encourage everybody to consider how they can help, there is movement, and the first working Cygwin/ARM64 is almost in sight.

given that the pipelines seem to be a lot more stable now, how do you feel about releasing a beta version of GfW arm64?

@dennisameling I think a really good first step would be to adjust the /git-artifacts command to build not only for i686 and x86_64, but also for aarch64.

That won't immediately publish the Windows/ARM64 artifacts with /release (which would require adjustments e.g. here and here), but we would probably want to adjust the ARM64 installer to say "Windows/ARM64 support is still in early preview" or some such long before that, anyway.

@Alovchin91
Copy link

I have noticed that recent ARM64 builds have been available in the Pacman repository (except for the latest 2.47.0.2). This is a very welcome change and I hope that it is there to stay. Thank you!

@dscho
Copy link
Member

dscho commented Nov 12, 2024

recent ARM64 builds have been available in the Pacman repository

This has been the hard (manual!!!) work of @dennisameling. Say thank you to Dennis ;-)

@Alovchin91
Copy link

Yes @dennisameling is definitely my hero! 😊 🙏

@dscho
Copy link
Member

dscho commented Nov 13, 2024

I am still a bit uneasy about the double-fork deadlock, even though there seems to be some movement on about a potential work-around.

This work-around has now been contributed to the cygwin-patches mailing list. I have updated git-for-windows/msys2-runtime#73 and am currently testing. So far, everything works!

Therefore you can add @jeremyd2019 to your heroes, @Alovchin91!

@Alovchin91
Copy link

Alovchin91 commented Nov 13, 2024

Therefore you can add @jeremyd2019 to your heroes, @Alovchin91!

Jeremy has been in my list of heroes for quite some time now 😉

Nice to see this workaround making its way into Git for Windows! I wonder though if Cygwin will be eager to merge it, or they'd rather wait for the ARM64 support in GCC to arrive to compile the whole thing to ARM64. In any case, hopefully this works for Git for now 🤞

BTW I did get some hangs even with the older version of Pacman a few times, but so far that has never happened with Git itself. And I have it set up with SSH commit signing, and have been using Git ARM64 from this repo for about a year now I think.

@shanselman
Copy link

shanselman commented Nov 14, 2024 via email

@Alovchin91
Copy link

Okay, looks like it's time to update my setup to msys2-runtime 3.5.4-4 and unblock Pacman upgrades. I got hangs on newer Pacman versions most of the time, so should be pretty straightforward to verify 😄

@Alovchin91
Copy link

Alovchin91 commented Nov 15, 2024

...aaand it works! Previously, running /update-via-pacman.bat, I would get hangs like 90% of the time. Now I ran it a number of times with the newest Pacman and it never hung, with no phantom processes staying behind either 👍

An official ARM64 release today? 😜 ('cause everybody knows that Friday is the best day for releases)

@dscho
Copy link
Member

dscho commented Nov 15, 2024

running /update-via-pacman.bat, I would get hangs like 90% of the time.

Could you please re-test with /update-via-pacman.ps1? The .bat file is still there for just one reason: time (lack thereof, to be precise).

An official ARM64 release today? 😜 ('cause everybody knows that Friday is the best day for releases)

Alas, I won't be able to do that. Hopefully I'll get around to it next week.

@Alovchin91
Copy link

Alovchin91 commented Nov 15, 2024

Could you please re-test with /update-via-pacman.ps1? The .bat file is still there for just one reason: time (lack thereof, to be precise).

Hmm, I'm using MSYS2 + the "MSYS2 proper" method, so I don't have that script. Does it need anything additional, or I can just copy it over from the SDK repo to test?

Alas, I won't be able to do that. Hopefully I'll get around to it next week.

No probs, it was a joke 😅 I expected it to be released with the next GfW update tbh.

@Alovchin91
Copy link

Alovchin91 commented Nov 15, 2024

I've copied it over from the Arm64 SDK repo. It ran without an issue, though it did install the asciidoctor package which, I believe, is only needed for development, and when re-installing git-extra it has shown a warning:

/tmp/alpm_KB3vFO/.INSTALL: line 6: : No such file or directory

I believe I've seen it before with the .bat script when the environment didn't contain a path to git.exe (I think).

I don't get the same warning when running the .bat version.

EDIT 1:

If I understand it correctly, this essentially boils down to this file and line:

https://github.com/git-for-windows/build-extra/blob/5d189481c541e1a9af47d20592d6fb7d4b1e19aa/git-extra/git-extra.install#L6

Running git -c core.editor=echo config --system -e 2>/dev/null gives me this: C:/msys64/clangarm64/etc/gitconfig. I suppose it could be something in the way the path is converted when Pacman is run from inside of PowerShell?

If that's of any importance, I ran the .ps1 script like this: powershell -ExecutionPolicy Bypass /update-via-pacman.ps1

EDIT 2:

Removing the line $env:MSYS2_PATH_TYPE = "minimal" removes the warning.

EDIT 3:

Please correct me if I'm wrong, but it seems like it makes no sense to set $env:PATH = "$(Get-Location)\usr\bin;" + $env:PATH and then set $env:MSYS2_PATH_TYPE = "minimal" and run bash -lc. I guess what is meant here is to set $env:MSYS2_PATH_TYPE to "inherit"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests