Fix error: "fatal: '$GIT_DIR' too big" when checking out in a long path with core.longpaths = true. #3877
Conversation
The Trace2 machinery wishes to parse the system and the global config early. To this end, it taps into the `common-main` framework to run first thing. There are more Git features that could benefit from such a handling, most notably the Windows-specific `core.longPaths` setting: If a user has worktrees whose path already requires long paths support, we cannot wait until we parse the the config settings in the usual way because the gitdir discovery needs to happen first and would fail because any `core.longPaths` setting in, say, `~/.gitconfig` would not have been parsed yet. To that end, let's refactor Trace2's early config parsing so that other users can tap into it, too. Signed-off-by: Kevin Worm <[email protected]> Signed-off-by: Johannes Schindelin <[email protected]>
Use case: Allow git to checkout a repository or submodule in a directory with a long path when core.longpaths = true. Example: > ./git.exe config --global core.longpaths true > ./git.exe clone https://github.com/git/git.git --recurse-submodules \ /c/eval/git_test/loooooooooooooooooooooooooooooooooooooooooooooooooooooooo\ oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo\ oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo\ oooooooong Context: $ sh --version GNU bash, version 4.4.23(1)-release (x86_64-pc-msys) $ ./git.exe --version --build-options git version 2.36.1.windows.1 cpu: x86_64 built from commit: e2ff68a sizeof-long: 4 sizeof-size_t: 8 shell-path: /bin/sh feature: fsmonitor--daemon Error: fatal: '$GIT_DIR' too big. Problem analysis: setup_explicit_git_dir in setup.c uses PATH_MAX to check if the git dir is to long. On Windows PATH_MAX is set by limit.h to 260 and setup_explicit_git_dir ignores core.longpaths. Solution: The implementation is based on the solution proposed by Johannes Schindelin, see: git-for-windows#3372 (comment) * Refactor the part of trace2_initialize() that reads the config. * Make tr2_sysenv_cb() a public function. * No longer calling it from trace2_initialize(), but from a static callback function in common-main.c. * Calling read_very_early_config() explicitly in main(), with that static callback function that calls into tr2_sysenv_cb(). * Extend the static callback function for Windows, and parse core.longPaths in that function. * Extend startup_info struct in cache.h with 'int max_long_path' so we can use it in setup_explicit_git_dir instead of PATH_MAX. This fixes git-for-windows#3372 Signed-off-by: Kevin Worm <[email protected]> Signed-off-by: Johannes Schindelin <[email protected]>
This currently fails, with:
++ p=/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef
++ p=y/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef
++ test_config_global core.longpaths true
++ test_when_finished 'test_unconfig --global '\''core.longpaths'\'''
++ test 0 = 0
++ test_cleanup='{ test_unconfig --global '\''core.longpaths'\''
} && (exit "$eval_ret"); eval_ret=$?; :'
++ git config --global core.longpaths true
++ git init y/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef
fatal: cannot chdir to y/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef: No such file or directory
error: last command exited with $?=128
not ok 7 - init with long path
Signed-off-by: Johannes Schindelin <[email protected]>
|
@kevin-worm thank you for working on this! I figured out the failures in 0210 (the early config must be parsed before initializing Trace2), split the commit in two and added a regression test. Please find the updated branch in your fork (thank you for allowing to update this PR that way, that makes things much more convenient than it would be otherwise). However, the test case I added fails because the I see that the directory exists, but as Can you verify this behavior? |
|
@dscho thanks for your quick response and edits! I see the same behavior.
|
|
The issue is caused by _wchdir in mingw_chdir which does not support the '\\?\' extended path-length prefix. I have tried to replace it with SetCurrentDirectory, but it requires the application to opt-in to remove the MAX_PATH limitation and I have not been able to get that to work. I tried updating the manifest 'compat/win32/git.manifest' to make the executable longPathAware, but that does not seem to do anything. |
As far as I remember, you also have to set Plus, you have to ensure that the code does not use |
|
That's correct, I have set the register entry to 1. According to the Microsoft documentation the following is required to circumvent the MAX_PATH limitation:
See: https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation I have added some printf statements to The result from I suspect that the reason why |
|
Maybe https://github.com/git-for-windows/git/blob/HEAD/compat/win32/git.manifest is incomplete? I do not see Windows 11 there, that's most likely an oversight on my part. |
|
Windows 11 shares the same supportedOS GUID as Windows 10 (https://docs.microsoft.com/en-us/windows/win32/sbscs/application-manifests#supportedos) so that should be fine, but we could update the comment for clarity. I have created a small C test app in visual studio 2022 (17.2.2) and in the test app it works with the updated manifest: Manifest: Project file Result: Removing longPathAware from the manifest causes the same error code that I also see in git: When I look for the manifest in git.exe I find 2 different manifests.
The second manifest is the updated manifest ( Only I have no idea where the first manifest is coming from as
|
|
Maybe it's coming from here via |
|
I got it working with a small change in git.rc and longPathAware enabled in the manifest, but another error popped up. So I guess the next step is going through |
This one might be as easy as replacing |
Wow, that must be a bug introduced by me. Sorry about that! Would you happen to know off the back of your head what the difference is between |
That is one more mole whacked, that fixes mingw_getcwd. The next issue is mingw_mktemp:
To be honest I'm not sure. In a fresh Visual Studio 2022 C project only a MANIFEST is added and the https://docs.microsoft.com/en-us/cpp/build/reference/manifest-create-side-by-side-assembly-manifest?view=msvc-170 mentions RT_MANIFEST, but does not explain why one would use it. |
https://docs.microsoft.com/en-us/previous-versions/bb756929(v=msdn.10) also mentions
Indeed. However, this comment gives me pause:
And indeed, the In I have not performed the same analysis for all callers, but the fact that the |
|
I think I understand what's going on with the manifest file. For the manifest to work it should be at I also found the header files: if I replace If I replace So it looks like |
|
I managed to get a bit further by fixing mingw_mktemp and mkstemp. Init now works with long paths: But cloning still has some issues: |
|
The issue is caused by CreateProcessW in
What I have seen so far indicates that there are no issues when changing the current dir to a long path after the process has started. The challenge seems to be that I will commit the changes I made so far. |
`t2031-checkout-long-paths.sh` runs successfully, but there are still issues remaining preventing submodules from working in a long path dir. Signed-off-by: Kevin Worm <[email protected]>
Did you see that |
|
Yes, but I when I traced the origin I found that When dir is
|
Hrm. That's odd. Can you pinpoint which call it is that is failing? E.g. by running the command with |
Earlier I had added some printf statements to check the current dir with |
One particular problem, I think, might be that |
|
The second-to-last log line does mention `git submodule': The call fails when Diff: Log: Only when it tries to call |
Is this really legitimate? |
Heh, I mistook this path to refer to the worktree you're checking out, but this is probably just your checkout of Git itself. In any case, I think we're hitting the issue here that is covered by https://stackoverflow.com/questions/51524910/subprocess-cwd-too-long-builtins-notadirectoryerror-winerror-267... |
Indeed that is my checkout of git itself.
I agree, I think it's the same issue. I think we have a few options to work around this issue:
|
To be honest, none of these really sound appealing ;-) The lease bad might be the third option, where we detect a too-long current working directory path and temporarily switch to the root directory of the corresponding drive. However, that will most likely not fix anything for |
One (admittedly nasty) approach that could get us slightly further along the call chain is to create the process in a suspended state in some short directory and then modify it's current directory before resuming, but that would then need similar patches to msys2-runtime to allow bash to run non-builtins from the dashed external in such a long working directory. This would also lead us deep into undocumented |
|
Another potential solution would be to teach the MSYS2 runtime the long paths trick, e.g. by introducing a The biggest problem with that is that some POSIX functions assume an implicit So we would need to require programs to be recompiled, and to indicate in some fashion that they use a longer |
|
Hi, @kevin-worm, @dscho, @rimrul first of all, thank you very much for diving into this topic. We want to migrate to git, but this issue is still of great concern for us. Unfortunately, there are tools involved, where we have no influence on the naming of files...and than comes git with its Metadata files :) (especially if LFS is involved). Is there any chance that this issue can be resolved (with our help)? Maybe something changed already in the past years for MSYS2? |
|
@skdsp could you provide a concrete example repository you need to work? |
|
@dscho Unfortunately, the concrete repositories are on our own servers. But the first example in the linked issue still shows the behavior: git clone -c core.longpaths=true https://github.com/git/git.git --recurse-submodules //?/D:/eval/Git_Test/looooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong |
|
@skdsp And there is no chance to get off of submodules, right? I frequently quip that: Friends don't let friends use submodules. There's more than a grain of truth behind that jab, though: Submodules pose many, many problems, and this here issue is just one of them. And no, nothing has changed in the MSYS2 runtime because it needs to be backwards-compatible to Windows 8.1, still (Windows 7 and 8 support was dropped recently, with minor efforts to keep those setups working in most scenarios, in some form or other). The best bet you'd have would be to convert the last remaining bits and pieces of
tl;dr there might not be any quick fix in sight. |
|
We are used to work with an integrated configuration management in our current VCS. Thus it would be convenient for us to work with submodules, especially since there is good tool support (e.g. using Git Extensions). |
This is a fix for issue: #3372
Use case:
Allow git to checkout a repository or submodule in a directory with a long
path when core.longpaths = true.
Example:
Context:
$ sh --version
GNU bash, version 4.4.23(1)-release (x86_64-pc-msys)
$ ./git.exe --version --build-options
git version 2.36.1.windows.1
cpu: x86_64
built from commit: e2ff68a
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon
Error:
fatal: '$GIT_DIR' too big.
Problem analysis:
setup_explicit_git_dir in setup.c uses PATH_MAX to check if the git dir is
to long. On Windows PATH_MAX is set by limit.h to 260 and
setup_explicit_git_dir ignores core.longpaths.
Solution:
The implementation is based on the solution proposed by Johannes
Schindelin, see:
#3372 (comment)
function in common-main.c.
callback function that calls into tr2_sysenv_cb().
in that function.
use it in setup_explicit_git_dir instead of PATH_MAX.
Signed-off-by: Kevin Worm [email protected]