Are we holding the GIL for most of the computation?

[siscia]

Hello everybody,

I am trying to run pygit2 in a multithread environment.

Each thread create a new repo handler with:

 repo = pygit2.Repository(repo_path)

and then try to compute the diff between two commits in a loop:

    for (commit1, commit2) in commits_queue:
        diff = repo.diff(commit1, commit2)
        diff.find_similar(flags = pygit2.GIT_DIFF_FIND_RENAMES|pygit2.GIT_DIFF_INCLUDE_TYPECHANGE)

and the result is pushed to a queue.

The hope was that repo.diff is a heavy, CPU bound call, executed by C code and that the python code would not acquire the GIL while executing it, so that multiple calls to repo.diff would run in parallel.

Of course, since I am asking help, it does not seems to be the case.

The python code is very happy to churn along at ~100% CPU (with peaks of 106%) but nothing more.

Was my assumption somehow corrrect?

Is there something we can do at the python layer, or maybe is a limitation of the underline libgit2 library?

Are we aware if moving to multiprocessing would help in any way?

Cheers,
S