Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stub doesn't work with multiprocessing #7

Open
wtfuzz opened this issue Jan 10, 2022 · 3 comments
Open

Stub doesn't work with multiprocessing #7

wtfuzz opened this issue Jan 10, 2022 · 3 comments
Labels
bug A crash or error in behavior.

Comments

@wtfuzz
Copy link

wtfuzz commented Jan 10, 2022

Describe the bug
When multiprocessing is used to fork a subprocess, the main program entrypoint is executed in the subprocess rather than the Process target.

To Reproduce
create a multiprocessing.Process() and start it using .start() The subprocess will call the parent process entry point.

Expected behavior
The target function is called as the entrypoint of the subprocess

Screenshots
If applicable, add screenshots to help explain your problem.

Environment:

  • Operating System: macOS
  • Python version: 3.10

Additional context
Add any other context about the problem here.

@wtfuzz wtfuzz added the bug A crash or error in behavior. label Jan 10, 2022
@wtfuzz
Copy link
Author

wtfuzz commented Jan 10, 2022

Note this requires beeware/Python-Apple-support#132 to include the _posixshmem package in the support package required by multiprocessing

@freakboy3742
Copy link
Member

freakboy3742 commented Jan 31, 2023

The _posixshmem issue should have been resolved as part of beeware/Python-Apple-support#161; it is now verified as part of the support testbed.

However, the more general issue with multiprocessing remains.

The issue is that Briefcase doesn't guarantee that there will be a python executable; as a result, multiprocessing will fail in spawn mode (which is the default as of Python 3.8).

fork and forkserver mode should both work. However, spawn mode assumes that sys.executable can be invoked with the same arguments as the python interpreter, which isn't true of a macOS stub binary.

This is definitely something that should be addressed; some sort of logic in the stub app is required that verifies if the parent process is "self"; and if so, changes the interpretation of command line arguments so that spawn calls behave like a "normal" Python interpreter. This will need to be fixed on the Xcode template (see beeware/briefcase-macOS-Xcode-template#20).

@mhsmith
Copy link
Member

mhsmith commented Feb 14, 2023

forkserver mode doesn't work either, because that calls sys.executable to start the server, plus what the multiprocessing documentation calls a "resource tracker process".

For example, if you replace the __main__ module of a Briefcase app with the following:

if __name__ == '__main__':
    import multiprocessing
    import operator
    import sys

    print(f"{sys.argv=}")
    if len(sys.argv) == 1:
        multiprocessing.set_start_method("forkserver")
        with multiprocessing.Pool(2) as p:
            print(p.map(operator.neg, [1, 2, 3]))

briefcase run will log the following command lines:

sys.argv=['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/MacOS/Hello World']
sys.argv=['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/MacOS/Hello World', '-B', '-I', '-c', 'from multiprocessing.resource_tracker import main;main(7)']
sys.argv=['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/MacOS/Hello World', '-B', '-I', '-c', "from multiprocessing.forkserver import main; main(17, 18, ['__main__'], **{'sys_path': ['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/support/python38.zip', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/support/python-stdlib', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/support/python-stdlib/lib-dynload', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/app_packages', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/app']})"]

If you remove the len(sys.argv) check, the app will replicate itself exponentially and lock up the whole system.

fork mode worked fine for me on macOS. It's no longer the default on macOS because of python/cpython#77906, but it looks like that only affects apps which call Objective-C APIs in the child processes.

Unfortunately, using fork mode isn't a cross-platform solution, because it isn't available on Windows. It's also unsafe to call fork in any process that has multiple threads (Python or otherwise), for the reasons explained here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A crash or error in behavior.
Projects
None yet
Development

No branches or pull requests

3 participants