Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace multiprocessing.Pool #12

Open
2 of 5 tasks
andreasvc opened this issue Dec 23, 2012 · 0 comments
Open
2 of 5 tasks

replace multiprocessing.Pool #12

andreasvc opened this issue Dec 23, 2012 · 0 comments

Comments

@andreasvc
Copy link
Owner

andreasvc commented Dec 23, 2012

multiprocessing pools work fine unless any kind of error condition arises...

  • properly detect segmentation faults, out of memory, &c.
    concurrent.futures does this, but doesn't take an initializer ... (works on both python 3.3 and backported python 2 version)
  • handle ctrl-C (resolved in python 3.3+ for both multiprocessing and concurrent.futures; cf. http://bugs.python.org/issue9205 )
  • robustness: retry failed jobs (but this might make things worse in certain cases, so better to design jobs so that half-finished jobs can be resumed later)
  • do not rely on copy-on-write memory inherited from parent process; load data in child processes.
  • Full Python traceback from children. parser.workerfunc() decorator does this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant