Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize regiment #15

Open
novemberkilo opened this issue Mar 21, 2017 · 1 comment
Open

Parallelize regiment #15

novemberkilo opened this issue Mar 21, 2017 · 1 comment

Comments

@novemberkilo
Copy link
Contributor

In the short term, support usage of regiment sort via xargs. So, Regiment.IO.merge should take a list of temp directories and merge across them.

In the longer term, regiment merge should mimic sort -m

@novemberkilo novemberkilo changed the title Regiment merge Parallelize regiment Mar 29, 2017
@novemberkilo
Copy link
Contributor Author

In order to support parallelized regiment, suggest the following approach:

  1. Implement regiment shard that takes the same arguments as regiment sort but only performs the operation of constructing a set of sorted temporary files (in the usual binary format) and returns the path to the directory in which these files are created.

  2. Implement regiment merge that takes a list of directories (that are the outputs of running regiment shard in parallel) and perform the merge operation on the collection of all temporary sorted files contained within those directories.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant