You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In order to support parallelized regiment, suggest the following approach:
Implement regiment shard that takes the same arguments as regiment sort but only performs the operation of constructing a set of sorted temporary files (in the usual binary format) and returns the path to the directory in which these files are created.
Implement regiment merge that takes a list of directories (that are the outputs of running regiment shard in parallel) and perform the merge operation on the collection of all temporary sorted files contained within those directories.
In the short term, support usage of
regiment sort
viaxargs
. So,Regiment.IO.merge
should take a list of temp directories and merge across them.In the longer term,
regiment merge
should mimicsort -m
The text was updated successfully, but these errors were encountered: