Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel PIRProcessDatabase #120

Merged
merged 1 commit into from
Oct 15, 2024

Conversation

glebfann
Copy link
Contributor

Asynchronous Processing:

  • Converted ProcessDatabase to support asynchronous execution by utilizing AsyncParsableCommand.
  • Speeded up the execution of PIRProcessDatabase by adding asynchronous processing enabled through the --parallel flag with Swift concurrency.
    Also added Sendable conformance to several types to ensure safe concurrent access.

Enhanced Logging:

  • Refined logging to include a prefix "Shard №(shardID):" for each log entry. This change addresses the non-sequential and interleaved nature of logs caused by asynchronous execution, making it easier to trace shard-specific activities.
  • Example log entries:
    --no-parallel: https://pastebin.com/gWH1n9B7
    --parallel: https://pastebin.com/QCrQMvJX

Performance Comparison, with 2 million numbers in identity.binpb:

  • Async Execution (time PIRProcessDatabase --parallel identity-config.json):
    1: real 0m11.638s, user 1m8.456s, sys 0m3.310s
    2: real 0m10.835s, user 1m8.368s, sys 0m3.242s
    3: real 0m10.833s, user 1m8.215s, sys 0m3.077s

  • Sync Execution (time PIRProcessDatabase --no-parallel identity-config.json):
    1: real 0m52.301s, user 0m49.667s, sys 0m1.936s
    2: real 0m52.117s, user 0m49.544s, sys 0m1.961s
    3: real 0m51.489s, user 0m49.378s, sys 0m1.789s

The asynchronous execution with --parallel demonstrates a significant reduction in real execution time compared to the synchronous approach.

P.S. identity-config.json:

{
    "inputDatabase": "identity.binpb",
    "outputDatabase": "identity_data/identity/identity-SHARD_ID.bin",
    "outputPirParameters": "identity_data/identity/identity-SHARD_ID.params.txtpb",
    "rlweParameters": "n_4096_logq_27_28_28_logt_5",
    "sharding": {
        "shardCount": 20
    },
    "trialsPerShard": 1,
    "cuckooTableArguments": {
        "hashFunctionCount": 2,
        "maxEvictionCount": 100,
        "bucketCount": {
            "fixedSize": {
                "bucketCount": 10000
            }
        },
        "maxSerializedBucketSize": 1024
    }
}

Copy link
Contributor

@karulont karulont left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR!

Overall it looks good, just a few minor comments.

Sources/PIRProcessDatabase/ProcessDatabase.swift Outdated Show resolved Hide resolved
Sources/PIRProcessDatabase/ProcessDatabase.swift Outdated Show resolved Hide resolved
Sources/PIRProcessDatabase/ProcessDatabase.swift Outdated Show resolved Hide resolved
@glebfann glebfann force-pushed the gleb032/parallel-database-processing branch from 36a80a3 to 2971bee Compare October 15, 2024 09:23
Sources/PIRProcessDatabase/ProcessDatabase.swift Outdated Show resolved Hide resolved
Sources/PIRProcessDatabase/ProcessDatabase.swift Outdated Show resolved Hide resolved
Sources/PIRProcessDatabase/ProcessDatabase.swift Outdated Show resolved Hide resolved
Sources/PIRProcessDatabase/ProcessDatabase.swift Outdated Show resolved Hide resolved
Sources/PIRProcessDatabase/ProcessDatabase.swift Outdated Show resolved Hide resolved
@glebfann glebfann force-pushed the gleb032/parallel-database-processing branch from 2971bee to 81b0ece Compare October 15, 2024 17:18
@glebfann glebfann force-pushed the gleb032/parallel-database-processing branch from 81b0ece to c8825e5 Compare October 15, 2024 21:43
@karulont karulont merged commit 99833cf into apple:main Oct 15, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants