P2P Sync Implementation Plan

[bausmeier]

P2P Sync Technical Implementation Plan

In order to implement P2P sync, a number of different modules will need to be updated or created.

Android FHIR Engine

The biggest challenge around implementing P2P sync comes from the way that server sync has been implemented in the Android FHIR engine. Changes to resources are synced to the server using the patch interaction rather than the update interaction which means that when a peer is syncing to the server after having synced from another peer, it needs the list of changes which were made to the resource rather than the latest version of the resource itself. So when it comes to implementing P2P sync it means that the resource and the local changes need to be synchronized between peers, and stored in the local database. To support this, some functions will need to be added to the Database interface.

The planned approach is to sync the locally updated version of the resource along with the local changes to the resource from the sender to the receiver, and then store them as-is on the receiver side.

On the sender side there will need to be a way to query for the local changes which have been made to particular resource, similar to the existing getAllLocalChanges function in the Database interface. The function signature would look something like suspend fun getLocalChanges(clazz: Class<R>, id: String): SquashedLocalChange.

On the receiver side there will need to be something similar to the existing insertRemote function in the Database interface, which bypasses the creation of local changes and simply overwrites the resource with the version received from the server. In this case the function signature would need to look something like suspend fun <R : Resource> updateRemote(resource: R, changes: List<LocalChangeEntity>) so that the resource and local changes can be stored atomically within the same transaction.

An alternative which was considered was keeping track of the version of the resource from the server alongside the locally updated version, and to send that original version of the resource along with the local changes so that they can be re-applied on the receiver side using a patch function without bypassing the existing mechanism to generate the local changes. The downside of this approach is the loss of efficiency due to having to store multiple versions of each resource, and having to re-apply the patches during sync. Storing the reverse patch for each update and applying the reverse patches to get back to the original resource may reduce the amount of additional storage needed, but it would increase the amount of processing required during sync.

P2P Sync Module

Payload

A new class is needed to encapsulate the data which is exchanged between the sender and receiver during sync.

The class needs to be serializable so that it can be sent over different communication channels such as a socket.

It will need to include:

• The locally updated version of resource.
• The list of local changes which have been made to the resource. The list could be empty for newly created resources, or resources which have not been updated since they were synced from the server. A new class containing only the relevant fields from the LocalChangeEntity should probably be used instead of the LocalChangeEntity class itself since fields other than type and payload are not relevant, but this can be decided during development.

data class SyncPayload(val resource: Resource, val patches: List<LocalChangeEntity>)

Sender

The module will provide a SyncSenderSession interface which can be implemented for different communication channels (Wi-Fi Direct, Nearby API, etc.). It is responsible for negotiating the parameters which are to be used for the sync, and for passing the sync payloads over the wire to the receiver.

interface SyncSenderSession {
  suspend fun initiate(): ResourceSyncParams
  suspend fun send(flow: Flow<SyncPayload>)
}

It uses kotlinx.coroutines.flow.Flow to provide a backpressure mechanism so that the receiver does not get overwhelmed when sending large numbers of resources.

To do the work of sending the resources, the module will also implement a SyncSender class which is responsible for retrieving the resources and local changes from a FhirEngine and a Database instance, and passing them to a SyncSenderSession instance. A sender will be instantiated by the app when a sync is performed, and it will report progress and completion back to the app.

The procedure which the sender will follow is:

1. Initiate a sync by calling initiate on the SyncSenderSession. This will get back the ResourceSyncParams from the receiver so that the sender knows which resources to send.
2. Select all of the resources which match the sync parameters from the receiver. The _lastUpdated timestamp is excluded from this matching because resources which haven't been updated from the server may still have local changes which need to be synced. Pagination or streaming should be used when selecting all of the resources so that they're not loaded into memory all at once since their could be a very large number of them.
3. Initialise sending of the resources by calling send on the SyncSenderSession with Flow builder block in which the sync payloads will be emitted.
4. For each resource, check if the last updated timestamp is equal to or more recent than the one specified by the receiver (or that there is no last updated timestamp in the case of new resources). If so, create a sync payload containing the resource as well as any local changes which have been made to that resource and emit it to the flow.

Receiver

To mirror the SyncSenderSession, the module will provide a SyncReceiverSession interface which can be implemented for different communication channels to handle the receiving of resources from the sender.

interface SyncReceiverSession {
  fun receive(): Flow<ResourcePayload>
}

The receive function will once again return a Flow to provide a backpressure mechanism.

The module will provide a ConflictResolutionStrategy interface which is used for determining how to deal with conflicts in the case where both peers have local changes to a resource.

interface ConflictResolutionStrategy {
  fun resolveConflict(theirs: SyncPayload, ours: SyncPayload): SyncPayload
}

Implementations of two strategies will be provided by default:

• TheirsConflictResolutionStrategy which accepts the other peer's version of the resource in the case of a conflict.
• OursConflictResolutionStrategy which accepts this peer's version of the resource in the case of a conflict.

These two strategies effectively apply directionality to the sync. The terminology of "theirs" and "ours" is borrowed from Git.

Additional custom strategies can be implemented within the apps that need them if merging of payloads is possible using some implementation-specified heuristics.

To do the work of receiving the resources, the module will also implement a SyncReceiver class which collects sync payloads from the sync receiver session and processes them by resolving conflicts using the specified strategy, and then persisting them to the local database. A receiver will be instantiated by the app when a sync is performed, and once the sync has been completed it will return a result containing references to the resources which were successfully processed, and to those which were not processed due to conflicts. These can then be reported back to the user at resource type or individual resource level.

The procedure which the receiver will follow is:

1. Call receive on the SyncReceiverSession to begin the resource synchronization process.
2. Find the most recent last-updated timestamp for each resource type which is configured to be synced, and add this to the existing ResourceSyncParams.
3. Respond to the sender's initiation request with the populated sync params.
4. For each resource which is received from the sender, check if there are local changes to the resource. If not, store the received resource and changes to the database. If so, resolve the conflict using the specified conflict resolution strategy, and store the resulting resource and changes to the database.

Wi-Fi Direct P2P Sync Module

After establishing a connection between peers, Wi-Fi Direct allows the peers to communicate with one another over a java.net.Socket. This is a low-level primitive, and some form of messaging protocol needs to be implemented on top of it in order to exchange data in a meaningful way. For the sake of simplicity this module will implement sync using JSON as the serialization format, and use a BufferedWriter to separate individual messages by newline characters. This can be made more efficient in future if the need arises, by implementing a more compact serialization format such as Protocol Buffers, and smarter message framing.

The module will exchange control messages over the socket to perform authentication, and to agree on sync parameters, and communicate state between sender and receiver.

It will provide a SocketSyncSenderSession implementation of the SyncSenderSession interface. When initiate is called by the SyncSender class, it will write a message to the socket, providing an authentication challenge and some metadata to the receiver such as the version of the protocol and the configured FHIR server URL. It will then wait for the receiver to respond by accepting the sync with the authentication response and the resource sync parameters, or by rejecting the sync with a reason (such as mismatched versions or FHIR server URLs). If the authentication response is valid it will wait for the send function to be called, otherwise it will notify the receiver and close the socket. When send is then called by the SyncSender class, it will collect the flow by serializing each payload and writing it to the underlying socket. Once the flow ends it will send a message to the receiver to notify it, and then close the socket.

It will provide a SocketSyncReceiverSession implementation of the SyncReceiverSession interface. When receive is called it will start reading from the socket and wait for the initiation message from the sender to be received. It will then compare the metadata from the sender with its own, prompt the user to provide an authentication code, and respond with the configured resource sync parameters. It will then wait to either receive an authentication failure message from the sender, or to being receiving sync payloads. For each payload received, it will deserialize it, and emit it to the flow so that it can be handled by the SyncReceiver class. Once the completion message has been received from the sender, it will close the socket and end the flow.

The module may also need to provide interface components for consumption in implementing apps as Fragments and Activities.

FHIR Core

Changes will need to be made to the FHIR Core repository to make use of the Wi-Fi Direct P2P Sync Module in the apps which require a P2P sync feature. This will entail injecting the required dependencies into the classes provided by the module, specifying some configuration, and making use of the UI components.

---

[pld]

Looks great a couple thoughts

1. you mention control messages for state, would this include things like total resources to sync?
2. is data compressed before send, does the bufferwriter/reader library handle that?

[bausmeier]

I hadn't planned for the total number of resources to sync to be part of the control messages, but it could conceivably be added. It would be difficult to know ahead of time though, because we'd have to perform a number of count queries for each resource type to determine which resources have been updated from the server, and which have local changes. A more feasible option would be to include the number of resources which match the sync parameters and are being considered for sync. Do you think that would be of value?

I also hadn't considered applying compression. This is definitely something we could add with very little effort using the GZIPInputStream and GZIPOutputStream classes which are part of the Java standard library. We'd probably want to measure whether or not the extra processing required for compression is worth it for transferring data over a local Wi-Fi connection. I'll keep track of that as an improvement to be applied after the initial implementation.

[pld]

Cool, yes I was thinking considered for sync, basically enough info so that the sender and receiver can present the user with a somewhat meaningful, but not exact, measure of progress and time estimation.

On compression that sounds good, it sounds like it's abstracted enough (or easy to extend to be) so that part of the improvement can be allowing the implementer to choose the stream class.

[bausmeier]

We've run into a challenge integrating the P2P sync functionality with the existing code. The plan was to implement P2P sync in its own p2p module, either within the existing Android FHIR SDK repo, or in its own repo. In order to implement P2P sync we need to make use of some new and existing lower-level functions from the Database interface. Currently the Database interface is marked as internal, so it can't be accessed from outside of the engine module.

We currently see 4 possible approaches to working around this:

1. Implement the core P2P sync functionality within the engine module instead of as a separate module. This adds more code and dependencies which need to be maintained to the engine, and ties the release lifecycle of the P2P functionality to that of the Android FHIR SDK.
2. Change the visibility of the Database interface from internal to public so that it can be used from outside of the engine module. The database could then be used directly from the p2p module, but this leaks some of the internals from the engine to all apps.
3. Expose the needed functions from Database interface on the FhirEngine interface. This means that the lower-level functions can be called on FhirEngine instance which the p2p module has access to, but leaks some of the internals of the engine to all apps.
4. Extend the FhirEngineImpl class from the p2p module, and only expose the functions on that subclass so that they're accessible within the p2p module. This will require the visibility of the FhirEngineImpl class to be changed from internal to public so that it can be extended, and for the visibility of the database field to be changed from private to protected so that it can be accessed within the subclass. Apps will then need to choose whether to initialise a normal FhirEngineImpl instance, or to initialise the one from the p2p module if they're making use of P2P sync.

[bausmeier]

@trevorgowing Please confirm that the above accurately captures our thinking from the discussion yesterday.

[trevorgowing]

@bausmeier Sounds good based on what we discussed yesterday. Trying to think of another solution that favours composition over inheritance.

[trevorgowing]

With transmitting changes

• Resource exists with sender but not receiver -> persist resource
• Resource exists with sender with no changes and with receiver with no changes -> conflict
• Resource exists with sender with changes and with receiver with no changes -> conflict
• Resource exists with sender with no changes and with receiver with changes -> conflict
• Resource deleted with sender and exists with receiver -> conflict
• Resource exists with sender and deleted with receiver -> conflict
• Resource deleted with sender and does not exist with receiver -> generate changes and persist resource and changes

Without transmitting changes

• Resource exists with sender but not receiver -> persist resource
• Resource exists with sender with no changes and with receiver with no changes -> generate changes and if exist, conflict else persist resource and changes
• Resource exists with sender with changes and with receiver with no changes -> generate changes and if exist, conflict else persist resource and changes
• Resource exists with sender with no changes and with receiver with changes -> generate changes and if exist, conflict else persist resource changes
• Resource deleted with sender and exists with receiver -> conflict
• Resource exists with sender and deleted with receiver -> conflict
• Resource deleted with sender and does not exist with receiver -> generate changes and persist resource and changes

[pld]

The ?s look like conflicts that would then follow the conflict strategy

[trevorgowing]

I tend to agree @pld.

[dubdabasoduba]

@trevorgowing @bausmeier

We should make sure we handle the following scenario.

• When the p2p sync is initiated the server sync should be disabled. This means the sync button and the period sync to the server should be disabled until the p2p sync is complete.

cc @pld