Sync and offline support

[f-odhiambo]

Save questionnaire resource to SQLite DB and ensure it syncs to FHIR server

• Sync offline data for patient registration and vaccine administration automatically sync to the FHIR server.
• Check for new patients/updates associated with that vaccine site or worker.

For ANC App

• The application will be able to support all key workflows offline including patient registration and ANC visit administration. The app will be able to automatically sync with a FHIR server. This will include checking for new patients/updates associated with that.

[pld]

From @fredhersch

Current approach: client constructs the query and sends to the HAPI restful end-point to perform an operation

Versus:

Does the SDK just point to an API end-point (passing in the authenticated user) that performs the operations server-side to determine what patients and resources the client should have access to?

The later makes sense to me because the server is what knows about permissions, and I believe that's what we're doing now.

@Andati @rkodev can you describe if this aligns w/what we're doing now?

CC @mberg

[vincent-karuri]

We need to coordinate with Google on this to flesh it out and figure out a strategy on sync. From this week's call, it seems like they would need our input.

[f-odhiambo]

Related to #42

[ekigamba]

Upload of new FHIR records and local modifications to the server

• Currently handled by FhirEngine#syncUpload. This gets all the local changes on the database and syncs to the server
  • We can improve by bulk uploading all the changes, currently each change is uploaded in it's own request
  • Fetch local changes from database by timestamp to make sure we maintain the event timeline for other devices which are going to be using the information/data
  • Batch the fetching and uploading of local changes to avoid performance issues
  • Order squashing of LocalChangeEntity for a single resource by timeline so that the latest changes are always on the right during the reduce function. Basically remove the possibility of new data being replaced with older data when squashing multiple changes made on a single Resource.
  • periodicSync only downloads periodically but needs to also upload first before downloading

Download of FHIR records periodically

• Currently handled byFhirEngine and periodicSync

  • On FhirEngine initialization, it triggers an initial download followed by scheduled subsequent syncs(data downloads) based on the SyncConfiguration timing. The downloads happen periodically based on the sync configuration.
  • Is SyncData#LAST_UPDATED_KEY similar to SyncData#LAST_UPDATED_ASC_KEY ?
  • We might need to define the no of retries in the SyncConfiguration. This will require a discussion on this and modification of android-fhir
  • The SyncConfiguration almost directly imitates the current OpenSRP filter functionality. It provides syncing of each resource type from a date X ordered by the last-updated-key. However, the SyncConfiguration requires a "location mapping" filter and the HAPI FHIR server should perform an authorisation and authentication check for each request. The server can also add extra filters depending on the user's authorisation
  • We need to add the location-mapping filters to the SyncConfiguration and update it in case the user changes their current focus location/care-team/organisation or their assignment changes
  • We need to decide on how to filter FHIR resources based on the location mapping conversation

Next steps

• Decide how FHIR core will filter FHIR resources. This is dependent on the result of the location mapping conversation
• Options for filtering when requesting data
  • Option 1: Use the search API to define custom parameters on how to download the data
  • Option 2: Add custom APIs on the server side to support complicated JOINs
  • Option 3: In case the filters are simple enough and properties of the resources, we can extend the current SyncConfiguration to support custom resource property filters
  • Option 4: We can implement a custom interceptor for Retrofit, this is assuming the sync params are simple might not need any changes to the SDK side
• Add a FHIRCoreApplication that implements generating the SyncConfiguration with the appropriate filters, resource types to sync and timing between periodic syncs
• Add FHIREngine#syncUpload as the first call during periodic sync in FHIR to ensure upload happens before the download
• Fix bugs in android-fhir upload functionality that would case loss of data and upload of data in the wrong chronological order - Possible bug when squashing of multiple local changes on a single resource during upload
• Optimize the upload functionality in android-fhir to bulk upload resources. HAPI FHIR server allows bulk uploading by resource type

[rkodev]

We can send a batch request by grouping by Resource Type and Request Type, the ordinality of the group can be determined combination of all POST requests = 0 and everything else = 1 + and the minimum change in the set of resources.

On the event downloads not exactly sure why the order will matter since the SDK seems not to care about the downloads, my understanding from the codebase looks like all data is saved regardless of the relationship and as long as sync is complete there will be Eventual Consistency.

On Sync :
Option 4 We can implement a custom interceptor for Retrofit, this is assuming the sync params are simple might not need any changes to the SDK side

General comment:
I think adding access control on the server-side instead of only relying on the client as we currently do in OpenSRP will great.

[ekigamba]

@rkodev I have updated the strategy based on your recommendations

[vincent-karuri]

@rkodev what do you mean by:

I think adding access control on the server-side instead of only relying on the client as we currently do in OpenSRP will great.

How does the client perform access control on OpenSRP?

[rkodev]

@vincent-karuri the older implementations of OpenSRP would return data without performing an authorization check to check whether the user requesting the data has access to the data.
The server relied on OpenSRP clients to request only the data the client has access to i.e provide the filter parameters

[f-odhiambo]

Currently blocked by location mapping tasks

[pld]

what's remaining here?

[ekigamba]

Update

• Some tasks on this issue are complete
  • Pushing immunization and patient resources to the server
  • Automatically checking for global immunization and patient resource updates on the server and downloading them
• The remaining tasks
  • Downloading updates per vaccine site/worker site - Requires work on location mapping to be completed
  • Support all key workflows required for the ANC app - This needs to more specific and broken down into smaller issues
  • Sync API updates based on discussions -> Not yet done. These need to be created as separate issues for better tracking
    • Optimizing sync by using bulk resource upload
    • Investigate squashing of LocalChangeEntity to confirm that new updates are not ignored
  • Offline support needs some updates to work correctly -> Needs research and probably updates on Android FHIR SDK

---

We probably want to change this issue into a discussion

[fredhersch]

Great to see the progress here.

Couple of questions:

1. Are you planning to use the sync API to construct the queries to determine which resources to sync or will you be using a custom end-point that you will point the SDK too?
2. What are the offline support updates required?

[ekigamba]

1. @f-odhiambo @dubdabasoduba Can you help answer # 1 ⬆️
2. Required offline updates - It seems this was fixed. I'd noticed that resources to be synced were not available on the resources table until synced to the server. This was probably fixed with the new engine update 😄 👍

I have updated my comment and updated the details also

[f-odhiambo]

@fredhersch @ekigamba Yes. This is what we want to achieve with the Single APK approach.
We will also tie in Keycloack to facilitate Role-Based access control (RBAC). Also, we have the location hierarchy (Sync Location/Team) piece that will aid in unblocking this once done i.e. #188 & #189

Lastly, converting this to a discussion point