FDA Feedback: Closes #152 Derive LOCF only considering ANL01FL=Y

[kaz462]

tlf-primary.r has been rerun after the update, the new output matches the one generated by pilot1

There are still some differences in the dataset adadas due to the following issue with CDISC data (NO impact for the output):

• CDISC SDTM/qs: 818 records for QSTESTCD=ACTOT

• CDISC ADaM/adadas: 1040 records for PARAMCD=ACTOT, 799 (directly from qs, should be 818) + 241 imputed records (DTYPE=LOCF)

• adadas generated by R: 1040 records for PARAMCD=ACTOT, 818 (directly from qs) + 222 imputed records (DTYPE=LOCF)

New diffs:

  =============================
   Variable  No of Differences 
  -----------------------------
    VISIT           33         
   VISITNUM         33         
     ADY            33         
     ADT            33         
     AVAL           19         
     CHG            19         
     PCHG           19         
    DTYPE           19         
   AWTDIFF          33         
    QSSEQ           33         
  -----------------------------

[robertdevine]

Hi @kaz462, the lintr is failing due to commented out code, mutate( ) function etc. Please let me know if you want me to correct. thx.

[kaz462]

Hi @kaz462, the lintr is failing due to commented out code, mutate( ) function etc. Please let me know if you want me to correct. thx.

@robertdevine that would be great, thanks! I'm not sure if this PR will be merged though, let's check with FDA this Thursday?

Thank you, @kaz462, will do on the lintr check item, and best as you noted, to confirm FDA Reviewers feedback this Thursday before decision to merge the PR. thx.

All set @kaz462, lintr check was failing due to (i)commented out code at mutate( ) and (ii)superfluous white spaces at adas_locf2 (line 129) and 'order = exprs(AVISITN, AVISIT),' (line 139). thx.

[laxamanaj]

Hi, @kaz462 . Thanks again for working on this and showing that ANL01FL ='Y' is what matches the original Pilot 1 results. We discussed this in a smaller meeting, but just noting here that we'll touch base with FDA on our findings to then see which approach would be ok to move forward. In the end, we may just want to match with Pilot 1 though may not be the approach to take, but we want to show that our R submission works including ADaMs. I'll confirm a merge after meeting with FDA.

[kaz462]

qc findings are updated here: https://github.com/RConsortium/submissions-pilot3-adam/wiki/QC-Findings#adadas
@laxamanaj please update ADRG accordingly, thx!

[robertdevine]

@laxamanaj, @kaz462, @bms63 - should the SAP be updated with a statement about the visit window rule used to tidy the output and match the original pilot1? - it appears to follow from the earlier observation regarding analysis visits.

The visit window rule approach specified in the SAP is translated into the programming specifications for Analysis Data Model (ADaM). Visit window rules sometimes appear as a separate tab in the ADaM specs Excel spreadsheet, or as a part of the specs for each corresponding ADaM dataset. thx.

[bms63]

Could we do a comment on why we this derivation is the way it is - feedback from FDA on need to match Pilot 1 - and maybe even provide github wiki qc finding link?

[kaz462]

@bms63 good point! updated

[bms63]

I updated the PR title a bit. Just a comment to insert in the code is suggested.

I think I changed it to Squash and Merge so the PR title is the commit message. Feel free to merge in - just make sure it is Squash and Merge

[bms63]

@laxamanaj, @kaz462, @bms63 - should the SAP be updated with a statement about the visit window rule used to tidy the output and match the original pilot1? - it appears to follow from the earlier observation regarding analysis visits.

The visit window rule approach specified in the SAP is translated into the programming specifications for Analysis Data Model (ADaM). Visit window rules sometimes appear as a separate tab in the ADaM specs Excel spreadsheet, or as a part of the specs for each corresponding ADaM dataset. thx.

@robertdevine I don't think we should update the SAP - we should just put it in the findings/feedback notes

@bms63 - maybe we'll add a 'Determination' paragraph onto the findings/feedback notes here under Issue#2 following the Action Item, and Observation entries. Doing so may provide further detail about the issue determination for visitors observing the pilot series and the regulatory feedback process. 21 CFR Part 11 comes to mind regarding clinical data - reliable, consistent, and accurate for regulatory review trace purposes. thx.

[robertdevine]

@laxamanaj, @bms63, @kaz462 - Pilot3 Team should be aware that pilot1wrappers remains in the pilot3 reproducible environment. A very simple test regarding the Primary Endpoint Analysis: ADAS Cog (11) - Change from Baseline to Week 24 - LOCF might be to use the pilot3 repository and pilot1wrappers separately from pilot3 (and its pilot3utils dependency) to see if the discrepancies pointed out by FDA Reviewers can be reproduced (contained within) the pilot3 repository (given the existing inclusion of pilot1wrappers). There are only six steps in the primary program and a simple test may tease out if pilot1wrappers inclusion in pilot3 repository can reproduce the pilot1 primary outputs (using pilot1wrappers only) running from the pilot3 repository. If so, removing pilot1wrappers (keeping everything else the same in the pilot3 repository) with the exception of changing primary output library dependency to pilot3 and its dependency on pilot3utils (then removing pilot1wrappers from the pilot3 reproducible environment) to check if the primary outputs match. Has anyone on Pilot3 Team performed this test? Seems reasonable and compelling to do so given the FDA feedback regarding primary output discrepancies between pilot1 and pilot3 and this outstanding issue about pilot1wrappers and refactoring the lock file here. thx.

[bms63]

@laxamanaj, @bms63, @kaz462 - Pilot3 Team should be aware that pilot1wrappers remains in the pilot3 reproducible environment. A very simple test regarding the Primary Endpoint Analysis: ADAS Cog (11) - Change from Baseline to Week 24 - LOCF might be to use the pilot3 repository and pilot1wrappers separately from pilot3 (and its pilot3utils dependency) to see if the discrepancies pointed out by FDA Reviewers can be reproduced (contained within) the pilot3 repository (given the existing inclusion of pilot1wrappers). There are only six steps in the primary program and a simple test may tease out if pilot1wrappers inclusion in pilot3 repository can reproduce the pilot1 primary outputs (using pilot1wrappers only) running from the pilot3 repository. If so, removing pilot1wrappers (keeping everything else the same in the pilot3 repository) with the exception of changing primary output library dependency to pilot3 and its dependency on pilot3utils (then removing pilot1wrappers from the pilot3 reproducible environment) to check if the primary outputs match. Has anyone on Pilot3 Team performed this test? Seems reasonable and compelling to do so given the FDA feedback regarding primary output discrepancies between pilot1 and pilot3 and this outstanding issue about pilot1wrappers and refactoring the lock file here. thx.

Hi Robert -

pilot1 versus pilot3

I thought all the pilot1 wrapper programs were put into the pilot3utils package. If not, then we should add remaining and then remove any of these pilot1 references from our code going forward. Hopefully, this clean up any lingering issues. Do you have time for this Robert? Hi @bms63, yes, the pilot1wrappers references should be removed - removing does require changes to several programs as noted in #133. Will do. thx @robertdevine

@bms63, @kaz462, @laxamanaj - searching some of the CDISC pilot replication sources - for example here,
the 14-3.01.rtf (which corresponds to Pilot1/3 primary outputs) matches the pilot1 output with regard to the highlighted
discrepancies observed during regulatory review of the pilot3 primary output - the CDISC study is from several years
ago however given the pilot1 output match and no discussion about output specific details here
great care should be taken in the pilot3 efforts to assure correctness and consistency regarding the primary outputs.
(please see below) The CDISC Pilot replication should be notable as the program it uses to generate the primary
output does not use pilot1wrappers (or RCWG pilot package) - thus may provide independent confirmation
regarding the interpretation found in QC Findings. thx.

FDA Reviewers Observations - pilot1 and pilot3 (highlighted)

CDISC Replication Effort (matches Pilot 1 primary output) - compare with highlighted values in pilot3 output

fda findings

My impression of the discussion was how we selected the observations in pilot3 was the primary driver in the differences in pilot1 versus pilot3. This update form @kaz462 should bring us into alignment with pilot1. As far I knew, the code in the pilot3utils package is identical to the pilot1 helper code.

Thank you, @bms63

[laxamanaj]

Minor comment regarding comments. Otherwise, agree with the updates here.

[laxamanaj]

For the comments regarding the FDA feedback and why we're making these changes, I wonder if this needs to be included here since we're already capturing this conversation in git comments.

[laxamanaj]

Referring to Lines 118 - 122.

[robertdevine]

@laxamanaj, @bms63, @kaz462 - commented lined 118 - 122 removed per requested changes. thx