Add VA/VRS GraphQL API #1426

phildarnowsky-broad · 2024-02-26T18:52:54Z

This allows querying of VA/VRS data (see https://vrs.ga4gh.org) through the gnomAD browser API.

phildarnowsky-broad · 2024-02-26T18:53:28Z

rileyhgrant

Just one teeny tiny nitpick, and one request:

Since there's a new config file that should be present for successful deployments, can we add to our CONTRIBUTING.md to include this information? Maybe a la these lines, or in some other way that seems fitting?

I'm not entirely sure how places normally handle this, but it would be good to get ahead of the issue of someone deploying without this file on accident.

rileyhgrant · 2024-02-28T16:39:21Z

.gitignore

@@ -29,3 +29,5 @@ hail-*.log

 # vscode settings
 **/.vscode/
+
+graphql-api/unversioned_config.json


nitpick (non-blocking): It looks like the other env/config type files bundled into various dockerfiles starting at line 16, maybe this is a better fit to put in there?

phildarnowsky-broad · 2024-06-03T21:41:59Z

@rileyhgrant @mattsolo1 ready for another look. This is basically a full overhaul so probably best just to approach it fresh.

rileyhgrant

Lots of good things in this PR, the commitment to strong types and the inclusion of tests to confirm behavior are very nice.

2 main changes requested, and one other small thing:

I don't appear to be able to compile this on my machine, I fail with this error from ts.

❯ ELASTICSEARCH_USERNAME=elastic ELASTICSEARCH_PASSWORD=$ELASTICSEARCH_PASSWORD ./start.sh
/Users/rgrant/dev/work-broad/gnomad-browser/node_modules/.pnpm/ts-node@10.9.1_@types+node@20.7.0_typescript@5.2.2/node_modules/ts-node/src/index.ts:859
    return new TSError(diagnosticText, diagnosticCodes, diagnostics);
           ^
TSError: ⨯ Unable to compile TypeScript:
src/graphql/schema.ts:28:8 - error TS1192: Module '"/Users/rgrant/dev/work-broad/gnomad-browser/graphql-api/src/graphql/resolvers/va"' has no default export.

28 import vaResolvers from './resolvers/va'
          ~~~~~~~~~~~
...

pnpm typecheck also complains about this import:

❯ pnpm typecheck

> @ typecheck /Users/rgrant/dev/work-broad/gnomad-browser
> pnpm tsc --noEmit

graphql-api/src/graphql/schema.ts:28:8 - error TS1192: Module '"/Users/rgrant/dev/work-broad/gnomad-browser/graphql-api/src/graphql/resolvers/va"' has no default export.

28 import vaResolvers from './resolvers/va'
          ~~~~~~~~~~~


Found 1 error in graphql-api/src/graphql/schema.ts:28

ELIFECYCLE  Command failed with exit code 2.

I'd guess this was mistakenly introduced when cleaning up the code to get it PR ready.

The variant pipeline has huge swaths of the pipeline commented out, most likely to aid in speedy development of the pipeline by ensuring you're annotating on top of the previous finished variant table. Can we un comment this, and add the step to the end of the pipeline, now that it has been run based off of the previous output in your development of this feature?

The types look well defined, and I like the solution of annotating the IDs in our own pipeline rather than waiting for someone upstream to do that -- just a few clean up things on this PR to my eyes.

rileyhgrant · 2024-06-04T20:02:52Z

data-pipeline/src/data_pipeline/data_types/variant/annotate_variants.py

+    vrs = exome_vrs.union(genome_vrs)
+    vrs = vrs.group_by(vrs.locus, vrs.alleles).aggregate(vrs=hl.agg.collect(vrs.vrs))
+    ds = ds.join(vrs)
+    ds.describe()


Can this describe() be removed now that this is production ready?

rileyhgrant · 2024-06-04T20:10:08Z

data-pipeline/src/data_pipeline/pipelines/gnomad_v4_variants.py

+#   pipeline.add_task(
+#       name="prepare_gnomad_v4_variants",
+#       task_function=prepare_gnomad_v4_variants,
+#       output_path=f"{output_sub_dir}/gnomad_v4_variants_base.ht",
+#       inputs={
+#           "exome_variants_path": "gs://gcp-public-data--gnomad/release/4.1/ht/exomes/gnomad.exomes.v4.1.sites.ht",
+#           "genome_variants_path": "gs://gcp-public-data--gnomad/release/4.1/ht/genomes/gnomad.genomes.v4.1.sites.ht",
+#           "variants_joint_frequency_path": "gs://gcp-public-data--gnomad/release/4.1/ht/joint/gnomad.joint.v4.1.sites.ht",
+#       },
+#   )


Since this pipeline has been run with the new task, can these steps be un-commented again?

I imagine that the purpose of commenting these out was to ensure that these steps didn't run, so that you could take the old output and annotate on top of that. It appears as though that has happened. As such, could we return the pipeline to the previous, un-commented state, now with the inclusion of your additional step?

indeed, I meant to do so but apparently either forgot or accidentally reverted it

rileyhgrant · 2024-06-04T20:44:22Z

graphql-api/src/graphql/schema.ts

@@ -25,6 +25,7 @@ import variantResolvers from './resolvers/variants'
 import variantFieldResolvers from './resolvers/variant-fields'
 import variantCooccurrenceResolvers from './resolvers/variant-cooccurrence'
 import cnvCoverageResolvers from './resolvers/cnv-coverage'
+import vaResolvers from './resolvers/va'


I get a typescript error from here (as ./resolvers/va doesn't have a default export).

❯ pnpm typecheck > @ typecheck /Users/rgrant/dev/work-broad/gnomad-browser > pnpm tsc --noEmit graphql-api/src/graphql/schema.ts:28:8 - error TS1192: Module '"/Users/rgrant/dev/work-broad/gnomad-browser/graphql-api/src/graphql/resolvers/va"' has no default export. 28 import vaResolvers from './resolvers/va' ~~~~~~~~~~~ Found 1 error in graphql-api/src/graphql/schema.ts:28 ELIFECYCLE Command failed with exit code 2.

Is it possible that some refactoring happened and this error got mistakenly introduced? I was going to start a local API to check the VA fields that become queryable by this PR, and got this error.

As something not directly related:

I'm not sure why the graphql-api CI didn't run on this PR, the yaml file makes it look like it should have run.

... pull_request: paths: - 'graphql-api/**' - 'dataset-metadata/**' - 'package.json' - 'pnpm-lock.yaml' - '.github/workflows/**' ...

I removed some unused code at the very end but missed a few references to it due to negelecting to run typecheck, fixed

phildarnowsky-broad · 2024-06-05T18:54:49Z

@rileyhgrant ready for another look here, it just needed some post-rebase cleanup

rileyhgrant

LGTM!

Started a local API and queried a few variants to check out the new VA data, all was looking good to my eyes.

rileyhgrant · 2024-06-06T16:45:53Z

data-pipeline/src/data_pipeline/pipelines/gnomad_v4_variants.py

-    validate_step4_output(pipeline)
+#       write_schemas(
+#           [pipeline],
+#           os.path.expanduser("~/schemas"),


Ooh nice one, os.path.expanduser() is a neat function

phildarnowsky-broad · 2024-06-14T19:59:59Z

@rileyhgrant couple of updates here for your review, the big items are, we now support search by VRS ID, and the HT data gets massaged into a more pleasant shape before we export to ES. If you want to check this out you can use the index gnomad_v4_variants-2024-06-13--14-27; note that that is downsampled to only 100 variants, though. LMK if you want help finding variants to check this against.

rileyhgrant

Started a local API and ran some test queries against a variant.

With the assumption that the temporary indice change will be reverted in favor of aliasing the more recent variant index, and with the comment about including caids in the load if possible, lgtm!

rileyhgrant · 2024-06-17T16:46:00Z

data-pipeline/src/data_pipeline/pipelines/export_to_elasticsearch.py

@@ -132,6 +132,7 @@ def add_liftover_document_id(ds):
                "locus",
                "transcript_consequences.gene_id",
                "transcript_consequences.transcript_id",
+                "vrs.alt.allele_id",


Not directly related, but I realized that caid is still commented out a few lines above this.

Might be worth it to uncomment that before (as I think this is still before) we do the next full load of the variants, as the table has that information in it already.

phildarnowsky-broad requested review from mattsolo1 and rileyhgrant February 26, 2024 18:53

rileyhgrant requested changes Feb 28, 2024

View reviewed changes

rileyhgrant assigned phildarnowsky-broad Mar 5, 2024

phildarnowsky-broad force-pushed the va_api branch 2 times, most recently from dc41bdf to ee902e1 Compare May 6, 2024 16:29

phildarnowsky-broad force-pushed the va_api branch 4 times, most recently from f32ef00 to 6733d19 Compare June 3, 2024 15:48

rileyhgrant self-requested a review June 4, 2024 02:08

rileyhgrant requested changes Jun 4, 2024

View reviewed changes

rileyhgrant linked an issue Jun 4, 2024 that may be closed by this pull request

Expose VA API via GraphQL #1132

Closed

phildarnowsky-broad force-pushed the va_api branch from 000c99f to 6439818 Compare June 5, 2024 18:02

mattsolo1 mentioned this pull request Jun 5, 2024

Try making va_api graphql response directly from elasticsarch data #1537

Closed

rileyhgrant self-requested a review June 6, 2024 16:43

rileyhgrant approved these changes Jun 6, 2024

View reviewed changes

rileyhgrant self-requested a review June 17, 2024 15:31

rileyhgrant approved these changes Jun 17, 2024

View reviewed changes

phildarnowsky-broad force-pushed the va_api branch 4 times, most recently from 31a79e2 to 9f6bea8 Compare June 20, 2024 17:10

phildarnowsky-broad added 3 commits June 24, 2024 15:58

Add VRS data to v4 variant data pipeline

747900b

Add VRS data to GraphQL API

83cb92c

Add some annotations to document new VA fields in API

ef8be18

phildarnowsky-broad force-pushed the va_api branch from 9f6bea8 to ef8be18 Compare June 24, 2024 19:58

phildarnowsky-broad merged commit f451fb8 into main Jun 24, 2024
6 checks passed

mattsolo1 mentioned this pull request Jul 8, 2024

GA4GH GKS API review #1472

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add VA/VRS GraphQL API #1426

Add VA/VRS GraphQL API #1426

phildarnowsky-broad commented Feb 26, 2024

phildarnowsky-broad commented Feb 26, 2024

rileyhgrant left a comment

rileyhgrant Feb 28, 2024

phildarnowsky-broad commented Jun 3, 2024

rileyhgrant left a comment

rileyhgrant Jun 4, 2024

rileyhgrant Jun 4, 2024

phildarnowsky-broad Jun 5, 2024

rileyhgrant Jun 4, 2024

phildarnowsky-broad Jun 5, 2024

phildarnowsky-broad commented Jun 5, 2024

rileyhgrant left a comment

rileyhgrant Jun 6, 2024 •

edited

Loading

phildarnowsky-broad commented Jun 14, 2024

rileyhgrant left a comment

rileyhgrant Jun 17, 2024 •

edited

Loading

Add VA/VRS GraphQL API #1426

Add VA/VRS GraphQL API #1426

Conversation

phildarnowsky-broad commented Feb 26, 2024

phildarnowsky-broad commented Feb 26, 2024

rileyhgrant left a comment

Choose a reason for hiding this comment

rileyhgrant Feb 28, 2024

Choose a reason for hiding this comment

phildarnowsky-broad commented Jun 3, 2024

rileyhgrant left a comment

Choose a reason for hiding this comment

rileyhgrant Jun 4, 2024

Choose a reason for hiding this comment

rileyhgrant Jun 4, 2024

Choose a reason for hiding this comment

phildarnowsky-broad Jun 5, 2024

Choose a reason for hiding this comment

rileyhgrant Jun 4, 2024

Choose a reason for hiding this comment

phildarnowsky-broad Jun 5, 2024

Choose a reason for hiding this comment

phildarnowsky-broad commented Jun 5, 2024

rileyhgrant left a comment

Choose a reason for hiding this comment

rileyhgrant Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

phildarnowsky-broad commented Jun 14, 2024

rileyhgrant left a comment

Choose a reason for hiding this comment

rileyhgrant Jun 17, 2024 • edited Loading

Choose a reason for hiding this comment

rileyhgrant Jun 6, 2024 •

edited

Loading

rileyhgrant Jun 17, 2024 •

edited

Loading