Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre vs berkeley DO NOT MERGE HERE #275

Draft
wants to merge 30 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3d4820b
WIP: Generate `refgraph` visualizations via GHA workflow
eecavanna Sep 27, 2024
9852362
Constrain Python version and regenerate `poetry.lock` file
eecavanna Sep 27, 2024
5ecdcd5
WIP: Inject `refgraph` pages into documentation preview site
eecavanna Sep 27, 2024
ea3794b
Install `refscan` via `pipx` instead of via `poetry`
eecavanna Sep 27, 2024
27754b1
Regenerate `poetry.lock` file (by running `$ poetry lock --no-update`)
eecavanna Sep 27, 2024
9c22098
Use correct directory name
eecavanna Sep 27, 2024
aa4843c
Remove comment to simplify PR diff
eecavanna Sep 27, 2024
80d509d
Remove unnecessary commands from GHA workflow
eecavanna Sep 27, 2024
d8ed9b3
Use `pipx run` to avoid installing anything
eecavanna Sep 27, 2024
4e753b9
Revert unnecessary changes to `poetry.lock` file
eecavanna Sep 27, 2024
9b750e1
WIP: Update site navigation bar so it contains links to graphs
eecavanna Sep 27, 2024
8915b2b
Restrict which `refscan` version will be used
eecavanna Sep 27, 2024
53a7875
Remove comment unrelated to other changes on branch
eecavanna Sep 27, 2024
168a3cc
Standardize comments between doc-related GHA workflows
eecavanna Sep 27, 2024
4da13d4
WIP: Populate project version number before generating derivative files
eecavanna Sep 27, 2024
69abcc6
pre-vs-berkeley
turbomam Oct 15, 2024
39eaa8e
remove documentation changes @eecavanna from thsi branch
turbomam Oct 15, 2024
9b0be8d
examples and abstracts
turbomam Oct 15, 2024
6ad4374
surprise added a makefile
turbomam Oct 15, 2024
3f944cb
Update pre_vs_berkeley.md
turbomam Oct 16, 2024
ca84684
Update pre_vs_berkeley.md
turbomam Oct 16, 2024
e69e585
define root classes and explain vulnerability
turbomam Oct 16, 2024
4a90747
title: Table of classes removed, added or refactored...
turbomam Oct 16, 2024
193dde8
Conventions
turbomam Oct 16, 2024
6ae9d08
schema doesn't classify StorageProcess as a PlannedProcess
turbomam Oct 16, 2024
58a9fba
updates and add slot_usage etc section
turbomam Oct 21, 2024
5ffbb26
new and "in v11" are redundant
turbomam Oct 21, 2024
84814e8
configuration subclasses illustrated in inlined image
turbomam Oct 21, 2024
9e363b0
see also release notes
turbomam Oct 21, 2024
d7b2407
pre_vs_berkeley folder
turbomam Oct 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added Activity.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added AttributeValue10.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added AttributeValue11.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Configuration.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added InformationObject.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added MaterialProcessing.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added MobilePhaseSegment.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added PortionOfSubstance.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added ProtocolExecution.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
78 changes: 78 additions & 0 deletions pre_vs_berkeley.Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
pre_schema.yaml:
poetry run python -c 'from linkml_runtime.utils.schemaview import SchemaView; \
from linkml_runtime.dumpers import yaml_dumper; \
schema_url = "https://raw.githubusercontent.com/microbiomedata/nmdc-schema/refs/tags/v10.9.1/src/schema/nmdc.yaml"; \
sv = SchemaView(schema_url, merge_imports=True); \
yaml_dumper.dump(sv.schema, "pre_schema.yaml")'

berkeley_schema.yaml:
poetry run python -c 'from linkml_runtime.utils.schemaview import SchemaView; \
from linkml_runtime.dumpers import yaml_dumper; \
schema_url = "https://raw.githubusercontent.com/microbiomedata/nmdc-schema/refs/tags/v11.0.1/src/schema/nmdc.yaml"; \
sv = SchemaView(schema_url, merge_imports=True); \
yaml_dumper.dump(sv.schema, "berkeley_schema.yaml")'

pre_study.yaml: pre_schema.yaml
yq '.classes.Study' $< | cat > $@

pre_biosample.yaml: pre_schema.yaml
yq '.classes.Biosample' $< | cat > $@

pre_types.yaml: pre_schema.yaml
yq '.types' $< | cat > $@

pre_settings.yaml: pre_schema.yaml
yq '.settings' $< | cat > $@

pre_prefixes.yaml: pre_schema.yaml
yq '.prefixes' $< | cat > $@

pre_enums.yaml: pre_schema.yaml
yq '.enums' $< | cat > $@

pre_subsets.yaml: pre_schema.yaml
yq '.subsets' $< | cat > $@

berkeley_study.yaml: berkeley_schema.yaml
yq '.classes.Study' $< | cat > $@

berkeley_biosample.yaml: berkeley_schema.yaml
yq '.classes.Biosample' $< | cat > $@

berkeley_types.yaml: berkeley_schema.yaml
yq '.types' $< | cat > $@

berkeley_settings.yaml: berkeley_schema.yaml
yq '.settings' $< | cat > $@

berkeley_prefixes.yaml: berkeley_schema.yaml
yq '.prefixes' $< | cat > $@

berkeley_enums.yaml: berkeley_schema.yaml
yq '.enums' $< | cat > $@

berkeley_subsets.yaml: berkeley_schema.yaml
yq '.subsets' $< | cat > $@

pre_vs_berkeley_study.yaml: pre_study.yaml berkeley_study.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_biosample.yaml: pre_biosample.yaml berkeley_biosample.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_types.yaml: pre_types.yaml berkeley_types.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_settings.yaml: pre_settings.yaml berkeley_settings.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_prefixes.yaml: pre_prefixes.yaml berkeley_prefixes.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_enums.yaml: pre_enums.yaml berkeley_enums.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_subsets.yaml: pre_subsets.yaml berkeley_subsets.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@


640 changes: 640 additions & 0 deletions pre_vs_berkeley.md

Large diffs are not rendered by default.

121 changes: 121 additions & 0 deletions pre_vs_berkeley.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
import pprint

from linkml_runtime import SchemaView
from linkml_runtime.dumpers import yaml_dumper

from linkml_runtime.linkml_model.meta import ClassDefinition, SlotDefinition

# from terminado.management import preexec_fn

pre_path = "https://raw.githubusercontent.com/microbiomedata/nmdc-schema/refs/tags/v10.9.1/src/schema/nmdc.yaml"

berkeley_path = "https://raw.githubusercontent.com/microbiomedata/nmdc-schema/refs/tags/v11.0.1/src/schema/nmdc.yaml"

pre_view = SchemaView(pre_path)

berkeley_view = SchemaView(berkeley_path)

pre_classes = pre_view.all_classes()
pre_class_names = list(pre_classes.keys())
print(len(pre_class_names))

berkeley_classes = berkeley_view.all_classes()
berkeley_class_names = list(berkeley_classes.keys())
print(len(berkeley_class_names))

pre_only = [c for c in pre_class_names if c not in berkeley_class_names]

berkeley_only = [c for c in berkeley_class_names if c not in pre_class_names]

pprint.pprint(pre_only)

pprint.pprint(berkeley_only)

pre_roots = pre_view.class_roots()
pre_roots.sort()
pprint.pprint(pre_roots)
print(len(pre_roots))

berkeley_roots = berkeley_view.class_roots()
berkeley_roots.sort()
pprint.pprint(berkeley_roots)
print(len(berkeley_roots))

pre_only = [c for c in pre_roots if c not in berkeley_roots]
pprint.pprint(pre_only)
print(len(pre_only))

berkeley_only = [c for c in berkeley_roots if c not in pre_roots]
pprint.pprint(berkeley_only)
print(len(berkeley_only))

####

pre_slots = pre_view.all_slots()
pre_slot_names = list(pre_slots.keys())
print(f"{len(pre_slot_names)}")

berkeley_slots = berkeley_view.all_slots()
berkeley_slot_names = list(berkeley_slots.keys())
print(f"{len(berkeley_slot_names)}")

pre_only = [s for s in pre_slot_names if s not in berkeley_slot_names]
pre_only.sort()

berkeley_only = [s for s in berkeley_slot_names if s not in pre_slot_names]
berkeley_only.sort()

pprint.pprint(pre_only)
print(f"{len(pre_only)}")

pprint.pprint(berkeley_only)
print(f"{len(berkeley_only)}")

####

# ChromatographicSeparationProcess, type: <class 'linkml_runtime.linkml_model.meta.ClassDefinition'>
# OmicsProcessing, type: <class 'linkml_runtime.linkml_model.meta.ClassDefinition'>
# CreditAssociation, type: <class 'linkml_runtime.linkml_model.meta.ClassDefinition'>
# total_bases, type: <class 'linkml_runtime.linkml_model.meta.SlotDefinition'>
# members_id, type: <class 'linkml_runtime.linkml_model.meta.SlotDefinition'>
# bin_name, type: <class 'linkml_runtime.linkml_model.meta.SlotDefinition'>

pre_abstract = []
pre_abstract_classes = []
pre_abstract_slots = []
pre_elements = pre_view.all_elements()
for ek, ev in pre_elements.items():
if isinstance(ev, ClassDefinition) and ev.abstract:
pre_abstract_classes.append(ek)
if isinstance(ev, SlotDefinition) and ev.abstract:
pre_abstract_slots.append(ek)

print(f"pre_abstract_classes: {len(pre_abstract_classes)}")
print(f"pre_abstract_slots: {len(pre_abstract_slots)}")

berkeley_abstract = []
berkeley_abstract_classes = []
berkeley_abstract_slots = []

berkeley_elements = berkeley_view.all_elements()

for ek, ev in berkeley_elements.items():
if isinstance(ev, ClassDefinition) and ev.abstract:
berkeley_abstract_classes.append(ek)
if isinstance(ev, SlotDefinition) and ev.abstract:
berkeley_abstract_slots.append(ek)

print(f"berkeley_abstract_classes: {len(berkeley_abstract_classes)}")
print(f"berkeley_abstract_slots: {len(berkeley_abstract_slots)}")

print(berkeley_abstract_classes)
print(pre_abstract_classes)

####


pre_study = pre_view.get_class("Study")
yaml_dumper.dump(pre_study, "pre_study.yaml")

berkeley_study = berkeley_view.get_class("Study")
yaml_dumper.dump(berkeley_study, "berkeley_study.yaml")
Binary file added pre_vs_berkeley/Activity.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pre_vs_berkeley/AttributeValue10.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pre_vs_berkeley/AttributeValue11.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pre_vs_berkeley/Configuration.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pre_vs_berkeley/InformationObject.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pre_vs_berkeley/MaterialProcessing.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pre_vs_berkeley/MobilePhaseSegment.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pre_vs_berkeley/PortionOfSubstance.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pre_vs_berkeley/ProtocolExecution.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
78 changes: 78 additions & 0 deletions pre_vs_berkeley/pre_vs_berkeley.Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
pre_schema.yaml:
poetry run python -c 'from linkml_runtime.utils.schemaview import SchemaView; \
from linkml_runtime.dumpers import yaml_dumper; \
schema_url = "https://raw.githubusercontent.com/microbiomedata/nmdc-schema/refs/tags/v10.9.1/src/schema/nmdc.yaml"; \
sv = SchemaView(schema_url, merge_imports=True); \
yaml_dumper.dump(sv.schema, "pre_schema.yaml")'

berkeley_schema.yaml:
poetry run python -c 'from linkml_runtime.utils.schemaview import SchemaView; \
from linkml_runtime.dumpers import yaml_dumper; \
schema_url = "https://raw.githubusercontent.com/microbiomedata/nmdc-schema/refs/tags/v11.0.1/src/schema/nmdc.yaml"; \
sv = SchemaView(schema_url, merge_imports=True); \
yaml_dumper.dump(sv.schema, "berkeley_schema.yaml")'

pre_study.yaml: pre_schema.yaml
yq '.classes.Study' $< | cat > $@

pre_biosample.yaml: pre_schema.yaml
yq '.classes.Biosample' $< | cat > $@

pre_types.yaml: pre_schema.yaml
yq '.types' $< | cat > $@

pre_settings.yaml: pre_schema.yaml
yq '.settings' $< | cat > $@

pre_prefixes.yaml: pre_schema.yaml
yq '.prefixes' $< | cat > $@

pre_enums.yaml: pre_schema.yaml
yq '.enums' $< | cat > $@

pre_subsets.yaml: pre_schema.yaml
yq '.subsets' $< | cat > $@

berkeley_study.yaml: berkeley_schema.yaml
yq '.classes.Study' $< | cat > $@

berkeley_biosample.yaml: berkeley_schema.yaml
yq '.classes.Biosample' $< | cat > $@

berkeley_types.yaml: berkeley_schema.yaml
yq '.types' $< | cat > $@

berkeley_settings.yaml: berkeley_schema.yaml
yq '.settings' $< | cat > $@

berkeley_prefixes.yaml: berkeley_schema.yaml
yq '.prefixes' $< | cat > $@

berkeley_enums.yaml: berkeley_schema.yaml
yq '.enums' $< | cat > $@

berkeley_subsets.yaml: berkeley_schema.yaml
yq '.subsets' $< | cat > $@

pre_vs_berkeley_study.yaml: pre_study.yaml berkeley_study.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_biosample.yaml: pre_biosample.yaml berkeley_biosample.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_types.yaml: pre_types.yaml berkeley_types.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_settings.yaml: pre_settings.yaml berkeley_settings.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_prefixes.yaml: pre_prefixes.yaml berkeley_prefixes.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_enums.yaml: pre_enums.yaml berkeley_enums.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@

pre_vs_berkeley_subsets.yaml: pre_subsets.yaml berkeley_subsets.yaml
poetry run deep diff --ignore-order $^ | yq -P | cat > $@


Loading
Loading