PR 235: Sn ExternalQualityMetric submission template * In commands/write_submission_spreadsheets.py:
- Add --eqm [dsa duplexseq] argument that grabs tooltip, key, and derived_from from the appropriate ExternalQualityMetric template and writes out to a new tab in the spreadsheet
- Remove properties from DonorSpecificAssembly that are now on ExternalQualityMetric
PR 263: React bootstrap v2, React 18, Redux and Misc. Npm Packages Upgrade
- Upgrade: React v17 to v18
- Upgrade: Redux v4 to v5 (there are breaking changes in store and dispatchers. SPC is updated to support both new and legacy usage)
- Upgrade: auth0-Lock v11 to v12
- Upgrade: gulp.js v4 to v5
- Upgrade: react-workflow-viz (animation updates to eliminate findDOMNode errors)
- Fix: User Content updates to fix markdown, jsx, and HTML static section rendering
- Upgrade: Bootstrap v5
- Upgrade: React-Bootstrap v2
- Upgrade: FontAwesome v6
- Node 18 to 20 upgrade including GitHub actions
PR 276: Updates for Statistics Legend Section & Mobile UI
- Adjust legend's items size for sm to xl displays
- Truncate long text for dropdown text for lg and md displays
- Fix toggle display for upcoming xxl displays upon react-bootstrap v2 upgrade
- Make property enzymes in LibraryPreparation optional if fragmentation_method does not contain "Transposase" or "Restriction Enzyme", otherwise required
- 2024-10-11/dmichaels
- Updated dcicutils version (8.16.1) for vulnerabilities.
PR 273: GA4 & Usage Statistics Follow-up
- New/Updated charts:
- File downloads
- Top file set downloads by file type - file format
- File detail views by file type - file format
- File search result appearances by file type - file format
- File search result clicks by file type - file format
- Metadata.tsv included file counts by location
- New options:
- 6-12-18 months and All date interval options added
- Toggle for charts
PR 224: feat: error state for homepage timeline
- Implement error state for homepage timeline
- Bugfix on Submission Status page. Could not set tags from Review File Group Qc modal when there were no tags present
PR267: SN Add target_read_count
- Add target_read_count to File data_generation_summary
- Added scripts/opensearch-dashboard-start.bash and Makefile target opensearch-dashboard-starto now that we are using OpenSearch rather than ElasticSearch we seem to need this rather than Kibana.
- Simplified Makefile for deploy1/1a/1b/2 for dcicsnovault 11.21.1 changes; related to allowing smaht-portal and cgap-portal to run at the same time locally.
- Added /files/upload_file_size endpoint (types/file.py); for use by smaht-submitr to determine if a file to upload has already been uploaded; and get its size as a side-effect; returns HTTP 200 if found otherwise 404); in particular we want to check if it has been uploaded but is still marked as 'uploading' because its md5 is still in the process of being computed (i.e. otherwise we could simply look at the file status an be done with it).
- New protection of /ingestion-status endpoint used by smaht-submitr (in ingestion/ingestion_status.py) for authenticated users only; and also limit /ingestion-status/{keys,keys_sorted,flush} to admin users only.
- Add decorator link_related_validator to edit validators as well as add
PR260: SN Add assay and vcf properties
- Add property 'category' to Assay
- Add property mode to VariantCalls
- Fix age maximum to 89 for Donor
- Add property rna_seq_protocol to LibraryPreparation and adjust enums for insert_selection_method
PR 231: Bm file overview updates
- Change warning banner on COLO829
- Improve status indicators to File Overview Page
- Add support for tsv notes
- Ajax in related files to display status indicator warning for files with obsolete & retracted statuses and available notes
- Add COLO829 Truth Set Tab to benchmarking page
- Add a way to change facets and columns from BenchmarkingDataMap
PR264: SN remove tissue from cell culture * Remove tissue link from CellCulture as part of switch to TissueSample link
PR261: Bring QC information to the Submission Status page * New version of Submission Status page
PR256: SN Write automated example
- In commands/write_submission_spreadsheets.py:
- Add argument --example that will pull metadata from a template uuid to fill out an example submission spreadsheet for the current schema
PR257: SN Update DSA calc_prop
- Change DSA calc prop for non-fasta files supplementary_files
PR238: SN Add valid_molecules required
- Add valid_molecules as a required property for Assay
PR 252: feat: add manifest type to button
- Add manifest type to download manifest button
PR247: feat: add downsampled tab
- Add downsampled hapmap tab
- Update benchmarking descriptions
- Adds data table view
- Links to data table view to related tracking-item
- Add to file_group calc_prop to accommodate FileSets with multiple samples
- If samples are homogenate tissue samples, sample_source is the Tissue
- If samples are cell culture samples, sample_source is the CellCulture
- If samples are intact tissue samples, sample_source is TissueSample (not mergeable)
- Add property on_target_rate to Sequencing that is included in the anyOf with target_coverage and target_read_count
- Update links to go to newest documentation
- Add a tip with a warning about submissions
- Add "Resources" to navigation
- Add "Genome Reference & Related Data" page
- Add property tpc_submitted for whether or not the donor was submitted by the TPC
- Include a maximum value of 90 for age to remove identifiable information
PR 240: fix: SN TPC samples annotated_filenames
- In commands/create_annotated_filename.py:
- Grab unique aliquot_ids so that files with tissue samples from TPC are not considered to have merged aliquots e.g. file has identical samples ST001-1A-001A1 from GCC and ST001-1A-001A1 from TPC. The TPC item is used for metadata cross-checking, and shouldn't be considered when checking if samples were merged for analyte extraction.
PR249: SN Revert FileSet calcprop
- Revert PR 244, removing files_status_retracted calc_prop from FileSet
- Effectively disable server-side validators which reference linked objects for smaht-submitr, when the skip_links=True.
- Create calcprop for FileSet, files_status_retracted, that returns "True" if a file in files has the status of obsolete or retracted
- Embed file_set.files_status_retracted in File
- Allow CellLine to link to itself with parent_cell_lines
- Allow CellLine to link to TissueSample (Sample) with tissue_samples
- Allow CellCulture to link to TissueSample. Keeping Tissue link for now, to remove later.
- Add a calculated property to CellLine, source_donor, that gets the donor information, if present, from donor, tissue_samples, or parent_cell_lines
PR 245: SN Make antibody an array * In Library, make antibody and array of strings * Small fix to submission template delimiter description
- Add operator is_type to QC thresholds schema
PR241: SN Add properties to library
- Add properties dna_target, guide_sequence, and antibody to Library schema
- Change property restriction_enzymes to enzymes in LibraryPreparation (recently added property so it is empty in the portal) to apply for transposase enzymes as well and update associated tests.
PR239: Add mei_detection_challenge_data enum ot dataset
- Add mei_detection_challenge_data enum to dataset
- Update manifest to prioritize filename over display_title
PR222: feat: add dataset target coverage row * Add dataset target coverage row to File Overview's Data Information card
PR 232: SN RE property * Add a property restriction_enzymes to LibraryPreparation * Add columns to SubmissionCenter search page for identifier and code * Add description property to VariantCalls
- Add new section to Data nav for challenge results
- Add COLO829 SNV/Indel V1 Challenge Page + tables
`PR 219: SN Metadata Custom Validators<https://github.com/smaht-dac/smaht-portal/pull/219>`_
- Add custom validators for metadata audit checks with tests for POST and PATCH: * Update assay.json`` schema to include properties valid_molecules and valid_sequencers to assist in validation. Will need to patch current assay items for this to function * In FileSet, check that the combination of libraries.analytes.assay and sequencing.sequencer are compatible * In Library, check that assay-specific properties are compatible with the linked assay (e.g. bulk_mas_iso_seq and target_monomer_size) * In Library and Analyte , check that molecule is compatible with molecule-specific properties * In UnalignedReads, check that paired_with is only present for R2 files and that the linked file is R1. Also check the the R1 and R2 files are linked to the same FileSet * Add more specific error messages to help with submission
PR 230: SN Update annotated_filenames
- In commands/create_annotated_filenames.py: * For annotated filenames, update aliquot_id to be [aliquot_id]MC if multiple benchmarking or production tissue samples from the same tissue aliquot but multiple cores (e.g. ST001-1A-001A1 and ST001-1A-001B2) and` MAMC` if from multiple tissue samples from different tissue aliquots * Remove the variant type from the end of annotated filenames for vcfs
- In commands/write_submission_spreadsheets.py * Add notes for properties that allow commas for large numbers and allow abbreviations (e.g. 100M or 3.1 Gb)
- In schemas/file.json * Add a version property for front-end
- Update to file_group to evaluate homogenate samples as if they are cell lines (using sample_source)
PR 225: SN Improvements to automated submission spreadsheet
- In write-submission-spreadsheet: * clean up args messages * Add functionality for` --item` working with --google * Adjust comment descriptions to clarify | delimiter and add search links for submitted items
- In schemas/: * Include schema description for GCC-submitted items
PR 229: SN Cell Culture upgrade #229
- Change CellCulture.cell_line property to be an array of strings linking to CellLine, rather than a string. * Adds an upgrader with test for cell_culture
- Updates file_group calcprop to properly resolve analyte.samples only when computing the sample_source_part
PR226: SN Add tissue link to cell_culture * Add parent_samples link in cell_culture_sample.json * Add tissue link in cell_culture.json * Make anyOf for requiring cell_line and/or tissue
PR 207: feat: about page visualization updates
- Reorganize Awardees Visualizations
- Provide links to individual PI's labs
- Reformat/reword copy
- Added validators endpoint to validate (initially) submitted_id for smaht-submitr.
- Added allow_multiplier_suffix and allow_commas properties (both true) to these schemas properties: - Sequencing.target_read_length - DonorSpecificAssembly.contig_n50 - DonorSpecificAssembly.genome_size - DonorSpecificAssembly.largest_contig_size - DonorSpecificAssembly.scaffold_n50 - DonorSpecificAssembly.total_ungapped_length
PR 209: Statistics Y-Axis Scale
- Usage stats: Add y-axis linear/pow/log scale options
- Usage stats: Split file downlods into two sections as file downloads and top file downloads (/w top 10/25/50/100 options) for clarity
- Usage stats: Exclude index file downloads (external_files)
- Submission stats: Fix custom date range filtering bug
- Misc: Improve charts and filters rendering in small and mid-size devices
- Misc: Improves warning messages displayed when the charts fail to render
PR216: Add ExternalQualityMetric
- Add ExternalQualityMetric item type which is a submitted item inheriting some properties from pre-existing QualityMetric
- Create new linking property external_quality_metrics in file.json
- Add new properties to variant_calls.json schema: comparator, external_databases, and filtering_methods
PR221: Include target_coverage in file data_generation_summary
- Embed file_sets.sequencing.target_coverage` in file.py and adds "Target Group Coverage" to data_generation_summary` used in the File Overview Page
PR 220: fix: move modal styles into search.scss
- fix: styling issue in File Overview metadata download modal
- Move modal styles into _search.scss file
PR 218: fix: typerror in ItemDetailList's ShouldUseTable
- fix: typerror, support number value being first
PR 213: Bm homepage drawer update
- Change defaultActiveKey and activeKey settings to get different home page drawer-tiers to stay open once selected
- Reorganize CLI commands components
- Utilize bootstrap Tabs component
- Styling updates for the command container
- Disable tagging and commenting functionality for non-admins on Submission Status page
PR212: Add functionality for arrays of objects
- Updates write_submission_spreadsheets to write out columns for arrays of objects
- Currently relevant for CellCultureMixture and the components property which has two nested properties, ratio and cull_culture
- Changes to DonorSpecificAssembly and SupplementaryFile schemas to reflect GCC feedback: * BUSCO score properties created for each haplotype (e.g. percent_single_copy to percent_single_copy_hap1 and percent_single_copy_hap2) * Change property name from percent_duplicate to percent_multi_copy for clarity * Change property name from largest_contig to largest_contig_size for clartiy * Change enums for data_type and data_category specific to SupplementaryFile fasta and chain files * Fix descriptions of some properties
- Remove Doug user
PR 195: fix: tablet navigation collapse
- Fix collapsing navigation bar for tablet size screens
- Switch to mobile navigation component early
- Show initials for username in tablet screen size
- Fix mobile styling bugs
- Minor documentation updates for how to install awscli in docs/source/{file_download,download_cli}.rst.
- Update to dcicutils 8.13.3.
- Bug fix: consider loadxl order in staggered reindexing
- Add
retractedstatus to Files with submission center only view
PR 188: feat: increase benchmarking table visibility
- Increase height of benchmarking tables
- Support toggling the benchmarking page descriptions
- Upgrade SPC to v0.1.86
PR 205: Update submission schema from TPC and GCC feedback
- Adds two arguments to write-submission-spreadsheets that create submission_workbook templates with a subset of ordered tabs consistent with the submission templates shared with TPCs and GCCs/TDDs. * --tpc outputs tabs for the TPC submission * --gcc outputs tabs for the GCC/TDD submissions
PR 206: SN Remove tissue recovery_interval
- Removes recovery_interval property from Tissue schema
- Includes upgrader for tissue schema from version 2 to 3 with test
- All existing recovery_interval values for Tissue items have already been transferred to TissueCollection items linked to Donor
PR 204: Update submission schema from TPC and GCC feedback
- For MedicalHistory, change enum from "Not done" to "Not Done" for consistency
PR 198: Bm accessibility font fix
- Convert pixel font sizes to use rem
PR 189: Update submission schema from TPC and GCC feedback
- Create DonorSpecificAssembly item type that contains information and QC metrics, and links to FileSet and Software used to generate the assembly.
- Create a submittable SupplementaryFile that can contain files as reference fasta and chain files. These can link to DSA.
- Add notes_to_tsv property to file.json schema for including static banners for notes on file pages.
- Add test for DonorSpecificAssembly rev_links
- Follow-up changes to the TPC submission schema:
- Add preservation_type and preservation_medium properties to Tissue.
- Make preservation_type an enum for Tissue and Sample
- Make core_size a string enum
- Add size_unit to Tissue to account for tissues measured in cm^2 instead of cm^3
- Update
/homepageto include counts for benchmarking tissues - Update
file_groupcalcprop to account for tissue data
- Documentation for smaht-submitr binary (pyinstaller-based) installation.
- Updates related to Python 3.12.
- New (and commented out by default) elasticsearch.server.actual_port property in base.ini to facilitate running a local ElasticSearch proxy to observe traffic (resquests/responses) between the portal and ElasticSearch with a tool like mitmproxy or mitmweb; see comments in base.ini; and see in snovault/dev_server for where this is handled.
- Quick fix to hide validation errors facet
- Update /homepage to include counts for benchmarking tissues
PR 196: Bm primary tissue fixes
- Hide unexpected facets
- Fix tab title counts
- Fix for home page link to primary/benchmarking tissue page
- Fix access_status enum options case in item_utils
PR 190: Bm primary tissue ui 2
- Add brain table for donors 3 & 4
- Remaining update to use dataset=tissue
PR 192: fix: include missing TPC and GCC in awardees table
- Fix missing consortium entries in the awardees table
- Add /routes endpoint to return all available routes
PR 187: Update file naming and release commands
- Refactor and improve commands release_file and create_annotated_filenames * Add test coverage, especially for annotated filenames * Add to item_utils and use extensively in commands * Fix command names in pyproject.toml * Update commands for tissue data
- Add tissue to dataset enum for benchmarking files + update benchmarking searches
PR 184: Update submission schema from TPC and GCC feedback
- Remove SamplePreparation item and merges the property homogenization_method with AnalytePreparation
- Add properties to Analyte based on GCC Benchmarking Extraction input
- Change to properties of Library based on GCC feedback
- Include more detailed descriptions and examples of some fields that are technology-specific (i.e. Kinnex)
- Minor change to TCC submission schema related to units and changing enums to suggested enums
- Update documentation for setting up Google Sheet API credentials for automated submission templates
- Enable benchmarking tables for Donor 1 and Donor 2 primary tissues
- Fix bug with Donor 2 data table
- Move "credentials for submission" to Access and rename it to "access key generation"
- Set up an additional re-direct, update old redirect
PR 182: Bm nomenclature fix jun2024
- Update PDF with brain and change version
- Update RST file with brain
- Adds a new API /download_cli that accepts a resource path as a URL or POST param and returns federation token for use with awscli
- Updates /metadata to allow download_cli links
PR 181: Submission and Usage Statistics Follow-up Updates
- Filtering * Previous 60 days added to date range options * Cumulative sum checkbox added to switch between individual bars and cumulative sum
- File downloads * File downloads splitted into two charts (count and volume) * Assay Type and Dataset views added
- File views * Assay Type and Dataset views added
- Page Views * Page views, Uniqe users by country/city views added * Page title and URL views added
- Schema and Style * tracking_item` schema fields that are still mapped to Universal Analytics naming convention are renamed/replaced * glitches in mobile view fixed
- Add new command write-submission-spreadsheets to generate submission spreadsheets (Excel or Google sheets) for submittable items
- Schema updates * Update descriptions for many properties with standardized units formatting * Misc. updates to TPC-related properties * Breaking change: Tissue location renamed to anatomical_location; upgrader included
- Fix minor issue on Submission Status page
Submission Status page updates: * Add filters for CellLine and CellCultureMixture * Some refactoring of the React component * Color filesets by file group
- Added src/encoded/tests/data/demo_inserts with (well) demo inserts for objects needed by the demo metadata spreadsheet (for annual meeting 2024): bcm_formatted_hapmapmix.xlsx ... https://docs.google.com/spreadsheets/d/1qCm0bY-vG4a9uiaOvmKHZ12MvhmMKKRfEpgAm-7Hsh4/edit#gid=1472887809 FYI: To cause these to be loaded at startup when running locally, edit development.ini and set load_test_data = snovault.loadxl:load_local_data (rather than the default load_prod_data).
- Updated dcicutils to 8.10.0 (mostly merge support in structured_data).
- Changed workbook-inserts/assay.json bulk_wgs item to code 002 (to match data/staging/wolf).
- Changed workbook-inserts/sequencer.json code from A to X (interfering with demo testing).
- Support merge in ingester for partial object updates from metedata. - Removed ref_lookup_strategy references for structured_data; refactored/internalized in dcicutils.
- Added rclone (Google-to-AWS) related documentation.
- Some make lint fixups.
- Ensure
docutilsis a true dependency
- Add new command
check-insert-consistencyto quickly detect errors on live environments related to inconsistencies withmaster-inserts - Add new command
load-data-from-localto allow interactive updates frommaster-inserts
- Add File Overview Page for improved view of additional details for File items
- Upgrade SPC to v0.1.85
PR 179: Bring in inserts command fix
- Bring in snovault with fix for help strings for command to update inserts
PR 178: Clean up poetry commands
- Clean up commands in pyproject.toml * Remove commands not present in repo * Use snovault commands where possible * Reorganize commands by source repo
PR 175: Fix command to load inserts
- Update snovault and use its updated command to load inserts
- Submission statistics charts for metadata submitted, data submitted and data released files with various group by options
- Add benchmarking tables for primary tissue data, organized by Donor, then Tissue type
- Update navbar and navbar links
PR 173: Multiple analytes per library
- Breaking change: Remove analyte from Library and replace with analytes array of linkTos
- Includes corresponding changes to item_utils, commands, calcprops, embeds, front-end, and tests
- Add a bunch of new inserts to serve as the new links to these various pages
- Rework the old inserts into redirect-only pages, so that people who have bookmarked old links don't lose their place
- Permission and order tweaks for sanity's sake
PR 141: Link FileSets to Samples
- Add samples linkTo to FileSet to track related samples for single-cell data
- Update item_utils to handle new linkTo (+ misc. cleanup)
- Added Sarah Nicholson to user.json master-inserts
- Update file columns and facets for benchmarking tables
- Add released_date to file calcprop for display in benchmarking tables
- Update Donor, Tissue, and TissueSample columns and facets for searches
- Add Donor and Tissue links from doc page
- Load search tables on tab open (instead of all at once on initial page load)
- Cleanup of context and fix for select all button and checkbox bugs
- Enable SPC fix of selected items clearing on filter by passing props to EmbeddedSearchView (spc v0.1.82b1)
- Extend release script to handle obsolete files
- 2024-05-08/dmichaels/PR-163
- Added option to ingester to skip validation on submit. With smaht-submitr/submit-metadata-bundle --validate-remote-skip flag; this flag previously only prevented server-side validation from being kicked off by submitr, but on actual submit server-side validation was still being done before loadxl; this flag now prevents that as well. This is hooked on a (submission folio) validate_skip flag; and this is ONLY allowed for admin users; if non-admin validate_skip forced to False.
PR 162: Broaden derived_from link
- Update derived_from linkTo from SubmittedFile to File
- Change File unique_key to submitted_id to allow finding SubmittedFiles by submitted_id
- Add accession to default lookup field for all collections, i.e. /<collection>/<accession>/ will work for all collections with accessions
PR 165: Update preparation items
- Move common preparation links to parent item and share props with all children
- Remove stale properties from preparation items
- Update workbook inserts to reflect changes
PR 164: Add Tissue code for table search
- Add code property to Tissue to be used in benchmarking table search
PR 160: Add N50 prop + in silico data type
- Add N50 property to UnalignedReads and AlignedReads
- Add in silico enum to File data_type
PR 161: feat: file download doc page
- Add file download doc page
PR 159: feat: colo829 benchmarking page banner
- Add callout banner for colo829 dataset benchmarking pages
- Add paired_with property to OutputFiles
- Add in silico mix tab to COLO829 benchmarking
- Add calcprop file_merge_group as a tag on file sets to help determine which file sets contain files that are candidates for merging
- Add additional fields to manifest files
- Documentation on manifest files
- Documentation on data release via status
- Adjust access key expiration down to 30 days
PR 151: fix: HapMap description formatting / Access key button text update
- Fix formatting issue with HapMap description
- Remove empty div element when BamQCLink not provided
- Update the access key button text
- Fix for front page links and update to tab titles for iPSC
PR 153: New dataset for in silico Colo829 mixes
- Add new option for in silico Colo829 mixtures to File dataset enum
PR 129: Categories for RNASeq pipelines
- Add category enums for RNASeq pipelines to MWF, Workflow, Software, and File
- Create new item (Basecalling) to track basecalling details
- Add link from Sequencing to Basecalling
PR 149: File overview sample summary
- Complete sample summary calcprop for file overview page
- Refactor utils modules to enable test app integration
- Fix bugs + type hints and add more functionality to item_utils modules
- Improvements to Status Submission page (refactoring + new filters)
- Improvements to SMaHT Nomenclature Page
- Update SMaHT Nomenclature PDF document
- Add properties to capture flow cell lane data on UnalignedReads and AlignedReads
- Add tagging and direct fileset search to Submission Status page
PR 145: ReferenceGenome build info
- Add properties to capture build information for ReferenceGenome
- Add URL property to ReferenceFile to track source of files
PR 144: Add parent samples to TissueSample
- Add parent_samples to TissueSample to track related samples
- Add scripts to release files and create annotated filenames
PR 138: feat: total file counts for benchmarking page tabs
- Fetch total number of files and render as a badge on benchmarking tabs (next to the title)
- Improve Submission Status page styling
PR 134: Updates for TPC metadata
- Reorganize donor-related items and fields following discussions/feedback from the TPC * Add new item types: FamilyHistory and MedicalTreatment * Delete unused item types: Therapeutic and MolecularTest * Removal, addition, and renaming of many properties
- Add external_id mixin to multiple SubmittedItem types for tracking identifiers from submitters
- Improve Submission Status page styling
- Add Submission Status page
- Add various embeddings to FileSet
- Add
file_status_trackingcalc prop to File
- Dumb typo/mistake in ingestion.loadxl_extensions (portal.is_file_schema -> is_schema_file_type)
- Added expiration for access-keys on user profile page.
- New version of dcicutils with minor fix in structured_data for smaht-submitr progress monitoring.
- Added support to get the version of the latest smaht-submitr Google Sheets metadata template; added endpoint /submitr_metadata_template_info (defined ingestion.metadata_template); added GOOGLE_API_KEY in development.ini.template, smaht_any_alpha.ini, and dcicutils.deployment_utils.
PR 131: feat: Benchmarking Page navigation toggle functionality
- Enable toggle for benchmarking page navigation
- Fix file embeds by removing sub-type LinkTos from Samples to SampleSources and deletion of unused embed
PR 127: File Overview Calcprops + Item Utils
- Add calcprops for file overview page
- Add calcprops for associated items on files
- Add item_utils module for common item functions
- 2024-03-25
- Changes to support tracking ingestion progess for smaht-submitr (via Redis).
- Minor documentation tweaks.
PR 139: feat: format latest release date string
- Format latest release date
- Fix sex not appearing on donor (and other) detail pages
- Dynamic homepage count support
- Isolate workbook and non-workbook tests
- Add facets to HapMap and iPSC
- Fix HapMap and iPSC links
2024-03-14: This is a temporary branch (extra_files_plus_main_20240314) which is Will's extra_files branch with main merged in (2024-03-14), and also Utku's utk_es_max_hit branch (PR-114) with documentation changes merge in. FYI: Branch utk_es_max_hit was merged into main 2024-03-19.
Added missing import of calculated_property from snovault to types/submitted_file.py.
Documentation changes.
Changes to support "resuming" smaht-submitr submission after a server validation "submission" timed out while waiting (via submit-metadata-bundle).
In this (server validation timeout) case the user can then run check-submission with the UUID for the validation submission, and if/when it is complete and successful, the user will be allowed to continue on to do the actual submission. Slightly tricky because the metadata file was uploaded (to S3) as a part of the validation submission, and/but when check-submission is run we don't want the user to have to specify this file again, partly because it is an odd user experience, but mostly because when we do the actual submission we want to make sure we use the EXACT file that was validated; and so to do this we grab the file from where it was uploaded as part of the validation submission (i.e. under an S3 key with the validation UUID) and copy it over to where it would normally be (i.e. under an S3 key with the submission UUID); and from there things continue as normal. Note also that both of the IngestionSubmission objects have a pointer to the other; i.e. the validation submission object has "submission_uuid" and the actual submission object has a "validation_uuid" (in the "parameters"); this hookup is done by the smaht-submitr code.
The "resuming" scare-quotes are because this is not really resuming a submission but rather resuming the process the submit-metadata-bundle was doing, i.e. where it does a server validation then then, if successful and okay with the user, it continues on to do the actual submission. The "submission" scare-quotes for the server validation is because this is a submission in the sense that an IngestionSubmission object is created, but not an actual submission because it is a validate_only submission.
Added display_title calculated property to IngestionSubmission to display either Validation or Submission depending on validate_only (followed by colon and uuid).
PR 114: ES total count, HealthView and rst content updates
- Adds UI updates implemented in #114
- Ports HealthView page fixes previously implemented in 4DN for package-lock.json v3
- Adds admonition support for rst-to-html conversion in static content
- Add upgraders for Sequencing and CellCulture for properties changed or removed in prior PRs
- Updating ethnicity/nationality descriptions for HapMap
- Breaking scheme change: Assay link removed from file set and required on library
- Create FileSet upgrader to remove assay link for schema version 1 items
- Update loadxl order to account for new assay linkTo location
- Add common_fields to MetaWorkflowRun and sequencing_center to WorkflowRun and QualityMetric
- Add new option to Software category enum
- Removed master-inserts/file_format.json.
- Changes for new skip_links (snovault.loadxl) mode for smaht-submitr.
- Added submits_for to master-inserts/users.json.
- Documentation related to smaht-submitr updates.
- FYI: For the record some merging complications (2024-03-09 ~ 15:35) ... Merged in some changes from commit c67d442e for __init__.py and server_defaults.py as issues with make deploy1a load errors related to user_submission_centers, after merging in from main, which had issues with make deploy1b WRT circular dependencies like ImportError: cannot import name 'test_accession' from partially initialized module 'snovault.server_defaults' (most likely due to a circular import).
- Updates nginx version to latest as of 03/13/2024 to resolve security alerts
- Add assay linkTo to library in preparation for future removal from current location on file set
- Add anyOf requirement on sequencing for coverage or read count
- Update select item properties to match those desired for "automated" submission
- Only documentation updates (related to smaht-submitr) from add_valid_item_types_to_fileformat_in_masterinserts branch.
- Permissions update: support for
submits_forandrestrictedstatus
PR 82: UI Dev 5 - March 1st Benchmarking Release
- Feedback and bugfixes from v1 release (see trello for full list: https://trello.com/c/2TSRUHWT/880-feedback-from-feb-1-release)
- v2 benchmarking with support for cell line pages
- Updated/improved alluvial, etc. visualizations
- New documentation page for SMaHT nomenclature PDF download
- Merged PRs: - #101 - #85 - #97 - #103 - #86
- Update small sections of documentation for referencing existing items
PR 105: GA4 file sequencing center updates
- Replaces submission center with sequencing center in file views/downloads GA4 analytics
PR 104: Automated submission alignment
- Update select property names to align with those in the 'manual' submission template
- Remove select properties suggested by feedback from submitters
- Create defaults and remove requirements for properties that currently only have one enum value
- Fix UUIDs in master-inserts to match items in the database
- Add documentation page for finding data relevant to submissions
- Add templates for submissions
- Improve columns + facets for searching on collections
- Add calcprops for searching on collections
- Fix file upgrader for handling additional enum values
- Add upgrader for MetaWorkflow custom_pf_fields
- Updated robots.txt to allow search engines, disallow known bots, and block /ingestion_status & /*-files downloads
- Update SEO utilities to reflect SMaHT-specific text & branding (will need further adjustments in future, most likely)
- Add small square SMaHT logo for use in search engines
- Remove unnecessary file_format.json & insert from master-inserts
- Remove duplicate code key from smaht-dac submission center in master-inserts
PR 92: Schema updates for submission
- Add new properties suggested by previous submitters
- Add fields to MetaWorkflow custom_pf_fields to bring in metadata from pipelines to files required for release
- Breaking property requirement changes with upgraders to clean up schema changes from benchmarking data model release
PR 95: Tracking Item and Misc. Google Analytics Updates
- Fixed a bug that prevents collecting submission center and file type dimensions in file views
- Fixed the incorrect links in top files download statistics tooltip
- Adds tracking_item py test
- Removes/Replaces legacy 4DN-specific reports and styles
- Documentation updates related to submission.
- Test fixes for test_structured_data related to date/time type handling.
- Update for smaht-submitr to suppress reference (linkTo) errors on validate_only, iff not reference errors according to structured_data; has dependent changes in snovault loadxl and schema_validation (version >= 11.11.0.1b2).
- Update for smaht-submitr to support --validate-first option, which means we do a validate_only check first before loading the data (both via loadxl).
- Schema additions to facilitate automation
- Hooks in QC Download API
PR 84: More Benchmarking Data Sets
- Update enums File dataset to include all expected cell line benchmarking data sets
PR 57: Submitter ID Validation
- Validate submitter_id for all submitted items * Validation includes: SubmissionCenter code, item type, and unique identifier
- Repair test namespacing in unit tests
- Adds last_modified to all items for edit tracking
PR 74: Table of Content improvements for RST content
- user_content.py is updated to support multi-level TOC generation for RST content
- Static_section.json in master-inserts is updated to correct text and navigation URL in Next - Previous links under the TOC
- Level 1 titles are disabled under Documents in top navigation bar
- Serkan Utku Öztürk added to users
- Update derived_from linkTo from File to SubmittedFile
- Add lifecycle properties to File schema
- Update SPC from 0.1.76b1 to 0.1.76
PR 45: UI Dev 4: End of January Release w/Benchmarking
- UIs for Benchmarking Data
- Google Analytics implementation
- Navigation edits to accommodate new documentation, about, data pages
- Various other front-end tweaks to home page, user pages, etc
- Note: many PRs included in this one - see link above for breakdown, commit history
PR 63: Benchmarking release data model
- Add two new item types: Sequencer + Assay
- Add code property to multiple item types to store file naming conventions
- Share SubmittedFile release properties with OutputFile
- Update file facets + columns and embed fields for search
- Repair small bug in extra file line generation, implement/test field fallbacks
- Repair permissions issues coming from snovault
- Update extra file names in encoded-core
- Adds a command to load users from a master OC spreadsheet (not tracked in git)
- Version updates to dcicutils, dcicsnovault, encoded-core. Changes to itemize SMaHT submission ingestion create/update/diff situation and deletes.
- Removed Rahi from admin group for submission ingestion testing purposes (master-inserts/user.json).
- Adds /peak-metadata support for retrieving facet information from the metadata.tsv
- Added gitinfo.json to buildspec.yml to make available to the app basic git info (branch/commit).
- Updated test_structured_data.py to (optionally - default for now) insulate itself from change to the schemas while they are undergoing a lot of modification, leading to frequent/annoying test breakage.
- Fixed load(xl) error handling for ingestion submission to report back to submitr properly.
- Adds /metadata support
PR 58: Culture mixture parent types
- Include CellCulture as parent item of CellCultureMixture for resolving reference during submissions
PR 56: Implement submittable item API
- Add functionality and tests for submittable item api to smaht portal
- update lockfile with latest snovault that contains the primitive for this
PR 50: Upgrader implementation
- Add upgrader functionality from encoded-core + tests
- Update dcicutils with schema_utils module
- Add portal side validation for QC rulesets
- Fix in ingestion.loadxl_extensions.load_data_into_database to handle errors correctly.
- Added record (uw_gcc) to master-inserts/submission_center.json for testing.
- Added test_structured_data.py back in after resolved GitHub Actions (only) failure.
PR 47: Fix admin affiliation validation
- Fix and test item affiliation validation for admins
- Allow any non-object JSON type for QC values instead of just strings
- Final adjustments to documentation, namely help desk email references
- Rework broken editable fields on User Page
- Add consortia and submission centers to User Page
- Broaden software version pattern pending further discussion with bioinformatics.
- Add some adapted user facing documentation
- Merge in ui-dev3 branch: PR: 39: UI-Dev 3 * [Cfm homepage updates] (#39) * Additional UI changes for V1 pre-release, including updates to user page, registration modal, nav, etc.
- SMaHT ingestion related work.
- Add first cut at "submittable" data model
- Fix default collection ACLs
- Add inserts for all item types to workbook-inserts
- Increase test coverage for schemas, types, item creation permissions, and various calcprops
- Remove redundant tests now covered by workbook inserts or elsewhere
- Update encoded-core with fix to extra_files property on File items
- Refactor schemas for increased sharing and less duplication
- Update bioinformatics-related schemas based on feedback + testing
- Update identifying properties in schemas for loadxl handling
- Clean up unique keys for item types
- Improve test coverage for schemas and types
- Fix statuses in inserts
- Fix ingester permissions
- Expanded permissions implementation
- Remove statuses no longer in use
- Refine the state each status corresponds to
- Update and add additional tests for this functionality
- Repair various schema and core data model issues associated with bioinformatics processing
- Merge in ui-dev2 branch: PR 28: UI-Dev 2 * Fixes for UI broken by data model update (including SPC update to [v0.1.73b1] (https://github.com/4dn-dcic/shared-portal-components/releases/tag/0.1.73b1)) * [Cfm map popover #27] (#27)
- Merge in drr_shared_schemas branch.
- Merge in ui-dev branch: PR 19: UI-Dev * Remove Splash Page + [re-add various features](#19) * Series of Fixes for Static Sections and Pages * [Bm homepage and more inserts #22] (#22) * [Bm cypress + studio #16] (#16) * [Cfm data viz integration #23] (#23) * [Cfm homepage figure #25] (#25)
- Update dcicutils to ^8.2.0.
- 2023-11.02
- Upgrade to Python 3.11.
- Adding ingestion processor. * Added/implemented ingestion_processor.py. * Added generate-local-access-key script (from snovault) to pyproject.toml. * Added view-local-object script (from snovault) to pyproject.toml. * Changed metadata_bundles_bucket to smaht-production-application-metadata-bundles in development.ini.template.
- Removed types/access_key.py and schemas/access_key.json as the ones in snovault are sufficient.
- What it says on the tin: ran npm run format to reformat JS/JSX files
PR 17: Webpack 5 Config Fixes + Prettier Install & Config
- Remove direct Terser-Webpack-Plugin from dev-dependencies
- Fix NODE_ENV warning on webpack build
- Add some comments for profiling webpack easily
- Also included changes from: #15
- Update Docker's MakeFile to use Node version 18.17.0
- Update SPC to 0.1.69
- Fix for search view error
- Fix for auth0 bug when SPC symlinking
- Hook in and test DRS implementation
- Lock newer snovault, utils versions with bug fixes
- Made scripts/psql-start.bash and bin/macpoetry-install.bash executable.
- Makefile invokes macbuild rather than build if this looks like Mac (uname -s contains Darwin).
- Corrected Makefile to refer to psql-start.bash and macpoetry-install.bash (with the .bash suffix).
- Pinned PyYAML version in pyproject.toml to 5.3.1 (as Mac M1 really wants this one, not 5.4.1).
- Fix to src/encoded/__init__.py for allowedConnections in /auth0_config endpoint.
- Removes
jsonschema_serialize_fork, use new schema draft version - Implement
$mergereferential schema fields
- Improve testing by porting relevant tests as needed
- Changes to accommodate working search/other tests
- Allow testing with ES in GA with smaht-development credentials
- Build Docker as part of GA
- Implementation of SMaHT splash page UI as temporary placeholder
- Some additional clean up of front end described in more detail here: #5
- Update to use Webpack 5
- Do some light adjustments to be more compatible with Google Analytics 4 down the line
- Implements various changes across repos to allow deployment of the smaht-portal
- Implement base level permissioning scheme working with
consortiaandsubmission_center
- Initial version
- TODO: update base.ini, various other ini file templates once new AWS is provisioned
- TOOD: build GLOBAL_ENV_BUCKET for testing (conftest.py)