Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix time data types in contexts getting reformatted #88

Merged
merged 51 commits into from
Oct 12, 2023
Merged

Fix time data types in contexts getting reformatted #88

merged 51 commits into from
Oct 12, 2023

Conversation

yangsophieee
Copy link
Collaborator

Addresses #49

  • Time data types are getting reformatted with read_csv, because it detects the column data type when reading in, such that e.g. "9:00:00" becomes "09:00:00". This is not too much of an issue if it's consistent with the output of metadata_add_contexts and process.R. However, metadata_add_contexts was rewriting "09:00:00" as total number of seconds: "32400.0"
  • I've fixed it so that context values are character type so that as.yaml in write_metadata does not convert times to number of seconds
  • See this comment about the rewriting from "9:00:00" to "09:00:00" (I've added a note to the user when time data types get reformatted)
  • Also added a fix for some of issue [traits.build workflow] Add "time" category of traits, in addition to "categorical" and "numeric"? #87, where substitutions for time variables (e.g. flowering_time, fruiting_time) fail dataset_test

ehwenk and others added 30 commits September 16, 2023 19:28
@ehwenk
Add repeat_measurements_id
587c7d2 
* Gets added to metadata files, where the default is set to false (otherwise this will mask the need for contexts to pivot)
* Not yet tested in process.R- waiting to merge in other branches first
@ehwenk
Merge branch 'develop' into Add-repeat_measurements_id
5aecfd8
@ehwenk
bit more work
668ce6a 
- repeat_measurements_id is now a column of NA's in austraits$traits
- however, don't have it working to be triggered at either the dataset or trait level
- in schema
- at this point need to change line ` i <- !is.na(data$value) & !is.na(data$repeat_measurements_id)` (or elsewhere) to have it recognise when `repeat_measurements_id: TRUE` (or `y`) is added to either metadata$dataset or specific traits in metadata
@yangsophieee
Update comments a bit and fix pivot_wider test
b8a42c9
@yangsophieee
Add new test dataset
d1c56fc
@yangsophieee
Add some dummy traits for testing curve data
03c8187
@yangsophieee
Add test dataset 7
c68e452
@yangsophieee
Implement repeat_measurements_id for the trait level
cd7c9f7 
- Set `repeat_measurements_id` to TRUE in the metadata
@yangsophieee
Merge branch 'develop' into Add-repeat_measurements_id
c6713c2
@yangsophieee
Update test dataset keys
b62eb6e
@yangsophieee
Update process.R
0b3d4cf
@yangsophieee
Update test dataset READMEs and pivot_wider test
21b2873
@yangsophieee
Fix variables that are used to make observation_id and `repeat_meas…
5069ab2 
…urements_id`
@yangsophieee
Reformat
ea5c1fa
@yangsophieee
Fix austraits_curators and context categories
7a2c84b
@yangsophieee
Merge branch 'develop' into Add-repeat_measurements_id
89cc22c
@yangsophieee
Update metadata and traits.yml
1cfafb5
@yangsophieee
Add new long dataset for testing response curve data
64cbc04
@yangsophieee
Add a bit more work
896b085
@yangsophieee
Add long format and wide format testing for repeat_measurements_id
a0cbdec
@yangsophieee
Allow repeat_measurements_id to be specified as TRUE and FALSE when…
5af2872 
… a trait is entered twice
@yangsophieee
Update comments
dd4d1d7
@yangsophieee
Added more tests
a59ccfe
@yangsophieee
Added repeat_measurements_id option to metadata_create_template
dccc0c4
@yangsophieee
Update test expected outputs
227b42d
@yangsophieee
Fix bug
a47e463
@yangsophieee
Add expected outputs for Test_2023_7 and Test_2023_8
0ac9b60
@yangsophieee
Change col_types argument to read in columns as character type by d…
e7b1951 
…efault
@yangsophieee
Update documentation and remove warnings/notes from R CMD Check
18502a1
@yangsophieee
Add collection_date to traits metadata schema
bd70b9e
@yangsophieee
Add Test Dataset 3 to testing
080fcd5
@yangsophieee
Update READMEs
523ba96
@yangsophieee
Update READMEs
e877806
@yangsophieee
Update data and metadata files
2345b0a
@yangsophieee
Update READMEs
a545072
@yangsophieee
Update some output files
33a945f
@yangsophieee
Update Test_2023_1 output
9ff8b21
@yangsophieee
Update output files
42f897f
@yangsophieee
Add expected outputs for Test_2023_3 and update process.R
413ae6b
@yangsophieee
Regenerate output
6b2af63
@yangsophieee
Move processing of traits to after where metadata fields are filled i…
ea0d585 
…n from locations

Otherwise population_id, observation_id, etc. are made before `entity_type` and other metadata fields are finalised
@yangsophieee
Remove browser
c3eb5f6
@yangsophieee
Fix output IDs
0e98e8e
@yangsophieee
Add test for substituting a time value
3be60db
@yangsophieee
Update context categories for metadata_add_contexts and convert con…
45cbcff 
…text values to character type

- The latter prevents time data types from converting weirdly when converted to yaml
@yangsophieee
Add expected output for testing substituting time value
53d56ef
@yangsophieee
Remove "time" values from dataset_test checks of allowable categori…
c9984ae 
…cal values
@yangsophieee
Add a note when time data types get reformatted
1f75e0f
@yangsophieee
Copy link
Collaborator Author

@dfalster This is ready for review :) Thank you!

@yangsophieee yangsophieee merged commit 96b6abc into develop Oct 12, 2023
3 checks passed
@yangsophieee yangsophieee deleted the time_data_types branch October 12, 2023 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dfalster dfalster dfalster approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@yangsophieee @dfalster @ehwenk