Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Module meta.yml restructure #3028

Closed

Conversation

mirpedrol
Copy link
Member

@mirpedrol mirpedrol commented Jun 19, 2024

Continuation of #2789

This PR adds an automated way of generating the right format meta.yml for modules.

 input: 
   - - meta: 
           type: map 
           description: | 
             Groovy Map containing sample information 
             e.g. [ id:'test', single_end:false ] 
       - scaffold: 
            type: file 
            description: Fasta file containing scaffold 
            pattern: "*.{fasta,fa}" 
   - - fasta: 
           type: file 
           description: FASTA reference file 
           pattern: "*.{fasta,fa}" 

Note that the structure proposed in nf-core/modules#4983 (comment) is not possible if we want to automate the creation of this file, as comments are ignored when reading a yaml file with Python.

Example of outputs formatting:

output:
  - versions:
    - "versions.yml":
        type: file
        description: File containing software versions
        pattern: "versions.yml"
  - bam:
    - meta:
        type: map
        description: Groovy Map containing sample information
  
    - "*.bam":
        type: file
        description: Sorted BAM/CRAM/SAM file
        pattern: "*.{bam,cram,sam}"

In this PR we also add an option --update-meta-yml --fix to fix existing files automatically.
To be changed to --fix as suggested in #2789 (comment)

Pytests are also missing for this functionality.
A test was added for the command nf-core modules lint --fix. ⚠️ This test will fail until the JSON schema is updated (nf-core/modules#5837)

Together with this PR, there are other actions which must happen at the same time:

POC in modules: nf-core/modules#5867

@mashehu
Copy link
Contributor

mashehu commented Jun 24, 2024

Note that the structure proposed in nf-core/modules#4983 (comment) is not possible if we want to automate the creation of this file, as comments are ignored when reading a yaml file with Python.

Looked into the pyaml issue, and looks like some people are using ruamel instead of pyaml. Should we do the same?

@mirpedrol
Copy link
Member Author

Looked into the pyaml issue, and looks like some people are using ruamel instead of pyaml. Should we do the same?

Sounds good!
I changed to ruamel in the follow-up PR because it was the best way to keep the proper indentation. But the comment is still not ideal because it moves it to the line above.

# tuple
- 
    - meta

You can see how the changes in some modules will look like here.

@mirpedrol mirpedrol force-pushed the poc-module-yaml-restructure-continue branch 2 times, most recently from 34d5828 to e6b4c85 Compare July 4, 2024 10:32
@mirpedrol mirpedrol force-pushed the poc-module-yaml-restructure-continue branch from e6b4c85 to ef7364d Compare July 5, 2024 10:14
@mirpedrol
Copy link
Member Author

I will close this PR because I fixed the merge conflicts in the follow-up one #3032, #3032 is the definitive PR 😄

@mirpedrol mirpedrol closed this Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants