Prepare input files for FINEMAP

[hopedisastro]

This script will prepare the z and ld files for FINEMAP, based on outputs from associaTR (meta-analysis) and corr_matrix_maker.py.

The z file contains the effect size and standard error estimates for each variant associated with a gene.
The ld file contains the correlation matrix calcualtions for each variant associated with a gene.

[MattWellie]

not a biggie, but be aware of your job scaling (type * chrom * gene), and you're creating 2 existence checks per combination. All that happens in the driver job so you're delaying the start of the actual work.

I'd experiment with a change here -

# get all files in the output folder, recursively, in a single query
all_files = list(to_path(output_path('finemap_prep', 'analysis')).glob('**'))

# check whether your intended outputs are in that list
ld_file = output_path(f'finemap_prep/{celltype}/{chrom}/{gene_name}.ld', 'analysis')
z_file = output_path(f'finemap_prep/{celltype}/{chrom}/{gene_name}.z', 'analysis')
if all (filepath in all_files for filepath in [z_file, ld_file]):
    continue

I thiiiiink this should scale a lot better, by posting one large query instead of thousands of individual ones.

This also builds the full output file names, so you can pass them to the relevant methods (you pass the celltype, chrom, and gene name to your methods, but you already made the full path here to check if it exists)