Question on interpreting the permutation results

[MingL196]

Hi Jeff,

I would like to make sure that I understand the permutation results correctly.

From the website, http://www.jeffbio.com/methylaction/docs/#permutation-and-bootstrap-testing the permutation results has the following examples:
(Note: I know that the example is a bootstrap results. I am currently using permutations, so I'd like to use your results as a hypothetical permutation result.)

pattern	type	nDMRs	permMean
010	all	486	6.33
all	all	1263	819.33

From these results, can we deduce that
for pattern 010, the number of false positive DMRs in 468 is very small because the permMean is very low (6.33), and
for pattern "all", the number of false positive DMRs in 1263 is very large because the permMean is very high (819.33)?

Thanks,
Ming

[jeffbhasin]

Hello Ming,
Yes that is correct. The number of DMRs for the pattern discovered in the mean across the permutations (sampling without replacement) or bootstraps (sampling with replacement) divided by the number of DMRs found in the real data establishes a false discovery rate. We would find that some patterns had low FDRs but others had high, and in studies with > 2 groups and many patterns, this helped us narrow it down and be interested in certain exceptional (low FDR) patterns and what those DMRs are.

Jeff

[MingL196]

Thank you for your confirmation!

Ming

[MingL196]

I have one more question:

Is the interpretation of variable FDRpercent the same for both bootstrap (with replacement) and permutation (without replacement)?

Thanks for your detailed answers,
Ming

[jeffbhasin]

Yes the calculation and interpretation should be the same. With my data, I found they often gave similar results. I implemented both so it can easily be done either way. The only subtlety to the interpretation is that the bootstrap mode is attempting to "redraw" more samples (to empirically estimate the sample means) , whereas the permutation mode is just shuffling the groups to create a null. For a larger sample size, the bootstrap could be interpreted as an estimate of the sampling distribution.

[MingL196]

Just to make sure I am understanding this correctly, the example:

pattern	type	nDMRs	permMean	permSD	permCV	FDRpercent
011	other	98	176.33	251.24	1.42	179.93

means that 179.93 % of the obeserved 98 DMRs are false positives? Does this mean that all 98 DMRs are false positives?

[jeffbhasin]

Yes. In this case, there can be an FDR reported over 100 when the permutations had more DMRs than in the real data. In these cases, our interpretation is the pattern is unlikely to have support for any real DMRs given the data, since the null is capable of finding just as many or more on average.

[IRECG]

Hi, I have also doubts about the results after bootstrapping. First of all, I would like to know the appropiate number of permutations to choose. I have only 2 groups and I have used nperms=3 following the how-to example. With this approach I have obtained:

Does it mean that I can only trust in those with a low FDR percentage? Let's say less than 10%... and consider just these as actual DMRs?
Thank you

[jeffbhasin]

Hello IRECG.
We recommend as many permutation iterations as are computationally feasible. I would say 100 to 1000 is a good range to try. I believe in some of our papers we ended up doing over 2000. To actually establish that we had done enough iterations, we subsampled our permutations to plotted how the estimates changed as we increased the number of permutations. We would see a tapering off towards a mean and know that at that point more iterations had diminishing returns.

There's an example of this plot in Figure S1E and S1F of our Bhasin et al, Cell Reports, 2015 paper (https://www.cell.com/cell-reports/pdfExtended/S2211-1247(15)01286-3).

I would say 3 is probably not enough and ideally you could try to do 100 or more.

The interpretation is correct - that we only expect those patterns with a low FDR percentage to be real. Otherwise, DMRs of that pattern are abundant even in the null case.

What is your sample size? The permutations are more relevant in cases of large sample sizes and many groups. They may not be informative if there are not enough degrees of freedom to permute in the first place.

Jeff

[IRECG]

Hello Jeff,
thanks for your answer.
I have two groups of 12 samples each. Do you think that in my case is not relevant doing permutations? If so, how can I filter my DMRs for the following analysis? By log2fc?

Thanks

[jeffbhasin]

In that case the permutation should have enough to work with.

Yes log2fc can be used to filter.

[IRECG]

So, if I run methylaction with "perm.boot=FALSE" what is occuring is permutation, isn't it? But in that case I don't have information of FDR, and I should use log2fc to select DMRs, is it correct?
Thanks

[jeffbhasin]

perm.boot is an option that determines if they should be permutations (without replacement) for bootstraps (with replacement). In both cases, there will be FDRs generated. Regardless of if you determine FDR or not, filtering by log2fc is still recommended.

[jeffbhasin]

From the documentation for methylaction() command - perm.boot is only relevant if nperms > 0. The setting for nperms determines if permutations/bootstraps and hence FDR are being computed at all.

Can also consider maPerm() to generate bootstraps/permutations after a call to methylaction(). This is how I would run it, and also allows using multiple computers to compute the permutations in parallel if run separately on the same original output from methylaction().

nperms Optional, perform this number of permutations after calling DMRs. Will create a
data.table called "FDR" in the output list. See also maPerm(), maPermMerge(),
and maPermFdr() for manual permutation running and FDR calculation.

perm.boot If nperms > 0 and if TRUE, perform bootstrapping (sampling with replacement).
Otherwise, perform permutations (sampling without permutations)

[IRECG]

I'm sorry if I did not explain me clearly, after reading the methylaction documentation I was a bit confused, because I am not an expert in all these terms. My question is: should I do permutation/bootstraping or not (perm.boot=FALSE) due to my sample size? In the case of not doing any I would use only log2fc to filter, in the other case after FDR I would select those that I can trust as a group and after log2fc to select individually the DMRs.

[jeffbhasin]

In our tests, we found permutation and bootstrap usually gave similar results. I would start with permutation (perm.boot=FALSE) and see what the results are.

[IRECG]

Hi,
I have done permuation, but in this case I don't have information about FDR when I call ma$fdr, but in the summary I can see "All bellow FDR window". Can I have acces to FDR if I don't use bootstrap? Thanks

[jeffbhasin]

There must be permutations (or bootstraps) done in order for FDR to be calculated.

[jeffbhasin]

If there were not permutations (or bootstraps) run, you can use the maPerm() and maPermFdr() functions to run them and compute the FDRs after the fact. Please see the function documentation for a description: http://www.jeffbio.com/methylaction/docs/methylaction-manual.pdf