Science metrics for photo-z PDF approximation performance assessment

[aimalz]

Re: conversation with @drphilmarshall today, it's time to give #36 some context. In order to make #39 submittable to a journal, we'll need to perform an end-to-end test on a science case, and it would be appropriate for it to answer the question that inspired qp to be written: "What is the best way to store photo-z PDFs?" @janewman-pitt-edu, we are thinking of the simplest scientifically relevant end product to which inaccuracy in PDF representation could be propagated. n(z) seems like an obvious choice, but I think this application would be best approached after CHIPPR is submitted. Thoughts?

[janewman-pitt-edu]

n(z) may not be ideal as you could have small storage errors that all average to zero and leave n(z) untouched. You thus may want to focus more on applications where individual object p(z)'s matter. Examples include cluster finding, intrinsic alignment mitigation in weak lensing, and strong lensing analyses. Of course Phil is an expert on the latter...

[drphilmarshall]

I think the strong lensing environment requirements will be similar to those of cosmic shear, and be (more or less) that the n(z) comes out accurately. One cheap alternative to checking the hierarchical inference of n(z) is to use the "stacked p(z)" approximation, just to quantify how n(z) estimation accuracy depends on p(z) approximation.

I like the idea of checking some alternatives to n(z) on scientific grounds. Alex, let's talk to Eli Rykoff today about the cluster-finding application, and we can ask around on Slack for input on the IA application. On the galaxies side, are there uses for p(z)'s that suggest particular combinations of them? Thanks Jeff!

[drphilmarshall]

Reports from Slack:

• David Alonso @damonge says:

"In the simplest scenario, we're interested in the global n(z) of a given sample (e.g. a photo-z redshift bin for a particular population), but ideally including the possible uncertainties on that n(z). It would be great to understand whether the latter can be provided and if so, in what format.
This is for the simplest analysis scenario we're envisaging right now, but other options might need individual p(z)'s."

• Michael Troxel @matroxel suggested we look at the mean inverse lensing sigma_crit, between pairs of redshift bins (and pointed to the test code he is using in DES here as a guide). He says:

Averaging sigma crit of individual pairs in two bins [tells] you something different [from n(z)].
You can produce sufficient signal to noise to directly measure the inferred bias without rescaling your covariance. We did both [this and the n(z) test] in the DES SV photo-z paper."

Thanks both! We'll look into these.

[janewman-pitt-edu]

For strong lensing, I thought you'd be interested in basically making a foreground mass map, which would require p(mass, z) for each potential foreground object?

For galaxies science I think there are few applications that would ever need very high precision, I'd look towards cosmology instead.

[drphilmarshall]

Good, that was our sense too. For strong lensing, it's true that for a 3D mass reconstruction we're interested in P(M*,z) or P(L,z), to get more information on the halo masses - but the 2D problem is beyond the scope of this paper! I think we'll learn something useful from thinking about 1D P(z) approximation for WL and LSS, and then next time around see what'd be different in the 2D case.

[aimalz]

@drphilmarshall Are we still thinking of including metrics on Sigma_crit^-1? I think it's nontrivial to calculate without point estimators of redshift and don't want to muddy the waters with straying too many layers beyond the PDFs -- wouldn't this involve calculating metrics on quantities derived from point estimators derived from PDFs?

[drphilmarshall]

To include metrics on Sigma_crit^-1 we'd need to understand what this is (one number per lens bin - source bin pair? a weighted integral over the two n(z)'s?) and how it enters into the cosmological parameter inference. I could then imagine defining a set of Sigma_crit^-1 parameters whose posterior PDF given the photo-z data needs to be sampled, and then looking at summary statistics of that PDF (eg mean bias, mean variance etc) as our metrics. It's not clear that it needs point estimators, but I would expect that some integration(s) over n(z) would be in order. I'd say focus on the moments of n(z) (mean, variance) first, and have in mind returning to Sigma_crit^-1 later. What clues do Krause et al 2017 (DES YR1 methods paper) give about the key issues with n(z)? Joe De Rose mentioned the mean of n(z) being the most important feature.

[aimalz]

I'm closing this along with #57 now that #54 is done.