SPR-T-SHE-000010-CTI-gal-001

Summary

Validated Requirement(s): R-SHE-PRD-F-140

Relevent test or test case(s): T-SHE-000010-CTI-gal (TC-SHE-100028-CTI-gal-SNR, TC-SHE-100029-CTI-gal-bg, TC-SHE-100030-CTI-gal-size, TC-SHE-100031-CTI-gal-col, and TC-SHE-100900-CTI-gal-tot)

Test dataset: TD-SHE-000001-SHE-analysis

Test procedure: Estimated shear values (binned for each test case) for a given observation were regressed against the distance of the object from the readout register. The slope of the best-fit regression was compared against the expected value of zero.

Test result: OK with warnings

Requirement Interpretation

This test validates Requirement R-SHE-PRD-F-140, which states:

R-SHE-PRD-F-140: The contribution to the error on the multiplicative bias (μ) due to the residuals of the CTI correction shall be smaller than 5x10^-4 (1σ).

Imperfect CTI correction is in fact more likely to contribute to additive bias than to multiplicative. Given that other requirements (e.g. R-SHE-CAL-F-080) impose constraints on additive bias, we interpret this requirement to mean generally that imperfect CTI correction should contribute no statistically signifant amount of bias above a minimal threshold.

T-SHE-000010-CTI-gal tests for any statistically significant presence of a slope between measured shear and readout register distance, which is likely to occur if there is sufficient error in the CTI correction. We note imperfect CTI correction is not the only possible cause of such a slope: An imperfect telescope model fit used for the generation of PSF models could potentially cause a similar. The PSF model is tested by multiple other unit tests though, and so if this test fails, the results of those tests can be used to inform if the failure might be do instead to issues with the PSF model.

Detailed test results

The test was run on data for a single fiducial observation, with data binned depending on the test case. We used the following definitions of parameters for binning the data:

• TC-SHE-100028-CTI-gal-SNR: Signal-to-noise ratio, defined as FLUX_VIS_APER/FLUXERR_VIS_APER, using data from the Final Catalog data product provided by PF-MER
• TC-SHE-100029-CTI-gal-bg: Sky background level, defined as value of the background map provided by PF-VIS as the stated position of the object, in ADU
• TC-SHE-100030-CTI-gal-size: Size, defined as size of the object's region in the segmentation map provided by PF-MER, in pixels
• TC-SHE-100031-CTI-gal-col: Colour, defined as 2.5*log10(FLUX_VIS_APER/FLUX_NIR_STACK_APER), using data from the Final Catalog data product provided by PF-MER
• TC-SHE-100900-CTI-gal-tot: All data is used in a single bin

The following bin limits were used for each test case, using the above parameter definitions and units:

• TC-SHE-100028-CTI-gal-SNR: 0, 3.5, 5, 7, 10, 15, 30, 1e99
• TC-SHE-100029-CTI-gal-bg: 0, 30, 35, 35.25, 36, 50, 1e99
• TC-SHE-100030-CTI-gal-size: 0, 30, 45, 75, 140, 300, 1e99
• TC-SHE-100031-CTI-gal-col: -1e99, -2.5, -2, -1.5, -1, -0.6, 1e99
• TC-SHE-100900-CTI-gal-tot: (all data)

These bin limits were selected to provide significant subsamples of the data. The test is to be deemed failed if, for any bin in which there is sufficient data to perform the test (generally at least 3 objects with well-measured shear), the slope of the shear versus the distance of objects from the readout register differs from zero by a statistically-significant margin.

To assess statistical significance, we must account for the fact that since we are checking multiple bins for each parameter, this increases the chance that the results for one bin will cross a given threshold of significance by chance. We choose to account for this through the following procedure:

1. Decide upon a fiducial threshold of statistical significance we wish to use: p_f = 0.05.

2. Calculate a threshold for each bin, p_b such that the probability of the results for any bin crossing the threshold is equal to the probability of the results for a single bin crossing the threshold p_f through:

   p_b = 1 - (1-p_f)^1/N

3. Calculate the corresponding two-tailed sigma value of a Gaussian distribution from this probability

This results in the following threshold probabilities and sigma values for each test case:

While not the focus of the test, the measured intercept value was tested for consistency with zero as well. A failure in this aspect does not indicate a likely issue with CTI, and so does not constitute a failure of this test, but does indicate the presence of some other issue, and so is raised as a warning if it occurs.

The test results for each test case and bin are detailed in the table below:

As all test cases pass for all bins where sufficient data is available, all test cases are thus considered passed. However, we do note some minor issues:

1. In the case of the test case TC-SHE-100029-CTI-gal-bg, all data resides in a single bin. This is likely due to the fact that the bin limits were chosen based on the distribution of background levels at object positions in a different observation, and in this observation, the variance around the mean background level is not large enough for data to be present in more than one bin. It is recommended that a separate approach be considered for bins in the background level test case instead of using fixed bin limits, such as setting bin limits based on the quartiles of the data found in each observation.

2. The scatter of measured slope values appears low relative to the error values. Using a chi-squared test, this doesn't reach statistical significance (p=0.94 for TC-SHE-100028-CTI-gal-SNR, for instance), but it is close enough that it is worth investigation to determine if it might indicate a bug in the error calculation.

3. In the case of TC-SHE-100900-CTI-gal-tot (and TC-SHE-100029-CTI-gal-bg, where all data resided in a single bin), the measured intercept value of (5.71 +/-1.03) x 10^-3 differed from zero by 5.52σ. This is not an expected effect of improper CTI correction, and so it does not constitute a test failure, but it does warrant a warning and further investigation.