Description
Steps in the various pipeline modules calculate variances due to different sources of noise or modify variances that were computed by previous steps. In some cases the variance arrays are only used internally within a given step. For several steps, these arrays must be propagated to subsequent steps in the pipeline. Anytime a step creates or updates variances, the total error (ERR) array values are always recomputed as the square root of the quadratic sum of all variances available at the time. Note that the ERR array values are always expressed as standard deviation (i.e. square root of the variance).
The table below is a summary of which steps create or update variance and error arrays, as well as which steps make use of these data. Details of how each step computes or uses these data are given in the subsequent sections below.
Step |
Stage |
Creates arrays |
Updates arrays |
Step uses |
---|---|---|---|---|
ramp_fitting |
1 |
VAR_POISSON, VAR_RNOISE |
ERR |
None |
gain_scale |
1 |
None |
ERR, VAR_POISSON, VAR_RNOISE |
None |
flat_field |
2 |
VAR_FLAT |
ERR, VAR_POISSON, VAR_RNOISE |
None |
fringe |
2 |
None |
ERR |
None |
barshadow |
2 |
None |
ERR, VAR_POISSON, VAR_RNOISE, VAR_FLAT |
None |
pathloss |
2 |
None |
ERR, VAR_POISSON, VAR_RNOISE, VAR_FLAT |
None |
photom |
2 |
None |
ERR, VAR_POISSON, VAR_RNOISE, VAR_FLAT |
None |
outlier_detection |
3 |
None |
None |
ERR |
resample |
3 |
None |
None |
VAR_RNOISE |
wfs_combine |
3 |
None |
ERR |
None |
Stage 1 Pipelines
Stage 1 pipelines perform detector-level corrections and ramp fitting for individual exposures, for nearly all imaging and spectroscopic modes. Details of the pipelines can be found at Stage 1 Pipelines.
The Stage 1 pipeline steps that alter the ERR, VAR_POISSON, or VAR_RNOISE arrays of the science countrate data are discussed below. Any step not listed here does not alter or use the variance or error arrays in any way and simply propagates the information to the next step.
ramp_fitting
This step calculates and populates the VAR_POISSON and VAR_RNOISE arrays in the ‘rate’ and ‘rateints’ files, and updates the ERR array as the square root of the quadratic sum of the variances. VAR_POISSON and VAR_RNOISE represent the uncertainty in the computed slopes (per pixel) due to Poisson and read noise, respectively. The details of the calculations can be found at ramp_fitting.
gain_scale
The gain_scale
step is applied after ramp_fitting
, and applies to both the
rate and rateints products. The gain correction is applied to the ERR,
VAR_POISSON, and VAR_RNOISE arrays. The SCI and ERR arrays are multiplied by the
gain correction factor, and the variance arrays are multiplied by the square of
the gain correction factor. More details can be
found at gain_scale.
Stage 2 Pipelines
Stage 2 pipelines perform additional instrument-level and observing-mode corrections and calibrations to produce fully calibrated exposures. There are two main Stage 2 pipelines: one for imaging calwebb_image2 and one for spectroscopy calwebb_spec2. In these pipelines, the various steps that apply corrections and calibrations apply those same corrections/calibrations to all variance arrays and update the total ERR array.
flat_field
The SCI array of the exposure being processed is divided by the flat-field reference image. The VAR_FLAT array is created, computed from the science data and the flat-field reference file ERR array.
For all modes except GUIDER, the VAR_POISSON and VAR_RNOISE arrays are divided by the square of the flat. The science data ERR array is then updated to be the square root of the sum of the three variance arrays.
For the GUIDER mode, their are no VAR_POISSON and VAR_RNOISE arrays. The science data ERR array is updated to be the square root of the sum of the variance VAR_FLAT and the square of the incoming science ERR array.
For more details see flat_field.
fringe
For MIRI MRS (IFU) mode exposures, the SCI and ERR arrays in the science exposure are divided by the fringe reference image. For details of the fringe correction, see fringe.
barshadow
This step is applied only to NIRSpec MSA data for extended sources. Once the 2-D correction array for each slit has been computed, it is applied to the science (SCI), error (ERR), and variance (VAR_POISSON, VAR_RNOISE, and VAR_FLAT) arrays of the slit. The correction values are divided into the SCI and ERR arrays, and the square of the correction values are divided into the variance arrays. For details of the bar shadow correction, see barshadow.
pathloss
The pathloss
step corrects NIRSpec and NIRISS SOSS data for various types of
light losses. The correction factors are divided into the SCI and ERR arrays of
the science data, and the square of the correction values are divided into the
variance arrays. For details of this step, see pathloss.
photom
The calibration information for the photom
step includes a scalar flux conversion
constant, as well as optional arrays of wavelength and relative response (as a
function of wavelength). The combination of the scalar conversion factor and any 2-D
response values is applied to the science data, including the SCI and ERR arrays,
as well as the variance (VAR_POISSON, VAR_RNOISE, and VAR_FLAT) arrays. The flux
calibration values are multiplied into the science exposure SCI and ERR arrays,
and the square of the calibration values is multiplied into all variance arrays.
For details of the photom correction, see photom.
Stage 3 pipelines
Stage 3 pipelines perform operations that work with multiple exposures and in
most cases produce some kind of combined product. The operations in these
pipelines that either use or modify variance/error arrays that are propagated
through the pipeline are outlier_detection
and wfs_combine
.
outlier_detection
The outlier_detection
step is used in all Stage 3 pipelines. It uses the ERR array to
make a local noise model, based on the readnoise and calibration errors of earlier
steps in the pipeline. This step does not modify the ERR array or any of the VAR
arrays.
resample/resample_spec
The resample
and resample_spec
steps make use of the VAR_RNOISE array to
compute weights that are used when combining data with the weight_type=ivm
option selected. The step also resamples all of the variance and error arrays,
using the same output WCS frame as the science data.
wfs_combine
The wfs_combine
step is only applied in the Stage 3 Wavefront Sensing and Control
(calwebb_wfs-image3) pipeline for dithered pairs of WFS&C exposures.
This step can modify variance/error arrays, but only if the optional
“do_refine” parameter is set to True (the default is False). In this
case the algorithm to refine image offsets will be used and the ERR array values will be
altered on output, using a combination of the input image errors.
See the step documentation at wfs_combine for
more details.