Reference File

The extract_1d step uses an EXTRACT1D reference file and an APCORR reference file.

EXTRACT1D Reference File

The EXTRACT1D reference file contains information needed to guide the 1D extraction process. It is a text file, with the information in JSON format for Non-IFU data and in ASDF format for IFU data.

Reference Selection Keywords for EXTRACT1D

CRDS selects appropriate EXTRACT1D references based on the following keywords. EXTRACT1D is not applicable for instruments not in the table. All keywords used for file selection are required.

Instrument

Keywords

MIRI

INSTRUME, EXP_TYPE, DATE-OBS, TIME-OBS

NIRISS

INSTRUME, EXP_TYPE, DATE-OBS, TIME-OBS

NIRSpec

INSTRUME, EXP_TYPE, LAMP, OPMODE, DATE-OBS, TIME-OBS

Standard Keywords

The following table lists the keywords that are required to be present in all reference files. The first column gives the FITS keyword names. The second column gives the jwst data model name for each keyword, which is useful when using data models in creating and populating a new reference file. The third column gives the equivalent meta tag in ASDF reference file headers, which is the same as the name within the data model meta tree (second column).

FITS Keyword

Data Model Name

ASDF meta tag

AUTHOR

model.meta.author

author

DATAMODL

model.meta.model_type

model_type

DATE

model.meta.date

date

DESCRIP

model.meta.description

description

FILENAME

model.meta.filename

N/A

INSTRUME

model.meta.instrument.name

instrument: {name}

PEDIGREE

model.meta.pedigree

pedigree

REFTYPE

model.meta.reftype

reftype

TELESCOP

model.meta.telescope

telescope

USEAFTER

model.meta.useafter

useafter

NOTE: More information on standard required keywords can be found here: Standard Required Keywords

Type Specific Keywords for EXTRACT1D

In addition to the standard reference file keywords listed above, the following keywords are required in EXTRACT1D reference files, because they are used as CRDS selectors (see Reference Selection Keywords for EXTRACT1D):

Keyword

Data Model Name

EXP_TYPE

model.meta.exposure.type

Reference File Format for non-IFU data

EXTRACT1D reference files for non-IFU data are text files, with the information stored in JSON format. All the information is specified in a list with key apertures. Each element of this list is a dictionary, one for each aperture (e.g. a slit) that is supported by the given reference file. The particular dictionary used by the step is found by matching the slit name in the science data with the value of key id. Key spectral_order is optional, but if it is present, it must match the expected spectral order number.

The following keys are supported for non-IFU data (see below for IFU keys). Key id is the primary criterion for selecting which element of the apertures list to use. The slit name (except for a full-frame input image) is compared with the values of id in the apertures list to select the appropriate aperture. In order to allow the possibility of multiple spectral orders for the same slit name, there may be more than one element of apertures with the same value for key id. These should then be distinguished by using the secondary selection criterion spectral_order. In this case, the various spectral orders would likely have different extraction locations within the image, so different elements of apertures are needed in order to specify those locations. If key dispaxis is specified, its value will be used to set the dispersion direction within the image. If dispaxis is not specified, the dispersion direction will be taken to be the axis along which the wavelengths change more rapidly. Key region_type can be omitted, but if it is specified, its value must be “target”. The remaining keys specify the characteristics of the source and background extraction regions.

  • id: the slit name, e.g. “S200A1” (string)

  • spectral_order: the spectral order number (optional); this can be either positive or negative, but it should not be zero (int)

  • dispaxis: dispersion direction, 1 for X, 2 for Y (int)

  • xstart: first pixel in the horizontal direction, X (int) (0-indexed)

  • xstop: last pixel in the horizontal direction, X (int) (0-indexed)

  • ystart: first pixel in the vertical direction, Y (int) (0-indexed)

  • ystop: last pixel in the vertical direction, Y (int) (0-indexed)

  • src_coeff: this takes priority for specifying the source extraction region (list of lists of float)

  • bkg_coeff: for specifying background subtraction regions (list of lists of float)

  • independent_var: “wavelength” or “pixel” (string)

  • smoothing_length: width of boxcar for smoothing background regions along the dispersion direction (odd int)

  • bkg_fit: the type of background fit or computation (string)

  • bkg_order: order of polynomial fit to background regions (int)

  • extract_width: number of pixels in cross-dispersion direction (int)

  • use_source_posn: adjust the extraction limits based on source RA/Dec (bool)

Note

All parameter values that represent pixel indexes, such as xstart, xstop, and the src_coeff and bkg_coeff coefficients, are always in the frame of the image being operated on, which could be a small cutout from a larger original image. They are also ZERO-indexed and the limits are inclusive (e.g. 11-15 includes 5 pixels).

See Extraction for 2D Slit Data for more details on how these parameters are used in the extraction process.

Editing JSON Reference File Format for non-IFU data

The default EXTRACT1D reference file is found in CRDS. The user can download this file, modify the contents, and use this modified file in extract_1d by specifying this modified reference file with the override option (override in python or override in strun). The format for JSON files has to be exact, for example, the format of a floating-point value with a fractional portion must include at least one decimal digit, so “1.” is invalid, while “1.0” is valid. The best practice after editing a JSON reference file is to run a JSON validator off-line, such as https://jsonlint.com/, and correct any format errors before using the JSON reference file in the pipeline.

Reference File Format IFU data

For IFU data the reference files are stored as ASDF files. The extraction size varies with wavelength. The reference file contains a set of vectors defining the extraction size based on wavelength. These vectors are all the same size and are defined as follows:

  • wavelength: wavelength in microns.

  • radius: the radius of the circular extraction aperture (arcseconds, float)

  • inner_bkg: of the inner edge of the background annulus (arcseconds, float)

  • outer_bkg: of the outer edge of the background annulus (arcseconds, float)

In addition following general keys are also in the ASDF reference file:

  • subtract_background: if true, subtract a background determined from an annulus with inner and outer radii given by inner_bkg and outer_bkg (boolean)

  • method: one of “exact”, “subpixel”, or “center”, the method used by photutils for computing the overlap between apertures and pixels (string, default is “exact”)

  • subpixels: if method is “subpixel”, pixels will be resampled by this factor in each dimension (int, the default is 10)

See Extraction for 3D IFU Data for more details on how these parameters are used in the extraction process.

Example EXTRACT1D Reference File

The following JSON was taken as an example from reference file jwst_niriss_extract1d_0003.json:

{
    "REFTYPE": "EXTRACT1D",
    "INSTRUME": "NIRISS",
    "TELESCOP": "JWST",
    "DETECTOR": "NIS",
    "EXP_TYPE": "NIS_SOSS",
    "PEDIGREE": "GROUND",
    "DESCRIP": "NIRISS SOSS extraction params for ground testing",
    "AUTHOR": "M.Wolfe, H.Bushouse",
    "HISTORY": "Build 7.1 of the JWST Calibration pipeline. The regions are rectangular and do not follow the trace.",
    "USEAFTER": "2015-11-01T00:00:00",
    "apertures": [
      {
      "id": "FULL",
      "region_type": "target",
      "bkg_coeff": [[2014.5],[2043.5]],
      "xstart":    4,
      "xstop":  2044,
      "ystart": 1792,
      "ystop":  1972,
      "dispaxis": 1,
      "extract_width": 181
      },

      {
      "id": "SUBSTRIP256",
      "region_type": "target",
      "bkg_coeff": [[221.5],[251.5]],
      "xstart":   4,
      "xstop": 2044,
      "ystart": 20,
      "ystop":  220,
      "dispaxis": 1,
      "extract_width": 201
      },

      {
      "id": "SUBSTRIP96",
      "region_type": "target",
      "bkg_coeff": [[1.5],[8.5],[92.5],[94.5]],
      "xstart":   4,
      "xstop": 2044,
      "ystart":  10,
      "ystop":   92,
      "dispaxis": 1,
      "extract_width": 83
      }]
}

APCORR Reference File

REFTYPE:

APCORR

The APCORR reference file contains data necessary for correcting extracted imaging and spectroscopic photometry to the equivalent of an infinite aperture. It is used within the source_catalog step for imaging and within the extract_1d step for spectroscopic data.

Reference Selection Keywords for APCORR

CRDS selects appropriate APCORR references based on the following keywords. APCORR is not applicable for instruments not in the table. All keywords used for file selection are required.

Instrument

Keywords

FGS

INSTRUME, EXP_TYPE, DATE-OBS, TIME-OBS

MIRI

INSTRUME, EXP_TYPE, DATE-OBS, TIME-OBS

NIRCam

INSTRUME, EXP_TYPE, DATE-OBS, TIME-OBS

NIRISS

INSTRUME, EXP_TYPE, DATE-OBS, TIME-OBS

NIRSpec

INSTRUME, EXP_TYPE, FILTER, GRATING, LAMP, OPMODE, DATE-OBS, TIME-OBS

Standard Keywords

The following table lists the keywords that are required to be present in all reference files. The first column gives the FITS keyword names. The second column gives the jwst data model name for each keyword, which is useful when using data models in creating and populating a new reference file. The third column gives the equivalent meta tag in ASDF reference file headers, which is the same as the name within the data model meta tree (second column).

FITS Keyword

Data Model Name

ASDF meta tag

AUTHOR

model.meta.author

author

DATAMODL

model.meta.model_type

model_type

DATE

model.meta.date

date

DESCRIP

model.meta.description

description

FILENAME

model.meta.filename

N/A

INSTRUME

model.meta.instrument.name

instrument: {name}

PEDIGREE

model.meta.pedigree

pedigree

REFTYPE

model.meta.reftype

reftype

TELESCOP

model.meta.telescope

telescope

USEAFTER

model.meta.useafter

useafter

NOTE: More information on standard required keywords can be found here: Standard Required Keywords

Type Specific Keywords for APCORR

In addition to the standard reference file keywords listed above, the following keywords are required in APCORR reference files, because they are used as CRDS selectors (see Reference Selection Keywords for APCORR):

Keyword

Data Model Name

Instruments

EXP_TYPE

model.meta.exposure.type

All

NON-IFU APCORR Reference File Format

APCORR reference files for non-IFU data are in FITS format. The FITS APCORR reference file contains tabular data in a BINTABLE extension with EXTNAME = ‘APCORR’. The FITS primary HDU does not contain a data array. The contents of the table extension varies for different instrument modes, as shown in the tables below.

Data model:

FgsImgApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

FGS

Image

eefraction

float

scalar

unitless

radius

float

scalar

pixels

apcorr

float

scalar

unitless

skyin

float

scalar

pixels

skyout

float

scalar

pixels
Data model:

MirImgApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

MIRI

Image

filter

string

12

N/A

subarray

string

15

N/A

eefraction

float

scalar

unitless

radius

float

scalar

pixels

apcorr

float

scalar

unitless

skyin

float

scalar

pixels

skyout

float

scalar

pixels
Data model:

MirLrsApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

MIRI

LRS

subarray

string

15

N/A

wavelength

float

1D array

micron

nelem_wl

integer

scalar

N/A

size

integer

1D array

pixels

nelem_size

integer

scalar

N/A

apcorr

float

2D array

unitless

apcorr_err

float

2D array

unitless
Data model:

NrcImgApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

NIRCam

Image

filter

string

12

N/A

pupil

string

15

N/A

eefraction

float

scalar

unitless

radius

float

scalar

pixels

apcorr

float

scalar

unitless

skyin

float

scalar

pixels

skyout

float

scalar

pixels
Data model:

NrcWfssApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

NIRCam

WFSS

filter

string

12

N/A

pupil

string

15

N/A

wavelength

float

1D array

micron

nelem_wl

integer

scalar

N/A

size

integer

1D array

pixels

nelem_size

integer

scalar

N/A

apcorr

float

2D array

unitless

apcorr_err

float

2D array

unitless
Data model:

NisImgApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

NIRISS

Image

filter

string

12

N/A

pupil

string

15

N/A

eefraction

float

scalar

unitless

radius

float

scalar

pixels

apcorr

float

scalar

unitless

skyin

float

scalar

pixels

skyout

float

scalar

pixels
Data model:

NisWfssApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

NIRISS

WFSS

filter

string

12

N/A

pupil

string

15

N/A

wavelength

float

1D array

micron

nelem_wl

integer

scalar

N/A

size

integer

1D array

pixels

nelem_size

integer

scalar

N/A

apcorr

float

2D array

unitless

apcorr_err

float

2D array

unitless
Data model:

NrsFsApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

NIRSpec

FS

filter

string

12

N/A

grating

string

15

N/A

slit

string

15

N/A

wavelength

float

1D array

micron

nelem_wl

integer

scalar

N/A

size

integer

2D array

arcsec

nelem_size

integer

scalar

N/A

pixphase

float

1D array

N/A

apcorr

float

3D array

unitless

apcorr_err

float

3D array

unitless
Data model:

NrsMosApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

NIRSpec

MOS

filter

string

12

N/A

grating

string

15

N/A

wavelength

float

1D array

micron

nelem_wl

integer

scalar

N/A

size

integer

2D array

arcsec

nelem_size

integer

scalar

N/A

pixphase

float

1D array

N/A

apcorr

float

3D array

unitless

apcorr_err

float

3D array

unitless

Row Selection

A row of data within the reference table is selected by the pipeline step based on the optical elements in use for the exposure. The selection attributes are always contained in the first few columns of the table. The remaining columns contain the data needed for the aperture correction. The row selection criteria for each instrument+mode are:

  • FGS Image:

    • None (table contains a single row)

  • MIRI:

    • Image: Filter and Subarray

    • LRS: Subarray

  • NIRCam:

    • Image: Filter and Pupil

    • WFSS: Filter and Pupil

  • NIRISS:

    • Image: Filter and Pupil

    • WFSS: Filter and Pupil

  • NIRSpec:

    • MOS: Filter and Grating

    • Fixed Slits: Filter, Grating, and Slit name

Note: Spectroscopic mode reference files contain the “nelem_wl” and “nelem_size” entries, which indicate to the pipeline step how many valid elements are contained in the “wavelength” and “size” arrays, respectively. Only the first “nelem_wl” and “nelem_size” entries are read from each array.

IFU APCORR Reference File ASDF Format

For IFU data the APCORR reference files are in ASDF format. The aperture correction varies with wavelength and the contents of the files are shown below. The radius, aperture correction and error are all 2D arrays. Currently the 2nd dimension does not add information, but in the future it could be used to provide different aperture corrections for different radii.

Data model:

MirMrsApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

MIRI

MRS

wavelength

float

1D array

micron

radius

float

2D array

arcsec

apcorr

float

2D array

unitless

apcorr_err

float

2D array

unitless
Data model:

NRSIFUApcorrModel

Instrument

Mode

Column name

Data type

Dimensions

Units

NIRSpec

MOS

filter

string

12

N/A

grating

string

15

N/A

wavelength

float

1D array

micron

radius

float

2D array

arcsec

apcorr

float

2D array

unitless

apcorr_err

float

2D array

unitless