CLASSIC Data Reduction Pipeline


The CLASSIC / CLIMB data reduction software is maintained by Theo ten Brummelaar. Please see his website for download and installation instructions.

Example CLASSIC data on Kappa Ser from 2018_05_11

The steps below show an example of how to process data using the CLASSIC/JOUFLU reduction pipeline called redfluor.  Typing "redfluor -V" at the command line prompt will give the version of the code and list the available options that can be used with the code.

$ redfluor -V
VERSION: V3.1 Wed Feb 28 14:48:15 PST 2018
usage: redfluor [-flags] ir_datafile
Flags:
-a        Toggle apodize for FFT (ON)
-A        Use shutter sequence A for noise (TRUE)
-b        Bootstrap to estimate the Median Error (OFF)
-c        Force this to be treated as CLASSIC data (OFF)
-d[0,1,2,3,4]    Set display level(1)
-D[Dir]        Directory for results (Basename)
-e        Toggle edit scans (ON)
-E[min_weight]    Edit scans by fringe weight (OFF)
-f        Force this to be treated as JOUFLU data (OFF)
-F[env_mult]    Change # of envelopes to include (3)
-g        Toggle GTK (OFF)
-h        Print this message
-H[n]        Set percentage definition of high frequency (20)
        Use 0.0 to force using upper integration limit.
-i        Toggle manual integration range (OFF)
-I[start-stop]    Set integration range of data (AUTO)
-j        Toggle adjusting filter width using Vg (ON)
-J        Toggle ignoring photometric data (OFF)
-k[0,1]    Set method of calculating Kappa (KAPPA_BY_SCAN)
            0 - Calculate for each scan.
            1 - Calculate for total mean.
-l[freq-fwhm]    Use low pass instead of Wiener filter (0).
-L        Toggle use photometry for noise estimate (OFF)
-O        Toggle Photometry Only (OFF)
-o[n_sigma]    Number (float) of standard deviations for outlier removal (OFF, <=0 to turn off)
-M        Use weighted mean (ON)
-p        Toggle use postscript (OFF)
-P[smooth_noise_size]    Change noise PS +-smooth size (5)
-q        Toggle ignore off star data (OFF)
-Q        K band shutter background percentage (3.63%)
-R        Toggle remote mode (OFF)
-s[spec-chan]    Change spectral channel (0=ALL)
-S[smooth_signal_size]    Change +-smooth size (1)
-t[start,stop]    Truncate scans (OFF)
-u        Toggle noise PS multiplier (OFF)
-U[freq]    Set DC suppression frequency (20.0 Hz)
-v        Toggle PLPLOT verbose mode (OFF)
-V        Toggle REDFLUOR verbose mode (OFF)
-x        Toggle use dither freq for fringes (OFF)
-X[stddevmult]    Set stddev multiplier (3)
-y        Toggle use fringe signal for waterfall (OFF)
-Y        Toggle use filter for waterfall (ON)
-z[pixmult]    Set pixel multiplier (2)
 

The README file in the reduceir software package includes a detailed description for these options.  Redfluor uses the PLPLOT package to produce the plots it makes, and therefore will also understand any standard Plplot command line arguments. For example, it would be very common to use:

redfluor -dev xwin 

on an X windows machine, or

redfluor -dev psc 

to produce a color postscript output.

A common way to invoke redfluor is:

redfluor -dev xwin -D/home/schaefer/chara/classic/2018/2018_05_11 -d2 -o3.0 2018_05_11_HD_139087_ird_001.fit 

where -D indicates the directory path to save the reduced files, -d2 displays an intermediate amount of plots, and -o3 removes outliers that are more than 3 standard deviations away from the median visibility.  The data file to be processed is 2018_05_11_HD139087_ird_001.fit.  This command would need to be run for every data file collected.

The first step of the redfluor pipeline will show a waterfall plot of the fringe envelope over time (with time running vertically).  On a night of good seeing, the fringe waterfall will be concentrated in the center of the scan window.  On a night with bad seeing, the waterfall will look more like a scatter plot, with the position of the fringes bouncing back and forth across the scan window.  If there are any gaps in the fringes (vertical regions where the fringes disappear), then these scans should be removed by entering "e" on the command line and clicking on the region to edit.

Fringe editing.

 

# Processing file 2018_05_11_HD_139087_ird_001.fit
# Vel = 425.0 Lambda = 2.1 BP = 199.2
Move on, zoom in, Zoom out, Edit, Redraw or Clear (m/z/Z/e/r/c)? e
Click on the start of the edit.
Click on the end of the edit.
Move on, zoom in, Zoom out, Edit, Redraw or Clear (m/z/Z/e/r/c)? m

Next the routine will show plots of the photometry (raw and background subtracted).  The first three segmented areas show the light from Beam 5, darks (shutters closed), and the light from Beam 6.  The next ~ 200 scans are the fringe data (light from both telescopes with fringes).  The second set of ~ 200 frames are light from both telescopes but without fringes (carts moved away from the fringe position).  The final three segmented areas used to be another shutter sequence, but are now sky frames where the telescopes are moved off the star and the sky background is recorded.  The sky frames were added to the observing sequence after it was found that the shutters inside the lab contribute thermal heat in the K-band, which is particularly important when observing faint targets.  The last shutter sequence should be checked to make sure that the telescope moved off the star during the sky frames.  If you see light during the last sequence and the data were recorded using the new off-fringe and star sequence (e.g., the fits header keyword CC_SEQ = 'OFF_FRG_AND_STAR'), then you will need to run redfluor using the -A flag to use only the first shutter sequence instead.

Photometry and Kappa Matrix.


Hitting enter in the terminal window will bring up the next window showing the power spectra for the dark frames, fringe frames, telescope A, and telescope B.  In the example below, the fringe peak shows up at 160-240 Hz.  Peaks in the dark frames would indicate the presence of electronic noise, whereas peaks in the shutter A or shutter B frames would likely indicate an oscillation of the telescope (e.g., the small peak at ~ 260 Hz in shutter B).

Power Spectra.


Hitting enter in the terminal window will bring up the next window showing the off-fringe power spectra (light from both telescopes but without fringes), the signal/noise estimate, and the noise subtracted power spectrum of the signal.  The noise subtracted power spectrum will also show outlines of the fringe integration regions selected by the Gaussian fit to the fringe peak.

Noise Subtracted Power Spectra.


By default, the integration range is set by fitting a Gaussian to the fringe power spectrum peak.  If you want to select the integration region by hand, then set the -i flag.  Or if you want to define a fixed integration region for all stars in the data set, use the -I[start-stop] flag.  The automated integration region will be output to the terminal:

# Vel = 413.9 Lambda = 2.1 BP = 194.0 (193.0, 195.1)
# To get this integration range use
# -I146.82-241.30

Next is a plot of the low pass filter used to normalize the fringe signal.

Low pass filter.


Next the routine applies the sigma clipping algorithm if the -o flag is set.  The results from the outlier rejection are displayed.  If any data points are rejected they will be removed from the plot on the right.

Outlier Rejection.

 

# Calculating .........
# Rejected 0 scans due to low photometry.
# Rejected 0 scans due to being too close to the edge.
# Rejected 0 scans due to low weight.

The final plot will show histograms of the correlation and the fringe weights, the fringe waterfall, and the power spectra waterfall.

Histograms of the correlation and fringe weights.

 

# Results from FLUOR PS calculation method:
N_SCANS 201
# Detector 1 Detector 2 Combined
# Mean StdDev Mean StdDev Mean StdDev
Vg 404.268 15.575 407.158 15.698 405.880 15.234
T0_SCANS 21.8 21.6 22.3
T0_500NM 3.8 3.8 3.9
V2_SCANS 0.22248 0.09699 0.21254 0.10369 0.20418 0.09331
V2_CORR -1.18262
V2_CHI2 0.00225 0.00492
V2_SQRT 0.47168 0.20563 0.46102 0.22492 0.45187 0.20650
V_SCANS 0.45902 0.10883 0.44529 0.11969 0.43844 0.10960
V_NORM 0.46011 0.17201 0.44672 0.18679 0.43964 0.17224
V_LOGNORM 0.46157 0.09710 0.44893 0.10488 0.44133 0.09702
V2_MEDIAN 0.14754 0.09873 0.13858 0.09644 0.12813 0.09599
V_MEDIAN 0.38411 0.12971 0.37226 0.14757 0.35796 0.14248
Move on, toggle Log, zoom in, zoom out or zoom = 1 (m/l/z/Z/1)?


The final results are output to the screen and stored in the .info files in the directory created for each data file.  The results consist of a variety of visibility estimators.  Currently, it is recommended to use V2_SCANS which is the weighted mean of the visibilities for all scans.  The three sets of numbers listed for each estimator give the visibility and error recorded for pixel 1, pixel 2, and the difference signal.  The default is to use the difference signal, however, comparing the values obtained individually from the two readout pixels could provide an estimate of systematic offsets.

Calibration

Use calibir to use the calibrator stars to calibrate the system visibility and correct the visibilities measured for the science target.  The calibir program reads the INFO files in the directories in the current location.  The routine calibrates the science target using the nearest neighbor calibrators observed before and after the science target.  If the alignment changes between a calibrator and neighboring science target, then these observations should be moved to different directories to avoid calibrating across a jump in the system visibilities.  If NIRO alignments occurred between two calibrators, then it should be OK to calibrate all of the data in one directory since calibir uses only the nearest neighbors.

A common way to call calibir is as follows:

calibir -F -i -s1.1368-0.1069,1.0644-0.1001 -BV2_SCANS HD_141477 HD_139087 HD_142244 

where -F saves the output to an OIFITS file based on the name of the object, -i selects objects based on the "ID" name listed in the INFO file, -s sets the calibrator diameters (uniform disk diameters in the K-band of θ = 1.1368 ± 0.1069 mas for HD 139087 and θ = 1.0644 ± 0.1001 mas for HD_142244 estimated by JMMC/Searchcal), and -BV2_SCANS sets the visibility estimator to "V2_SCANS" for both the target and the calibrators.  After the flags, the science target ID is listed, followed by a listing of the calibrators in the same order as their diameters in the -s flag.  Calibir outputs a plot of the calibrator visibilities (green, blue points), and the raw (red) and calibrated (yellow) visibilities of the science target.  In this example sequence, a NIRO alignment was done after every C1-O-C2 set.

Calibir calibration plot.


By default, the calibrated data are saved to an oifits file with the name 2018_05_11_HD_141477_001.fits. The oifits files can be fit using various data analysis software or read directly into programming languages like IDL.


OIFITS Utitlities

There are a few oifits utilities included in the reduceir package that can be used to manipulate, view, and analyze data.  For instance, oifits-merge can be used to merge multiple OIFITS files on the same target (e.g., merge observations from different nights) and oifud can be used to fit simple geometries like a uniform disk to the visibility data:

oifud 2018_05_11_HD_141477_001.fits 

 

oifud interface.

 

Uniform disk fit.