CLASSIC Data Reduction Pipeline


The CLASSIC / CLIMB data reduction software was written by Theo ten Brummelaar.  It is recommended that the code is run using the michelson computer on the mountain or using the Remote Data Reduction machine through the CHARA server in Atlanta.  These computers will have the most recent versions of the code installed.  The reduction software is maintained through the CHARA gitlab repository.  A compiled version of the software is usually available for download on Theo's website.  However, this static copy does not always include the most up-to-date version of the code, so it is recommended that the software is run from one of the CHARA computers.

Example CLASSIC data on Kappa Ser from 2018_05_11

The steps below show an example of how to process data using the CLASSIC/JOUFLU reduction pipeline called redfluor.  Typing "redfluor -V" at the command line prompt will give the version of the code and list the available options that can be used with the code.

$ redfluor -V
VERSION: V3.2 Tue Jun  4 17:05:37 PDT 2019
usage: redfluor [-flags] ir_datafile
Flags:
-a        Toggle apodize for FFT (ON)
-A        Use shutter sequence A for noise (TRUE)
-b        Bootstrap to estimate the Median Error (OFF)
-c        Force this to be treated as CLASSIC data (OFF)
-d[0,1,2,3,4]    Set display level(1)
-D[Dir]        Directory for results (Basename)
-e        Toggle edit scans (ON)
-E[min_weight]    Edit scans by fringe weight (OFF)
-f        Force this to be treated as JOUFLU data (OFF)
-F[env_mult]    Change # of envelopes to include (3)
-g        Toggle GTK (OFF)
-h        Print this message
-H[n]        Set percentage definition of high frequency (20)
        Use 0.0 to force using upper integration limit.
-i        Toggle manual integration range (OFF)
-I[start-stop]    Set integration range of data (AUTO)
-j        Toggle adjusting filter width using Vg (ON)
-J        Toggle ignoring photometric data (OFF)
-k[0,1]    Set method of calculating Kappa (KAPPA_BY_SCAN)
            0 - Calculate for each scan.
            1 - Calculate for total mean.
-l[freq-fwhm]    Use low pass instead of Wiener filter (0).
-L        Toggle use photometry for noise estimate (OFF)
-m        Toggle compute V2 from weighted mean power spectrum (ON)
-M        Use weighted mean (ON)
-n        Use new data sequence for Fluor (OFF)
-N        Toggle correct PS for linear slope after noise subtraction (ON)
-O        Toggle Photometry Only (OFF)
-o[n_sigma]    Number (float) of standard deviations for outlier removal (OFF, <=0 to turn off)
-p        Toggle use postscript (OFF)
-P[smooth_noise_size]    Change noise PS +-smooth size (5)
-q        Toggle ignore off star data (OFF)
-Q        K band shutter background percentage (3.63%)
-R        Toggle remote mode (OFF)
-s[spec-chan]    Change spectral channel (0=ALL)
-S[smooth_signal_size]    Change +-smooth size (1)
-t[start,stop]    Truncate scans (OFF)
-u        Toggle noise PS multiplier (OFF)
-U[freq]    Set DC suppression frequency (20.0 Hz)
-v        Toggle PLPLOT verbose mode (OFF)
-V        Toggle REDFLUOR verbose mode (OFF)
-x        Toggle use dither freq for fringes (OFF)
-X[stddevmult]    Set stddev multiplier (3)
-y        Toggle use fringe signal for waterfall (OFF)
-Y        Toggle use filter for waterfall (ON)
-z[pixmult]    Set pixel multiplier (2)
-Z        Toggle mean PS for noise estimate (ON)
 

The README file in the reduceir software package includes a detailed description for these options.  Redfluor uses the PLPLOT package to produce the plots it makes, and therefore will also understand any standard Plplot command line arguments. For example, it would be very common to use:

redfluor -dev xwin 

on an X windows machine, or

redfluor -dev psc 

to produce a color postscript output.

A common way to invoke redfluor is:

redfluor -dev xwin -D/home/schaefer/chara/classic/2018/2018_05_11 -d2 -o3.0 2018_05_11_HD_139087_ird_001.fit 

where -D indicates the directory path to save the reduced files, -d2 displays an intermediate amount of plots, and -o3 removes outliers that are more than 3 standard deviations away from the median visibility.  The data file to be processed is 2018_05_11_HD139087_ird_001.fit.  This command would need to be run for every data file collected.

The first step of the redfluor pipeline will show a waterfall plot of the fringe envelope over time (with time running vertically).  On a night of good seeing, the fringe waterfall will be concentrated in the center of the scan window.  On a night with bad seeing, the waterfall will look more like a scatter plot, with the position of the fringes bouncing back and forth across the scan window.  If there are any gaps in the fringes (vertical regions where the fringes disappear), then these scans should be removed by entering "e" on the command line and clicking on the region to edit.

Fringe editing.

 

# Processing file 2018_05_11_HD_139087_ird_001.fit
# Vel = 425.0 Lambda = 2.1 BP = 199.2
Move on, zoom in, Zoom out, Edit, Redraw or Clear (m/z/Z/e/r/c)? e
Click on the start of the edit.
Click on the end of the edit.
Move on, zoom in, Zoom out, Edit, Redraw or Clear (m/z/Z/e/r/c)? m

Next the routine will show plots of the photometry (raw and background subtracted).  The first three segmented areas show the light from Beam 5, darks (shutters closed), and the light from Beam 6.  The next ~ 200 scans are the fringe data (light from both telescopes with fringes).  The second set of ~ 200 frames are light from both telescopes but without fringes (carts moved away from the fringe position).  The final three segmented areas used to be another shutter sequence, but are now sky frames where the telescopes are moved off the star and the sky background is recorded.  The sky frames were added to the observing sequence after it was found that the shutters inside the lab contribute thermal heat in the K-band, which is particularly important when observing faint targets.  The last shutter sequence should be checked to make sure that the telescope moved off the star during the sky frames.  If you see light during the last sequence and the data were recorded using the new off-fringe and star sequence (e.g., the fits header keyword CC_SEQ = 'OFF_FRG_AND_STAR'), then you will need to run redfluor using the -A flag to use only the first shutter sequence instead.

Photometry and Kappa Matrix.


Hitting enter in the terminal window will bring up the next window showing the power spectra for the dark frames, fringe frames, telescope A, and telescope B.  In the example below, the fringe peak shows up at 160-240 Hz.  Peaks in the dark frames would indicate the presence of electronic noise, whereas peaks in the shutter A or shutter B frames would likely indicate an oscillation of the telescope (e.g., the small peak at ~ 260 Hz in shutter B).

Power Spectra.


Hitting enter in the terminal window will bring up the next window showing the off-fringe power spectra (light from both telescopes but without fringes), the signal/noise estimate, and the noise subtracted power spectrum of the signal.  The noise subtracted power spectrum will also show outlines of the fringe integration regions selected by the Gaussian fit to the fringe peak.

Noise Subtracted Power Spectra.


By default, the integration range is set by fitting a Gaussian to the fringe power spectrum peak.  If you want to select the integration region by hand, then set the -i flag.  Or if you want to define a fixed integration region for all stars in the data set, use the -I[start-stop] flag.  The automated integration region will be output to the terminal:

# Vel = 413.9 Lambda = 2.1 BP = 194.0 (193.0, 195.1)
# To get this integration range use
# -I146.82-241.30

Next is a plot of the low pass filter used to normalize the fringe signal.

Low pass filter.


Next the routine applies the sigma clipping algorithm if the -o flag is set.  The results from the outlier rejection are displayed.  If any data points are rejected they will be removed from the plot on the right.

Outlier Rejection.

 

# Calculating .........
# Rejected 0 scans due to low photometry.
# Rejected 0 scans due to being too close to the edge.
# Rejected 0 scans due to low weight.

The next plot will show histograms of the correlation and the fringe weights, the fringe waterfall, and the power spectra waterfall.

Histograms of the correlation and fringe weights.

 

# Results from FLUOR PS calculation method:
N_SCANS    201
#               Mean    Stddev
FRINGE_WEIGHT    17.0262    16.687727
#                  Detector 1     Detector 2      Combined
#               Mean    StdDev  Mean    StdDev  Mean    StdDev
Vg        404.500    14.914    407.346    14.683    405.985    14.860
T0_SCANS        22.8        23.1        22.8
T0_500NM        4.0        4.1        4.0
V2_SCANS    0.23061    0.09691    0.22202    0.10360    0.20814    0.09328
V2_CORR        -1.28396
V2_CHI2        0.00161 0.00368
V2_SQRT        0.48022    0.20180    0.47119    0.21987    0.45622    0.20446
V_SCANS        0.46840    0.10612    0.45678    0.11592    0.44328    0.10813
V_NORM        0.46930    0.16916    0.45788    0.18314    0.44437    0.17075
V_LOGNORM    0.47060    0.09563    0.45978    0.10306    0.44595    0.09627
V2_MEDIAN    0.15592    0.09842    0.14831    0.09635    0.13216    0.09607
V_MEDIAN    0.39487    0.12535    0.38511    0.14176    0.36354    0.13971
Move on, toggle Log, zoom in, zoom out or zoom = 1 (m/l/z/Z/1)? 
 


The results for a variety of visibility estimators are printed to the screen.  Hitting enter in the terminal window will bring up the final window showing the normalized mean power spectrum of the signal and the noise on the left and the noise-subtracted mean power spectrum on the right.  In addition to the visibility estimators listed above, the code will also compute the visibility from the mean power spectrum rather than taking the mean of the visibilities computed from individual scans as done above.  This can improve the noise subtraction, particularly for low S/N data.  The uncertainty in the visibility of the mean power spectrum is computed through a bootstrap approach.  The bootstrap loop can take a few minutes to run, so if you are not interested in computing the visibility from the mean power spectrum, you can turn off this visibility estimator by setting the -m flag.  The mean power spectrum computation also includes a linear fit to remove any excess noise to flatten the background in the noise-subtracted mean power spectrum.  This feature may not work correctly if there are noise peaks near the fringe integration window.  This excess noise correction can be turned off using the -N flag.  Another useful flag is -U which can be used to set the DC suppression frequency so that frequencies below the specified value are not included when scaling and correcting the noise.

The plot on the left shows the mean power spectrum for the fringe signals and the noise.  The plot on the right shows the noise-subtracted mean power spectrum.  The vertical red lines show the fringe integration region.  Note that for the example observations on 2018_05_11, we ran redfluor using the -N flag to turn off the excess noise correction because the noise spikes near the fringe window corrupted the fit. 

 

Finished computing visibility from mean power spectrum.
Hit enter to continue.
Calculating bootstrap uncertainties for mean PS... V2_MEAN_PS 0.21835 0.00855 0.20785 0.00939 0.19931 0.00821


After the bootstrap loop finishes, the results from the mean power spectrum are printed to the screen.  The final results from all of the visibility estimators are output to the screen and stored in the .info files in the directory created for each data file.  A few of the most useful visibility estimators include:

  • V2_MEAN_PS: Visibility computed from the mean power spectrum.  This can improve the noise subtraction, particularly for low S/N data.
  • V2_SCANS: Weighted mean of the visibilities computed from each scan individually.
  • V_LOGNORM: Visibilities computed assuming log-normal statistics which can better represent changes in correlation caused by atmospheric turbulence.

The three sets of numbers listed for each estimator give the visibility and error recorded for pixel 1, pixel 2, and the difference signal.  The default is to use the difference signal, however, comparing the values obtained individually from the two readout pixels could provide an estimate of systematic offsets. 

After running through all of the data files for a given sequence of measurements, it is often useful to compare the integration ranges determined for each file.  This can be done by running the command line routine "extractir INT_RANGE" in the results directory.  Extractir will pull the values for the specified keywords from the .info files in each sub-directory.  You might decide to process all data files in a given sequence using a fixed integration range using the -I[start-stop] flag in redfluor.

Calibration

After reducing the data, the calibrator stars are used to calibrate the system visibility (loss in coherence caused by the atmosphere and the instrument) and correct the visibilities measured for the science target.  There are three options for running the CLASSIC calibration software:

  • calibir - This calibrates the science data based on the nearest neighbor calibrators in each Cal1-Obj-Cal2 sequence.
  • calibirgtk - A user-friendly interface for calibir that allows the user to select which stars are calibrators and objects and automatically looks up the angular diameters of calibrators through JMMC.
  • calibir_linfit: A version of calibir that computes a linear fit to the calibrator visibilities.  This is useful when observing in a Cal1-Cal2-Obj-Cal1-Cal2 sequence.  It also allows the user to define breaks in the calibration sequence when alignments were done.

Each of these routines is described in more detail below.

 

calibir

The calibir program reads the INFO files in the processed data directories at the current location.  The routine calibrates the science target using the nearest neighbor calibrators observed before and after the science target.  If the alignment changes between a calibrator and neighboring science target, then these observations should be moved to different directories to avoid calibrating across a jump in the system visibilities (or use the -J flag to define a time range for the calibration).  If NIRO alignments occurred between two calibrators, then it should be OK to calibrate all of the data in one directory since calibir uses only the nearest neighbors.

A common way to call calibir is as follows:

calibir -F -i -s1.1368-0.1069,1.0644-0.1001 -BV2_MEAN_PS HD_141477 HD_139087 HD_142244 

where -F saves the output to an OIFITS file based on the name of the object, -i selects objects based on the "ID" name listed in the INFO file, -s sets the calibrator diameters (uniform disk diameters in the K-band of θ = 1.1368 ± 0.1069 mas for HD 139087 and θ = 1.0644 ± 0.1001 mas for HD_142244 estimated by JMMC/Searchcal), and -BV2_MEAN_PS sets the visibility estimator to "V2_MEAN_PS" for both the target and the calibrators.  After the flags, the science target ID is listed, followed by a listing of the calibrators in the same order as their diameters in the -s flag.  Calibir outputs a plot of the calibrator visibilities (green, blue points), and the raw (red) and calibrated (yellow) visibilities of the science target.  In this example sequence, a NIRO alignment was done after every C1-O-C2 set.

Calibir calibration plot.


By default, the calibrated data are saved to an oifits file with the name 2018_05_11_HD_141477_001.fits. The oifits files can be fit using various data analysis software or read directly into programming languages like IDL.

The full listing of options available in calibir are listed below:

$ calibir -V
Version: V3.1 Mon Jun 10 09:51:29 PDT 2019
usage: calibir [-flags] {OBJ CAL1 CAL2 CAL3...}
Flags:
-b[beta]        Visibility multiplier (1.0)
-B[Vis Type]        Set Vis estimator for Object and Calibrator (V2_SCANS)
-c            Use CHARA number for identifier (OFF)
-C[Cal Vis Type]    Set Vis estimator for Calibrator (V2_SCANS)
-D[spec chan]        Change spectral channel 
-f[oif]            Set OIFITS filename (From object)
-F            Toggle saving OIFITS file (OFF)
-h            Print this message
-H            Use HD number for identifier (OFF)
-i            Use ID/Name for identifier (OFF)
-J[mjdmin,mjdmax]    Restrict MJD range (All)
-l            Assumes a linear variation of the cals (OFF). Otherwise also takes into account the cal errors
-n            Use standard error instead of standard deviation (ON)
-N[min_n_Scans]        Minimum number of scans (0)
-o            Invert the sign of the UV coords (OFF)
-O[Obj Vis Type]    Set Vis estimator for Object (V2_SCANS)
-p[detector chan]    Change detector channel (0=diff, 1, 2, 3=weighted mean) 
-P{dev}        Toggle plotting, or set device (ON/xwin)
-r            Print raw data (ON)
-s[diam1-err,diam2-err,...]    Size of calibrators in mas (0.0)
-t            Includes variability of the co-Transfer in error.
-S            Self Calibrate mode (OFF)
            If error left out it is set to zero.
-V            Verbose mode (OFF)
-2            Output V^2 table based on V estimator (ON)

Visibility estimators available are:
V_CMB V_FIT V_ENV V_MEAN_ENV_PEAK 
V_MEAN_ENV_FIT BINARY_V_A BINARY_V_B BINARY_ENV_V_A 
BINARY_ENV_V_B V2_SCANS V_SCANS V_NORM 
V_LOGNORM V2_MEDIAN V_MEDIAN V2_SCANS_DIR 
V_SCANS_DIR V_NORM_DIR V_LOGNORM_DIR V2_MEAN_PS 
 

If using V2_MEAN_PS as the visibility estimator, then the -n flag (use standard error instead of standard deviation) is automatically turned off in the software. This is because the uncertainty in V2_MEAN_PS is determined from the standard deviation of the bootstrap distribution, so there is no reason to divide by the square-root of the number of scans.

 

calibirgtk

Running "calibirgtk" will open a graphical user interface listing all of the calibrators and objects observed on a given night.

calibirgtk interface.

The user can select which stars to use as calibrators, which to use as the science object, or whether to ignore the star.  The routine will query the JMMC catalog to look up the estimated angular diameters for each star.  These values can also be input by hand into the user interface.  Some of the most common visibility estimators (SCANS, LOGNORM, MEDIAN, MEAN PS) are available for selection, as well as whether to use the difference signal, the pixel 1 signal, or the pixel 2 signal.  The MJD range can be set if you want to limit the calibration to a smaller subset of the data.  Clicking the "SAVE OIFITS" button will save the output to an oifits file.

When the user clicks the "RUN CALIBIR" button, the code will call calibir based on the options selected in the GUI.  If the user clicks the "OIFUD" button it will open a dialog box to select an oifits file and run the oifud program to compute a fit for a uniform disk diameter (see below).

 

calibir_linfit

The calibir_linfit routine computes a linear fit to the calibrator visibilities when determining the system visibility.  This is useful when observing in a Cal1-Cal2-Obj-Cal1-Cal2 sequence, as it will include all of the calibrator observations in the fit (whereas calibir uses only the nearest two calibrators).  It also allows the user to define breaks in the calibration sequence when alignments were done.  Calibir_linfit uses the same options as calibir.  An example of how to call the program is as follows (see description of calibir for an explanation of the flags):

calibir_linfit -i -F -BV2_MEAN_PS -s0.3189-0.0078,0.2553-0.0063 HIP_51317 HD_87301 HD_88725 

This will open up a plot showing the raw visibilities of the object and the calibrators.

First plot in calibir_linfit where the user can specify breaks in the calibration sequence.


The code will then ask the user if they want to enter a break in the alignment sequence. The user can enter one or more time breaks and a vertical line will appear in the visibility plot at the time of each break.

Do you want to want to define an alignment sequence (y/n)?
y
Enter time of alignment in XX.X hours:
13.2
Time of break 13.20
Do you want to want to define another alignment sequence (y/n)?
n
Hit return to continue.
Number of time breaks: 1
Time break: 0
tmin: 12.27,  tmax: 13.20
Number of CAL observations 6 
Number of OBJ observations 3 
Time break: 1
tmin: 13.20,  tmax: 13.87
Number of CAL observations 4 
Number of OBJ observations 2 
 

The code will then compute a linear fit to determine the system visibility as a function of time for each calibration sequence.  It will compute the calibrated visibility of the science target based on the system visibility at the time of science observation based on the linear fit.  It will then open a plot showing the calibrated visibilities, print a summary of the results to the screen, and save the results to an oifits file (if the -F flag is set).

Second plot in calibir_linfit that shows the calibrator visibilities corrected for their angular diameters (Tcal1, Tcal2, etc), the raw visibility of the target (Raw), the system visibility computed at the time of the science observation (Tobj) based on the linear fit to the calibrator visibilities, and the calibrated visibility of the science object (Cal).


OIFITS Utitlities

The calibrated oifits files can be analyzed using various data analysis software or read directly into programming languages like IDL.

There are a few oifits utilities included in the reduceir package that can be used to manipulate, view, and analyze data.  For instance, oifits-merge can be used to merge multiple OIFITS files on the same target (e.g., merge observations from different nights) and oifud can be used to fit simple geometries like a uniform disk to the visibility data:

oifud 2018_05_11_HD_141477_001.fits 

 

oifud interface.

 

Uniform disk fit.


Recent Revisions:

  • 2018Feb: Use weighted mean of the visibilities computed from the individual scans.  The fringe weights are computed from a ratio of the power on and off of the fringes.  The option to use the weighted mean is ON by default and can be turned OFF using the -M flag.

  • 2019Feb: Modification to prevent negative visibilities from being produced by background subtraction of low S/N data.  This option uses the mean power spectrum of the noise scaled to the low frequency region of the fringe power spectra of each individual scan.  It is ON by default and can be turned OFF using the -Z flag.

  • 2019May: Compute the visibility from the mean power spectrum which will have higher signal than the power spectra of the individual scans.  This can improve the noise subtraction, particularly for low S/N data.  The uncertainties for the mean power spectrum are computed through a bootstrap approach.  The mean power spectrum visibility estimator can be turned off using the -m flag (for faster run times).  This feature also includes a linear fit to the background around the fringe peak to remove any excess noise.  The excess noise correction can be turned off using the -N flag.  The -U flag can be used to set the DC suppression frequency to remove low frequencies from the fit.  A detailed description of the mean power spectrum method and examples of how to use this feature are included in the following technical report.