DATA SLURPING COOKBOOK
version 18 MAY 2009 by Jen Winters and John Subasavage
This is used to retrieve data from the 0.9m at CTIO and to do basic bias subtraction and flat-fielding.
**********************************************************************
***** YOU BEGIN BY SLURPING DATA FROM CTIO *****
**********************************************************************
Step 1: (in one terminal window)
A) cd /nfs/recons3/SMARTS
B) mkdir YYYYMMDD--- call the directory by a eight character string of
numbers that corresponds to the observation date, e.g. 20110922
Step 2: (in a second terminal window)
A) Type in 'huygens' to ensure that you are on the machine huygens,
which is the only one cleared for entry to computers at CTIO.
B) Type ssh v12@ctio36.ctio.noao.edu and use our CTIOPI password to
get in. The data you want are typically found in a directory with
format YYYYMMDD-09.
C) v12% will be displayed on the screen; you are now using unix.
D) Type "ls" to see where you are and verify that the data you are
looking for are there.
If you don't find the data there, try these places on the ctio36
machine:
ssh -l v12 ctio36.ctio.noao.edu
/tmp_mnt/usr/remote/u363/v12/MIRANDA
ctio36
/usr/remote/u362/iraf/v12/MIRANDA
/u362/iraf/MIRANDA
/ua45/v12/YYYYMMDD.09
***** INSERT SMARTS STEPS NOTED AT END OF FILE IF YOU ARE DOING SMARTS *****
E) Change directories into the one that you want by typing cd YYYYMMDD-09.
F) Type 'ls' to see what's in that directory; there should be files
ending in ".imh" and ".fits".
G) To see how many entries you will slurp, do a word count by typing
ls *.fits | wc
H) Type rsync -avz *.fits username@ssh.chara.gsu.edu:/nfs/recons3/incoming.0.9m/YYYYMMDD/
i.e., rsync -avz *.fits winters@ssh.chara.gsu.edu:/nfs/recons3/incoming.0.9m/20110922/
I) You will be asked for login name, use your chara account and
password; that will bring you into your home directory.
J) Logout to come back to GSU and check data.
K) When it finishes, if you "ls", it will give you a directory listing.
L) Check the file sizes to make sure they are all the same by typing
ls -l *.fits
M) Type ls *.fits | wc to make sure that all the files were
successfully transferred.
**********************************************************************
***** PROCESS THE DATA AT GSU *****
**********************************************************************
A) cd into the file that you created for the slurping process
(i.e. /nfs/recons3/SMARTS/20030327) type in "xgterm -sb &". In
xgterm, and type "cl".
A1) Sometimes the observer will name the .fits files with *.CTIOPI.*
instead of *.09.*. If this is the case, you will need to rename
the image files.
A2) To do this, type 'ls *.fits >> old'. This will create a file
called 'old' that contains all the slurped image files. copy the
'old' file into another file called 'new'. Then edit the 'new'
file by replacing CTIOPI with 09.
A3) Use 'imrename' in iraf to change all image files names: 'imrename
@old @new'.
B) type "fixname"; it will ask you for the root name; the root name is
the string of eight characters that is the date you are working on
(ie, 20070303). This step will generate a list; wait until it's
done to go to next step.
C) type "gof"
D) type "proc". You will then be asked to enter the bias section; trim
section, ccd readout noise, and ccd gain in turn; the numbers
should be provided. (The numbers in each column are identical all
the way down).
Make sure that the numbers given for you to enter match the ones in
the column. If they do, hit enter; otherwise, enter the numbers
manually in the form specified (including the brackets), and then
hit enter.
Then it will say "now reducing V band star images", "now reducing R
band star images", "now reducing I band star images". Once the
whole process is finished, you will see a "cl>" appear; this lets
you know that you're done with this set of data.
E) "ccdlist *.fits" to make sure that the filter is correct and
included.
**********************************************************************
***** SORT DATA INTO DIRECTORIES *****
**********************************************************************
PHOTOMETRY DIRECTORIES (/nfs/recons3/phot.0.9m/)
A) If there is a photometry night, please make sure that you create a
directory for that night and then COPY all frames into that
directory BEFORE you sort the frames into the parallax directories.
A) If there are new objects and the frames are photometry frames,and
you don't know what the parallax filter is, just copy them to the
photometry directory (no need to create a new parallax directory
for them. A parallax filter probably has not yet been selected for
them.).
ARALLAX DIRECTORIES (/nfs/recons4/CTIOPI/regions/)
A) hselect *.o.fits 'ra, $I, object' yes >> sort.file
B) If there are objects that are not ours (they are not on the
observing lists): EXO planet data go into exoplanet.dir under
special targets; BIN data go into the binary directory under
special targets.
C) Then, after you have sorted all the non-parallax frames where they
are supposed to go, type 'astrometry.processor sort.file'. This
will sort all the frames into the parallax directories.
D) If the directory does not exist, the code will ask you if you want
to create the directory and then will sort it into that newly
created directory if you reply 'yes'. If you reply 'no', it will
ask you for an alternative directory destination. If you simply
hit enter, it will do nothing.
E) The code might also complain if the name of the directory contains
capital letters. If this is the case, simply rename the directory
so that all letters are lower case.
Once the frames have been sorted, we move the entire directory (with
the raw frames only -- no .o.fits files should be left in it, unless a
photometry night was begun and then scrapped) to
/nfs/whiting1/CTIOPI/SMARTS.raw.2007
**********************************************************************
***** UPDATE THE REDUCTION LOG *****
**********************************************************************
(/nfs/morgan3/thenry/public_html/CTIOPI/protected.dir/master.astr.status)
**********************************************************************
***** SMARTS SPECIFIC STEPS *****
**********************************************************************
3) Check to see that the data are in the expected directory. This
step can usually be omitted as the service observer always stores
the data here but if there is ever a problem with the next step,
come back to this one.
cd /ua45/v12/"YYYYMMDD"-"PI NAME"/ (i.e., 20090402-Ciardi)
4) Return to original directory and run smartsproc script.
cd
./smartsproc (the './' is not explicitly necessary)
5) Answer questions using all lower case.
PI name: ciardi
Date (YYYYMMDD): 20090402
6) If it doesn't barf, all is well. Log into IRAF from the current directory.
cl
7) Change directory to where the input and output file names were
written (and where the fits data will be stored).
cd /u363/v12/SMARTS/"PI NAME"/"YYYYMMDD"/
(i.e., /u363.etc/ciardi/20090402/)
wfits @infile @outfile
8) Once all data are written to fits, generate an info file
hselect *.fits $I,object,filters yes > "YYYYMMDD"."PI NAME".inf
(i.e., 20090402.ciardi.inf).
9) Exit IRAF, go into directory data are stored, and ftp back to GSU server
logout
cd /u363/v12/SMARTS/"PI NAME"/"YYYYMMDD"/
rsync -avz 2009* "your
username"@ftp.chara.gsu.edu:/ftp/pub/SMARTS/"PI
NAME"/"YYYYMMDD"/
(i.e., rsync -avz 2009*
subasavage@ftp.chara.gsu.edu:/ftp/pub/SMARTS/ciardi/20090402/)
(NOTE: if the PI name does not exist on the GSU server, this will
barf. You'll need to log onto joy and change directory to
/ftp/pub/SMARTS/ and add the PI name in that directory).
Check the slurping process every few minutes as you will undoubtedly
lose the link and have to kill the process and up arrow to restart the
rsync command.
That's it!