I'm ready to advance to a highly technical and in-depth discussion on processing Spitzer data.
STILL UNDER DEVELOPMENT... working on it
The basic idea here is the same as the basic idea for making a simple mosaic. The difference here is the following: whereas before you were combining products from our pipeline (which takes into account a variety of things, nearly all of which are instrumental artifacts), now you are starting from individual frames, so it becomes much more of your responsibility to take into account instrumental artifacts. You need to tell MOPEX not just the individual frames and errors, but also the individual mask files, including masks for permanently damaged pixels (which change with time as the telescope ages) and masks for instrumental artifacts (like cosmic rays and scattered light). This process has MANY MORE buttons and dials than the prior, simpler process - which is good once you're an expert, because you can remove the artifacts and leave a great image. But all these options also means you can really easily shoot yourself in the foot if you are not careful.
For IRAC, to do this properly, for most star-forming regions, not only do you have 4 channels, but you also have a short and a long exposure (from the HDR setting in the AOR) for each pointing. So you have 8 mosaics to make (and artifacts to remove separately from all the components of those 8 mosaics).
For MIPS, you may or may not have to remove the artifacts manually for MIPS-24. The pipeline automatically produces two sets of BCDs for 70 and 160, and you need to decide which you should use, based on your goals for the final image. Artifact removal from these two longest bandpasses can be quite tricky indeed.
- Formal, complete data filenaming conventions for these Spitzer data (not at all sugar-coated)
- Data Handbooks for these Spitzer data (tells you which files are important, but also not at all sugar-coated)
high-level overview, plus general cryptic notes put here until we have time to turn them into something more user-friendly.
File:Processingoverview.doc -- more wordy doc file with more description, waiting to be translated into here.
run sean's artifact mitigation software on irac data. uses idl.
requires a flat directory structure that is just a nightmare!
after it finished: moved all log files to sean_intermed moved final files to sean_final
over_cor_sfx_SPITZER_I[1-4]..._bcd.fits
cor_sfx_SPITZER_I[1-4]..._bunc.fits
SPITZER_I[1-4]..._bimsk.fits
moved all other intermediate files to sean_intermed
mv over_cor_sfx_SPITZER_I1_*_bcd.fits sean_final mv over_cor_sfx_SPITZER_I2_*_bcd.fits sean_final mv over_cor_sfx_SPITZER_I3_*_bcd.fits sean_final mv over_cor_sfx_SPITZER_I4_*_bcd.fits sean_final mv cor_sfx_SPITZER_I1*bunc.fits sean_final mv cor_sfx_SPITZER_I2*bunc.fits sean_final mv cor_sfx_SPITZER_I3*bunc.fits sean_final mv cor_sfx_SPITZER_I4*bunc.fits sean_final mv SPITZER_I1*bimsk.fits sean_final mv SPITZER_I2*bimsk.fits sean_final mv SPITZER_I3*bimsk.fits sean_final mv SPITZER_I4*bimsk.fits sean_final
cp *list sean_final cp *list sean_intermed mv *log sean_intermed
mv *I1*fits sean_intermed mv *I2*fits sean_intermed mv *I3*fits sean_intermed mv *I4*fits sean_intermed
irac_preprocess is still subdir for all code, and make_corrected_bcds.tcsh is still in this main dir.
all *list files got copied to sean_intermed, sean_final.
- list files in this dir need to be updated to point to sean_final.
awk '{printf("sean_final/%40s\n",$1)}' ch1234.list > ch1234.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_long.list > ch1_long.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_long.list > ch2_long.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_long.list > ch3_long.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_long.list > ch4_long.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_med.list > ch1_med.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_med.list > ch2_med.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_med.list > ch3_med.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_med.list > ch4_med.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_long_dmasks.list > ch1_long_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_long_dmasks.list > ch2_long_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_long_dmasks.list > ch3_long_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_long_dmasks.list > ch4_long_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_med_dmasks.list > ch1_med_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_med_dmasks.list > ch2_med_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_med_dmasks.list > ch3_med_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_med_dmasks.list > ch4_med_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_long_sigmas.list > ch1_long_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_long_sigmas.list > ch2_long_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_long_sigmas.list > ch3_long_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_long_sigmas.list > ch4_long_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_med_sigmas.list > ch1_med_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_med_sigmas.list > ch2_med_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_med_sigmas.list > ch3_med_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_med_sigmas.list > ch4_med_sigmas.list.txt
sscsci101% which mosaic.pl mosaic.pl: Command not found. sscsci101% source ../rebull/mopex/mopex-script-env.csh sscsci101% which mosaic.pl /ssc/sci/pleiades/rebull/mopex//bin/mosaic.pl
Generate the FIF as a separate step (not required to be separate, but i am getting strange errors now UNLESS i make it a separate step) if you do this, though, turn off FIF generation for individual mosaics, since it will save you time (and, evidently, errors). edit the file cdf/justfif_gui.nl
mosaic.pl -n justfif_gui.nl
make the mosaic: mosaic.pl -n mosaic_long_sr.nl > mosaic8.log &
maybe useful to make one just of the first year to compare
with the mosaics you made mon/tues:
grep 13375232 ch1_long.list.txt > ch1_long.list235.txt grep 13375232 ch2_long.list.txt > ch2_long.list235.txt grep 13375232 ch3_long.list.txt > ch3_long.list235.txt grep 13375232 ch4_long.list.txt > ch4_long.list235.txt grep 13375232 ch1_med.list.txt > ch1_med.list235.txt grep 13375232 ch2_med.list.txt > ch2_med.list235.txt grep 13375232 ch3_med.list.txt > ch3_med.list235.txt grep 13375232 ch4_med.list.txt > ch4_med.list235.txt grep 13375232 ch1_long_dmasks.list.txt > ch1_long_dmasks.list235.txt grep 13375232 ch2_long_dmasks.list.txt > ch2_long_dmasks.list235.txt grep 13375232 ch3_long_dmasks.list.txt > ch3_long_dmasks.list235.txt grep 13375232 ch4_long_dmasks.list.txt > ch4_long_dmasks.list235.txt grep 13375232 ch1_med_dmasks.list.txt > ch1_med_dmasks.list235.txt grep 13375232 ch2_med_dmasks.list.txt > ch2_med_dmasks.list235.txt grep 13375232 ch3_med_dmasks.list.txt > ch3_med_dmasks.list235.txt grep 13375232 ch4_med_dmasks.list.txt > ch4_med_dmasks.list235.txt grep 13375232 ch1_long_sigmas.list.txt > ch1_long_sigmas.list235.txt grep 13375232 ch2_long_sigmas.list.txt > ch2_long_sigmas.list235.txt grep 13375232 ch3_long_sigmas.list.txt > ch3_long_sigmas.list235.txt grep 13375232 ch4_long_sigmas.list.txt > ch4_long_sigmas.list235.txt grep 13375232 ch1_med_sigmas.list.txt > ch1_med_sigmas.list235.txt grep 13375232 ch2_med_sigmas.list.txt > ch2_med_sigmas.list235.txt grep 13375232 ch3_med_sigmas.list.txt > ch3_med_sigmas.list235.txt grep 13375232 ch4_med_sigmas.list.txt > ch4_med_sigmas.list235.txt
grep 13375232 ch1234.list.txt > ch1234.list235.txt
make a mosaic just of the 235 files.
update a (new copy of a) namelist and invoke it.
mosaic.pl -n mosaic235.nl > m.log
mosaic.pl -n justfif_gui.nl > fif.log mkdir p235pbcd3.6long mkdir p235pbcd3.6short mkdir p235pbcd4.5long mkdir p235pbcd4.5short mkdir p235pbcd5.8long mkdir p235pbcd5.8short mkdir p235pbcd8long mkdir p235pbcd8short cp FIF.tbl p235pbcd3.6long cp FIF.tbl p235pbcd3.5short cp FIF.tbl p235pbcd4.5long cp FIF.tbl p235pbcd4.5short cp FIF.tbl p235pbcd5.8long cp FIF.tbl p235pbcd5.8short cp FIF.tbl p235pbcd8long cp FIF.tbl p235pbcd8short
mosaic.pl -n mosaic235i1l.nl > m1.log mosaic.pl -n mosaic235i2l.nl > m2.log mosaic.pl -n mosaic235i3l.nl > m3.log mosaic.pl -n mosaic235i4l.nl > m4.log
mosaic.pl -n justfif_gui.nl > fif2.log mkdir pbcd3.6long mkdir pbcd3.6short mkdir pbcd4.5long mkdir pbcd4.5short mkdir pbcd5.8long mkdir pbcd5.8short mkdir pbcd8long mkdir pbcd8short cp FIF.tbl pbcd3.6long cp FIF.tbl pbcd3.6short cp FIF.tbl pbcd4.5long cp FIF.tbl pbcd4.5short cp FIF.tbl pbcd5.8long cp FIF.tbl pbcd5.8short cp FIF.tbl pbcd8long cp FIF.tbl pbcd8short
mosaic.pl -n mosaicfulli1l.nl > m1lfull.log mosaic.pl -n mosaicfulli1s.nl > m1sfull.log mosaic.pl -n mosaicfulli2l.nl > m2lfull.log mosaic.pl -n mosaicfulli2s.nl > m2sfull.log mosaic.pl -n mosaicfulli3l.nl > m3lfull.log mosaic.pl -n mosaicfulli3s.nl > m3sfull.log mosaic.pl -n mosaicfulli4l.nl > m4lfull.log mosaic.pl -n mosaicfulli4s.nl > m4sfull.log
grep 13375232 ch1_long.list > ch1_long.list235 grep 13375232 ch2_long.list > ch2_long.list235 grep 13375232 ch3_long.list > ch3_long.list235 grep 13375232 ch4_long.list > ch4_long.list235 grep 13375232 ch1_med.list > ch1_med.list235 grep 13375232 ch2_med.list > ch2_med.list235 grep 13375232 ch3_med.list > ch3_med.list235 grep 13375232 ch4_med.list > ch4_med.list235 grep 13375232 ch1_long_dmasks.list > ch1_long_dmasks.list235 grep 13375232 ch2_long_dmasks.list > ch2_long_dmasks.list235 grep 13375232 ch3_long_dmasks.list > ch3_long_dmasks.list235 grep 13375232 ch4_long_dmasks.list > ch4_long_dmasks.list235 grep 13375232 ch1_med_dmasks.list > ch1_med_dmasks.list235 grep 13375232 ch2_med_dmasks.list > ch2_med_dmasks.list235 grep 13375232 ch3_med_dmasks.list > ch3_med_dmasks.list235 grep 13375232 ch4_med_dmasks.list > ch4_med_dmasks.list235 grep 13375232 ch1_long_sigmas.list > ch1_long_sigmas.list235 grep 13375232 ch2_long_sigmas.list > ch2_long_sigmas.list235 grep 13375232 ch3_long_sigmas.list > ch3_long_sigmas.list235 grep 13375232 ch4_long_sigmas.list > ch4_long_sigmas.list235 grep 13375232 ch1_med_sigmas.list > ch1_med_sigmas.list235 grep 13375232 ch2_med_sigmas.list > ch2_med_sigmas.list235 grep 13375232 ch3_med_sigmas.list > ch3_med_sigmas.list235 grep 13375232 ch4_med_sigmas.list > ch4_med_sigmas.list235
cp *list235 COMEGETME cp p235pbcd3.6long/Combine/mosaic.fits COMEGETME/p235i1long.fits cp p235pbcd4.5long/Combine/mosaic.fits COMEGETME/p235i2long.fits cp p235pbcd5.8long/Combine/mosaic.fits COMEGETME/p235i3long.fits cp p235pbcd8long/Combine/mosaic.fits COMEGETME/p235i4long.fits cp p235pbcd3.6long/Combine/mosaic_cov.fits COMEGETME/p235i1long_cov.fits cp p235pbcd4.5long/Combine/mosaic_cov.fits COMEGETME/p235i2long_cov.fits cp p235pbcd5.8long/Combine/mosaic_cov.fits COMEGETME/p235i3long_cov.fits cp p235pbcd8long/Combine/mosaic_cov.fits COMEGETME/p235i4long_cov.fits cp p235pbcd3.6long/Combine/mosaic_unc.fits COMEGETME/p235i1long_unc.fits cp p235pbcd4.5long/Combine/mosaic_unc.fits COMEGETME/p235i2long_unc.fits cp p235pbcd5.8long/Combine/mosaic_unc.fits COMEGETME/p235i3long_unc.fits cp p235pbcd8long/Combine/mosaic_unc.fits COMEGETME/p235i4long_unc.fits
p pbcd3.6long/Combine/mosaic.fits COMEGETME/i1long.fits cp pbcd4.5long/Combine/mosaic.fits COMEGETME/i2long.fits cp pbcd5.8long/Combine/mosaic.fits COMEGETME/i3long.fits cp pbcd8long/Combine/mosaic.fits COMEGETME/i4long.fits cp pbcd3.6long/Combine/mosaic_cov.fits COMEGETME/i1long_cov.fits cp pbcd4.5long/Combine/mosaic_cov.fits COMEGETME/i2long_cov.fits cp pbcd5.8long/Combine/mosaic_cov.fits COMEGETME/i3long_cov.fits cp pbcd8long/Combine/mosaic_cov.fits COMEGETME/i4long_cov.fits cp pbcd3.6long/Combine/mosaic_unc.fits COMEGETME/i1long_unc.fits cp pbcd4.5long/Combine/mosaic_unc.fits COMEGETME/i2long_unc.fits cp pbcd5.8long/Combine/mosaic_unc.fits COMEGETME/i3long_unc.fits cp pbcd8long/Combine/mosaic_unc.fits COMEGETME/i4long_unc.fits
cp pbcd3.6short/Combine/mosaic.fits COMEGETME/i1short.fits cp pbcd4.5short/Combine/mosaic.fits COMEGETME/i2short.fits cp pbcd5.8short/Combine/mosaic.fits COMEGETME/i3short.fits cp pbcd8short/Combine/mosaic.fits COMEGETME/i4short.fits cp pbcd3.6short/Combine/mosaic_cov.fits COMEGETME/i1short_cov.fits cp pbcd4.5short/Combine/mosaic_cov.fits COMEGETME/i2short_cov.fits cp pbcd5.8short/Combine/mosaic_cov.fits COMEGETME/i3short_cov.fits cp pbcd8short/Combine/mosaic_cov.fits COMEGETME/i4short_cov.fits cp pbcd3.6short/Combine/mosaic_unc.fits COMEGETME/i1short_unc.fits cp pbcd4.5short/Combine/mosaic_unc.fits COMEGETME/i2short_unc.fits cp pbcd5.8short/Combine/mosaic_unc.fits COMEGETME/i3short_unc.fits cp pbcd8short/Combine/mosaic_unc.fits COMEGETME/i4short_unc.fits
tim found the overlap correction had gone bad in ch3. redoing, using the SSC's overlap corrector. overlap.pl -n overlap_long_sr.nl > overlap.log
