C-WAYS source matching work

Big Picture Introduction -- an analogy that might be too fanciful?

When westerners first discovered the Americas, they had largely set out with the goal of finding gold or other treasure. They were coming at the new continent from the perspective of someone in a boat, with minimal information about what the landforms really were, except for what they could see with their own eyes. Their maps look strange to those of us used to seeing images of these landforms from space, but we have a whole lot more information now than they did then.

The first thing that these early Western explorers were able to attempt to map was the coasts, because that's what they had the most information about...and the most immediate need to know. They needed to know where coral reefs were that might damage their ships, and where the big rivers emptied into the sea so that they could take on more fresh water. They also could learn about more land faster when boating up the rivers rather than walking.

As more and more boats explored the coasts, the maps got better, but they still seem distorted compared to the landforms we know today. In some of these early maps of the Americas, when Europe and Africa were included, even the African coast on the Mediterranean side doesn't look all that realistic, compared to what we know now.

As the westerners pushed further and further into the land (in the Americas or for that matter Africa) from the coasts, their knowledge deepened about what the continent actually looked like, aided by improvements in technology (such as more accurate ways of measuring longitude). Their knowledge of the land started in clumps around the rivers, again because that's what they needed that best enabled them to travel the furthest. But, their knowledge expanded as fast as they could expand. And their goals changed too -- certainly some were still looking for treasure (or freedom from persecution, religious or otherwise), but more in the earliest years were just trying to survive (here I'm thinking of Jamestown or Roanoke). They explored to find more food to eat (critters or plants).

The Native Americans, of course, had a perfectly good understanding of what their land looked like, but even so, most likely, one tribe only knew the land near them -- my guess is that the Powhatan tribe (in VA) had no idea whatsoever what the Sioux tribe's lands looked like, even if trade routes were such that items could move from the Dakotas to Virginia. But the Native Americans were observing the land in a different way, having lived there for a while and having their own methods of exploration. Once westerners realized that they could learn from the Native Americans (here I'm thinking of Lewis and Clark), their knowledge could expand even faster once they paid attention to what was already known.

There were, at nearly every stage of these early maps, regions that were sort of hazy and unexplored, e.g., "here be dragons". Someone might have a vague idea of what was there, but maybe only on the edges; no one (at least no one known to the map maker) had any detailed knowledge of what was there.

SO NOW.... here is a perhaps tortured analogy... We would like to go exploring in a particular region, making note of where the big landforms are, and we have a specific goal of finding edible animals and plants to support our efforts in further explorations. Some people have spent some time exploring parts of this region before. It will make our knowledge expand faster if we pay attention to what is already known before setting off on our own journey. Others have mapped different parts of the region using different methods of exploration before -- some on foot and some on horseback, and still others in boats. Some people just ran through this region identifying the big trees and big rocks. Some people wrote down what they learned in Algonquin (what the Powhatan spoke) and we need to translate it before it makes sense in the language we speak. Some people cared a lot about their tiny camp next to the river and they know that region really well, but beyond the borders of their camp, "here be dragons." There is some information about the area around the camp that we can obtain from other people and from exploring on our own, but we can also come back to this very well-known region and check what we think we know about the rest of the area by testing it on the well-known region. The people who know the region right next to the river really well also know that this animal or that plant is good food and won't make us sick. When we go exploring out further from the camp, if we find a critter or a plant that we think is the same as the stuff that the people next to the river know is ok to eat, we can bring it back to that camp to compare it and see if it is, in fact, the same or something new. We can also, among the animals and plants we find, put them in groups of apparently similar things -- these are all 4-footed furry critters, and those have feathers and wings.

Explicitly drawing lines between that analogy and reality:

analogy	reality translation
We would like to go exploring in a particular region, making note of where the big landforms are, and we have a specific goal of finding edible animals and plants to support our efforts in further explorations.	We have a goal of exploring a region (well, really 3) on the sky, specifically looking for young stars.
Some people have spent some time exploring parts of this region before. It will make our knowledge expand faster if we pay attention to what is already known before setting off on our own journey.	We need to read and understand the literature.
Other people have mapped different parts of the region using different methods of exploration before -- some on foot and some on horseback, and still others in boats.	Other people have used different wavelengths to explore this region before.
Some people just ran through this region identifying the big trees and big rocks.	Some people just identified the bright young stars, or the things bright in the wavelengths they were using.
Some people wrote what they learned down in Algonquin (what the Powhatan spoke) and we need to translate it before it makes sense in the language we speak.	Some people wrote down poorly constrained coordinates in epoch B1950 coordinates and we need to translate it to accurate J2000 coordinates.
Some people cared a lot about a tiny patch next to the river and know that region really well, but beyond the borders of their camp, "here be dragons."	The NITARP team last year spent their year worrying about the 5'x5' patch with 4-band IRAC coverage, but did not care about anything else.
There is some information about the area around the camp that we can obtain from other people and from exploring on our own...	We can comb the literature and use the 2MASS+WISE data to help guide us in exploring the region.
...but we can come back to this very well-known region and check what we think we know about the rest of the area by testing it on the well-known region.	We can use WISE to identify things with YSO-like colors in the region. Did we rediscover the YSOs that other people found, particularly last year's team using Spitzer data? If not, why not? Do the objects with YSO-like colors look like point sources in 2MASS or do they look like galaxies?
The people who know the region right next to the river really well also know that this critter or that plant is good food and won't make us sick. When we go exploring out further from the camp, if we find a critter or a plant that we think is the same as the stuff that the people next to the river know is ok to eat, we can bring it back to that camp to compare it and see if it is, in fact, the same or something new.	We have a region of space that is very well studied with Spitzer, and serendipitous Spitzer data nearby. We will use WISE over the whole region to find things that we think might be YSOs. We can check our hunch that some of the objects are in fact YSOs by comparing what we get to the Spitzer data where we can, and including that data in our analysis.
We can also, among the animals and plants we find, put them in groups of apparently similar things -- these are all 4-footed furry critters, and those have feathers and wings.	Among the objects we find, we can put them in groups based on the shape of their SED, from 'really embedded' (class 0-I) down to 'not much of an IR excess' (class II-III).

We will not be able to get a comprehensive be-all-end-all understanding of the region (e.g., in the analogy, we will not go straight from Columbus or even Lewis and Clark to weather satellite views of the continent with a GPS in our car as we drive). We can, however, do the best that we can with the information we have, by learning from those who have gone before, learning as we go, and making intelligent guesses about what we don't know.

And, of course, we need to do this in all of the regions we care about.

More specific introduction to source matching from the literature

Several people have done prior studies in BRC 27, BRC 34, and BRC 38 before, but they have NOT found all the young stars! You worked hard to find all of these articles in the context of the proposal. Certainly investigator A working in BRC 27 in year X saw some of the same sources as investigator B working in that same region in year Y, as did investigator C in year Z. Now we actually have to do the work of figuring out which specific sources are which in all the papers - are the sources called out in paper 1 the same or different sources as paper 2?, etc., until all the papers are exhausted, and we have a complete catalog of all the previously studied sources in the region.

The thing that makes this complicated is that, even though everyone is reporting in RA and Dec, not everyone is using exactly the same system (some are 1950 coordinates and some are 2000 coordinate), and not everyone has the same coordinate accuracies (some are working off of photographic plates, and some are working off large-format CCDs). And, what does it mean to have "the same" coordinates -- is within an arcsecond ok? 5 arcseconds? an arcminute? This is where it gets tricky, and where you have to apply your brain! Spitzer, WISE, and 2MASS are all using exactly the same, high-accuracy coordinate system -- it's all tied to 2MASS's J2000 coordinates -- but even then the position of the same object will not be EXACTLY the same in each image, in each catalog, because there is a limit to the precision with which we can identify the coordinates. Where possible, we need to update the old coordinates by comparing what the old papers say to the 2MASS data. Then we need to fold in the objects with newer coordinates into our collection of sources.

Part of the challenge here is bookkeeping -- writing down coordinates correctly, keeping track of which sources are which, and getting the correct data matched to the correct source.

Last year, I thought this would be a relatively simple project that could be done before the summer visit. However, it turned out not to be the case. Part of this is, I think, the complexity of the region, BRC 27 in particular (which is of course still a problem this year), and part of it was I think my ability to explain it! I've pulled out and updated all of my best(?) explanations and descriptions here. IF IT DOESN'T MAKE SENSE, PLEASE ASK QUESTIONS. If this is done wrong, or only halfway done, it will make for a LOT of pain downstream. Trust me.

1950 vs. 2000 Coordinates and Yet Another Analogy

Review the differences between the 1950 and 2000 coordinate systems. REMINDER: THE STARS ARE NOT MOVING. (Several people last year kept saying things like "wow the stars move a lot" and if you say that, I WILL correct you!) The coordinate system is moving, not the stars. (Well, technically the stars may really be moving, but we don't have that information, and the size of any such motion will be on the order of a tenth of an arcsec or less; the change due to the change in coordinate systems is much greater.)

An analogy can provide an example. Through Google Maps, I can see that there is a Baskin Robbins near Ms. Linahan's school (from the NITARP 2011 class). I can give you the position of that Baskin Robbins in any of a number of ways:

• 346 North Lake St, Mundelein, IL
• 8 long blocks roughly west of the school
• 4 minutes west of the school (if you drive)
• 20 minutes west of the school (if you walk)
• latitude 42.269711, longitude -88.004308

Or, I could be really pathological and/or vague and say:

• the 3rd oldest Baskin Robbins in Mundelein (NB: i'm just making this up)
• the 5th oldest ice cream store in Munelein (NB: i'm just making this up)
• An ice cream store on Lake, south of Loch Lomond

Are the coordinates different in these systems? Yes, but they are pointing to the same physical location. Is the Baskin Robbins really moving? No.

That is what is going on in these BRCs. The most obvious difference is between B1950 coordinates and the J2000 coordinates -- these are different coordinate systems, and we are trying to translate one into the other, but we are also trying to understand the intrinsic errors of the earlier studies, and figure out which object they were 'really' talking about. In our example above, we want to translate each of the bullets in the above into "latitude 42.269711, longitude -88.004308". The added complication (in the BRCs and in the list above) is that some of the previous authors were not working in particularly precise coordinate systems. Given the list above, your steps in finding the 'true location' of this store would be slightly different for each bullet, and in the end, might include identifing all the ice cream stores on Lake, south of Loch Lomond, figuring out how old the ice cream stores are in Mundelein, figure out how far you could get in radius in a 4min drive from the school, figure out how far you could get in radius in a 20 min walk from school, etc. In this case, you know that all of these pointers are trying to indicate the same physical location, so you could then look at the commonalities in all of those results, and then, eventually, assert with confidence that all of them point to lat/long (42.269711, -88.004308). In the case of the BRCs, we don't know for sure that each object in each paper actually does have a match. Sometimes they don't. Sometimes there really are two objects close to each other, not just one.

So all of this is what we have to do with the BRCs. We have lots of lists of objects, all in slightly different systems and coming from different ways of searching for young stars. We have to apply our brain and figure out which objects each paper is talking about -- which ones are new objects and which ones are the same as objects previously identified.

Venn Diagrams and Bookeeping

One of the difficulties we will have during this project is keeping all the source lists straight. It happens every year, and I don't know how to make it easier, except for warning you that it will happen! Here is a Venn diagram explaining, roughly, the various source lists we will have before we are done, at minimum. This Venn diagram is meant to be a "big picture" sort of thing; this page on the source matching is meant to address just, well, the previously identified sources.

The source lists include:

• All "bright enough" sources seen in the WISE maps (a conceptual list only)
• Sources in the WISE catalog of photometry (to which we will add photometry from 2MASS, Haleakala, and Spitzer in the places where we have that data)
• Sources in this general direction studied by anyone else, ever (the majority of those reported are also YSOs, but not all of them) -- this is the list we are trying to assemble here.

Out of those sets, our ultimate scientific goals mean that we are striving to identify:

• YSO candidates we select from IR excess
• YSOs that others identify that do not appear to have an IR excess.

The Venn diagram is even trying to correctly represent the relative sizes of the circles in that "all bright enough sources" ought to be darn close to "sources in the catalog" and that there will be some "sources in this general direction..." not in the regions we care about, and some of those sources that do not have IR excesses.

NOW we are going to work on the list of "Sources in this general direction studied by anyone else, ever" for BRC27.

For this diagram, I tried to spatially represent the concepts behind what we're doing now (on this page), but I admit the circles are not as carefully constructed/laid out as the first one!

Each of the 5 papers studying things in the region of BRC27 (Ogura et al 2002, Gregorio Hetem et al. 2009, Chauhan et al 2009, Shevchenko et al 1999 and Wiramihardja et al 1986) looked in the direction of BRC27. Surely, then, they saw some of the same sources as each other, and as what we are seeing. For example, the Gregorio-Hetem survey covered a HUGE area, and we care only about a part of it. Ogura saw some of the same sources that Gregorio-Hetem did, but not all of them -- they were not covering the same areas, but even within the same area, they did not see the same sources, because one survey was X-ray driven, and one was Halpha driven. They *will* see different sources, not only because they're looking at different wavelengths, but also because each survey is not infinitely deep -- the sensitivity of the surveys is limited, and as such will not see every source in this direction. Same for each other pair of papers, and our survey.

This is hardest for BRC 27 because it has the most previous work. This is easiest for BRC 34 because there have been so few papers done on this region.

The Goal

The goal here is to construct a list that is as clean as possible for each of the objects that these other folks studied, identifying which objects are truly the same between surveys, and identifying which of these objects are ones that those authors thought were actually young stars (as opposed to, e.g., background giants). We also want to carry along each of the relevant bits of information that these other authors provided -- the object is a lot easier to identify as clearly a young object or a contaminant if there is optical data, so if the other authors reported any optical measurements, we should keep track of those and tie them to the correct object in our analysis. We should also make note of any spectral types or other relevant information. The aim of this part of the project is thus:

• Which objects from paper x are also seen in paper y?

and then, the next step we will take is

• Which of these objects are seen in the WISE data?

The Challenges

This would be an easy task if:

• everyone provided their original images, either as a figure or as a fits file
• everyone worked in the same coordinate system, by which i mean not just "J2000" vs "B1950" but "J2000 tied to 2MASS" as opposed to "J2000 tied to the pulsars seen by NRAO" or "J2000 as calibrated as best I can based on the HST Guide Stars I happen to see in my image".
• the objects were all greater than 5 arcseconds apart from each other on the sky, such that each source that is detected was cleanly and uniquely detected in each survey.
• and, of course, if we were guaranteed a match between surveys.

Working backwards up that list...

We've already talked above about how we are not guaranteed a match between surveys, because stars are different brightnesses at different bands, and because the surveys have limited sensitivity.

There are plenty of sources that are very close together. Even among just the YSO candidates, some are very close to each other, closer than 5 arcseconds.

If we had fabulous coordinates for everything, we could let the computer match them all up and not worry about it. But we don't. And it's not just a matter of precessing the B1950 ones to J2000 ones either - there are inherent errors in those old coordinates which will not necessarily smoothly map into a clean match to other objects.

If we had images, we could line them up by eye and identify the same objects in each frame. I don't necessarily mean "line them up in ds9" (which would be the ideal case). But also, you can identify the objects simply by comparison between images they publish and images to which you have access (IRAC, 2MASS, POSS). This is what we are going to have to retreat to, in the tough cases.

Here are some notes on the 5 BRC 27 papers, in no particular order:

• Gregorio-Hetem - finding charts for a few complex fields, which may or may not be in our region. J2000 coordinates tied to 2mass, though, so less worried about these. note that their discussion includes this very conceptual problem -- they are trying to tie X-ray sources to optical sources.
• Wiramihardja - no finding charts; a few very coarse-scale ones, one with just YSOs. Not a lot of help here. Also just 1950 coordinates read off a photographic plate. Likely to be large and unsystematic errors.
• Shevchenko - finding chart provided (fig 1). 1950 coordinates. big field (bigger than the region we care about).
• Ogura - finding chart provided; J2000 coordinates but not necessarily tied to 2MASS.
• Chauhan - no real finding charts provided; there are a few coarse-scale ones. J2000 coordinates tied to 2mass, though, so less worried about these.

The mechanics of what we need to do

For each of these 5 papers, we need a machine-readable (read as "plain text file that the computer can parse into individual numbers rather than images of numbers") version of the relevant data tables. This was either:

• obtained from the journal itself, in which case the data table is much longer than we need
• obtained by typing in the coordinates of the objects in our fields from these older papers and try to get updated coordinates.

Here is a video I made for last year's team on getting updated coordinates. This approach SHOULD work in MOST cases but did not work in every case; more on this momentarily. What we need to do is :

• type the 1950 coordinates into the 2mass point source archive, making sure that it knows that it is 1950 coordinates; or type in the 2000 coordinates (and it will assume that they are 2000 coordinates).
• look at what comes back, and take the closest *bright* object.

Then we will have 5 lists of UPDATED, HIGH QUALITY coordinates, one per paper, and we can let the computer run through the list, finding the matches between papers. We then can generate one file that purports to have one line per literature object, with all the relevant data on that line. The difficulty comes in that inevitably, a few sources during this process end up tied to the same object, or identified in other ways as duplicates or incorrect matches.

The approach above to get updated coordinates for targets works ON THE ASSUMPTION THAT THERE ARE NOT "TOO MANY" SOURCES NEARBY, AND THAT THE PRIOR SURVEYS, BEING SHALLOW, ARE MOST LIKELY TO MATCH TO THE BRIGHT 2MASS SOURCE. As I say, this should work in MOST cases but not ALL of them.

If the original authors are reporting more than one source within one paper, we should not consolidate them into one unless we are REALLY sure that the original authors were wrong. (This does happen; see my North American Nebula paper for examples.) Within each paper, this is what I would do:

• Go do the 2MASS archive search (Go here: http://irsa.ipac.caltech.edu/applications/Gator/ -- pick 2MASS then on the next page pick 2mass point source catalog (PSC), then use that search page, making sure to specify which coordinate system in which you are working. Use the coordinates from the original paper to avoid transcription errors, as opposed to any subsequent notes or xls files. Be VERY careful to copy the coordinates exactly; that's the most common problem.)
• Look at the sources returned. Is there just one within 5-10 arcsec? that's probably your match; go on to the next source. If there is more than one, is there just one BRIGHT one within 5-10 arcsec? Is there another source from that same paper within 5-10 arcsec? If the answer to either of those questions is yes, go on to the next step.
• Go get the images in another window. Compare the images (if provided) from the original papers. We can use Finder Chart (http://irsa.ipac.caltech.edu/applications/FinderChart/) to get DSS and 2MASS images of the region where there is a confusing match. Compare this to the images provided in the papers. Identify which object is the one(s) in the paper.
• With that information, then go back to your PSC, and then use the 2MASS point source catalog to find the actual high-precision coordinates of that specific dot you have identified in the images as the match to the literature object.

Here is a list of all the papers we have assembled as part of the lit review, as well as a list of their coordinate list status. First priority items (e.g. those for you to tackle first in May/June) are in bold in the last column:

paper	notes	coordinate list status	merging status and pending items
Getman et al 2007	X-rays. BRC 38 only. coordinates likely just fine.	got all tables from journal. LMR has merged t1,t2,t3 together	does it really have all of Nisini et al. (2001) already in t3? need to merge it to rest of catalogs.
Beltran et al. 2009	NIR. 38 only. coordinates likely just fine.	got all tables from journal. (link)	need to merge relevant bits of tables together, and then merge that to rest of catalogs.
Choudhury et al. 2010	MIR. 38 only. coordinates likely just fine.	got all tables from journal. (link)	need to merge relevant bits of tables together, and then merge that to rest of catalogs.
Chauhan et al. 2009	NIR, MIR (IRAC only). 27 and 38. coordinates ... odd. look at carefully.	got all tables from journal. (link)	need to merge their tables together (will take some care! - do this if the rest of the bold stuff here is done and you're chomping at the bit for more), then merge that to rest of catalogs.
Barentsen et al. 2011	iprime,rprime, halpha. BRC 34, 38.	got all tables from journal.	LMR has already merged it to rest of catalogs.
Nakano et al. 2012	iprime, halpha, some AKARI. BRC 34, 38.	got all tables from journal. (link)	LMR has already merged it to rest of catalogs.
Ogura et al. 2002	BRC 27, 34, 38. turns out will probably need 26 too!! there are finding charts!!!	34 is ~done. 27 only done in center (see rebull et al 2012 below). 26, 38 not touched. link for data files as downloaded from journal	COORD NEED **UPDATING** 34 is done (see below). 27 is done only in center (see below). 26, 38 are not touched. need to obtain list of objects as published, find match in 2mass, assemble list of matches betw this and 2mass. you probably will need the finding charts in the paper to sort out the matches.
Gregorio-Hetem et al. 2009	BRC 27 only. X-rays. coordinates probably ok.	have whole tables from journal (link). merged together. done in center (see below). over whole region, LMR beat against 2mass in region and took nearest within 2 arcsec. several still don't have matches. region we care about is small compared to region they worked on.	COORD NEED CHECKING AND POSSIBLY UPDATING. File:Gregorio 20120514.txt here is my file of the sources which has the published RA and Dec written in two ways (both J2000) and the proposed 2mass match (done blindly by the computer). if it has an asterisk (*) in front of the name, it is in the region we care about. Is it correctly matched to the right 2mass source? if not, what should it be??
Shevchenko et al. 1999	BRC 27 only. antiquated coordinates.	have whole table from journal. done in center (see below). over whole region, LMR beat against 2mass in region and took nearest within 5(!!) arcsec. several still don't have matches. region we care about is small compared to region they worked on.	COORD NEED CHECKING AND POSSIBLY UPDATING. File:Shevchenko 20120514.txt here is my file of the sources which has the published RA and Dec (B1950) and then a converted-to-J2000 version of the published coordinates, both written in two ways (degrees and hh:mm:ss, dd:mm:ss) and the proposed 2mass match (done blindly by the computer). if it has an asterisk (*) in front of the name, it is in the region we care about. Is it correctly matched to the right 2mass source? if not, what should it be??
Wiramihardja et al. 1986	BRC 27 only. antiquated coordinates.	have nothing from journal (it's too old).	COORD NEED **UPDATING** done in center (see below). need to obtain list of objects as published, find match in 2mass, assemble list of matches betw this and 2mass. NB: region we care about is probably smaller than region they published. no need to find matches for objects outside of our region but may be easier to just blow through and do all of them?
Rebull et al. 2012	BRC 27, 34 only. coordinates and crossmatches to literature should be good. I hope!!	have tables, of course!	should hopefully be ok! File:Rebull2012t1brcxids 20120514.txt -- file of all literature information, crossids. note that technically this is in latex format, so "&" divides columns and \nodata means, well, "no data". some people provided errors, some didn't, so that is reflected in this table. last object listed for BRC 27 is actually off the edge of the IRAC 4-band region, and no longer appears in the table. The cross-ids should actually be ok, though. File:Rebull2012t2 20120518.txt -- file of new measurements discussed in Rebull et al. 2012. Note that there are both limits and errors in this table.

Last (next) steps

Once we have updated high-precision lists of coordinates from each paper, we can merge them together, and come up with a new-and-improved list of all the previously identified objects in this region in one place, one line per object with all the relevant information, in a plain text file.

Then we will compare this list to our WISE catalog. There will be some with IR excesses, and some without IR excesses. I expect that we will probably detect them all, but there might be some we do not detect.

Then we will also compare this list to where there is Spitzer data, and obtain Spitzer data for those sources where we can.

Then we will also compare this list to the optical data, where possible. There WILL be some we WILL NOT DETECT in the optical data.