Add ability to indicate that an ancestor is a sample

[hyanwong]

In some cases, we want to insert a proxy ancestor and actually treat it as a sample. There is a relatively easy way to do this: during match_ancestors each "real" ancestor gets assigned to an individual in the ancestors tree sequence. Then when we match_samples, any nodes that have an associated individual are not simplified away, but marked as samples instead.

There are some intricacies, such as how to map the proxy ancestors to samples during match_ancestors, and how to fill out e.g. singleton mutations in these historical samples, but I think the general idea is sound.

[hyanwong]

After an initial discussion with @jeromekelleher, we think that we should (a) add a sample_id column to the ancestors format, which is only set when using insert_proxy_samples, and then (b) if we want the sample data file to contain the sample in question as an ancestor, we port the sample information from the sample data file to the ancestors tree sequence during match_ancestors.

We need to think about how we indicate, during match_ancestors, that a particular ancestor is a sample, and therefore we should mark it as such. My suggestion is that we do this if the ancestor is placed at exactly the same timepoint as the sample (i.e. during insert_proxy_samples was inserted with epsilon=0). That way we can combine sample ancestors and proxy ancestors in the same ancestor file, and we will end up doing the right thing.

Another issue is that in an ancestors tree sequence, every node is marked as a sample, so we can't us this to flag up which nodes really are samples! So when creating the ancestors tree sequence we should flag up these nodes using something like constants.NODE_IS_HISTORICAL_SAMPLE, with metadata indicating the sample ID. The appropriate metadata, individuals, populations, etc can be ported into the final tree sequence and associated with the correct node during match_samples.

[hyanwong]

There is, in fact, a decision to make here. Do we slurp in the details of the historical samples (metadata etc) during match_ancestors or match_samples? There are pros and cons both ways:

Doing it in match_ancestors: (pro) that's where we have access to the ancestors with their sample ids, so we can read off the metadata etc very easily; (pro) it seems like the logical place to do it because we are creating tree sequence information about the ancestors themselves in this routine (con) we would need to store information (such as individuals & populations) in the ancestors_ts and then unify the tables from the ancestors_ts with the final ts - this is quite a lot more code and might cause problems if e.g. the sample data file used for each if different, with different population definitions etc.

Doing it in match_samples: (pro) that's where we do all the creation of populations etc anyway, so we would map sample_data to tree sequence info in one go (con) we would need to pass the routine the information about which row in the sample data file to use for each historical sample node (e.g. we could encode it in the node metadata).

I'm unsure which is best. Discussion welcome.

[hyanwong]

To be reconsidered when reworking the matching pipeline, therefore I'm assigning this to 0.5 for the time being