Add ability to create taxonomy terms that have additional fields

[mjordan]

Currently, Workbench can create taxonomy terms but only the term name is supported. We need a way to allow creation of terms that have additional fields, such as the Person vocabulary created by the Controlled Access Terms Defaults module:

• Alternate Name
• Authority Sources
• Birth Date
• Death Date
• Relationships

[mjordan]

This came up using real live user data over in #106.

[mjordan]

I've determined that if a field in a vocabulary is required, Drupal won't let you create the term without populating that field. Workbench will write the following entries to its log (where "Test term" belongs to a vocabulary that has a required field):

04-Oct-20 09:49:13 - WARNING - Term 'Test term' not created, HTTP response code was 422.
04-Oct-20 09:49:13 - ERROR - Node for CSV record 05 not created, HTTP response code was 422.

[mjordan]

Islandora Workbench has already established a pattern in how to populate CSV data to accommodate complex taxonomy data. Typed Relation fields are a good example of this. Islandora Defaults creates the Linked Agent field, which references three vocabularies, Person, Family, and Corporate Body. The Typed Relation field adds to fields that reference vocabularies the ability to define how the thing described in the taxonomy term (such as a person's name) is related to the Islandora object. For example, items from the Person vocabulary can be applied to Islandora objects using "typed" relationships such as photographer, author, editor, or bookbinder. Authority lists of these types exist; in this example, the MARC Relators list defines standard codes for these types.

Because Typed Relation fields encode the type of relationship between the values in a vocabulary and the Islandora object node, in order to assign them using Workbench, we need to express three different pieces of information, 1) a namespace indicating the vocabulary the relation is from, 2) a relation type (e.g. author, photographer, editor), and 3) a term ID. Currently in Islandora Workbench, to populate a Typed Relation field with a taxonomy term ID, you need to put a value like this into your CSV: relators:art:30. This says term 30 (which is the term ID for the Person term "Mark Jordan") is the artist (art) and this code comes from the relators authority list (MARC Relators). This is a very complex bit of data to put in a single CSV cell.

To further complicate things, if a Typed Relation field is multivalued, we need a way to put multiple values in one cell. In Workbench, we do this by separating the values with a pipe (|), e.g. relators:art:30|relators:art:45.

And it gets worse! For example, the Linked Agent field references three different vocabularies: Person, Family, and Corporate Body. Another pattern in Islandora Workbench, for indicating which vocabulary in a multivocabulary field the CSV value applies to, prepends the vocabulary's machine name to the term ID or term name, for example cats:Tabby, which means create the term "Tabby" in the "cats" vocabulary in a field that references vocabularies in addition to "cats".

And it continues to get worse! Vocabularies can have custom fields. In Islandora, we have a few of those vocabularies. Islandora uses the Linked Data principle of providing URIs for vocabulary entries, the "Things not strings" principle, so they have a globally shared identifier. The Islandora Display, Islandora Media Use, and Islandora Model vocabularies all have an External URI field that holds the URIs for terms. The External URI for the "Digital Document" term from the Islandora Models vocabulary is https://schema.org/DigitalDocument. The fact that we use taxonomy terms that have external URIs is awesome (Linked Data FTW). I am using them here as examples of Drupal vocabularies that have fields in addition to the basic term name. You can create your own vocabularies that have custom fields as well, if you need them.

Lost the thread? I don't blame you. This is the problem we have: being able to use CSV data to create taxonomy terms while bulk ingesting nodes, or even assigning preexisting taxonomy terms, will require in some cases fairly complex data structures like those described above. But how do we put complex data into a simple container?

We need to figure out a way to express this complex taxonomy data in CSV while keeping the CSV data simple enough that humans can read and populate it, and simple enough that user interfaces like a desktop spreadsheet editor can be used to read and populate it.

[mjordan]

Another thought. If a vocabulary has fields, the terms need to be created prior to their use. But, we add a create_taxonomy_terms task that creates terms from a CSV with the necessary fields. This would inconvenience the small number of users who need to create fielded terms but retain the relative simplicity of the current CSV structure.

Once the terms exist in the target Drupal, they could be referenced in the existing ways (id, name, uri) in the create and update task CSV.

[mjordan]

I can think of two options for adding field data to taxonomy terms:

1. Separate CSV file per vocabulary

This is the approach I am recommending.

If allow_adding_terms is true, each vocabulary can have its own data file, identified in the following config option:

vocab_csv:
  - person: person.csv
  - tags: /tmp/mynewtags.csv

Within each CSV file, the only required field is "term_name", which contains names of new terms used in your main CSV file. Term ames in this column must strictly match new term names used in node CSV columns (so in effect the "term_name" column is the key joining the node and each vocabulary CSV). "parent" is an optional field that will contain either a term name, a term ID, or a term URI (required for address #236).

term_name,parent,field_1,field_2,field_3

--check will validate that:

• vocabularies that require additional fields have an entry in vocab_csv
• each field in the vocab CSV is present in the target vocabulary
• all required fields in the vocabulary are present in the vocab CSV
• vocab CSV values conform to each field type's requirement (e.g. maximum length, cardinality) and field-type specific requirements (e.g. typed relation, etdf)

During execution, each time a new term is created within the main (node) input CSV, that term is looked up in the vocabulary-specific CSV, and all fields present in the vocabulary CSV for that term are included when the term is created.

I think this is the most practical and flexible approach.

2. Include vocabulary fields in the main input CSV

In this approach, we would need to use special column headers in the main (node) CSV file that indicate they are for a vocabulary, something like:

id,file,title,field_model,field_date_issued,vocabulary:person:field_alternate_name,vocabulary:person:field_relationships

However, this option will not allow for adding additional fields to multiple terms at one time (that is, where multiple new terms are in a node CSV, subdelimited by a pipe), unless we have a convention that the subdelimited values in the term field columns are in the same order as the new terms in the node field columns. I doubt this order-based structure will be easy to populate or reliable from a data integrity perspective.