Collections Discussion

[cmheazel]

This issue attempts to pull the various /collections discussions into a single issue.

[dblodgett-usgs]

(EDIT: 6/4/20) This issue is essentially agreed to. See: #140 (comment)

Actions here are still

1. Work through documenting the outcome in these slides
2. Triage the open issues and make sure we haven't missed anything.
3. Open new issues that describe how to implement the outcome here.

---

Both API-Coverages and API-Environmental Data are in an awkward position pending progression of this discussion. Here's my attempt at summarizing to get us moving.

I think there are two issues at play here, let's call them data-resource and items.

The data-resource issue comes down to:

1. Will the collections end point be a container (catalog) for flexibly typed resources that are ostensibly datasets?
2. Will the collections end point be reserved for collections of things that are ostensibly features?

See #17, #36, #39, #45, #47, #74, #86, #99, #105, #106, #111, #116, #120, #122, #128, #130

The items issue comes down to:

1. Will a consistent approach to metadata for sets of items (at whatever path) and an items API path literal be used?
2. Will an approach to "sets of items" be left to each individual API?

See #45, #80, #82, #87, #83, #107, #110, #128 (and likely others)
Broadly, this wiki post by @cmheazel discusses this wiki post.

The status of addressing these issues for now:

Collections and items have been moved to its own specification part so core can move forward: http://docs.opengeospatial.org/DRAFTS/20-024.pdf

In the current (5-21-20) collections spec, the /collections path literal is used for "A body of resources that belong or are used together. An aggregate, set, or group of related resources."

The items path literal is used for: "the individual member resources that make up the collection".

Narrative:

Some are advocating for a flexible definition of collection to allow spatial data resources to be the resources that make up the collections path literal resources. Some spatial data resource types in this approach would use the items path literal but not all.

Others are advocating for strict typing of the collections path literal. Essentially saying that the things you get from a collections resource should all be the same (e.g. feature collections). I don't think there are issues with using items in this approach as long as a different path literal is used high up in the chain.

A path forward:

In follow up comments here, please take care to focus on the issues and seek to find ways to communicate unique characteristics of proposals with clarity. Please make fully-fledge proposals and try to name them such that we can discuss with clarity.

References OGC-API specifications of interest:

There is some discussion of the API-Features approach to collections in the core requirements class.

Coverages discusses their approach in collection access

EDR discusses their approach in environmental resources

Records discusses their approach in collection access

Processes does not use the collection endpoint

There may be other relevant specs (styles?) to consider, but I think these are the core that we need to consider at this juncture.

[dblodgett-usgs]

To give the group something to get started, I want to propose that we

1. do NOT add a collections literal path to API-Common, but rather,
2. include a /{spatialResource} at the root of an OGC API which API-Features /collections would be conformant with.
3. Have a reusable set of conformance classes for items that could be used on any set of resources in an OGC API. API-Features items would also be conformant with this common items.

[jerstlouis]

As recently discussed in #11 and #111, I believe the most controversial aspect of the Collections problem is its generic name which sows confusion, while what we are trying to define here is something better described as OGC API - Common (Geospatial data).

As such, and in line with what both @dblodgett-usgs and @jeffharrison are suggesting, let's entertain the idea that we can drop the literal 'collections' from being fixed, and that a compliant client must instead rely on finding the list of geospatial data resource by following "rel" : "data" from the landing page. As far as compatibility with OGC API - Features is concerned, this would either require that any server offering Features (which may be served along with other APIs) stick to "/collections", or that the Features standard is revised with a breaking change where clients can no longer rely on "rel" : "data" pointing to "/collections". Let's assume we are okay with this for now and continue.

Now "OGC API - Common - Part 2: Geospatial data" could say:

• The "rel" : "data" of the "OGC API - Common - Part 1: Core" links to a URL we will call {dataResourcesURL}
• A GET request on {dataResourcesURL} will return a list of data resources, with a schema matching what is currently returned by Features as the reponses to /collections
  (NOTE: This does imply a "collections" : property at the top of that JSON response, there's sadly no way I can think of around that.).
• In that list of data resources, as per the OGC API - Features specifications, and the current draft OGC API - Common (Collections) specifications, two key link relations are defined inside the "collections" property array:
  • "items" -- This is where you actually retrieve your data (In Features, this links to /collections/{collectionID}/items, in Coverage, that currently links to /collections/{collectionID}/coverage/all I believe. For 3D Tiles, this could point to your tileset.json (linking to one or more node of the BVH). For i3s, this could point to your root BVH node.
  • "self" -- In Features, this links to /collections/{collectionID} where you normally retrieve the exact same content of that element for the one collection object of the "collections" array, all by itself. Same here, this links to {dataResourcesURL}/{dataResourceID} with same response.

The Tiles API could also be tied here by having a "tiles" link relation within each element of that array.

I also foresee the need for additional optional conformance classes to be able to arbitrarily retrieve data using bounding boxes and resolution, without the client having to know anything about the type of geospatial data.

[akuckartz]

Maybe some ideas from https://www.w3.org/TR/ldp/#ldpc can be used?

[dblodgett-usgs]

@akuckartz Please note the "A Path Forward" section above.

Can you please flesh out what you think is of value from the Linked Data Platform Containers list?

[akuckartz]

@dblodgett-usgs That same comment ends with "There may be other relevant specs (styles?) to consider, ..." LDP is such a spec - and even a standard.

Can you please flesh out what you think is of value from the Linked Data Platform Containers list?

I will try, but can not guarantee that I find enough time.

[dblodgett-usgs]

I see -- that closing comment was meant to close out the list of OGC API Specifications that have work in progress related to the data-resource and items issues. If you think there are elements of the LDP specification that are useful in bringing closure, please present them here, but that comment was not intended to be an open-ended ask for additional concepts from outside the current baseline.

[dblodgett-usgs]

@jerstlouis -- why carry rel: data forward from Features at all? Couldn't rel: data be an OGC API Features link relation that gets you to a feature collection view of the data-distribution at / ?

Your response does not address the two aspects of the issue as posed in my top-level summary and seems to be mixing thing up.

Re: the "data-resource" issue, which seems to be the one you are addressing, you are clearly interested in collections being a container for flexibly typed resources that are datasets. But your examples just describe a design pattern -- not why it is better than another design pattern.

Your:

(NOTE: This does imply a "collections" : property at the top of that JSON response, there's sadly no way I can think of around that.).

Indicates to me that it is a bit of a hack and is probably not a solution that we want to pursue.

Specifically, what is wrong with:

• / as a data-resource
• /collections as a feature-collection spatial-resource
• /coverages as a coverage spatial-resource
• /tiles as a tiles spatial-resource
• etc. etc.

Where API-Common would specify the common aspects of a spatial-resource but not the path semantics -- only that a given spatial-resource view over a data-resource should have a literal path for its API.

[jeffharrison]

I don't think there's anything wrong with it. There should be room in OGC API for straightforward geospatial resources too.

Best Regards,
Jeff

[cportele]

In general, I see two general approaches for moving forward, if the existing Collections resource does not work for the SWGs that specify other spatial data resources.

Let me start with the proposal from @dblodgett-usgs, which I would characterise as follows:

• Restrict an API that shares spatial data to a single dataset.
  • Note: There will also be APIs with no dataset involved, e.g. to share processing capabilities, styles, etc.
  • Note: A metadata catalogue is a dataset, too.
  • Note: An API that implements Records and that catalogs data APIs is essentially an API that provides access to multiple datasets.
• Group spatial data resources so that they provide reliable access patterns and do not require that code does a detailed analysis of the resources to determine how to access sub-resources. Use a fixed token for the path element.
• Each spatial data resource represents one or more distributions of the dataset (one per supported media type).
• The same dataset may be shared via multiple spatial data resources ("views") with different access patterns. For example, the data may be shared as features, as a coverage or as vector tiles, each with their own access patterns (sub-resource structures, applicable query parameters, etc.).

That is, /collections would be restricted to cases, where the data items are accessed at /collections/{collectionId}/items with support for paging and filtering via bbox/datetime. Example: features and records. Neither /collections nor an "items" link relation type would be discussed in Common Part 2. Their specification would remain in Features.

For spatial data resources with other access characteristics, other resource types with other tokens should be defined / used in the respective specifications. For example, Tiles, Coverages, EDR.

Note that I don't see how Common Part 2 could define "a reusable set of conformance classes for items that could be used on any set of resources in an OGC API." The items resource is a specific sub-resource of a collection and other spatial data resources may have different access patterns (I assume, for example, coverages).

One aspect that would need more thought are the link relation types. In Features, the "data" link relation type references the Collections resource at /collections. The link relation type is defined as "refers to the root resource of a dataset in an API." Since we cannot register something like "data" with IANA, we need a fresh start in Common anyhow. I see two options:

• define a link relation type for each spatial data resource type (e.g., Features could define "ogc:collections"), or
• define something general like "ogc:spatial-data-resource" or "ogc:distributions" in Common Part 2.

The second option works, if/since we use fixed tokens like "collections" or "tiles" for the spatial data resource types. However, for clients navigating the API by following links, the first option should be easier to use.

The second approach that I see is to also move away from fixed paths like /collections in Common Part 2, but define a flexible architecture for other resource types that represent dataset distributions. This should work, too, is maybe a cleaner architecture, but also more complex to develop client code.

For simplicity, let's assume that we still restrict an API that shares spatial data to a single dataset. If this changes, the solution would become more complex.

Without fixed paths we need to rely on other mechanisms so that humans and software can understand an API, both from the API definition and from navigating the resources:

1. For a general solution we would need a richer set of link relation types to distinguish links to different types of data resources - individual data items and aggregations of them with datasets and distributions having a special role. The downside is that we would need to develop a sound resource model and I doubt that we have enough experience yet to standardize one.

2. Since the starting point is that not all spatial data resources are under /collections, we need to be able to link to other spatial data resources and be able to understand what they are. We probably would need to a) register a JSON media type for each resource type representation in the API and b) should include a "type" member in each JSON object (in the current Collection(s) resource types these members would have to be optional for backwards compatibility). The structure of these resources would be out-of-scope for Common Part 2, which to me also would exclude any discussions about "items".

A potential risk with all that is that the flexibility on the server side comes also at a cost for client developers. Developing a generic client that works out-of-the-box would require good knowledge about all these concepts. (NB: there is also more work for document editors / OGC as more and more IANA registrations would be the likely result.)

One of the key drivers behind the WFS 3.0 / OGC API Features activity, and I hope also behind the OGC API idea in general, was/is to reduce the learning curve and the complexity for developers compared to many of the standards from the OWS/XML stack. Yes, we also want to improve the overall architecture in the OGC baseline in this process, but we should avoid approaches that add complexity/flexibility that is not needed by the majority of the deployed APIs.

If we go down a path with a very flexible resource structure, there should be agreement that OGC API standards (e.g., Features) can remove flexibility for "their" resource types. In Features we ended up with the current structure after implementation feedback and intensive discussions (see, e.g., issues 90, 64 and others) and that approach has proven to work well for Features.

[dblodgett-usgs]

Thanks for this @cportele.

I am admittedly out on a limb with the items idea. I was thinking that the link relation for items and at least limit/paging could be reused in other places that are not under a collections API path literal?

The reason I'm leaning toward an approach where each API access pattern gets its own literal path is largely what you point to as a key driver for OGC API.

... to reduce the learning curve and the complexity for developers compared to many of the standards from the OWS/XML stack.

Your notes are really important to what I'm seeing as a path forward:

Note: A metadata catalogue is a dataset, too.
Note: An API that implements Records and that catalogs data APIs is essentially an API that provides access to multiple datasets.

In this world view, an OGC API can be cataloged in an OGC API Records and referenced as a dataset in its own right. I've used ISO19139 (services metadata) to integrate dataset services into processing workflows very successfully and see this as putting the complexity in the right place but keeping it "in band".

Where are others at on this? @cmheazel @joanma747 What would you suggest as a path forward? I am pushing here because of how much work is bound up in EDR and Coverages pending this discussion. Coverages and EDR folks: @Schpidi @pebau @chris-little @m-burgoyne where do you stand on this?

[jerstlouis]

@cportele @dblodgett-usgs @cmheazel @joanma747 What I was hoping to see in OGC API - Common Part 2: Geospatial data is the following...

1. A common mechanism (regardless of the type of geospatial data or available views) by which to list all data layers within a dataset (starting from its landing page), including common information useful to a client, using a Common schema (though it can be extended for the specific module). This includes:

• Identifier
• Title
• Spatial & Temporal extent
• Intended scale / resolution

/collections/{collectionID} in both OGC API - Features and the current draft of OGC API - Coverage satisfies this for the most part.

2. A common mechanism by which links are provided within this schema for each of these data layers, which return the data in one or more forms in which it is being distributed. Based on the link properties ("rel", "type" and/or other properties), a client knows what it will get when it follows that link.

Currently, the "rel" : "items" of OGC API - Features, also used in the current draft of OGC API - Coverage, linking respectively to /collections/{collectionID}/items and /collections/{collectionID}/coverage/all also satisfies this.

Relations could be changed as needed, additional properties for the links could be added, but this is the functionality I hope ends up in this Common approach to Geospatial data.

3. Then I also hope for conformance classes supporting a simple retrieval mechanism from BBOX+resolution to retrieve the data, either from that same link relation, or from a separate "rel" if necessary.
   /collections/{collectionID}/items?bbox=30,40,50,60 -- Returns me a GeoJSON for my bbox for a vector layer
   /collections/{collectionID}/coverage/all?bbox=30,40,50,60 -- Returns a CoverageJSON for my bbox for my raster layer
   /collections/{collectionID}/coverage/all?bbox=30,40,50,60&f=geotiff -- Returns a GeoTIFF for my bbox for my raster layer

Similarly, rather than BBOX+resolution, one might use the Tiles API instead in a consistent manner, for retrieving the data either as vector and/or raster. And one might use the Maps API to render that data in a consistent way, and one might refer to this data layer the same way as an input to a Process.

These are the use cases I care the most about, and so far the newer proposals seem to move away from this and I see it as a major setback in terms of having a common approach to geospatial data.

If we can resolve this, then we could discuss about how one might represent a hierarchical structure both within a single dataset, and as a way to organize multiple datasets, and whether that capability could be one and the same, or implemented in a similar manner, but that is largely a separate issue.

My rationale for wishing to have this functionality is also entirely based on reducing the learning curve and the complexity for developers. By implementing these simple capability once, clients automatically handle the generic aspect of working with any type of geospatial data, and can gradually implement additional support for the special handling or capabilities specific to a particular data type or retrieval mechanism.

As a practical example of the value of this, based on the current draft Coverage specifications, the only thing currently missing in our Features & Tiles API client from supporting Coverages is parsing CoverageJSON, because the current generic common geospatial data approach already allows it to follow the links all the way to /collections/{collectionID}/coverage/all which returns the data as CoverageJSON. Without writing any special code for Coverage, it could already see the titled coverages and their geospatial and temporal extents.

[cportele]

@jerstlouis - You have lost me now. I thought you wanted to get rid of /collections as a root resource for dataset distributions, but now you seem to say that every data API should use /collections?