Support/standardize semantic linking of input parameters with their output objects

[KiaraGrouwstra]

PR 654 enabled using schema (and therefore $ref) in non-body parameters, allowing one to $reference definitions to allow sharing them between input parameters and existing definitions for response objects, preventing duplication, as suggested in #301 by @MrTrick and @jasonh-n-austin.

Some ways to do that:

• @MrTrick in Standardize Parameter Models #301 suggested "schema": { "$ref": "#/definitions/person_id" }, with the Person definition's id property also $refing that.
• idea 2: "schema": { "$ref": "#/definitions/Person/properties/id" }. Clear, but verbose.
• 3: have the person_id definition itself be a reference to #/definitions/Person/properties/id so it could function as a shorthand to $ref to.

I think this in itself is great, and should be encouraged to those upgrading.
And I think especially the clarity of idea 2 would enable an interesting extension: allowing one to semantically link input parameters to such object fields, to allow tools to recognize API endpoint input/output properties as related.

So I'd like to discuss how tooling vendors could reliably (without standardization of $ref usage) find semantic links between:

• references to definitions in response objects and the definitions they refer to ("this response includes a reference to the User definition")
• references to definition (fields) in input parameters, and which field of which definition they refer to ("this input parameter would be the id field of a User instance")

... with the intent of using the above information so as to suggest input information of one API endpoint could be provided by the output of another.

If one considers $ref purely as a pointer to be resolved, then the above ideas (even 2) contain one piece of information: anything contained at the path it resolves to.
However, this proposed semantic linking essentially requires a bit more information. Note that after resolving $refs, this information goes lost:

• "the input parameter correlates to a property of the object at #/definitions/Person"
• "the input parameter to the property of that structure is at its sub-path ./properties/id"
• ideally: "the structure it is part of should probably be referred to as Person". Note Swagger Editor solves this by injecting these titles into the definitions using their keys, which seems like a fair approach.

I'd then wonder if Swagger Editor had solved point 1 as well, so as to relate response objects to their used definitions, which would obviously be similar. From what I can see though, it didn't. (Swagger Editor shows Product and its fields in the Uber /products response, but this is a resolved link, rather than also still being aware of the Product definition it took the fields from.)

Thoughts?

[KiaraGrouwstra]

edit: tl;dr I don't think we need spec changes, see my post below for a PoC.

[MrTrick]

It's going to be difficult.

1. The underlying json-schema concept for $ref does not restrict where it can point.
   (relative vs absolute vs external as a separate issue)
2. There's no technical reason restricting where fields, parameters, responses, etc are defined. Convention dictates that shared response definitions are in the #/responses/ area, but one could just as easily (and a bit confusingly) define a response inline in one of the #/paths/... and $ref it from another path.
3. The $ref address has no intrinsic meaning right now. I don't think tools do anything with it other than de-referencing, and maybe a little bit of validation to restrict where in the spec it may point.

So, semantic linking is going to be an uphill battle!

Also, convention is usually a better approach than restriction. Many many bug tickets and discussions start with "I have a reason to do this, but the spec won't let me!"

One more issue with an implementation guessing how parameters link to response attributes, is the plenty of cases where it's not clear;

• A parameter has to be a person id. Is it the primary key, the supervisor id, or the creating user's id?
• A parameter has to be a date. Is it related to created/modified/etc, or something else entirely?
• A parameter has to be a date. Is it an exact filter param, or a 'earlier than', or a 'later than' filter?
• Cases where an input parameter relates to mulitple response attributes.
• Cases where an input parameter relates to a different response attribute, depending.
• More complex cases that any simple spec will be unable to capture.

You'll need more than $ref. At the very least, response attrs will probably need to be linked somehow to filter parameters, not the other way round.

[KiaraGrouwstra]

I'm cool with settling for convention where alternatives fail. In that event I guess OpenAPI could still add value there by encouraging good conventions.

On ambiguities:

A parameter has to be a person id. Is it the primary key, the supervisor id, or the creating user's id?

• I suppose secondary keys in definitions could similarly be defined + semantically linked by a $ref (or whatever) to the original location/definition of the primary key. If we'd be able to link both this secondary key as well as the parameter back to the original primary key, I think that should give sufficient information to allow suggesting either id or supervisor_id fields of previously obtained User instances (including perhaps the creating User you mentioned) as potentially relevant inputs for this parameter. We can't read minds, but suggesting relevant options seems like a good start.

A parameter has to be a date. Is it related to created/modified/etc, or something else entirely?

• Use refs to whatever is appropriate -- the api spec writer has the freedom to distinguish such cases if needed.

A parameter has to be a date. Is it an exact filter param, or a 'earlier than', or a 'later than' filter?

• I'll concede additional info could well go lost as you're showing here. That said, I'd argue it's still better to be able to make whatever suggestions can be drawn from the info available.

Cases where an input parameter relates to multiple response attributes.
Cases where an input parameter relates to a different response attribute, depending.

• Hm, maybe say anyOf with references...

You'll need more than $ref. At the very least, response attrs will probably need to be linked somehow to filter parameters, not the other way round.

• If I were writing an API description, it would seem easier if I could add an endpoint by being able to copy-paste parts from an existing one, rather than having to add to an array halfway across the document as well.

[MrTrick]

You'll need more than $ref. At the very least, response attrs will probably need to be linked somehow to filter parameters, not the other way round.

This differs from my conclusion, so might I ask you to elaborate on this further?

• That $ref is an existing construct already used in all kinds of ways. Overloading and restricting it by way of adding semantic meaning is likely to make life harder for spec designers.
• Abstracting for a moment the specific keyword used, there are two ways to have that link in the spec; "in param define effect on response" or "in response define how param affects it". I'll amend my previous comment, and say that either are doable.

I do applaud your idea, but I'm not exactly sure whether you are trying to define a filtering spec, or define a mechanism by which a filtering spec may itself be defined?

(If the former, then remember that the OpenAPI spec is meant to be API agnostic)
(If the latter, then consider just how difficult it will be to support any API design!)

[KiaraGrouwstra]

I'm thinking in terms of 'relevant suggestions' rather than 'filter'. This difference again sounds subtle, but the latter sounds like adding limitations, while I'd think of having better information as enabling rather than limiting.

[KiaraGrouwstra]

On $ref vs. alternatives:

• Does #/definitions/Person/properties/id contain sufficient info to infer (a) we're looking at a field of this model at #/definitions/Person, (b) that model is called Person, and (3) we're looking for its field at properties/id.
• Yes, presuming (I) json-schema keywords like items, additionalItems, properties, patternProperties, additionalProperties, allOf, anyOf, oneOf and similar can be seen as indications that we've started navigating a model, and (II) we can just take the last path slice of the model (Person) as its name. If these assumptions appear to be somewhat sane/reasonable, then I think syntax would need to be no more complex than that of $ref.
• alternatives: a new keyword e.g. links (array) or link (ref, optionally using anyOf).

[MrTrick]

@tycho01 I think the more fundamental issue is deeper than any $ref or specific keywords.

Consider the docs for the OpenAPI spec itself; "OpenAPI does not require you to rewrite your existing API. It does not require binding any software to a service--the service being described may not even be yours. It does, however, require the capabilities of the service be described in the structure of the OpenAPI Specification."
So let's take that as being "you should be able to implement a sensible API, (I know the docs go on to say it won't cover all APIs) and have the spec describe it".

"Linking input parameters with their output objects" sounds to me like potentially a large flexible space. API A might use input parameters in a completely different manner to API B.
Consider carefully what you'd like to achieve in an extension or enhancement here:

• Option 1: Define a mechanism that improves the setup, code generation etc process for "API A"-like APIs. Hooray! - unless you have an API that's not "API A"-like.
• Option 2: Define a mechanism that tries to improve the setup, code generation etc process for any kind of API - A, B, C... Much more difficult to do, and the creator of API D might still be unsatisfied with the options available within the spec.

This sounds to me like a miniature version of what I've been cross with Swagger/OpenAPI about in the last few months;

1. The "JSON-API" specification is one sensible way to design/implement an API.
2. I've built a system that meets the spec, and I want to define a corresponding swagger spec, for documentation, etc.
3. The Swagger/OpenAPI standard made it really difficult!
   • The existing json-schema describing JSON-API was considered invalid by strict swagger standards.
   • Many of the JSON-API standards weren't describable inside the swagger spec.
   • Much of parameters' semantic meaning (like `$ref:"#/definitions/person_id") was lost.
4. So I'm stuck with keeping two spec documents;
   • one that's got all the features I want around validation, sanity etc. and
   • one that's dumbed down, but that conforms to the OpenAPI spec.

Moral of the story? Don't go with Option 2 - that way lies disappointment confusion and anger. If not for the one defining that lovely mechanism, then for every author that comes around and is unable to use it to capture how their API works.

[KiaraGrouwstra]

Thanks, that's a fair point. I'm definitely okay with not covering every single edge case.
And perhaps by taking a few assumptions (notably (I), also (II)) I'm already trying to compromise a few edge cases for the sake of more common cases (tooling/description users who don't care and would rather have backward compatibility).

[KiaraGrouwstra]

I tried a quick PoC (i.e. I glossed over the complexity of JSON reference/pointer e.g. external/remote files) to demonstrate it's possible to gather the info with just $ref, so no changes to the OpenAPI spec, though @MrTrick's scenario ends up a bit different.
As stated above, one can't just run the existing de$referencer in your language over an api spec and expect that to get you this info, but I'm happy this seems viable without changes.