Feature exploration/request: nested data (nth-degree recursion)

[arxpoetica]

Two prior conversations triggered this ticket. This (closed) ticket #91 and this conversation in Slack https://graphql.slack.com/archives/general/p1478088194004298. Unlike the closed ticket, this should explicitly be considered a feature exploration/request, not necessarily a question about how to do nesting/recursion.

Understanding that GraphQL is about being explicit and declarative, how would/could one be declarative and specify recursion. This is a known limitation that would need to be addressed: https://facebook.github.io/graphql/#sec-Fragment-spreads-must-not-form-cycles One suggestion from the Slack thread was to relax the requirement a bit by allowing @recursive limits. There may need to be a default limit too.

Potential example.

fragment PersonFields on Person {
  name
  primaryAddress { street }
  parents @recursive(maxDepth: 8)
}
fragment EmployeeFields on EmployeeInterface {
  primaryAddress { street}
  boss @recursive(maxDepth: 4)
}

{
  person(id: 1) {
  … PersonFields
  … EmployeeFields
  }
}

One strong aspect of graphs is deep nesting. Is this a possibly worthy feature request?

[arxpoetica]

Example of the tedium that can be explicit recursion: https://github.com/graphql/graphql-js/blob/master/src/utilities/introspectionQuery.js#L74-L105 Family trees (32 nested deep? 64?) and hacker news/reddit-style comment threads are examples.

[benwilson512]

Minor note, since that was an example I originally sketched out (to be clear, I'm not actually for this idea). It needs to specify the fragment being recursed upon, IE

fragment PersonFields on Person {
  name
  primaryAddress { street }
  parents {
    ... PersonFields @recursive(maxDepth: 8)
  }
}
fragment EmployeeFields on EmployeeInterface {
  primaryAddress { street}
  boss  {
    ... EmployeeFields @recursive(maxDepth: 4)
  }
}

{
  person(id: 1) {
  … PersonFields
  … EmployeeFields
  }
}

[arxpoetica]

Yeah, sorry I didn't h/t. ;)

[samuelhorwitz]

A bonus to a declarative syntax is the ability to write better resolvers. I'm already using DataLoader but with recursion, for example in a Reddit style comment thread, every layer is going to need to hit the database for each table that layer needs no matter what batching is set up.

If there was a way to declare recursion then one could write a resolver that is recursion aware and makes one database hit for everything in the tree.

[smolinari]

This might be my own inexperience talking here, but why does the data need to arrive in the nested form?

Scott

[arxpoetica]

@smolinari Trees. Trees are messy data structures that don't really have very good working tools in the wild—this is a rash generalization, but think about it for a second. Is there a Google/Excel Docs equivalent for trees? Ironically, GraphQL is a declarative structure for (nested) trees (graphs), but it doesn't really do recursive nesting very well.

[smolinari]

That doesn't really answer my question. Let me ask a different way. How can the returned response of the data in the form of a tree help with the representation of that tree in the UI? Or asked another way, could the representation of the tree in the UI be just as simple (or hard), if the data for the tree is returned as a simple flat list?

Scott

[benwilson512]

This seems like an unrelated discussion. The return representations of the values are a general GraphQL thing when using JSON, this seems tangential at best to whether or not fragments should be allowed some form of limited recursion.

[smolinari]

Nothing tangential at all. Lee Byron made it fairly clear in the linked issue above that this suggestion won't be handled. If you want them to move on this at all, you'll have to come up with better reasoning than "GraphQL is hierarchical, thus it should also have fragment recursion on the server" or "GraphQL is explicit and declarative, thus recursion of fragments should be declarable". That reasoning isn't going to motivate the devs to work on this.

So, my intention is to get the answers to the questions, what is the real problem this suggestion is supposed to be solving? What is the pain you feel, when you don't get the fragment recursion?

Scott

[pcarrier]

@smolinari it's a bit silly to write tree flattening on the server and tree grafting on the client, and whilst it provides a good opportunity to revisit textbook algorithms, app developers probably have better things to do.

The argument for including that feature as a directive in the graphql spec is the same as @skip: it affects the constraints on the shape of the response, which are strictly defined in the specification.

Should graphql have kept numerics out of the spec on the basis that one can serialize and deserialize them as strings without native support? That's easier to implement with [insert standard library of choice]...

[pcarrier]

@smolinari I disagree with your assessment that @leebyron made "it fairly clear [that] this suggestion won't be handled".

He expressed concerns that, as far as I can tell, are specifically addressed by this request, and finished with "For now this limitation continues to exist for this balance of simplicity for the common case and protection from inadvertent infinite recursion. Perhaps we can revisit it in the future if a solid proposal to do what you suggest is compelling."

Shall we discuss whether this proposal is solid, and if not what would need to be addressed to move forward?

[smolinari]

@pcarrier -

it's a bit silly to write tree flattening on the server and tree grafting on the client

We are talking in the cases above about recursion on single fragments or data types. So to me, and correct me if I am wrong, there isn't any tree flattening being done on the server. It is flat, because the data is either in a single table or a single collection, like "comments" or "ancestors" or whatever. If we would be talking about tree flattening across different data types, then I'd agree with you. Because, doing that would be defeating the purpose of GraphQL.

Perhaps we can revisit it in the future if a solid proposal to do what you suggest is compelling.

Shall we discuss whether this proposal is solid, and if not what would need to be addressed to move forward?

You disagree with me, then add a question, that my questions are trying to accomplish? Nice. How about agreeing with me? 😄

So, to answer your question, yes. You all should be discussing what needs to be addressed to move this forward and as I keep saying, to do that, you need to explain what problems will this suggestion solve. If you had this suggestion done, what developer pain would be avoided? If you can make that clear, then Lee might feel your pain.

If you don't have any pain to avoid with this, then you aren't suggesting anything worthwhile to fix.

Scott

[pcarrier]

We are talking in the cases above about recursion on single fragments or data types. So to me, and correct me if I am wrong, there isn't any tree flattening being done on the server. It is flat, because the data is either in a single table or a single collection, like "comments" or "ancestors" or whatever. If we would be talking about tree flattening across different data types, then I'd agree with you. Because, doing that would be defeating the purpose of GraphQL.

Not every storage system, or backend being queried by the graphql server, will expose tree structures as tables or collections. Even if they did, that's an implementation detail that my graphql server might very well want to abstract from its client.

Take the structure of comments in reddit. That's a tree structure. Imagine that my graphql server queries a backend service over HTTP+JSON that returns something like this:

{ text: "hello", author: "pierre",
  responses: [
    { text: "hello back", author: "jean",
      responses: [
        { text: "how are you?", author: "pierre"},
        { text: "what's up", author: "paul" }
      ]
    },
    { text: "hello back", author: "jacques" }
  ]
}

That's a simple example with a single datatype being nested; this issue does cover other cases, but it should suffice for our purposes. It's not unrealistic: nesting for replies is exactly how Reddit's API exposes comments (example).

Now let's say that I want my graphql server to expose this to the graphql client, which will then represent those in the DOM. For the client, the shape above is the most convenient representation.

What do you suggest the query look like? With @recurse the answer is easy. However if you want the graphql server to return a flat list of comments, the graphql server needs to flatten the tree coming from the backend (eg by exposing an ID and a parent ID for every comment), and the graphql client needs to graft that list back into a tree.

That's the general problem we're describing here and trying to solve. Having to flatten tree models in the server, and grafting them back in the client. I really can't think of a better phrasing.

And to be clear, yes I can do { text author responses { text author responses { text author responses { text author responses { text author responses { text author } } } } } }, but I don't know how many levels to include, and if I suddenly want my component to display a new field (say, the comment field), I have to make many edits.

[smolinari]

If the data source returns the data already structured as a tree or a graph, does the GraphQL resolver need recursion?

Scott

[pcarrier]

@smolinari I'm not sure I understand the question, you might want to read #91 and this ticket again.

It might be helpful to refer specifically to the following parts of the spec (from this repo):

All GraphQL operations must specify their selections down to fields which return scalar values to ensure an unambiguously shaped response.

When querying an Object, the resulting mapping of fields are conceptually ordered in the same order in which they were encountered during query execution, excluding fragments for which the type does not apply and fields or fragments that are skipped via @Skip or @include directives. This ordering is correctly produced when using the CollectFields() algorithm.

So basically, the shape of the response will always match exactly the shape of the query, with a few exceptions (nullables being null, the effect of @skip & @include, fragments conditioned on types for selection sets on unions and interfaces), and some fields being arrays or arrays of arrays or [...] as exactly specified in the field definition (in the schema).

You can implement your own "JSON scalar" which returns arbitrary JSON, but then no traversal at execution, so no field selection, no aliases, no directives, no schema-based static typing, etc.; in short no GraphQL.