Proposed changes to NestedText that are not backward compatible

[KenKundert]

We are considering deprecating quoted keys. This will be a change that is not backward compatible. You can see the discussion that triggered this decision here. To summarize, the feeling is that:

1. Quoted keys add considerable complexity to both the implementation and support (ex: they added considerable complexity to implementation of Vim syntax highlighting) that is not consistent with the design philosophy of NestedText.
2. The approach taken is unique and unfamilar to everyone that encounters it.
3. Distinguishing the key from the value can be difficult in some cases.
4. The approach taken is not in keeping with the other concepts of NestedText.
5. Even with quoting, there are some strings that cannot be used as keys.
6. Quoted keys provide too little value given the above issues.

Eliminating quoted keys further limits the strings that can be used as keys. We are considering adding multi-line keys to replace quoted keys. It is felt that multi-line keys are more in keeping with the style of NestedText than quoted keys were, and they allow NestedText to accept any string as a key.

Multi-line keys are patterned after multi-line strings, except the string tag >␣ is replaced by the dict tag :␣ and a trailing indented value is required. For example:

: this is the first line of a multi-line key
: this is the second line
    > this is the value

This would be interpreted as:

{
    "this is the first line of a multi-line key\nthis is the second line": "this is the value"
}

Multi-line keys are not expected to be commonly used, but they are being considered because the fit naturally in the language and they make NestedText completely general, meaning that with multi-line keys NestedText can handle any combination of lists, dictionaries, and strings, where the leaf values are all strings. We could not say that previously.

Comments?

[LewisGaul]

Thanks for raising this issue for discussion - I'd be interested to hear other people's thoughts on this.

Summarising my position that I think I've already made clear in comments of #21 and #22:

• I'm happy to see quoted keys going away
• I'm not personally convinced by the need for supporting whitespace/colons in keys
• I think your proposed multiline key syntax has some nice properties (e.g. each line type still identifiable without the context of surrounding lines)
• However it does seem strange that the key is appearing after a colon, making it appear like a value at first glance
• I just looked up multiline keys in YAML and found that it is supported but uses a leading ? (according to a couple of stackoverflow answers at least, e.g. here). Maybe using ? instead of : to start lines intended as multiline keys would make more sense for consistency?

[joshgoebel]

Questions:

• Is it a syntax error if a key is a single line? I would presume not since this can also be used for pure escaping, and one might wish to escape a single line?
• If not then the following would be completely valid as well, yes?

: president
    : name 
        > Katheryn McDaniel
    : phone
        cell: 1-210-555-5297
        home: 1-210-555-8470
    : kids
        - Joanie
        - Terrance

[KenKundert]

Josh, a single line multi-line key is fine, so your example is perfectly valid. In fact, a multi-line key can be empty (this is the only way to get an empty key):

:
    > this value has an empty key

which becomes {"": "this value has an empty key"}.

[asb]

To share my 2 cents:

I think I have a similar point of view to @LewisGaul. Allowing a wider range of keys to be used sounds attractive, but I'm not sure the extra "oddness" of multiline keys is worth it. They're easily machine parseable, but I think potentially confusing to the human reader. Using a different prefix to : might be slightly better, but I'd still lean towards not adding it at all.

[kalekundert]

I'm late to this thread, but I thought it would be helpful to list some pros and cons (as I see them) of each proposed syntax, including one that @KenKundert and I have discussed but not mentioned yet:

• : syntax:

  Example:

  : lorem
  : ipsum
    > dolor sit amet
  

  Pros:

  • The : character is clearly and consistently associated with dictionaries (like - with lists and > with strings).

  Cons:

  • The keys look like values at first glance.

• < syntax (this is the one that hasn't been mentioned yet):

  Example:

  < lorem
  < ipsum
    > dolor sit amet
  

  Pros:

  • Symmetry with the multiline string syntax. You can also think of > as pointing to the right for values, and < pointing to the left for keys.

  Cons:

  • The keys look like strings at first glance. I actually think it's a bit hard to tell that there's a dictionary there at all.

• ? syntax:

  Example:

  ? lorem
  ? ipsum
    > dolor sit amet
  

  Pros:

  • Keys are not confused with values/strings.
  • Precedent in YAML.
  • A mnemonic could be that keys are for looking up values, which is kinda like asking a question, so it kinda makes sense to prefix keys with ?.

  Cons:

  • ? does not immediately suggest "dictionary" like : does (the above mnemonic notwithstanding).
  • Feels a little wrong to add ? as a syntax character given that it will be so rarely used compared to all the other syntax characters.

• No multiline keys:

  Pros:

  • No potential for confusion.

  Cons:

  • Some keys can't be represented.

I personally like the idea of adding a multiline key syntax, although I definitely appreciate the arguments against doing so. The ? syntax is growing on me more and more, too. I do think that we should at least make empty keys illegal, to keep open the possibility of adding the : syntax without breaking backwards compatibility again.

[LewisGaul]

Thanks for the summary, I'd agree with the pros/cons. I'm still -1 for multiline keys in general, looking at the options above.

I struggle to visually parse any of these as mappings, since I think YAML/JSON always require a colon separating the key and the value, which is missing from all of the above.

I think perhaps I'd be slightly less against it if the separating colon was added, e.g. (picking my preferred syntax from the options above):

? lorem
? ipsum
  : dolor sit amet

However, this doesn't play nice with values that aren't a simple single-line string:

? key with : unrestricted characters
  :
    > multi
    > line

I think maybe I still prefer this to a lack of requiring the colon though.

In fact, if the colon is still required, I guess in theory the keys could just be represented as regular multi-line strings (with >), but you'd be losing the "line type is context independent" invariant.

[KenKundert]

Okay, things seem like they have settled down. I have implemented some of the suggestions and checked them in. More work to be done, but just on completing these changes. No additional features are being considered.

The new version:

• removes quoted keys
• adds multi-line keys
• adds inline lists and dictionaries

Details are in the documentation.

Example:

: key 1: the first key
    [[11, 12, 13], [21, 22, 23], [31, 32, 33]]

: key 2: the second key
    {alpha: α, beta: β, delta: δ, omega: ω, pi: π, tau: τ}

becomes

{
    "key 1: the first key": [
        ["11", "12", "13"],
        ["21", "22", "23"],
        ["31", "32", "33"]
    ],
    "key 2: the second key": {
        "alpha": "α",
        "beta": "β",
        "delta": "δ",
        "omega": "ω",
        "pi": "π",
        "tau": "τ"
    }
}

My expectation is that these new features will not be heavily used, but would be very helpful on occasion and helps to complete the language. With these changes, NestedText becomes capable of handling any hierarchical combination of lists, dictionaries and strings.

Please try it out and give me your impressions.

-Ken

[LewisGaul]

Thanks for this work! Although it does add some complexity to the language (albeit removing the complexity of quoted keys) I think this is a positive set of changes, as you say giving more completeness to the language.

I'll have a go at implementing these changes in zig-nestedtext at some point (maybe next weekend) and provide any feedback I might have.

One minor point - shouldn't the new version be 1.4.0 rather than 1.3.2, given the backwards incompatibility and size of the changes?

[KenKundert]

My version numbers are interpreted as follows: <major>.<minor>.<patch>. Stable releases are those where <patch> is 0. Those get pushed on to pypi. Anything with a nonzero patch is considered tentative and subject to change. They are for developers and only found on GitHub. The major component of the version number advances with changes that are not backward compatible in some significant way. Since this version loses quoted keys and so represents a change that is not compatible with version 1.3, it will eventually become 2.0.0 rather than 1.4.0.

[torresjrjr]

I think the inline lists and dictionaries features add adding huge complexity to an already near-perfect format. I'm personally strongly against those features. It breaks expectation, which simply kills the format for me. If I wanted those feautres, I'd use yaml.

No-quoted-keys are a good idea. They vastly reduce the complexity.

As for the multiline keys, I don't find them a bad idea. However, they solve what I consider are non-problems.

Currently, unquoted keys cannot contain colons, so multilined-keys are invented to allow colons and much more. But we already constrain keys with whitespace-trimming rules. I think it would be much more sensible to then also disallow ': ' and '\n' character sequences.

With NestedText, their is already a precedent that validation and schemas are left to the developer. I think setting the expectation for developers to remove ': ' and '\n' character sequences from their keys is better than allowing them. It's probably not good to encourage those characters sequences in the first place. It could be considered bad practice. If they are necessary, there is the meta-solution of replacing them with literals like '\n\r\t\x0a\u0123'.

Notes that ':' would still be allowed afaik, meaning you could use URLs as keys, like in json-ld.

[kalekundert]

Would you mind elaborating on the complexity that inline lists and dicts would add? From my perspective, they add a moderate amount of complexity to the parser, but very little complexity for the end user. The syntax is intuitive (i.e. common to many programming languages) and maintains the property that the type of each line can be identified from its leading characters.

Regarding YAML, I think its problem is not that it supports inline lists/dicts, but that its rules for quoting and type-casting are complicated and frequently break expectations (to use your phrase). But I don't see how the proposed changes to NT would break expectations in a comparable way.

I do think that there are advantages and disadvantages to the inline syntax, I just think the former outweigh the latter (see #24 for more discussion):

Advantages:

• Discourages the use of ad-hoc inline list/dict formats, like ','.split(). These are bad for a bunch of reasons, including (i) the developer needs to anticipate the desire for inline syntax, (ii) the syntax between different fields may not be consistent, (iii) nesting is not trivial to support, etc.
• Allows authors more control over the "density" of information in the file. Normal NT files tend to be long and narrow, which reduces the amount of information that can fit on the screen and makes it harder to grok the whole file. Inline data structures could certainly be abused to go too far in the other direction, but on balance I think they give authors a useful tool to improve readability.

Disadvantages:

• Adds some complexity to the spec. I think the added complexity is minor, for the reasons given above, but I'd be happy to hear more about this.
• Makes it more difficult to load a file and dump it unchanged. This is already made non-trivial by comments and blank lines, though, so the increase in difficulty is relatively small.
• Conflicts visually with common templating languages. I saw this argument here, but I think it's a very minor consideration. You can always change the templating delimiters if it's too much of a problem.

Ambiguous:

• Allows empty data structures to be represented. I'm on the fence about this one. On one hand, the inability to represent such structures is a conspicuous gap in the current spec (i.e. it's one of the most common questions we get). On the other hand, I'm sympathetic to the view that a schema should be used to interpret empty strings as empty list/dicts as appropriate (since that puts the least burden on the author to specify type). But I'm also sympathetic to the view that the author should be on the hook for specifying list/dict types, since those are the only types that NT actually understands.

[LewisGaul]

I've tried out the new multiline object key syntax, and have come up with some edge cases that make me question the re-use of :.

As a user, what would you read the following as?

A :
 : B:
: C :
D :
: E

I think this could be interpreted in quite a range of ways, but as I understand it this is actually
{"A": {"B:": ""}, "C :": "", "D": "", "E": ""}

I'd still be in favour of using ? instead of : for multiline keys. Perhaps this could also be improved slightly by requiring the object value be given explicitly when using the multiline key syntax? An empty string can be represented using >:

A :
 : B:
  >
: C :
 >
D :
 >
: E

Also, there seems to be a bug(?) in the Python implementation that disallows starting the file with a multiline key, giving a "content must start with key" error.

Slight nitpick: the latest language reference has a section headed 'Inline Objects' that's talking about both inline objects and lists. I understand objects are being referred to as 'dictionaries' but this is quite a pythonism. E.g. my understanding is that JSON (JS Object Notation) refers to key-value pairs as 'objects'.