Update grammar format versioning rules et al. (#471)

* Refine grammar format versioning rules; a too new minor version now makes the grammar unparsable.

* Clean-up grammar format spec.

* Implement new versioning rules.

* Add tests for `Grammar.IsUnparsable` and `HasUnknownData`.

* Update release notes.

* Do not emit group end symbols in the main DFA.

This de facto makes the custom DFA group start states mandatory for parsers to understand. The format spec already requires it.
Oriskany

* Add specification for DFA state machines on bytes.
We won't support them in the library right now, but if they are added in the future, they won't be affected by the `Critical` flag.

Author: teo-tsirpanis

RELEASE_NOTES.md

@@ -4,6 +4,7 @@ The following changes were made after 7.0.0-preview.1:
 * __Breaking change:__ Removed the `IParserStateBox` interface and related APIs.
 * Added `BuildSemanticProvider` extension method to `IGrammarBuilder<T>`.
 * Added structures to the grammar file format that can make parsing groups two times faster or more. (https://github.com/teo-tsirpanis/Farkle/issues/153)
+* Updated the grammar file format specification to improve its version compatibility rules, and add specification for potential future features.
 * Fixed referencing the Farkle package in F# Interactive.
 * Fixed failures when rendering templates from very large grammars.
 * Fixed a bug where the `IsSingleTokenizerInChain` extension method would return wrong results.

designs/7.0/grammar-file-format-spec.md

@@ -2,12 +2,23 @@
 
 This document describes the binary format of Farkle 7's grammars. It is heavily inspired by the Common Language Infrastructure metadata format described in [ECMA-335][ecma].
 
+The current version of the format is __7.0__.
+
 ## Ground rules
 
 * The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
+* Non-normative text is written in blockquotes.
 * Numbers are stored in little-endian format.
 * The total size of a grammar file MUST NOT exceed 2<sup>31</sup> - 1 bytes.
 
+## Implementation requirements
+
+Implementations of this specification are divided into three categories:
+
+* _Writers_ create grammar files. They MUST implement the subset of the specification that is necessary to create valid grammar files according to the specification and their use case.
+* _Readers_ read grammar files for the purpose of displaying their content to users. They MAY ignore parts of the specification that dictate the behavior of parsers, or data structure specifications that they don't need to display.
+  * _Parsers_ are a subset of readers, and read grammar files for the purpose of parsing input according to the grammar's specification, either directly, or by generating code that performs the parsing. They are REQUIRED to implement the entire specification, unless specified otherwise. They MAY forgo implementing parts of the specification that they don't need, as long as they report an error when encountering data that would require such parts to be implemented.
+
 ## Abstract file structure
 
 From a top-level perspective, a grammar file is made of a _header_ and a sequence of _streams_.
@@ -25,10 +36,10 @@ A grammar file starts with the following data:
 
 * If the first eight bits of the file are different than the __Magic__ field's expected value, readers MUST NOT read past them and MUST report an error.
     > Note that the magic code ends with two zeroes. It will prevent [GOLD Parser][gold] grammar file readers from continuing reading it.
-* If the value of the __MajorVersion__ field is larger than the expected one or smaller than `7`, readers MUST NOT read past the __MinorVersion__ field and MUST report an error.
-    * When and if a file version after 7 gets specified, if the version of the file is smaller than the latest the reader supports, it MAY keep compatibility with the older format, or report an error.
+* If the value of the __MajorVersion__ field is outside the range of the major versions the reader supports, readers MUST NOT read past the __MinorVersion__ field and MUST report an error.
+* If the value of the __MajorVersion__ field is equal to the latest supported major version, and the value of the __MinorVersion__ field is larger than the latest supported major version, the grammar file MUST NOT be used for parsing.
 
-A different than expected __MajorVersion__ field indicates that the file cannot be read at all. A different than expected __MinorVersion__ field indicates that the file can be read, but might be incorrectly interpreted.
+> A different than expected __MajorVersion__ field indicates that the file cannot be read at all. A different than expected __MinorVersion__ field indicates that the file can be read, but might be incorrectly interpreted.
 
 ### Stream Definition
 
@@ -208,8 +219,8 @@ The following bit values are defined for the __Flags__ column:
 |Bit|Name|Description|
 |---|----|-----------|
 |0|`EndsOnEndOfInput`|The group can also end when the end of the input is reached, without encountering the token symbol specified in the __End__ column.|
-|1|`AdvanceByCharacter`|When inside this group, the parser should read the input without invoking the regular tokenizer.|
-|2|`KeepEndToken`|When the group ends, the parser should keep the token that ended the group in the input stream.|
+|1|`AdvanceByCharacter`|When inside this group, the parser must read the input without invoking the regular tokenizer.|
+|2|`KeepEndToken`|When the group ends, the parser must keep the token that ended the group in the input stream.|
 
 The following rules apply to the _Group_ table:
 
@@ -315,7 +326,7 @@ The following rules apply to the _Production_ table:
 * The value of the __FirstMember__ column of the first production MUST be equal to one.
 * If a production and all productions after it does not have any members, its __FirstMember__ column MUST be equal to the number of rows in the _ProductionMember_ table plus one.
 
-> Before accessing the members of a production, readers MUST ensure that it actually has members.
+> Before accessing the members of a production, readers MUST ensure that the production actually has members.
 
 ### _ProductionMember_ table
 
@@ -343,15 +354,20 @@ The following values are defined for the __Kind__ column:
 |3|LR(1) state machine.|
 |4|Generalized LR(1) (GLR(1)) state machine.|
 |5|Deterministic Finite Automaton (DFA) group start states on 16-bit character ranges.|
+|6|Deterministic Finite Automaton (DFA) on 8-bit character ranges.|
+|7|Deterministic Finite Automaton (DFA) on 8-bit character ranges with conflicts.|
+|8|Deterministic Finite Automaton (DFA) default transitions on 8-bit character ranges.|
+|9|Deterministic Finite Automaton (DFA) group start states on 8-bit character ranges.|
 |_anything else_|Reserved for future use by the Farkle project.|
 
 > Instead of GLR(1) we could have called it "LR(1) state machine with conflicts" for symmetry, but this kind of state machine has an established name. Currently there are no plans to support GLR parsing in the Farkle project.
 
 The following rules apply to the _StateMachine_ table:
 
 * The __Kind__ column MUST NOT contain duplicate values.
-* If both state machines of __Kind__ 0 and 1, or 3 and 4 exist, they MUST describe the same state machine, with their only difference being in the preferred values in case of conflicts.
-* If a state machine of __Kind__ 2 exists, a state machine of __Kind__ 0 or 1 MUST also exist.
+* If both state machines of __Kind__ 0 and 1, 3 and 4, or 6 and 7 exist, they MUST describe the same state machine, with their only difference being in the preferred values in case of conflicts.
+* If state machines of __Kind__ 2 or 5 exist, a state machine of __Kind__ 0 or 1 MUST also exist.
+* If state machines of __Kind__ 8 or 9 exist, a state machine of __Kind__ 6 or 7 MUST also exist.
 
 State machines with no states MUST be treated as if they do not exist.
 
@@ -369,7 +385,7 @@ The following rules apply to the _SpecialName_ table:
 * The __Symbol__ column MUST NOT contain duplicate values.
 * The __Name__ column MUST NOT contain duplicate values.
 
-> The use case for this table is to help custom code that integrates with parsers such as tokenizers. Since in Farkle symbols can be renamed and many can have the same name within a grammar, the special name provides a stable way to identify them (it sticks to the symbol's original name and duplicate names would cause build failures).
+> The use case for this table is to help custom code that integrates with parsers such as tokenizers. Since many symbols can have the same name within a grammar, the special name provides a guaranteed unique way to identify them.
 
 ## State machines
 
@@ -512,6 +528,19 @@ Readers SHOULD expose an API that indicates whether a grammar file contains data
 * A state machine whose __Kind__ is not known to the reader.
 * A table whose kind is not specified in this specification.
 
+### Summary of extensibility mechanisms
+
+The following table summarizes the extensibility mechanisms provided by the format, and whether third parties can use them:
+
+|I want to|How to achieve it|Available to third parties|
+|-|-|-|
+|Extend the format with data that does not affect parsing behavior and can be safely ignored|Add a custom stream|Yes|
+|Extend the format with data that might affect parsing behavior|Add a custom state machine|Yes|
+|Extend the format with data that might affect parsing behavior|Add a custom table|No|
+|Instruct parsers to refuse parsing if they do not understand your format extensions|Set the `Critical` flag in the _Grammar_ table|Yes|
+|Instruct parsers to refuse parsing if they do not understand your format extensions, while keeping the `Critical` flag available for third parties to use|Increase the __MinorVersion__ field in the file header|No|
+|Substantially change the format in a way incompatible for both readers and parsers|Increase the __MajorVersion__ field in the file header|No|
+
 [ecma]: https://www.ecma-international.org/publications-and-standards/standards/ecma-335/
 [rfc2119]: https://www.rfc-editor.org/rfc/rfc2119
 [gold]: http://goldparser.org

src/Farkle/Builder/GrammarBuild.cs

@@ -2,6 +2,7 @@
 // SPDX-License-Identifier: MIT
 
 using System.Collections.Immutable;
+using System.Diagnostics;
 using System.Numerics;
 using Farkle.Builder.Dfa;
 using Farkle.Builder.Lr;
@@ -50,6 +51,10 @@ private static string ExtractFirstPossibleCharacters(Regex regex)
         // Support only the subset of regexes the builder currently supports to start and end groups.
         // If we want to generalize groups in the future and have them be bounded by arbitrary regexes,
         // we might need to somehow run this inside the DFA builder.
+        while (regex.IsAccept(out Regex? r, out _, out _))
+        {
+            regex = r;
+        }
         if (regex == NewLineRegex)
         {
             return "\n\r";
@@ -67,9 +72,10 @@ private static string ExtractFirstPossibleCharacters(Regex regex)
     /// for when the tokenizer is inside a group. If a custom DFA cannot be used,
     /// this function will return <see langword="null"/>.
     /// </summary>
-    private static Regex? GetGroupRegex(string start, Regex endRegex, TokenSymbolHandle endSymbol, bool isRecursive,
-        GroupAttributes groupAttributes)
+    private static Regex? GetGroupRegex(string start, Regex endRegexWithAccept, bool isRecursive,
+        GroupAttributes groupAttributes, out bool addEndRegexToMainDfa)
     {
+        addEndRegexToMainDfa = true;
         if ((groupAttributes & GroupAttributes.AdvanceByCharacter) == 0)
         {
             // Token groups cannot use a custom DFA starting state by definition.
@@ -93,15 +99,16 @@ private static string ExtractFirstPossibleCharacters(Regex regex)
         bool keepEndToken = (groupAttributes & GroupAttributes.KeepEndToken) != 0;
         if (keepEndToken)
         {
-            prohibitedCharacters = prohibitedCharacters.AddRange(ExtractFirstPossibleCharacters(endRegex));
+            prohibitedCharacters = prohibitedCharacters.AddRange(ExtractFirstPossibleCharacters(endRegexWithAccept));
         }
         // Set HighPriorityInverted because, if a recursive group starts and ends with the same character,
         // we must fail and leave it to the main DFA to determine which of the two (or none) happened.
         // Set BreakOnAccept, in order to stop reading random text when the group end gets matched.
         Regex result = Regex.Chars(prohibitedCharacters, Regex.CharsFlags.HighPriorityInverted | Regex.CharsFlags.BreakOnAccept).ZeroOrMore();
         if (!keepEndToken)
         {
-            result += Regex.Accept(endRegex, endSymbol, lowestPriority: false);
+            result += endRegexWithAccept;
+            addEndRegexToMainDfa = false;
         }
         return result;
     }
@@ -352,15 +359,27 @@ void HandleGroup(string name, string start, string? endOrNewLine, GroupOptions o
             }
             else
             {
-                endHandle = GetOrCreateGroupEndLiteral(endOrNewLine, out endRegex);
+                endRegex = GetRegexForLiteral(endOrNewLine);
+                endHandle = GetOrCreateGroupEndLiteral(endOrNewLine);
             }
             bool isRecursive = (options & GroupOptions.Recursive) != 0;
             GroupHandle groupHandle = writer.AddGroup(writer.GetOrAddString(name), container, flags, startHandle, endHandle, isRecursive ? 1 : 0);
             if (isRecursive)
             {
                 writer.AddGroupNesting(groupHandle);
             }
-            groupDfaRegexes?.Add(GetGroupRegex(start, endRegex, endHandle, isRecursive, flags));
+            if (regexBuilder is not null)
+            {
+                Regex endRegexWithAccept = Regex.Accept(endRegex, endHandle, lowestPriority: false);
+                bool addEndRegexToMainDfa = true;
+                groupDfaRegexes?.Add(GetGroupRegex(start, endRegexWithAccept, isRecursive, flags, out addEndRegexToMainDfa));
+                if (addEndRegexToMainDfa)
+                {
+                    // The regex might be added multiple times, but it's OK since all accept
+                    // the same symbol, and won't cause any conflicts.
+                    regexBuilder.Add(endRegexWithAccept);
+                }
+            }
         }
 
         // Gets the handle to a group end literal symbol, creating it if it does not exist.
@@ -373,16 +392,14 @@ void HandleGroup(string name, string start, string? endOrNewLine, GroupOptions o
         // bookkeeping ourselves. By storing the strings inside the general symbol map, we
         // also avoid conflicts between group end symbols and literals, which was not possible
         // before.
-        TokenSymbolHandle GetOrCreateGroupEndLiteral(string content, out Regex regex)
+        TokenSymbolHandle GetOrCreateGroupEndLiteral(string content)
         {
-            regex = GetRegexForLiteral(content);
             if (symbolMap.TryGetValue(content, out EntityHandle existingHandle))
             {
                 return (TokenSymbolHandle)existingHandle;
             }
             TokenSymbolHandle handle = writer.AddTokenSymbol(writer.GetOrAddString(content), TokenSymbolAttributes.None);
             dfaSymbols?.Add(handle, content, TokenSymbolKind.GroupEnd);
-            regexBuilder?.Add(Regex.Accept(regex, handle, lowestPriority: false));
             symbolMap.Add(content, handle);
             return handle;
         }

src/Farkle/Grammars/Grammar.cs

@@ -23,6 +23,16 @@ public abstract partial class Grammar : IGrammarProvider
     internal readonly BlobHeap BlobHeap;
     internal readonly GrammarTables GrammarTables;
 
+    /// <summary>
+    /// The grammar file's format major version.
+    /// </summary>
+    public ushort FormatVersionMajor { get; }
+
+    /// <summary>
+    /// The grammar file's format minor version.
+    /// </summary>
+    public ushort FormatVersionMinor { get; }
+
     /// <summary>
     /// A read-only buffer to the <see cref="Grammar"/>'s binary data.
     /// </summary>
@@ -112,6 +122,8 @@ private protected Grammar(ReadOnlySpan<byte> grammarFile)
     {
         GrammarHeader header = GrammarHeader.Read(grammarFile);
         ValidateHeader(header);
+        FormatVersionMajor = header.VersionMajor;
+        FormatVersionMinor = header.VersionMinor;
 
         GrammarStreams streams = new(grammarFile, header.StreamCount);
 
@@ -433,6 +445,11 @@ internal bool IsUnparsable([NotNullWhen(true)] out string? errorResourceKey)
             errorResourceKey = nameof(Resources.Parser_UnparsableGrammar);
             return true;
         }
+        if (FormatVersionMajor == GrammarConstants.VersionMajor && FormatVersionMinor > GrammarConstants.VersionMinor)
+        {
+            errorResourceKey = nameof(Resources.Parser_UnparsableGrammar_TooNewFormat);
+            return true;
+        }
         if (HasUnknownData && (flags & GrammarAttributes.Critical) != 0)
         {
             errorResourceKey = nameof(Resources.Parser_UnparsableGrammar_Critical);

src/Farkle/Grammars/GrammarConstants.cs

@@ -22,6 +22,11 @@ internal static class GrammarConstants
     public const uint DfaOnCharDefaultTransitionsKind = 2;
     public const uint DfaOnCharGroupStartStatesKind = 5;
 
+    public const uint DfaOnByteKind = 6;
+    public const uint DfaOnByteWithConflictsKind = 7;
+    public const uint DfaOnByteDefaultTransitionsKind = 8;
+    public const uint DfaOnByteGroupStartStatesKind = 9;
+
     public const uint Lr1Kind = 3;
     public const uint Glr1Kind = 4;
 

src/Farkle/Grammars/GrammarStateMachines.cs

@@ -33,6 +33,12 @@ public GrammarStateMachines(ReadOnlySpan<byte> grammarFile, in BlobHeap blobHeap
                 case GrammarConstants.DfaOnCharGroupStartStatesKind:
                     AssignStateMachine(grammarFile, in blobHeap, kind, data, ref DfaOnChar.GroupStartStates, ref seenDfaOnCharGroupStartStates);
                     break;
+                case GrammarConstants.DfaOnByteKind:
+                case GrammarConstants.DfaOnByteWithConflictsKind:
+                case GrammarConstants.DfaOnByteDefaultTransitionsKind:
+                case GrammarConstants.DfaOnByteGroupStartStatesKind:
+                    // We don't currently support DFAs on bytes; ignore them but don't set HasUnknownData.
+                    break;
                 case GrammarConstants.Lr1Kind:
                     AssignStateMachine(grammarFile, in blobHeap, kind, data, ref Lr1, ref seenLr1);
                     break;

src/Farkle/Properties/Resources.cs

@@ -227,6 +227,8 @@ public static string GetEofString(IFormatProvider? formatProvider)
 
     public static string Parser_UnparsableGrammar_Critical => GetResourceString(nameof(Parser_UnparsableGrammar_Critical));
 
+    public static string Parser_UnparsableGrammar_TooNewFormat => GetResourceString(nameof(Parser_UnparsableGrammar_TooNewFormat));
+
     public static string Parser_GrammarLrMissing => GetResourceString(nameof(Parser_GrammarLrMissing));
 
     public static string Parser_GrammarLrProblem => GetResourceString(nameof(Parser_GrammarLrProblem));

src/Farkle/Properties/Resources.el.resx

@@ -72,6 +72,9 @@
   <data name="Parser_UnparsableGrammar_Critical" xml:space="preserve">
     <value>Η γραμματική δεν μπορεί να χρησιμοποιηθεί για συντακτική ή λεκτική ανάλυση επειδή περιέχει δεδομένα που δεν αναγνωρίζονται από αυτήν την έκδοση του Farkle</value>
   </data>
+  <data name="Parser_UnparsableGrammar_TooNewFormat" xml:space="preserve">
+    <value>Η γραμματική δεν μπορεί να χρησιμοποιηθεί για συντακτική ή λεκτική ανάλυση επειδή η έκδοση μορφής της είναι πολύ νέα</value>
+  </data>
   <data name="Parser_GrammarLrMissing" xml:space="preserve">
     <value>Η γραμματική δεν μπορεί να χρησιμοποιηθεί για συντακτική ανάλυση επειδή δεν έχει πίνακα καταστάσεων LR</value>
   </data>

src/Farkle/Properties/Resources.resx

@@ -72,6 +72,9 @@
   <data name="Parser_UnparsableGrammar_Critical" xml:space="preserve">
     <value>The grammar cannot be used for parsing or tokenizing because it contains data not recognized by this version of Farkle</value>
   </data>
+  <data name="Parser_UnparsableGrammar_TooNewFormat" xml:space="preserve">
+    <value>The grammar cannot be used for parsing or tokenizing because its format version is too new</value>
+  </data>
   <data name="Parser_GrammarLrMissing" xml:space="preserve">
     <value>The grammar cannot be used for parsing because it does not contain an LR state table</value>
   </data>