GH-41298: [Format][Docs] Add a canonical extension type specification for UUID

[rok]

Rationale for this change

Several users have expressed a need for a UUID type. This is to provide a canonical UUID extension type specification.

What changes are included in this PR?

This adds to documentation of canonical extension types.

Are these changes tested?

No as only docs are changed.

Are there any user-facing changes?

No.

• GitHub Issue: [Format][Docs] Add a canonical extension type specification for UUID #41298

[github-actions]

⚠️ GitHub issue #41298 has been automatically assigned in GitHub to PR creator.

[mbrobbel]

Isn't FixedSizeBinary a logical type? Is it specified somewhere what physical layout should be used? Unless I'm mistaken there are two options: fixed-size primitive layout (FixedSizePrimitive<[u8; 16]>) or fixed-size list layout (FixedSizeList<u8>(16))?

[rok]

I assumed FixedSizeBinary maps to a single physical type (fixed size primitive). If this is ambiguous we should indeed specify (preferrably one) physical type.

[jorisvandenbossche]

AFAIK we want to use a logical type here, since that are the concrete types that are actually used in the various specs like IPC and C Data Interface (the phsyical layouts are just described in the format docs, but it are the logical types that are listed in the flatbuffer Schema.fbs).
While "storage type" might sound like it points to a physical layout, the docs also mention that extension types annotate any built-in logical type: https://arrow.apache.org/docs/dev/format/Columnar.html#extension-types

Sidenote: there is not really such a thing as a "physical type", but only physical layout. Next to the various layouts, we just have "types" on top of that, which are sometimes called "logical types", but we are not very consistent about that terminology ..

[emkornfield]

it seems that specifying the version should be optional metadata?

[felipecrv]

UUID versioning seems like a concern for the generator and consumers. It could become very difficult for Arrow to guarantee anything about versions without validating the values one by one.

[felipecrv]

You should specify what byte (0 or 15) is most-significant byte.

For instance, Java treats the 0-th byte as the MSB [1] which is consistent with the convention that big-endian the network byte order [2]:

    private UUID(byte[] data) {
        long msb = 0;
        long lsb = 0;
        assert data.length == 16 : "data must be 16 bytes in length";
        for (int i=0; i<8; i++)
            msb = (msb << 8) | (data[i] & 0xff);
        for (int i=8; i<16; i++)
            lsb = (lsb << 8) | (data[i] & 0xff);
        this.mostSigBits = msb;
        this.leastSigBits = lsb;
    }

When porting this code to C/C++, be careful with the fact that Java's integer types are big-endian (inheritance from SPARC), unlike most architectures we use today that are little-endian.

An advantage of putting the MSB at byte 0 is that when you parse an UUID string you read the string from the MSB to the LSB and write the UUID data from 0 to 15.

[1] https://github.com/openjdk/jdk/blob/819f3d6fc70ff6fe54ac5f9033c17c3dd4326aa5/src/java.base/share/classes/java/util/UUID.java#L116-L126
[2] https://stackoverflow.com/questions/13514614/why-is-network-byte-order-defined-to-be-big-endian

[rok]

Thanks for the suggestion! I've added a comment (0eaa1b9) and reported to the vote thread.

[felipecrv]

LGTM. Merge depends on mailing list votes.

[rok]

@emkornfield do you have any concerns left or shall we proceed and merge?

[rok]

If there's no further comments I'll merge this today.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit c5be027.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.