Compression

[pnorman]

Currently the specification does not allow compression, but some PBFs seen in the wild are compressed, and not at the transport layer (HTTP).

The specification should be explicit on if compression is allowed.

[kkaefer]

Compression should always happen at the transportation/storage layer. Do you have an example of a PBF in the wild that is served compressed?

[pnorman]

quoting @ajashton when I sent him an uncompressed tile

Mapbox Studio expects PBF tiles to be compressed with either gzip or deflate. When served with gzip-encoding I was able to load and style the tile with the tilejson you provided.

cc @zerebubuth

[gravitystorm]

mapbox/tilelive-vector#71 (comment) and mapbox/tilelive-bridge#21 (comment) both suggest that the tiles should be compressed, even at rest on a plain disk.

[kkaefer]

Yeah, but the Spec doesn't provide a storage facility, so it is up to the storage mechanisms to transparently compress/decompress the data.

[zerebubuth]

What's confusing me is whether this should be happening at the transport (or storage) layer, as @kkaefer is saying, or is an intrinsic part of being a "mapnik vector tile" as @jfirebaugh is saying.

If it's handled at the transport layer, then an HTTP request for a mapnik vector tile which specifies no Accept-Encoding (or "Accept-Encoding: identity") should receive an uncompressed PBF. If it's intrinsic, then it should receive a gzip compressed PBF, despite still having "Encoding: identity".

[jfirebaugh]

The conventions are at the API level, not the specification level, and are driven by pragmatic constraints:

1. Vector tiles benefit from compression in transport and in storage.
2. We wish to offload decompression to the user agent in client side (GL rendering) scenarios.
3. We wish to avoid repetitive decompression/recompression when a VT travels through a tilelive pipeline.
4. gzip is the only reliable transport encoding for HTTP.

Together, these constraints led to the current solution: vector tiles are gzipped at rest, and served with Content-Encoding: gzip for HTTP transport. Browser user agents will automatically decompress HTTP bodies, satisfying #2. Non-browser user agents typically do not automatically decompress HTTP bodies, satisfying #3.

This means:

• vector-tile-js expects uncompressed vector tile data
• tilelive-* modules expect gzipped input and generate gzipped output

[pnorman]

cc @yohanboniface, as another author of vector tile consuming software

Leaving aside the storage what you're saying is vector tiles are uncompressed, but

• HTTP requests to feed tilelive-* modules will have an Accept-Encoding that only accepts gzip, e.g. Accept-Encoding: gzip;q=1, identity;q=0 and will need to not decompress the response before passing it to the tilelive-* module
• Servers based on tilelive-* modules will respond with a 406 error to Accept-Encoding: identity;q=1, gzip;q=0 as they only have a gzipped version of the file and do not decompress on the fly

Is this correct?

On-disk storage is an interesting question, as what is optimal can depend on the filesystem, the disks, and what is consuming the tiles.

[bcamper]

FWIW we ran into this issue recently when @ericfischer was trying to use the Mapzen vector tiles in Mapbox Studio. We were serving gzipped tiles when Accept-Encoding included gzip, but it looked like Studio was not including an Accept-Encoding header in its requests. We changed the service to always serve gzipped tiles for compatibility with the current Studio behavior, but ideally we would only do so when Accept-Encoding is present.

[pnorman]

We were serving gzipped tiles when Accept-Encoding included gzip, but it looked like Studio was not including an Accept-Encoding header in its requests.

Serving gzip if Accept-Encoding is not specified is okay. The HTTP spec says servers should use identity in such cases, not must. Serving gzip if identity is requested is broken.

I must admit that I haven't looked at the MBS HTTP requests - does it currently send an accept-encoding header which disallows identity encoding?

[bcamper]

I cannot recall offhand, I believe no Accept-Encoding was present, I'm sure a Mapboxer here could say :)