Compress the source when posting to build server

[foolip]

Just uploading the source accounts for the majority of the build time when you're on a slow network.

https://tools.ietf.org/html/rfc2388#section-5.1 says "do it yourself", so I see two options:

1. Add a sourcegz field or something that takes the compressed source instead.
2. Make the build server use HTTPS, and verify that the negotiated connection uses compression.

I think HTTPS makes sense, for other obvious reasons as well.

[foolip]

@domenic?

[domenic]

Yeah, I've been meaning to add HTTPS for a while, especially since AWS apparently makes that easy now.

[foolip]

Another thing one could do is to assume that the build server has a copy of the html repo, and only send a diff against the merge-base with whatwg/html. A bit elaborate, of course, and the next bottleneck would be sending the output back.

[zcorpan]

Could we use dropbox for this, or some similar service? https://www.dropbox.com/help/8

[foolip]

Unfortunately, as tested with Wireshark in whatwg/html-build#64 (comment) there is no automatic compression at the TLS layer.

[foolip]

So, just to break this down and see where the low hanging fruit is, here's what we send:

• source-whatwg-complete is 6055 kB and gzips to 1158 kB
• caniuse.json is 980 kB and gzips to 135 kB
• w3cbugs.csv is 245 kB and gzip to 96 kB

The returned wattsi-output.zip is 4218 kB and is already compressed using Defl:N (so says unzip -lv), which is the same algorithm that gzip uses.

Potential compression in total is from ~7.2 MB to ~1.4 MB, and we only need to fiddle with the posted data.

[foolip]

As for the imagined automatic compression of TLS, a colleague has educated me, and while it's part of the protocol it's been turned off in browsers because of the security issues, "adaptive chosen plaintext attacks." Even if we can get it to work with curl, it's probably not a long-term safe bet.

So, @domenic, do you think you could add support for a .gz variant of each field? Unless someone knows of a way to get the whole request body compressed at the HTTP level, since TLS is the wrong level. Maybe HTTP2 can do it?

[haavardmolland]

The TLS spec does support compression, but is considered bad as it opens up for attacks. Compression attacks belong in "adaptive chosen plaintext" category though, which means you need to somehow control part of what the agent sends. This is hard to conduct, and mostly browsers are vulnerable. However, since it's considered harmful I wouldn't use it, as it is or will most likely be turned off in whatever TLS stack you are using.

[foolip]

@haavardmolland informs me that HTTP2 compresses (almost) everything, so that would be an option. curl has a --http2 option that presumably works at least some of the time. @domenic, does your build server know how to speak HTTP2?

[domenic]

I would love to make my build server http2 aware. I will try to figure that out over the next day or two.