Not possible to do requests with pre-compressed data

[mwfrojdman]

Currently aiohttp always compresses the request data if the Content-Encoding header is set. If the data is already compressed, aiohttp will compress it a second time. Example:

url = 'http://some.host/'
deflated_data = zlib.compress(b'mydata')
headers = {'Content-Encoding': 'deflate'}
yield from aiohttp.request('POST', url, data=deflated_data, headers=headers)

Looking at client_reqrep, self.update_content_encoding() is called in ClientRequest.init() on line 88. The method first checks if the Content-Encoding header is set, and if it is, self.compress is set to that value and chunking turned on. Otherwise, self.compress (originally set in init() from the constructor's argument with the same name) is used.

Later in send(), request.add_compression_filter(self.compress) is called on line 460 if self.compress is set. This leads to the double compression.

Suggestion: Only do compression of data in aiohttp if init() is called with the compress argument. If the Content-Encoding header is set, leave it to the user to supply compressed data.

I ran into this user when looking at the raven-python's aiohttp transport (used to send exceptions & al to Sentry at https://github.com/getsentry/raven-aiohttp/blob/c35ca66379f00d58cb8404b51f845f0e754c91b5/raven_aiohttp.py#L41 which gets the data already compressed and Content-Encoding header set.

[asvetlov]

Well, we may assign ClientRequest.compress to False if Content-Encoding header is set and ClientSession.request() has compress=False parameter.
compress=True enforces compression, compress=None respects Content-Encoding.
The proposal keeps backward compatibility I believe.
The single line need to be changed: instead of https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/client_reqrep.py#L234 we can add something like:

if self.compress is not False:
    self.compress = enc

Is that satisfies your needs?
If yes -- would you provide a Pull Request?
PR requires test(s) and documentation update.

[mwfrojdman]

That would work. Though I'm also thinking that providing two ways in the API to instruct aiohttp to compress data makes its use more inconsistent among users: post(url, data=data, compress='deflate') does the exact same thing as post(url, data=data, headers={'Content-Encoding': 'deflate'}.

Backwards compatibility may be an issue though, so I'm preparing a pull req with the change you suggested.

[mwfrojdman]

Created pull req: #621

[asvetlov]

Done by #621

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.