Provide raw path / URI

[blueyed]

The path currently gets unquoted / percent-decoded, which results in unrecoverable loss of information: a request to /foo/bar is the same as /foo%2Fbar.

With WSGI there seems to exist workarounds via RAW_URI / REQUEST_URI (e.g. in gunicorn, etianen/aiohttp-wsgi#17).

I think it would be nice to have raw_path officially in the spec.
I am not sure if it should be a unicode string, or bytes (to avoid another field later on).

(I do not think that raw_uri (path + query info) is necessary, since query_string is already given as bytes)

Current spec: https://asgi.readthedocs.io/en/latest/specs/www.html#id1
(Via Kludex/uvicorn#354)

[andrewgodwin]

We already looked at this in #51 - I am curious if there is a use for the raw path other than those discussed there. Specifically the fact that while WSGI claims to give you the raw path, this is not always actually true.

[blueyed]

Oh, sorry for missing #51. I've only skimmed this quickly - you said there:

if we provided a "raw" one then people might rely on it only for it to break later on when a webserver in front did exactly the same thing (webservers: not good at adhering perfectly to the URI RFC)

So because others might break it it gets broken upfront? :)
It basically means that any proxy using ASGI is forced into the group of "webservers: not good at adhering perfectly to the URI RFC" then.

I can see that it is too late to change path now probably, but something like raw_path should be added.

I currently have to get it in via setting a custom header from nginx, updating the request.url from there then. This works for me, but should not be necessary really.

[andrewgodwin]

It's more that this falls into the bucket of "well, it's useful for 1% of people", and I'm not personally convinced the job of a spec is to get all those corner cases - e.g. ASGI also doesn't allow for certain kinds of responses that are valid in HTTP.

Is there an app you're developing that is not low-level HTTP but does need raw path? If I may ask, what specifically about the path do you need?

[blueyed]

I think it is more like "99% do not notice / are not affected".
The app I've noticed this with is a proxy after all.. :)

[blueyed]

And it is used e.g. with https://firebasestorage.googleapis.com/v0/b/noticeable-service.appspot.com/o/users%2F9s6zABHvXDeoEK3Pk0G3Z7gro513%2Fprojects%2FbYyIewUV308AvkMztxix%2Flogo?alt=media&token=9ad915fd-fd45-46fe-a82c-0d581d3113d8 - with ASGI the "users%2F" gets turned into "users/" then.

[davidism]

From what I remember of the pycon discussion that led to the original decision, basically we can't standardize this because we can't control what web servers return. So we have to standardize to the lowest common denominator, which is slash unquoting. ASGI servers can choose to send a raw url as an extension, but there's no guarantee it will be any more accurate. Ultimately the dev would need to configure their server correctly then choose the raw url.

[andrewgodwin]

Right, that's pretty much what I was getting at, thanks @davidism. I'd be open to defining this as an optional extra, but only if we think a decent number of servers are going to be able to supply it correctly, and with the caveat being "make it correct or don't supply it at all".

[blueyed]

Wouldn't the lowest denominator be to not touch it all, followed by turning it into unicode, and only then apply more processing (i.e. slash unquoting)?

(Just for some example, nginx will not touch it with proxy_pass http://upstream:8080, but with proxy_pass http://upstream:8080/ (trailing slash), so basically only when the path is adjusted (although "/" is a special case, but it applies to /mounted/ also))
Given that I could understand if servers would e.g. unquote an adjusted root_path, and then path there also maybe, but it should be available at the root level in the original form - this would also be useful for mounted apps then, too.

[andrewgodwin]

No, the main problem is that some servers auto-unquote for you - thus the lowest common denominator is actually the unquoted path. We went through a lot of revisions of path in the ASGI spec to get to this - I tried to keep it "raw", but there just kept being server bugs around things coming in already-unquoted.

[lovelydinosaur]

but there just kept being server bugs around things coming in already-unquoted.

For context are we talking about "server bugs" at the Python layer there, or does that appear to include behavior of HTTP intermediaries in the wild? Eg. fronting proxies, or whatevs.

Presumably some client implementations may also end up send unquoted values.

[lovelydinosaur]

Answering my own question by reference to #51

The path is subject to re-interpretation by both reverse-proxying webservers and downstream proxies