[json] when downloading JSON Schema files, do Conditional GETs

[madskristensen]

The requests coming in for JSON Schema files to SchemaStore.org don't check if the remote files have changed before downloading them. This puts unneeded pressure on the SchemaStore.org servers and it downloads more data than needed which will cost money for users that pay bandwidth by the megabyte.

By supporting Conditional GETs (HTTP 304) using either If-Modified-Since or If-None-Match request header, these issues are fixed.

[madskristensen]

@aeschli I was hoping this could get a higher priority. I've gotten the VS team to prioritize this as well on their end. The reason is that I pay for server traffic and the free tier is no longer enough to handle the growing load from VS, VSCode and other editors.

[aeschli]

I need to plan time for this, as it requires implementing http caching on the vscode/node side.
If you have any tips, let me know.

[madskristensen]

Thanks. I understand.

One way to implement this could be this flow:

1. VS Code was just installed. No cache exist yet
2. User opens JSON file triggering a download of schema file
3. Schema downloaded and HTTP Last-Modified response header is timestamped into the file when it was saved
4. User opens the JSON file again within 7 days. VS Code never downloads new schemas if the current cached one is younger than 7 days (or however many days are appropriate for VS Code users)
5. User opens JSON file again 10 days later, this time VSCode sends If-Modified-Since header containing the file's timestamp
6. SchemaStore.org sends HTTP 304 Not Modified empty response because schema didn't have an update
7. VS Code uses the schema on disk instead

You can send these headers using fetch from node.js and then check the HTTP status code from the response.

I believe this is the same conceptual pattern implemented by VS as well. Right @jimmylewis?

[jimmylewis]

Yes, that pretty much summarizes the VS implementation. Should note that:

• for an expired local cache, we'll load the local copy, make the request, then reload JSON schema evaluation when we get a response. This keeps user responsiveness high for slow/flaky connections.
• getting the 304 back should reset the local cache expiration for another 7 days. This minimizes bandwidth usage.
• we also have a gesture to forcibly clear the local cache and force all schemas to download again the next time they are used.

[aeschli]

I added a cache with a local cache expiration of 3 days (for now). After the cache expiration we check the content again with the last known etag.