Multiple uploads resulting in CANCELED

[Cyberes]

Description

I am attempting to upload two video files (each about 30 MB) to a room at the same time. This always fails with a generic error message popup (The file '[filename]' failed to upload.). There is no error in the console and the network of the debug window shows the fetch request as (canceled). Synapse logs don't show any errors and neither does Openresty. Uploading the files one at a time works, but uploading them together always fails.

• My upload limit in Synapse and Openresty (reverse proxy) is 40M. This has been an issue for a while and I have not discovered the root cause yet.
• My client is desktop and web Element on Ubuntu.
• I have a media worker with an S3 storage provider setup (see config below).

Steps to reproduce

• ???

Homeserver

matrix.evulid.cc

Synapse Version

1.86.0

Installation Method

pip (from PyPI)

Database

PostgreSQL

Workers

Multiple workers

Platform

Ubuntu Server LXC container

Configuration

media_store_path:                                 "/mnt/matrix-storage/media"

media_storage_providers:
- module:                                         s3_storage_provider.S3StorageProviderBackend
  store_local:                                    True
  store_remote:                                   True
  store_synchronous:                              True
  config:
      bucket:                                     matrix-synapse
      endpoint_url:                               'https://xxx.r2.cloudflarestorage.com/matrix-synapse'
      access_key_id:                              'xxx'
      secret_access_key:                          'xxx'

Relevant log output

No unusual logs.

Anything else that would be useful to know?

No response

[Cyberes]

Looks like the 30MB file fails when individually uploaded.

[MadLittleMods]

I know you mentioned that there aren't any relevant logs but seems like it could be related to #12937 -> #13009

Are you using Element to upload the file? Element will cancel the request if it doesn't see any progress for 30 seconds (see matrix-org/matrix-js-sdk -> src/http-api/index.ts#L102-L111)

It seems like TCP dumps from the client <-> server would be useful to debug this further.

Other related issues:

• Media upload fails with ValueError: seek of closed file #6192
• ValueError: seek of closed file when client aborts media upload at exactly the right time #11154 (comment)

[Cyberes]

I'm no web developer, but I checked the debugger and saw that I am indeed triggering the timeout. The upload only takes a few seconds but the bar sits at 100% for a bit before the timeout kills it.

I also moved the media store to an SSD and put /tmp on RAM. I don't think it's related to the file moving issue because its only a 30MB file.

[clokep]

Is your reverse proxy killing the upload? I think by default they usually have small limits.

[MadLittleMods]

@Cyberes Does the media end up on your server after the dust settles? (if so, feels like it's one of the issues I've already linked)

@clokep also has an interesting angle with the reverse-proxy.

Otherwise, seems like a case of tracing or adding logs to see where Synapse is stuck at. I've added some extra instrumentation so traces provide a lot more info when using the /upload endpoint -> #15850

Are you able to enable tracing for your Synapse instance (will need wait for the PR to merge or apply those changes manually)? Please share the trace JSON files if you get this going

[Cyberes]

There isn't anything in the nginx error logs. I have client_max_body_size 40M; in my config.

And find /mnt/matrix-storage/ -name "*[ID returned by /upload]*" does show the file in local_content.

I'll pull that branch and enable tracing.

[Cyberes]

Here's the trace.

evulid.cc media_worker UploadResource.json.txt

From the clients perspective, it was canceled after 33.68s of being in-flight: