Unable to use evaporate.js in a web worker

[m-andrew-albright]

Due to the way evaporate.js parses a url into a uri object (see https://github.com/TTLabs/EvaporateJS/blob/master/evaporate.js#L1981) by using document.createElement('a');, it cannot currently be used in a web worker without some modifications (such as pulling in jsdom, which I have not yet tried so I can't guarantee it works).

Currently, we are using AWS Signature V2, and calculating the MD5 on large file uploads causes our UI to stutter and stop responding until the uploading is finished; this is the reason we are trying to use evaporate.js in a web worker in the first place, so if there is some workaround to this performance issue, I would greatly welcome any recommendations.

Thanks for the awesome library!

[bikeath1337]

That makes send @m-andrew-albright but what I still don't understand is why your client is blocking when creating the checksum. I've not seen any delays with part sizes of 15MB or less. What is your part size?

I'm open to other solutions for parsing a URL. Using document.createElement seemed like a safe, reliable option.

[m-andrew-albright]

I believe we are using the default; which is 6 * 1024 * 1024.

Now I'm not so sure that this is the problem, because I was able to successfully modify evaporate to allow asynchronous md5 generation, which I delegated to a web worker, but my app still becomes very unresponsive and jittery while uploading.

I've only been doing front end dev for about eight months, and I still have a difficult time understanding the cpu profiler in chrome's devtools, but it seemed to me that the MD5 calculation was the culprit:

^ This is prior to my attempt to make the MD5 asynchronous and delegated into a web worker, and this is after:

It's my understanding that (program) is native code running, and if the jittering/unresponsiveness is due to that, it may be difficult or impossible to pinpoint exactly what is causing this.

Thanks for responding!

[bikeath1337]

Thank you. FWIW, Evaporate calls the MD5 method asynchronously already: a) when fetching the part bytes to digest b) through the Promise method that creates the digest.

Why don't you just diff two times in your MD5 method and verify for sure?

[m-andrew-albright]

By "asynchronous" I really meant to say "in another thread" which - the only way to do that in client JS is by using web workers. It really seems like the application's unresponsiveness corresponds to the timings of calculating the MD5, and that is one of the most CPU-intensive things happening during these file uploads. However, getting the MD5 to be calculating in the web worker didn't seem to have any impact on these issues we're having.

I'm not sure what you mean by "diff" two times in my MD5 method; the only "diff" I'm familiar with is diffing two files for changes.

Thanks for taking the time to discuss this with me.

[bikeath1337]

No problem. By "diff" i meant, take calculate the delta of the start time and end time within the method to verify the approximate time cost to create the digest.

[m-andrew-albright]

Ah yes, I did that prior to moving the MD5 into a web worker, and with AWS-SDK, it was something like 500ms per call; with SparkMD5, it's consistently 160ms. I will try calculating within the web worker and let you know what that looks like (if I understand correctly, I would use console.time() in the web worker before and after the MD5 function call)

[bikeath1337]

#312 proposes a small change to how the aws_url is parsed. Would this be an improvement for compatibility with node for you?

[m-andrew-albright]

That looks like it might help - I can't tell if URL is available in web workers though:

https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Functions_and_classes_available_to_workers#APIs_available_in_workers

It looks like it isn't available except in Firefox, as best I can tell according to that table.

[YoshiWalsh]

Hi,

I'm running EvaporateJS in a WebWorker too. I got it working without modifications by using a few shims and NPM packages.

var window = self; // For Worker-unaware scripts

// Shim to make Evaporate work in a Worker
var document = {
	createElement: function() {
		var href = undefined;
		
		var elm = {
			set href(url) {
				var obj = new URL(url);
				elm.protocol = obj.protocol;
				elm.hostname = obj.hostname;
				elm.pathname = obj.pathname;
				elm.port = obj.port;
				elm.search = obj.search;
				elm.hash = obj.hash;
				elm.host = obj.host;
				href = url;
			},
			get href() {
				return href;
			},
			protocol: undefined,
			hostname: undefined,
			pathname: undefined,
			port: undefined,
			search: undefined,
			hash: undefined,
			host: undefined
		};
		
		return elm;
	}
};

// Local storage shim, works around this bug: https://github.com/TTLabs/EvaporateJS/issues/316
(function() {
	var localStorageShim = {};
	window.localStorage = {
		getItem: function(item) {
			return localStorageShim[item];
		},
		setItem: function(item, value) {
			localStorageShim[item] = value;
		},
		removeItem: function(item) {
			localStorageShim[item] = undefined;
		}
	};
})();

importScripts("/lib/sha256/sha256.min.js");
importScripts("/lib/spark-md5/spark-md5.min.js");
importScripts("/lib/url-parse/url-parse.js");
importScripts("/lib/xmldom/xmldom.js");
importScripts("/lib/evaporate/evaporate.js");

DOMParser = self.xmldom.DOMParser;

xmldom and url-parse can be found on NPM and work in the browser after they've been Browserify'd. The easiest way to do this is to fetch them from wzrd.in (essentially Browserify-as-a-service):

https://wzrd.in/standalone/url-parse@latest

https://wzrd.in/standalone/xmldom@latest

I'm still running into the issue where the Worker is somehow blocking the main thread, but I only experience it in Chrome. Edge and Firefox don't have this problem. I made a StackOverflow post about it but haven't received any useful responses.

Best of luck getting it working, and let me know if you work out a solution to the blocking issue.

EDIT: I added an additional shim to work around #316

[m-andrew-albright]

@YMIndustries Thanks for chiming in; I actually came upon your SO post at one point in my journey to find the solution to this issue.

I have been pulled into other areas of our front end, but this bug is glaring at me from the top of the prioritized backlog, and I have no idea how to solve it if web workers don't do the trick.

[YoshiWalsh]

@m-andrew-albright I'm in the exact same position. I'm working on more pressing things at the moment, but our app allows users to type descriptions and such while their upload is running in the background, and the freezes every couple of seconds make the user experience pretty dreadful. It also really frustrates me because according to my understanding of Web Workers and all the documentation I can find, the way I've written it Should Work (tm). I actually think this is a Chrome bug, but in order to report it as such I'd need to create a much simpler way of reproducing it that I could share. I think I'll have to do that at some point, but I don't have the time right now.

[bikeath1337]

I don't see any performance glitches or hangs on any of my browsers: Chrome, Safari or Win 11 with a 6MB part size. Can you give me a test example I can try out locally (without web workers)? @m-andrew-albright

[bikeath1337]

Sorry the momentary lack of understanding. I think I now understand that you're using web works to offload the entire upload process. For some silly reason I was focusing on running just the checksum calculation.

I'm happy to address compatibility with Evaporate in a web worker and now understand that part of this discussion might be a bug in web workers that makes it not cooperative on Chrome.

[m-andrew-albright]

@bikeath1337 This is still on my backburner (but not off the stove) - as soon as I switch back over to trying to solve this issue, I will get you as much information as I can regarding our setup.

For now though, it's probably worth mentioning that we are using evaporate as part of a proprietary media system's client interface, and it is being used in an Aurelia SPA. It's possible that something Aurelia is doing is degrading the performance of evaporatejs; again, I'll get you as much information as I can when I switch back over to trying to solve this.

@YMIndustries I don't suppose you're using Aurelia, are you?