Faster test iteration

[alanknight-wk]

This is a proof of concept of faster test integration. Rather than run a serve loop internally and watch it, this runs test completely separately. But I discovered that package:test has a --pub-serve= option. So we can run a ddev serve. separately, and run a test that just points at it. Except that it doesn't give us all the stuff we need. So we can make it duplicate what happens when build_runner test creates a merged directory. The advantage of that is that that merged directory takes e.g. almost 30 seconds to build for graph_ui. So this avoids any of that startup overhead and also runs nicely in a loop. It does accept test arguments, so you can restrict it to a subset of tests.

So we run a ddev serve somewhere, and then run a trivial server that statically serves the current directory, the test sub-dir, and .dart_tool/build/generated/<package_name>, and also has a proxy to the ddev serve. And we spawn that and then just a simple shell loop that does a dart run test, waits for user input, and does it again.

This is held together with string, but it does seem to work. Also, it's based off 3.7.0 because I was testing mostly against graph_ui, and didn't want to fight with newer deps.

Among the issues

• When I was running this separately from the command line the tests ran faster. The shelf server was also simpler at that point. And it's written very quickly without really understanding shelf. It ought to be able to be faster, and if it would go faster, there's no reason it needs to be in Dart.
• If I don't run the tests with -j 1, some of them error out. This is quite possibly a conflict or race condition in the crappy shelf server. It didn't actually seem to make the tests any slower to do so than running with straight pbr test.
• It would be better if it could listen to the build loop instead of running
• ddev serve compiles stuff differently than running tests, and in particular it makes .browser_tests, and sticks debug into names. So there are hacky substitution rules that if you don't see x, look for y. There are tests in microfrontend that don't pass when run via ddev serve, so they don't pass this way either. I don't know why.
• ddev serve is also different between packages, presumably based on build configuration. I tried admin_client, which works, but in normal config it's only the aggregates that get built, which would make for slower iterations.
• This should know how to not run with aggregation for tests.
• The way it spawns the server is awful, and it needs to be manually killed after you stop the process.
• I copied in the contents of shelf_proxy verbatim because that required newer deps.
• The dart_dev tool part is a minimally hacked version of the TestTool.

[aviary-wf]

Security Insights

No security relevant content was detected by automated scans.

Action Items

• Review PR for security impact; comment "security review required" if needed or unsure
• Verify aviary.yaml coverage of security relevant code

---

Questions or Comments? Reach out on Slack: #support-infosec.