[REGRESSION]: Out of memory errors on 0.15.0

[dialupdev]

Context:

• GOOD Playwright Version: 0.14.0
• BAD Playwright Version: 0.15.0
• Operating System: Docker on linux using qawolf/playwright-ci:v0.9.0

Describe the bug

Hi team. I started seeing a lot of Out of memory errors in Chromium after upgrading to 0.15.0. I have a test that can make it fail consistently, but it's very large and I haven't yet narrowed down what part of it is causing the issue.

Here's the error I'm getting.

```sh [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err [0424/192554.150296:FATAL:memory.cc(40)] Out of memory. size=262144 [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #0 0x564ff73ae919 base::debug::CollectStackTrace() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #1 0x564ff7313723 base::debug::StackTrace::StackTrace() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #2 0x564ff7324080 logging::LogMessage::~LogMessage() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #3 0x564ff733ff79 base::internal::OnNoMemoryInternal() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #4 0x564ff845f921 viz::bitmap_allocation::AllocateSharedBitmap() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #5 0x564ff862f753 cc::BitmapRasterBufferProvider::AcquireBufferForRaster() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #6 0x564ff85c3cbc cc::TileManager::CreateRasterTask() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #7 0x564ff85c0e9b cc::TileManager::AssignGpuMemoryToTiles() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #8 0x564ff85bf319 cc::TileManager::CheckIfMoreTilesNeedToBePrepared() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #9 0x564ff735b5ab base::TaskAnnotator::RunTask() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #10 0x564ff736b648 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #11 0x564ff736b3e8 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #12 0x564ff7327b8a base::MessagePumpDefault::Run() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #13 0x564ff736bbf5 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #14 0x564ff7344fbe base::RunLoop::Run() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #15 0x564ff6bb18f5 blink::scheduler::WorkerThread::SimpleThreadImpl::Run() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #16 0x564ff73bf3be base::(anonymous namespace)::ThreadFunc() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #17 0x7fb0b54026db start_thread [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #18 0x7fb0aee7888f clone [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err Task trace: [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #0 0x564ff85c9f9e cc::UniqueNotifier::Schedule() [docker-2] 2020-04-24T19:25:54.439Z pw:browser:err #1 0x564ff85c6b30 cc::(anonymous namespace)::DidFinishRunningAllTilesTask::RunOnWorkerThread() [docker-2] 2020-04-24T19:25:54.439Z pw:browser:err [docker-2] 2020-04-24T19:25:54.439Z pw:browser:err Received signal 6 [docker-2] 2020-04-24T19:25:54.446Z pw:browser:err #0 0x564ff73ae919 base::debug::CollectStackTrace() [docker-2] 2020-04-24T19:25:54.457Z pw:browser:err #1 0x564ff7313723 base::debug::StackTrace::StackTrace() [docker-2] 2020-04-24T19:25:54.464Z pw:browser:err #2 0x564ff73ae4b5 base::debug::(anonymous namespace)::StackDumpSignalHandler() [docker-2] 2020-04-24T19:25:54.464Z pw:browser:err #3 0x7fb0b540d890 (/lib/x86_64-linux-gnu/libpthread-2.27.so+0x1288f) [docker-2] 2020-04-24T19:25:54.464Z pw:browser:err #4 0x7fb0aed95e97 gsignal [docker-2] 2020-04-24T19:25:54.464Z pw:browser:err #5 0x7fb0aed97801 abort [docker-2] 2020-04-24T19:25:54.475Z pw:browser:err #6 0x564ff73ad415 base::debug::BreakDebugger() [docker-2] 2020-04-24T19:25:54.483Z pw:browser:err #7 0x564ff7324520 logging::LogMessage::~LogMessage() [docker-2] 2020-04-24T19:25:54.495Z pw:browser:err #8 0x564ff733ff79 base::internal::OnNoMemoryInternal() [docker-2] 2020-04-24T19:25:54.505Z pw:browser:err #9 0x564ff845f921 viz::bitmap_allocation::AllocateSharedBitmap() [docker-2] 2020-04-24T19:25:54.525Z pw:browser:err #10 0x564ff862f753 cc::BitmapRasterBufferProvider::AcquireBufferForRaster() [docker-2] 2020-04-24T19:25:54.539Z pw:browser:err #11 0x564ff85c3cbc cc::TileManager::CreateRasterTask() [docker-2] 2020-04-24T19:25:54.546Z pw:browser:err #12 0x564ff85c0e9b cc::TileManager::AssignGpuMemoryToTiles() [docker-2] 2020-04-24T19:25:54.559Z pw:browser:err #13 0x564ff85bf319 cc::TileManager::CheckIfMoreTilesNeedToBePrepared() [docker-2] 2020-04-24T19:25:54.568Z pw:browser:err #14 0x564ff735b5ab base::TaskAnnotator::RunTask() [docker-2] 2020-04-24T19:25:54.577Z pw:browser:err #15 0x564ff736b648 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl() [docker-2] 2020-04-24T19:25:54.587Z pw:browser:err #16 0x564ff736b3e8 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() [docker-2] 2020-04-24T19:25:54.595Z pw:browser:err #17 0x564ff7327b8a base::MessagePumpDefault::Run() [docker-2] 2020-04-24T19:25:54.607Z pw:browser:err #18 0x564ff736bbf5 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run() [docker-2] 2020-04-24T19:25:54.615Z pw:browser:err #19 0x564ff7344fbe base::RunLoop::Run() [docker-2] 2020-04-24T19:25:54.623Z pw:browser:err #20 0x564ff6bb18f5 blink::scheduler::WorkerThread::SimpleThreadImpl::Run() [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err #21 0x564ff73bf3be base::(anonymous namespace)::ThreadFunc() [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err #22 0x7fb0b54026db start_thread [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err #23 0x7fb0aee7888f clone [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err r8: 0000000000000000 r9: 00007fb0a39a78e0 r10: 0000000000000008 r11: 0000000000000246 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err r12: 00007fb0a39a8bb0 r13: 00007fb0a39a7b40 r14: 00007fb0a39a8bc0 r15: aaaaaaaaaaaaaaaa [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err di: 0000000000000002 si: 00007fb0a39a78e0 bp: 00007fb0a39a7b30 bx: 00007fb0a39a8374 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err dx: 0000000000000000 ax: 0000000000000000 cx: ffffffffffffffff sp: 00007fb0a39a78e0 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err ip: 00007fb0aed95e97 efl: 0000000000000246 cgf: 6946000000000033 erf: 0000000000000000 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err [end of stack trace] [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err Calling _exit(1). Core file will not be generated. ```

I'm wondering if this is caused by #1956 since I don't see any other Chrome-specific changes between the two releases, but I honestly don't know.

Thanks for looking.

[pavelfeldman]

Viz is a compositor infrastructure in Chromium. According to the trace, it runs out of memory while allocating gpu? memory for the tiles rasterization. Your screen is divided into a grid of tiles that can repaint independently. If you can reliably reproduce it on 0.15 and not at 0.14 it does point at a change in Chrome. Having a repro would be of a great help.

[dialupdev]

Well this is awkward. It turns out this was 100% my fault. I actually had a memory leak in my code. I'm doing some weird stuff to get custom methods working (#1655) and it turns out one of my functions created an infinite loop. Classic. Not sure why it wasn't causing problems in 0.140, but I don't think it really matters.

Sorry for the noise and thanks for your commitment to taking care of issues!

Edit: see below.

[dialupdev]

After a lot more digging, I found a way to reproduce this crash without any of my sketchy creative code. I created a repo to demonstrate: https://github.com/celeryclub/playwright-crash-repro.

I originally saw this crash on a Linux host, but this repro works on macOS too. Here's my VM config.
VMware Fusion v11.5.1
Linux 4.14.154-boot2docker
16384 MB RAM

Just to clarify, this repro doesn't crash 100% of the time. It's more like 10-15%. And it turns out this also happens in 0.14.0, but not as often - maybe like 7% of the time. I included some logs in the repo that show the results of my runs.
Logs for 0.14.0
Logs for 0.15.0

One pattern I've notice while watching these runs is that the crash always seems to happen when opening a new tab.

[pavelfeldman]

Could you try passing docker run --shm-size=1gb?

[dialupdev]

Thanks @pavelfeldman. That does appear to fix it. Do you know why --shm-size=1gb would work but --disable-dev-shm-usage wouldn't? Seems like they should do about the same thing. I guess it might be dependent on my host system.

[pavelfeldman]

I have no idea - I did not look into why it was not regarded in this viz allocation.

@aslushnikov should we update our docker guide? with that, closing this issue?

[dialupdev]

Thanks @pavelfeldman. For future debuggers, docker run --ipc=host seems like the most solid fix I've found for this crash and others too. Reference: cypress-io/cypress#350 (comment).

I'd love to see this added to the docker guide. 👍🏻

[jknooks583]

Would this work for firefox as well or just Chrome?
Using --shm-size=1gb ?

[dialupdev]

Either that or --ipc=host would most likely help with FF too. Give it a try and see what happens!