[Bug]: wolfSSL_free crash on ESP32s3

[PaulMartinsen]

Contact Details

No response

Version

a026d84 (master branch ca 8:33 am, 15/7/2023)

Description

Calling wolfSSL_free to dispose of a TLS session from a 'cleanup' thread leads to an assertion failure in xQueueGenericSend in esp-idf/v5.0/components/freertos/FreeRTOS-Kernel/queue.c. This seems to occur only when wolfSSL_free is called from a different task to the one that created it. In both case all communication is finished and it is time to free the memory. And wolfSSL_free is only being called once for the session.

Stack trace (below), shows this occurs when esp_sha_hw_unlock calls esp_CryptHwMutexUnLock(&sha_mutex). The same application code operates correctly running on Windows (however, that's not using the Espressif port).

Currently Windows decided it can't find the ESP32s3 JTAG interface, so I haven't been able to look at what's happening in any more detail. Will add new info as it comes to hand.

I'm using this version which is a clean fork from a recent master branch with only trivial changes for building on Windows. @gojimmypi , do you think your new PR might make a difference?

Reproduction steps

No response

Relevant log output

0x4038171d: esp_system_abort at C:/Programming/Toolchains/VisualGDB/ESP32/v5/esp-idf/v5.0/components/esp_system/esp_system.c:135
0x40388e65: __assert_func at C:/Programming/Toolchains/VisualGDB/ESP32/v5/esp-idf/v5.0/components/newlib/assert.c:78
0x403820ec: xQueueGenericSend at C:/Programming/Toolchains/VisualGDB/ESP32/v5/esp-idf/v5.0/components/freertos/FreeRTOS-Kernel/queue.c:837 (discriminator 8)
0x42095433: esp_CryptHwMutexUnLock at k:/Lumos/Vendor/WolfSSL/wolfssl/wolfcrypt/src/port/Espressif/esp32_util.c:80
0x42094dd0: esp_sha_hw_unlock at k:/Lumos/Vendor/WolfSSL/wolfssl/wolfcrypt/src/port/Espressif/esp32_sha.c:819
0x42094eb5: esp_sha_init_ctx at k:/Lumos/Vendor/WolfSSL/wolfssl/wolfcrypt/src/port/Espressif/esp32_sha.c:260 (discriminator 15)
0x42094f62: esp_sha_init at k:/Lumos/Vendor/WolfSSL/wolfssl/wolfcrypt/src/port/Espressif/esp32_sha.c:99
0x4208cd9d: InitSha256 at k:/Lumos/Vendor/WolfSSL/wolfssl/wolfcrypt/src/sha256.c:771
0x4208d5a6: wc_Sha256Free at k:/Lumos/Vendor/WolfSSL/wolfssl/wolfcrypt/src/sha256.c:1855
0x4209715f: FreeHandshakeHashes at k:/Lumos/Vendor/WolfSSL/wolfssl/src/internal.c:6845
0x42097cc2: SSL_ResourceFree at k:/Lumos/Vendor/WolfSSL/wolfssl/src/internal.c:7929
0x42097e61: FreeSSL at k:/Lumos/Vendor/WolfSSL/wolfssl/src/internal.c:8470
0x420606ba: wolfSSL_free at k:/Lumos/Vendor/WolfSSL/wolfssl/src/ssl.c:1491
0x4204dfef: CTLSSession::Dispose() at k:/Lumos/DeviceWebservicesLibrary/Communications/TLSContext.cpp:77
0x4204e01d: CTLSSession::~CTLSSession() at k:/Lumos/DeviceWebservicesLibrary/Communications/TLSContext.cpp:69
0x4204d249: CTcpConnectionInfo::~CTcpConnectionInfo() at k:/Lumos/DeviceWebservicesLibrary/Communications/TcpConnectionInfo.h:13
 (inlined by) CTcpConnectionInfo::~CTcpConnectionInfo() at k:/Lumos/DeviceWebservicesLibrary/Communications/TcpConnectionInfo.h:13
0x4200bbdf: std::default_delete<CConnectionInfo>::operator()(CConnectionInfo*) const at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits/unique_ptr.h:85
 (inlined by) std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >::~unique_ptr() at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits/unique_ptr.h:361
 (inlined by) void std::destroy_at<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > >(std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >*) at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits/stl_construct.h:88
 (inlined by) void std::allocator_traits<std::allocator<std::_Fwd_list_node<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > > >::destroy<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > >(std::allocator<std::_Fwd_list_node<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > >&, std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >*) at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits/alloc_traits.h:533
 (inlined by) std::_Fwd_list_base<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >, std::allocator<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > >::_M_erase_after(std::_Fwd_list_node_base*, std::_Fwd_list_node_base*) at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits/forward_list.tcc:86
0x4204c65f: std::_Fwd_list_base<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >, std::allocator<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > >::~_Fwd_list_base() at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits/forward_list.h:346
 (inlined by) std::forward_list<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >, std::allocator<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > >::~forward_list() at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits/forward_list.h:591
 (inlined by) remove_if<CConnectionManager::DiscardOldConnections(const time_point&)::<lambda(const std::unique_ptr<CConnectionInfo>&)> > at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits/forward_list.tcc:330
0x4204c6c5: CConnectionManager::DiscardOldConnections(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> > > const&) at k:/Lumos/DeviceWebservicesLibrary/Communications/ConnectionManager.cpp:200
0x4204c779: CConnectionManager::Process() at k:/Lumos/DeviceWebservicesLibrary/Communications/ConnectionManager.cpp:161
0x4204e639: non-virtual thunk to CConnectionFactory::Process() at k:/Lumos/DeviceWebservicesLibrary/Services/ConnectionFactory.h:54
0x4216b0f5: CManualBackgroundTaskDispatcher::RunEachTaskOnce() at k:/Lumos/DeviceWebservicesLibrary/Services/ManualBackgroundTaskDispatcher.cpp:21
0x4200c354: app_main at C:\Users\Paul\Projects\BLS\FPH Projects\2022 SDC\Software\Lumos\Lumos32i\build\VisualGDB\Debug/../../../main/Lumos32i.cpp:544
0x42174c2b: main_task at C:/Programming/Toolchains/VisualGDB/ESP32/v5/esp-idf/v5.0/components/freertos/FreeRTOS-Kernel/portable/port_common.c:131 (discriminator 2)
0x403856f5: vPortTaskWrapper at C:/Programming/Toolchains/VisualGDB/ESP32/v5/esp-idf/v5.0/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:151

[gojimmypi]

Hi @PaulMartinsen - thank you for reporting this issue.

There's definitely an improvement in my #6624 regarding properly locking and releasing the hardware, and falling back to SW as needed.

Given that, if you disable the HW acceleration, do you see the same problem?

Note that I've recently refactored the macros in #6587, removing the WROOM32, so different settings are needed in user_settings.h to disable HW acceleration. It looks like you've already applied that update, so this comment is for other viewers.

Coincidentally I am now working on updating the ESP32 client and server examples. See also wolfssl-examples/ESP32. There are no changes to these apps in the open PR.

The work on I am doing is with regards to supporting a new cipher suite, but I've confirmed at least my current server works properly with a Linux client. If you use my example, just make sure you don't enable the SM Ciphers.

If you have some example code to reproduce the error, that would be very helpful. I didn't immediately see it in your fork.

Regarding your JTAG problems, I can certainly relate! I've found Zadig to be quite helpful when Windows decides to change the drivers. I set libusbK for the Tigard and for the ESP32-S3 on-board JTAG for my dev board, it uses WINUSB.

You are still using VisualGDB, eh?

For reference, here's my debug setting dialog box:

Other ideas: different USB cable, restart Windows, power cycle board. Exit/restart Visual Studio. VM running? Make sure the device does not get used there (yes, I've had that happen more than once!). Perhaps also try using Windows to completely remove the USB driver. Which exact board model are you using?

Thanks again for all your input! Cheers

[PaulMartinsen]

Thanks @gojimmypi . I was a little worried reporting the problem with so little info, but some very helpful suggestions there as always so I'm glad I did.

I'm using the ESP32-S3-DevkitC-1 v1.0 board with the P2N8R2 variant on it. I always overlook testing w/o the hardware acceleration, but that sounds like a good first step. And your new PR. This is part of a much larger project now, but I will try to create a simplified example that illustrates the problem, if the PR doesn't resolve, based on the library examples.

One odd thing I notice with zadig: I get two devices when I connect the boards jtag port: USB JTAG/serial debug unit (Interface 0) and USB JTAG/serial debug unit (Interface 2). No Interface 1. Interface 0 is using usbser (v10.0.22621.1778) and interface 2 is using WINUSB (v10.0.22621.608). Have you had any experience like that? Maybe switching one or both to WINUSB (v6.1.7600.16385) is a path worth trying?

[gojimmypi]

Hi @PaulMartinsen, I'm glad you found the suggestions helpful.

I'm using the ESP32-S3-DevkitC-1 v1.0

Ah yes, excellent, I have exactly the same board. The -R2 (2 MB PSRAM) is not enough to run Linux, but I also have a -R8 for that.

There are several options for allocating the external PSRAM. Which are you using?

I get two devices when I connect the boards jtag port

hm, that does sound odd. I only see one JTAG device. The UART is on a second physical connector. In contrast, my Tigard shows both USB devices on one USB connector.

If you just cannot get it working, I'd suggest going into Device Manager and removing the driver. Be sure to unplug the device & allow Windows to reinstall the driver. I've also seen VisualGDB itself detect bad driver problems and offer to correct automatically with their UsbDriverTool.

TLS session from a 'cleanup' thread leads to an assertion failure

I'll be looking into this more in the coming days.

One thing I've found helpful when troubleshooting odd problems is the ESP stack high water mark. I think it is much better to monitor the actual level, rather than just rely on the stack smashing monitor & compiler options. I have an example in my latest benchmark app.

Have a great weekend. Cheers

[PaulMartinsen]

Hi @gojimmypi , Just confirming I do not see the cross-task crash if I disable hardware acceleration (that is NO_ESP32_CRYPT, NO_WOLFSSL_ESP32_CRYPT_RSA_PRI, NO_WOLFSSL_ESP32_CRYPT_HASH and NO_WOLFSSL_ESP32_CRYPT_AES defined in user_settings.h).

Currently I don't think I'm using the PSRAM at all (i.e., not configured, free ram reports 73k). Next stage of the project is going to be the memory intensive bit and I'll investigate the different options for that. Do you have recommendations?

I vaguely recall reading the jtag connector can support serial comms too so I'm assuming that's why two devices are showing up with just that USB port plugged in. I know I spent a lot of time trying to get it going on my old machine; then it just worked on the new one through VisualGDB. Until now.

Thanks for the reminder about the high-water mark. That's a great tool too add to my health monitoring service.

You have a great weekend too.
paul

[PaulMartinsen]

Quick update: I tried PR6624, but the cross-task crashing problem with hardware acceleration enabled still occurs in that branch unfortunately.

[PaulMartinsen]

Hi @gojimmypi . Turns out it is an id10t programmer error. While building a example to demonstrate I noticed I'd failed to call wc_ShaFree after setting a server id for session resumption. Doing so resolves the crash both for PR6624 and the master branch mentioned.

I'll mark this closed with apologies for the distraction, thanks for your help and thanks to whomever wrote the comment in sha256.c that helped me connect the dots:

        /* probably due to unclean shutdown, error, or other problem.
         *
         * if you find yourself here, code needs to be cleaned up to
         * properly release hardware. this init is only for handling
         * the unexpected. by the time free is called, the hardware
         * should have already been released (lockDepth = 0)
         */
        (void)InitSha256(sha256); /* unlock mutex, set mode to ESP32_SHA_INIT */

[PaulMartinsen]

Hi @gojimmypi . It looks like I jumped to conclusions closing this one. The problem resurfaced when I tidied up my fix and remains persistent. I haven't been able to create a simplified version to share yet, but instrumented a couple of files (attached) to:

1. track ctx->lockDepth in esp32_sha.c
2. track the lock not being released in SendTls13ClientHello in tls13.c.

What I have noticed is that a lock on the sha hardware is taken but never released during SendTls13ClientHello. This leads to the crash when wc_Sha256Free tries to unwind the lock (to recover from the unreleased lock). SendTls13ClientHello → HashOutput → HashRaw first calls wc_Sha256Update, which gets the lock, then calls wc_Sha512Update which sees the lock in use and reverts to software calculation. They seem to be using different WC_ESP32SHA values though, so I'm not sure why wc_Sha512Update can't use the hardware. The relevant piece of the log file (line 115ff) is:

Client hello 4533
HashRaw, 9749: wc_Sha256Update of WC_ESP32SHA @ 3fcd8020d
Lock depth @ 779 = 1 for WC_ESP32SHA @ 3fcd8020d
HashRaw, 9770: wc_Sha512Update of WC_ESP32SHA @ 3fcd8110d
�[0;32mI (15005) wolf_hw_sha: 
Hardware in use for WC_ESP32SHA @ 0x3fcd8110d; Mode REVERT to ESP32_SHA_SW
HashRaw, 9794: EdDSA_Update
HashRaw, 9800
Client hello 4538

I've attached the log output I captured, the instrumented files and wolf config files. I'm using your PR6624 version of the WolfSSL library.

In this scenario:

1. server on the ESP32s3 receives a command to subscribe to an event (here an event is a message sent periodically with the ESP32s3 as the client and a .net program as the receiving server).
2. a connection service task opens a connection to the server, completes the TLS handshake, then signals the notification task that actually transmits the event messages. I call wolfssl_Debugging_On immediately after creating the session for the TLS handshake (i.e., after calling wolfSSL_new.
3. a couple of event messages are sent over the connection from the notification task.
4. the connection is closed by the notification task. It is this point where the assert described above fails.

I turn on logging immediately after creating the TLS handshake at step 2, which is in the connection service task:

  WOLFSSL* pNewSession = wolfSSL_new(m_Context.GetContext());
  CTLSSession NewSession(pNewSession);
  printf("Created session %0xd for %s\n", (unsigned)pNewSession, svServerId.data());
  wolfSSL_Debugging_ON();
  esp_log_level_set("wolfssl", ESP_LOG_INFO);
  esp_log_level_set("wolf_hw_sha", ESP_LOG_VERBOSE);

I also noticed when the SSL session for the server request is released, the lock depth is 1 one (line 62 in the log). In this case, wc_Sha256Free successfully unwinds the lock and recovers. Both the session creation and cleanup will be happening in a single task in this situation though. However, I'm not sure why the lock isn't released earlier.

Let me know if you'd like to do a video chat to discuss. It seems quite complex!

[gojimmypi]

Hi @PaulMartinsen

The problem resurfaced when I tidied up my fix and remains persistent.

It would be quite interesting to see the diff of before and after of the tidy. What changed after the problem went away?

They seem to be using different WC_ESP32SHA values

Let me explain in more detail some of the challenges. Perhaps I can inspire an "aha" moment and we can better isolate the root cause without me actually seeing your code.

The interesting thing in a multi-thread environment with multiple processes both between threads and within a given thread is the ctx object, and more importantly: if a copy has been made of the object.

That's what the initializer attempts to address: who am I and am I the one that called the HW encryption lock.

I keep track of the address of the object that was currently active when the lock was placed. That way, any copy of the object that may think it was doing HW encryption (due to object property settings), should be gracefully and quietly falling back to SW.

What's interesting is when a copy is made of the SHA ctx object. I suppose there's a non-zero chance of an in-progress SHA process that is abandoned before completion. I would think that the worst possible case scenario here though, would be an infinitely long lock. This seems quite unlikely in normal circumstances.

What might be interesting though is if in one thread with ctx HW in process, that before completion it is abandoned and at nearly the same time another process in another thread is able to allocate a new ctx in exactly the same place in memory as the one in the other thread that was just released. This too seems pretty unlikely, but I hope you can see where I'm going with these ideas: I'm looking for an errant copy of a ctx that is misbehaving.

The other place to look is why the unlock failed. It is very important to only call the wolfSSL locking. See the hack alert notes. See in particular the ESP-IDF periph_module_disable and periph_module_enable function implementations in periph_ctrl.c.

There's no protective logic in the ESP-IDF functions to prevent the caller from exceeding or omitting a call. For example, see the periph_module_disable that will happily decrement their lock depth counter, even if ref_counts[periph] is already zero with ref_counts[periph]--;.

What's even more curious is the tail end of your log. The "Bad lock depth" sounds like something I would have written, but I was not able to find those words in the code. Is that something you added? It does sound like something was not keeping track of who is actually in control of the hardware when the lock was released.

21:04:32.821	�[0;32mI (29495) wolfssl: Free'ing client ssl�[0m
21:04:32.821	Bad lock depth @ 1857 = 1 for 3fcd8020d
21:04:32.826	Lock depth @ 811 = 1 for WC_ESP32SHA @ 3fcd8020d
21:04:32.832	Lock depth @ 818 = 0 for WC_ESP32SHA @ 3fcd8020d
21:04:32.837	
21:04:32.837	assert failed: xQueueGenericSend queue.c:837 (pxQueue->pcHead != ((void *)0) || pxQueue->u.xSemaphore.xMutexHolder == ((void *)0) || pxQueue->u.xSemaphore.xMutexHolder == xTaskGetCurrentTaskHandle())

How to detect this?

So we need to be looking at code that may for some reason think it is in charge of the HW, but isn't. We should be looking for copies of the ctx object, or in particular anything else that may be using the HW directly and not via one of the wolfSSL functions.

Let me know if you'd like to do a video chat to discuss.

yes, I agree that might help. I'm available later today (late afternoon, early evening Pacific time), or we can schedule something during the week.

In the meantime I will study the information you provided in more detail & try to reproduce the problem.

btw, have you tried SINGLE_THREADED mode? (edit: I see you are not currently using this. Might be an interesting data point)

btw (2) - if you are using my branch, there's a new NO_WOLFSSL_ESP32_CRYPT_HASH to turn off just SHA HW acceleration.

• edit: try turning these on to ESP_LOGI or ESP_LOGW to help detect the possible object copy instances:

                case ESP32_SHA_HW_COPY:
                    /* This is an interesting but acceptable situation:
                    ** an anticipated active HW copy that will demote to SW. */
                    ESP_LOGV(TAG, "HW WC_ESP32SHA ctx mode = ESP32_SHA_HW_COPY.");
                    break;

                default:
                    /* this will frequently occur during new init */
                    ESP_LOGV(TAG, "ALERT: unexpected WC_ESP32SHA ctx mode. "
                                  "Uninitialized?");
                    ctx->mode = ESP32_SHA_INIT;
                    break;

And now that I look at that code, I'm wondering if there's a missing ctx->mode assignment for case ESP32_SHA_HW_COPY? That might be one of my first changes: ctx->mode = ESP32_SHA_INIT; there. In fact, the assignment should probably even be outside the entire switch statement, as any copy should be in initialization mode. Hm... unless the copy is of a SW mode in progress. That would be odd. Still, this is someplace I'd definitely look first in your code. Add some more verbosity here and see if this is indeed a problematic area, in particular to confirm if we ended up here, the ESP32_SHA_HW_COPY state is handled properly. What I'm thinking here is what if the next operation after the copy is not an initialization?

• edit(2) I'd be interested in seeing your call stack, single threaded if possible, during the error condition.

• edit(3) I see from your logs there's mention of a "subscribe": are you using wolfMQTT?

[PaulMartinsen]

@gojimmypi , removing the jtag driver & letting VisualGDB install a new one resolved the debugging issue for me. Sometimes the simple solutions are best.

This let me look into why the assert was failing in xGenericQueueSend. After a bit of mucking about I got the debugger to stop in the right place to inspect the values in the assert and found:

So the assert is failing because the pxQueue->u.xSemaphore.xMutexHolder is neither null, a mutex nor on the current thread. The current task handle matches the one for my Notify task (which is the one calling wolfSSL_Free) and the pxQueue->u.xSemaphore.xMutexHolder task handle matches the one for my Connect task (which is the one that created the WOLFSSL session). I copied the call stack to this location below.

This mutex comes from line 67 in esp32_sha.c: sha_mutex. It is initialized by esp_CryptHwMutexInit → wc_InitMutex → xSemaphoreCreateMutex.

Previously I thought we might have two different problems: something not releasing the hw and the cleanup not working. But the xSemaphoreCreateMutex doc says This type of semaphore uses a priority inheritance mechanism so a task ‘taking’ a semaphore MUST ALWAYS ‘give’ the semaphore back once the semaphore it is no longer required. So maybe if the hw remains locked by a different thread, then the lock can't be untangled because the mutex shoots itself in the foot?

Call stack to failing mutex (line number in queue.c is 6 higher than in the original issue due to extra code to trap the fault)::

xQueueGenericSend
	at C:\Programming\Toolchains\VisualGDB\ESP32\v5\esp-idf\v5.0\components\freertos\FreeRTOS-Kernel\queue.c(843)
esp_CryptHwMutexUnLock
	at K:\Vendor\WolfPR\wolfssl\wolfcrypt\src\port\Espressif\esp32_util.c(83)
esp_sha_hw_unlock
	at K:\Vendor\WolfPR\wolfssl\wolfcrypt\src\port\Espressif\esp32_sha.c(824)
esp_sha_init_ctx
	at K:\Vendor\WolfPR\wolfssl\wolfcrypt\src\port\Espressif\esp32_sha.c(260)
esp_sha_init
	at K:\Vendor\WolfPR\wolfssl\wolfcrypt\src\port\Espressif\esp32_sha.c(99)
InitSha256
	at K:\Vendor\WolfPR\wolfssl\wolfcrypt\src\sha256.c(773)
wc_Sha256Free
	at K:\Vendor\WolfPR\wolfssl\wolfcrypt\src\sha256.c(1858)
FreeHandshakeHashes
	at K:\Vendor\WolfPR\wolfssl\src\internal.c(6845)
SSL_ResourceFree
	at K:\Vendor\WolfPR\wolfssl\src\internal.c(7929)
FreeSSL
	at K:\Vendor\WolfPR\wolfssl\src\internal.c(8470)
wolfSSL_free
	at K:\Vendor\WolfPR\wolfssl\src\ssl.c(1492)
CTLSSession::Dispose
	at K:\DeviceWebservicesLibrary\Communications\TLSContext.cpp(87)
CTLSSession::~CTLSSession
	at K:\DeviceWebservicesLibrary\Communications\TLSContext.cpp(77)
CTcpConnectionInfo::~CTcpConnectionInfo
	at K:\DeviceWebservicesLibrary\Communications\TcpConnectionInfo.h(13)
CTcpConnectionInfo::~CTcpConnectionInfo
	at K:\DeviceWebservicesLibrary\Communications\TcpConnectionInfo.h(13)
std::default_delete<CConnectionInfo>::operator()
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\unique_ptr.h(79)
std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >::~unique_ptr
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\unique_ptr.h(361)
std::destroy_at<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > >
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\stl_construct.h(88)
std::allocator_traits<std::allocator<std::_Fwd_list_node<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > > >::destroy<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > >
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\alloc_traits.h(533)
std::_Fwd_list_base<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >, std::allocator<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > >::_M_erase_after
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\forward_list.tcc(86)
std::_Fwd_list_base<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >, std::allocator<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > >::~_Fwd_list_base
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\forward_list.h(590)
std::forward_list<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >, std::allocator<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > >::~forward_list
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\forward_list.h(591)
std::forward_list<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> >, std::allocator<std::unique_ptr<CConnectionInfo, std::default_delete<CConnectionInfo> > > >::remove_if<CConnectionManager::ReleaseInterest(const CConnectionInterest*)::<lambda(const std::unique_ptr<CConnectionInfo>&)> >(struct {...})
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\forward_list.tcc(330)
CConnectionManager::ReleaseInterest
	at K:\DeviceWebservicesLibrary\Communications\ConnectionManager.cpp(176)
CConnectionInterest::~CConnectionInterest
	at K:\DeviceWebservicesLibrary\Communications\ConnectionUser.cpp(22)
CSubscriptionRecord::~CSubscriptionRecord
	at K:\DeviceWebservicesLibrary\Storage\SubscriptionRecord.h(39)
std::destroy_at<CSubscriptionRecord>
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\stl_construct.h(88)
std::allocator_traits<std::allocator<std::_Fwd_list_node<CSubscriptionRecord> > >::destroy<CSubscriptionRecord>
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\alloc_traits.h(533)
std::_Fwd_list_base<CSubscriptionRecord, std::allocator<CSubscriptionRecord> >::_M_erase_after
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\forward_list.tcc(86)
std::_Fwd_list_base<CSubscriptionRecord, std::allocator<CSubscriptionRecord> >::~_Fwd_list_base
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\forward_list.h(590)
std::forward_list<CSubscriptionRecord, std::allocator<CSubscriptionRecord> >::~forward_list
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\forward_list.h(591)
std::forward_list<CSubscriptionRecord, std::allocator<CSubscriptionRecord> >::remove_if<CNotificationProvider::RemoveSubscriptions()::<lambda(const CSubscriptionRecord&)> >(struct {...})
	at c:\programming\toolchains\visualgdb\esp32\v5\tools\xtensa-esp32s3-elf\esp-2022r1-11.2.0\xtensa-esp32s3-elf\xtensa-esp32s3-elf\include\c++\11.2.0\bits\forward_list.tcc(330)
CNotificationProvider::RemoveSubscriptions
	at K:\DeviceServiceProviders\Communications\NotificationProvider.cpp(294)
CNotificationProvider::WorkerTask
	at K:\DeviceServiceProviders\Communications\NotificationProvider.cpp(179)
CWorkerTask::WorkerWrapper
	at K:\Espressif\Tasks\WorkerTask.cpp(47)
vPortTaskWrapper
	at C:\Programming\Toolchains\VisualGDB\ESP32\v5\esp-idf\v5.0\components\freertos\FreeRTOS-Kernel\portable\xtensa\port.c(151)