FileDescriptorTables *** glibc detected *** double free or corruption (out)

[cloudkey]

I use Lua to write a C++.So shared library, which contains protobuf.
My WebService is nginx, the request is dealt with the Lua, and at the end of request,write a Pb log. But when nginx worker exit, it will be core:
#0 0x000000302af2e2ed in raise () from /lib64/tls/libc.so.6
#1 0x000000302af2fa3e in abort () from /lib64/tls/libc.so.6
#2 0x000000302af62db1 in __libc_message () from /lib64/tls/libc.so.6
#3 0x000000302af6888e in int_free () from /lib64/tls/libc.so.6
#4 0x000000302af68bd6 in free () from /lib64/tls/libc.so.6
#5 0x000000302d3ae19e in operator delete(void*) () from /usr/lib64/libstdc++.so.6
#6 0x0000000000620298 in gnu_cxx::hashtable<std::pair<std::pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const>, std::pair<google::protobuf::EnumDescriptor const, int>, google::protobuf::(anonymous namespace)::PointerIntegerPairHash<std::pair<google::protobuf::EnumDescriptor const*, int> >, std::Select1st<std::pair<std::pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const> >, std::equal_to<std::pair<google::protobuf::EnumDescriptor const, int> >, std::allocator<google::protobuf::EnumValueDescriptor const*> >::clear (

this=0xa25760 <google::protobuf::FileDescriptorTables::kEmpty+160>)
at /usr/lib/gcc/x86_64-redhat-linux/3.4.5/../../../../include/c++/3.4.5/ext/new_allocator.h:86

#7 0x00000000006094e0 in ~hashtable (this=, __in_chrg=)

at /usr/lib/gcc/x86_64-redhat-linux/3.4.5/../../../../include/c++/3.4.5/ext/hashtable.h:327

#8 0x00000000006094e0 in ~hash_map (this=, __in_chrg=)
#9 0x00000000006094e0 in ~hash_map (this=0x242e, __in_chrg=)
#10 0x00000000006094e0 in google::protobuf::FileDescriptorTables::~FileDescriptorTables (this=0xa256c0 google::protobuf::FileDescriptorTables::kEmpty,

__in_chrg=<optimized out>)

#11 0x000000302af30c65 in exit () from /lib64/tls/libc.so.6
#12 0x00000000004e4420 in ngx_worker_process_exit (cycle=0xdcb1a0) at src/os/unix/ngx_process_cycle.c:1103
#13 0x00000000004e3cc6 in ngx_worker_process_cycle (cycle=0xdcb1a0, data=) at src/os/unix/ngx_process_cycle.c:844
#14 0x00000000004e1bab in ngx_spawn_process (cycle=0xdcb1a0, proc=0x4e3bcd <ngx_worker_process_cycle>, data=0x0, name=0x7da525 "worker process", respawn=-3)

at src/os/unix/ngx_process.c:198

#15 0x00000000004e33ce in ngx_start_worker_processes (cycle=0xdcb1a0, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:394
#16 0x00000000004e2dd8 in ngx_master_process_cycle (cycle=0xdcb1a0) at src/os/unix/ngx_process_cycle.c:166
#17 0x00000000004c70fe in main (argc=5, argv=) at src/core/nginx.c:412

I guess this is related to protobuf FileDescriptorTables?

[cfallin]

It's impossible to say more without actually looking at your source. A double free is often tricky to debug because you can do something incorrectly (free an object too early) and get a stack trace only much later, on the second free.

FileDescriptorTables is supposed to free the descriptors that it holds. That is correct behavior.

Likely what is happening is that you have some code that is calling delete on a descriptor, which is incorrect. This would then cause the double-free error with the program shuts down.

Maybe try running your program under Valgrind?

[cloudkey]

thanks first!
Yesterday, I finally found out the reason, a module which embedded in our nginx contains protobuf, compiled into nginx bin. But, my Lua code calls the C++.So also uses protobuf. I found, when I put the nginx to remove the protobuf, not core.
Also, I find some global static variable in descriptor.cc~ Whether it is for this reason ?

[cloudkey]

Valgrind information:
==22518== Invalid read of size 8
==22518== at 0x620287: gnu_cxx::hashtable<std::pair<std::pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const>, std::pair<google::prot
obuf::EnumDescriptor const, int>, google::protobuf::(anonymous namespace)::PointerIntegerPairHash<std::pair<google::protobuf::EnumDescriptor const*, int> >, std::Select1st<std::pair<std::
pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const> >, std::equal_to<std::pair<google::protobuf::EnumDescriptor const_, int> >, std::allo
cator<google::protobuf::EnumValueDescriptor const*> >::clear() (stl_vector.h:462)
==22518== by 0x6094DF: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (hashtable.h:327)
==22518== by 0x302AF30C64: exit (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x4E441F: ngx_worker_process_exit (ngx_process_cycle.c:1103)
==22518== by 0x4E3CC5: ngx_worker_process_cycle (ngx_process_cycle.c:844)
==22518== by 0x4E1BAA: ngx_spawn_process (ngx_process.c:198)
==22518== by 0x4E33CD: ngx_start_worker_processes (ngx_process_cycle.c:394)
==22518== by 0x4E2DD7: ngx_master_process_cycle (ngx_process_cycle.c:166)
==22518== Address 0x4d62570 is 0 bytes inside a block of size 1,544 free'd
==22518== at 0x490796C: operator delete(void*) (vg_replace_malloc.c:480)
==22518== by 0x6094F0: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (new_allocator.h:86)
==22518== by 0x302AF30F4A: __cxa_finalize (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x678F1C2: ???
==22518== by 0x694CBAC: ???
==22518== by 0x302AD0AA5F: _dl_catch_error (in /lib64/ld-2.3.4.so)
==22518== by 0x302B201551: _dlerror_run (in /lib64/libdl-2.3.4.so)
==22518== by 0x302B2010DE: dlclose (in /lib64/libdl-2.3.4.so)
==22518==

==22518== Invalid write of size 8
==22518== at 0x6202A9: gnu_cxx::hashtable<std::pair<std::pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const>, std::pair<google::prot
obuf::EnumDescriptor const, int>, google::protobuf::(anonymous namespace)::PointerIntegerPairHash<std::pair<google::protobuf::EnumDescriptor const*, int> >, std::Select1st<std::pair<std::
pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const> >, std::equal_to<std::pair<google::protobuf::EnumDescriptor const_, int> >, std::allo
cator<google::protobuf::EnumValueDescriptor const*> >::clear() (stl_vector.h:462)
==22518== by 0x6094DF: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (hashtable.h:327)
==22518== by 0x302AF30C64: exit (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x4E441F: ngx_worker_process_exit (ngx_process_cycle.c:1103)
==22518== by 0x4E3CC5: ngx_worker_process_cycle (ngx_process_cycle.c:844)
==22518== by 0x4E1BAA: ngx_spawn_process (ngx_process.c:198)
==22518== by 0x4E33CD: ngx_start_worker_processes (ngx_process_cycle.c:394)
==22518== by 0x4E2DD7: ngx_master_process_cycle (ngx_process_cycle.c:166)
==22518== Address 0x4d62570 is 0 bytes inside a block of size 1,544 free'd
==22518== at 0x490796C: operator delete(void*) (vg_replace_malloc.c:480)
==22518== by 0x6094F0: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (new_allocator.h:86)
==22518== by 0x302AF30F4A: __cxa_finalize (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x678F1C2: ???
==22518== by 0x694CBAC: ???
==22518== by 0x302AD0AA5F: _dl_catch_error (in /lib64/ld-2.3.4.so)
==22518== by 0x302B201551: _dlerror_run (in /lib64/libdl-2.3.4.so)
==22518== by 0x302B2010DE: dlclose (in /lib64/libdl-2.3.4.so)
==22518==

==22518== Invalid free() / delete / delete[] / realloc()
==22518== at 0x490796C: operator delete(void_) (vg_replace_malloc.c:480)
==22518== by 0x6094F0: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (new_allocator.h:86)
==22518== by 0x302AF30C64: exit (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x4E441F: ngx_worker_process_exit (ngx_process_cycle.c:1103)
==22518== by 0x4E3CC5: ngx_worker_process_cycle (ngx_process_cycle.c:844)
==22518== by 0x4E1BAA: ngx_spawn_process (ngx_process.c:198)
==22518== by 0x4E33CD: ngx_start_worker_processes (ngx_process_cycle.c:394)
==22518== by 0x4E2DD7: ngx_master_process_cycle (ngx_process_cycle.c:166)
==22518== Address 0x4d62570 is 0 bytes inside a block of size 1,544 free'd
==22518== at 0x490796C: operator delete(void_) (vg_replace_malloc.c:480)
==22518== by 0x6094F0: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (new_allocator.h:86)
==22518== by 0x302AF30F4A: __cxa_finalize (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x678F1C2: ???
==22518== by 0x694CBAC: ???
==22518== by 0x302AD0AA5F: _dl_catch_error (in /lib64/ld-2.3.4.so)
==22518== by 0x302B201551: _dlerror_run (in /lib64/libdl-2.3.4.so)
==22518== by 0x302B2010DE: dlclose (in /lib64/libdl-2.3.4.so)
==22518==

[cfallin]

Yes, there are some global variables, for example the "generated descriptor pool". But just using global variables doesn't make a program double-free. Do you have more specific theories or evidence?

You haven't given us any information on what your program is doing. There are many ways for a C++ program to corrupt memory, usually due to some API misuse, stray pointer, or lifetime issue. If you can give us a program that will reproduce this error, we will try to look into it. But right now there is not enough information to diagnose further. The fact that the final crash happens during FileDescriptorTables cleanup does not mean that FileDescriptorTables is doing something wrong (again, as I said before: it is correctly delete-ing its memory), only that something else in your program has corrupted memory.

[rinatz]

@cloudkey I sent the pull request to fix this problem and it was merged.