segfault with both php 7.4 and php 8.1 in _zend_is_inconsistent (ht=0x0, file=0x5616496c7238

[blacktek]

Hello,
I've random segfaults when using protobuf.
Same issue with both PHP 7.4.29 and PHP 8.1.6 on
$ uname -a
Linux xxxxxx.it 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Core was generated by `/usr/local/php81debug/bin/php /home/xxxx/domains/sf.xxxx.it/cron/proces'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0 0x0000561648c65d07 in _zend_is_inconsistent (ht=0x0, file=0x5616496c7238 "/usr/local/directadmin/custombuild/php-8.1.6/Zend/zend_hash.c", line=2469) at /usr/local/directadmin/custombuild/php-8.1.6/Zend/zend_hash.c:54

54 if ((HT_FLAGS(ht) & HASH_FLAG_CONSISTENCY) == HT_OK) {

[Current thread is 1 (Thread 0x7f3cb9586bc0 (LWP 3090694))]

(gdb) bt

#0 0x0000561648c65d07 in _zend_is_inconsistent (ht=0x0, file=0x5616496c7238 "/usr/local/directadmin/custombuild/php-8.1.6/Zend/zend_hash.c", line=2469) at /usr/local/directadmin/custombuild/php-8.1.6/Zend/zend_hash.c:54

#1 0x0000561648c6dd71 in zend_hash_get_current_data_ex (ht=0x0, pos=0x24) at /usr/local/directadmin/custombuild/php-8.1.6/Zend/zend_hash.c:2469

#2 0x0000561648a36502 in zif_current (execute_data=0x7f3cb9217190, return_value=0x7f3cb92170e0) at /usr/local/directadmin/custombuild/php-8.1.6/ext/standard/array.c:1184

#3 0x0000561648c8e616 in ZEND_DO_ICALL_SPEC_RETVAL_USED_HANDLER () at /usr/local/directadmin/custombuild/php-8.1.6/Zend/zend_vm_execute.h:1297

#4 0x0000561648d036db in execute_ex (ex=0x7f3cb9216020) at /usr/local/directadmin/custombuild/php-8.1.6/Zend/zend_vm_execute.h:55756

#5 0x0000561648d08f2f in zend_execute (op_array=0x7f3cb925f3c0, return_value=0x0) at /usr/local/directadmin/custombuild/php-8.1.6/Zend/zend_vm_execute.h:60123

#6 0x0000561648c51979 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/local/directadmin/custombuild/php-8.1.6/Zend/zend.c:1792

#7 0x0000561648bae644 in php_execute_script (primary_file=0x7ffc99f3f6b0) at /usr/local/directadmin/custombuild/php-8.1.6/main/main.c:2538

#8 0x0000561648dc5725 in do_cli (argc=4, argv=0x56164b7c19e0) at /usr/local/directadmin/custombuild/php-8.1.6/sapi/cli/php_cli.c:966

#9 0x0000561648dc6867 in main (argc=4, argv=0x56164b7c19e0) at /usr/local/directadmin/custombuild/php-8.1.6/sapi/cli/php_cli.c:1368

I can provide the coredump too (but it's > 200MB).

Thank you for help

[blacktek]

Hello,
is there anything that I can provide to help you with the debug of this issue?

Thank you.

[fowles]

What version of protobuf are you using?
Can you share an actual reproduction and not just the stack trace?

[blacktek]

Hello,
I was using the latest version 3.21.0.

It's not easy to share an actual reproduction because the bug seg fault is not deterministic. I use protobuf to download googleads reports using 3 processes in parallel (processes, not threads). Sometimes it segfaults after 20 reports, sometimes after 10; in the middle there is a huge amount of instructions processed.

What can I provide?

Thank you

[fowles]

Can you share the protos involved? Is it always the same process of the three (or are all three instances of the same binary)?

[blacktek]

what do you mean for "protos"? the protobuf messages exchanged?

When I see that there is a segfault while downloading report A, if I retry only the report A it works (it means that is not related to the real messages exchanged, but some data structure in memory/stack). Perhaps something related to garbage collection.
The segfaults never happen at first report.

all instances have the same code (not really a binary, it's a php script).

the same issue doesn't happen when I use the protobuf library from composer (of course much slower)

[fowles]

What .proto file?
If you download report A and parse it in a loop does that crash?

[blacktek]

I don't know the details of the .proto files. Probably too much encapsulated in Google Ads libraries. I do not work directly with protobuffer.

Is there any snippet of php code that you can send me and that I can test?

Thank you

[fowles]

I was going to ask you the same thing ;)

Do you have any php code that you can share?

[blacktek]

unfortunately not. GoogleAds library is huge and to test it you need access to google ads accounts, developer tokens etc etc etc

If somebody has a piece of code to repeat in a loop I can test. Otherwise I think it would be quite complicated to extract it from the libraries.

[fowles]

I will try and get more data on our end, but we are going into the holiday weekend so expect a bit of delay

[bobhancock]

@blacktek Let's close out the issue here since there is no indication that this is a protobuf issue. Please go to https://github.com/googleads/google-ads-php/issues and create an issue there. Be sure to include:

• Which version of the Google Ads library are you using?
• A snippet, without the Zend scaffolding, that show what type of call you are making?

[blacktek]

@bobhancock I use "googleads/google-ads-php": v15.0.0

I'm not sure why you say that ths issue is with protobuf.

everything is working properly using the protobuf from composer v3.21.1

If I use the compiled protobuf library I've segfaults.

As said I'm not doing direct calls to protobuf, it's the googleads library doing that.

if you can provide some script to run on the server in a loop I can try to see if it works.

I've the full dump of the segfault, that is huge too. If you give me instructions I could try to extract something from it.

I understand that with so little information it's very hard for you to pinpoint the problem (that is not clearly reproducible)

The only guarantee is that everything works with the composer protobuf, but segfaults randomly with the compiled version.

[blacktek]

Hello,
tested with I use "googleads/google-ads-php": v15.1.0 too, but I still have the same issue.

No luck trying to download the same report again and again in an infinite loop, it didn't crash.

I've a log of all the reports downloaded and the last reported for which php crashed with a segfault. If I retry such reports later they work.

I really can't figure out what is triggering the segfault.