Repository navigation
segfault in zend_mm_alloc_small triggered by call to unserialize #12351
Description
Description
This is a large WordPress website running some commonly available community plugins. The site was working fine on PHP 7.4 but since upgrading to 8.2.6 and then subsequent 8.2.10 to see if the issue was resolved we are seeing segfaults on multiple servers in the cluster servicing this website.
Every single fault is during a call to unserialize, here is the zbacktrace:
[0x7fb2ac2138e0] unserialize("a:5:{s:3:"AUD";a:8:{s:4:"name";s:3:"AUD";s:4:"rate";d:1;s:6:"symbol";s:3:"AU$";s:8:"position";s:4:"left";s:
9:"is_etalon";i:1;s:10:"hide_cents";i:0;s:11:"description";s:17:"Australian Dollar";s:4:"flag";s:88:"https://www.xxxxxxx.com/wp-content/p
lugins/wooco...") [internal function]
[0x7fb2ac213860] maybe_unserialize("a:5:{s:3:"AUD";a:8:{s:4:"name";s:3:"AUD";s:4:"rate";d:1;s:6:"symbol";s:3:"AU$";s:8:"position";s:4:"le
ft";s:9:"is_etalon";i:1;s:10:"hide_cents";i:0;s:11:"description";s:17:"Australian Dollar";s:4:"flag";s:88:"https://www.xxxxxxx.com/wp-con
tent/plugins/wooco...") /home/xxxxxxx/public_html/wp-includes/functions.php:650
[0x7fb2ac213650] get_option("woocs", array(2)[0x7fb2ac2136b0]) /home/xxxxxxx/public_html/wp-includes/option.php:253
[0x7fb2ac2135d0] WOOCS->get_currencies() /home/xxxxxxx/public_html/wp-content/plugins/woocommerce-currency-switcher/index.php:972
[0x7fb2ac213530] WOOCS->init_no_cents() /home/xxxxxxx/public_html/wp-content/plugins/woocommerce-currency-switcher/index.php:497
[0x7fb2ac213470] WOOCS->__construct() /home/xxxxxxx/public_html/wp-content/plugins/woocommerce-currency-switcher/index.php:82
[0x7fb2ac2133e0] (main) /home/xxxxxxx/public_html/wp-content/plugins/woocommerce-currency-switcher/index.php:2213
[0x7fb2ac213220] (main) /home/xxxxxxx/public_html/wp-settings.php:462
[0x7fb2ac2131a0] (main) /home/xxxxxxx/public_html/wp-config.php:72
[0x7fb2ac213100] (main) /home/xxxxxxx/public_html/wp-load.php:50
[0x7fb2ac213070] (main) /home/xxxxxxx/public_html/wp-blog-header.php:13
[0x7fb2ac213020] (main) /home/xxxxxxx/public_html/index.php:17
And the PHP backtrace
#0 zend_mm_alloc_small (bin_num=11, heap=0x7fb2ac200040) at ./Zend/zend_alloc.c:1313
#1 zend_mm_alloc_heap (size=<optimized out>, heap=0x7fb2ac200040) at ./Zend/zend_alloc.c:1384
#2 _emalloc (size=<optimized out>) at ./Zend/zend_alloc.c:2601
#3 0x0000556923c3962a in zend_string_alloc (persistent=false, len=88) at ./Zend/zend_string.h:152
#4 zend_string_init (persistent=false, len=88,
str=0x7fb2a6e89a92 "https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"NZD\";a:8:{s:4:\"name\";s:3:\"NZD\";s:4:\"rate\";d:1.07660000000000000142108547152020037174224853515625;s:6:\"symbol\";s:3:\"NZ$\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:18:\"New Zealand Dollar\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"GBP\";a:8:{s:4:\"name\";s:3:\"GBP\";s:4:\"rate\";d:0.502499999999999946709294817992486059665679931640625;s:6:\"symbol\";s:7:\"£\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:13:\"British Pound\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}}") at ./Zend/zend_string.h:174
#5 zend_string_init_fast (len=88,
str=0x7fb2a6e89a92 "https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"NZD\";a:8:{s:4:\"name\";s:3:\"NZD\";s:4:\"rate\";d:1.07660000000000000142108547152020037174224853515625;s:6:\"symbol\";s:3:\"NZ$\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:18:\"New Zealand Dollar\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"GBP\";a:8:{s:4:\"name\";s:3:\"GBP\";s:4:\"rate\";d:0.502499999999999946709294817992486059665679931640625;s:6:\"symbol\";s:7:\"£\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:13:\"British Pound\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}}") at ./Zend/zend_string.h:184
#6 php_var_unserialize_internal (rval=0x7fb2a6e45fa0, p=0x7fff469fb468, max=<optimized out>, var_hash=0x7fff469fb470)
at ./ext/standard/var_unserializer.c:1798
#7 0x0000556923c39110 in process_nested_array_data (elements=0, ht=0x7fb2a6f0bb28, var_hash=0x7fff469fb470, max=0x7fb2a6e89da2 "",
p=0x7fff469fb468, rval=0x7fff469fb2a0) at ./ext/standard/var_unserializer.c:511
#8 php_var_unserialize_internal (rval=0x7fff469fb2a0, p=0x7fff469fb468, max=0x7fb2a6e89da2 "", var_hash=0x7fff469fb470)
at ./ext/standard/var_unserializer.c:1742
#9 0x0000556923c39110 in process_nested_array_data (elements=2, ht=0x7fb2a6f0ba80, var_hash=0x7fff469fb470, max=0x7fb2a6e89da2 "",
p=0x7fff469fb468, rval=0x7fff469fb390) at ./ext/standard/var_unserializer.c:511
#10 php_var_unserialize_internal (rval=0x7fff469fb390, p=0x7fff469fb468, max=0x7fb2a6e89da2 "", var_hash=0x7fff469fb470)
at ./ext/standard/var_unserializer.c:1742
#11 0x0000556923c3ac47 in php_var_unserialize (rval=<optimized out>, p=<optimized out>, max=<optimized out>, var_hash=<optimized out>)
at ./ext/standard/var_unserializer.c:853
#12 0x0000556923c29fb4 in php_unserialize_with_options (function_name=0x556923db086e "unserialize", options=<optimized out>,
buf_len=1674,
buf=0x7fb2a6e89718 "a:5:{s:3:\"AUD\";a:8:{s:4:\"name\";s:3:\"AUD\";s:4:\"rate\";d:1;s:6:\"symbol\";s:3:\"AU$\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:1;s:10:\"hide_cents\";i:0;s:11:\"description\";s:17:\"Australian Dollar\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"USD\";a:8:{s:4:\"name\";s:3:\"USD\";s:4:\"rate\";d:0.71679999999999999271693695845897309482097625732421875;s:6:\"symbol\";s:3:\"US$\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:9:\"US Dollar\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"EUR\";a:8:{s:4:\"name\";s:3:\"EUR\";s:4:\"rate\";d:0.6449000000000000287769807982840575277805328369140625;s:6:\"symbol\";s:6:\"€\";s:8:\"position\";s:10:\"left_space\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:4:\"Euro\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"NZD\";a:8:{s:4:\"name\";s:3:\"NZD\";s:4:\"rate\";d:1.07660000000000000142108547152020037174224853515625;s:6:\"symbol\";s:3:\"NZ$\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:18:\"New Zealand Dollar\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"GBP\";a:8:{s:4:\"name\";s:3:\"GBP\";s:4:\"rate\";d:0.502499999999999946709294817992486059665679931640625;s:6:\"symbol\";s:7:\"£\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:13:\"British Pound\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}}", return_value=0x7fb2ac2138c0) at ./ext/standard/var.c:1406
#13 php_unserialize_with_options (return_value=0x7fb2ac2138c0,
buf=0x7fb2a6e89718 "a:5:{s:3:\"AUD\";a:8:{s:4:\"name\";s:3:\"AUD\";s:4:\"rate\";d:1;s:6:\"symbol\";s:3:\"AU$\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:1;s:10:\"hide_cents\";i:0;s:11:\"description\";s:17:\"Australian Dollar\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"USD\";a:8:{s:4:\"name\";s:3:\"USD\";s:4:\"rate\";d:0.71679999999999999271693695845897309482097625732421875;s:6:\"symbol\";s:3:\"US$\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:9:\"US Dollar\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"EUR\";a:8:{s:4:\"name\";s:3:\"EUR\";s:4:\"rate\";d:0.6449000000000000287769807982840575277805328369140625;s:6:\"symbol\";s:6:\"€\";s:8:\"position\";s:10:\"left_space\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:4:\"Euro\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"NZD\";a:8:{s:4:\"name\";s:3:\"NZD\";s:4:\"rate\";d:1.07660000000000000142108547152020037174224853515625;s:6:\"symbol\";s:3:\"NZ$\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:18:\"New Zealand Dollar\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}s:3:\"GBP\";a:8:{s:4:\"name\";s:3:\"GBP\";s:4:\"rate\";d:0.502499999999999946709294817992486059665679931640625;s:6:\"symbol\";s:7:\"£\";s:8:\"position\";s:4:\"left\";s:9:\"is_etalon\";i:0;s:10:\"hide_cents\";i:0;s:11:\"description\";s:13:\"British Pound\";s:4:\"flag\";s:88:\"https://www.xxxxxxx.com/wp-content/plugins/woocommerce-currency-switcher/img/no_flag.png\";}}", buf_len=1674, options=<optimized out>, function_name=0x556923db086e "unserialize")
at ./ext/standard/var.c:1334
#14 0x0000556930e24644 in ?? ()
#15 0x0000000000000001 in ?? ()
#16 0x0000000000000003 in ?? ()
#17 0x00000004ac200040 in ?? ()
#18 0x00007fff469fb4f0 in ?? ()
#19 0x0000000000000001 in ?? ()
#20 0x0000000000000001 in ?? ()
#21 0x00007fb2a6e45480 in ?? ()
#22 0x0000556900000003 in ?? ()
#23 0x0000000000000002 in ?? ()
#24 0x0000000000000001 in ?? ()
#25 0x0000556925ed44c0 in ?? ()
#26 0x0000000000000001 in ?? ()
#27 0x0000556925ed44c0 in ?? ()
#28 0x3743e960f3c58a00 in ?? ()
#29 0x00007fff469fb870 in ?? ()
#30 0x00007fb2ac277000 in ?? ()
#31 0x00007fb2ac213020 in ?? ()
#32 0x0000556923f3f500 in ?? ()
#33 0x0000000000000000 in ?? ()
And the PHP modules loaded
[PHP Modules]
calendar
cgi-fcgi
Core
ctype
curl
date
dom
exif
FFI
fileinfo
filter
ftp
gd
gettext
gmp
hash
iconv
igbinary
json
libxml
mbstring
memcache
memcached
msgpack
mysqli
mysqlnd
openssl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
random
readline
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
xml
xmlreader
xmlrpc
xmlwriter
xsl
Zend OPcache
zip
zlib
[Zend Modules]
Zend OPcache
Setting I had set the environment variable improperly, re-testing to see if this prevents the fault.USE_ZEND_ALLOC=0 does not change this behaviour at all.
Edit: USE_ZEND_ALLOC=0 completely eliminates the issue.
PHP Version
8.2.10
Operating System
Debian 11.7
Activity
gnif commentedon Oct 3, 2023
Further information, here is the content of
heap:iluuu1994 commentedon Oct 3, 2023
Hi @gnif! Segfaults in Zend memory manager are usually caused by buffer overflows rather than bugs in ZMM itself. Are you using JIT? If so, could you try disabling it? It not, it would be helpful to build PHP with the
./configure --enable-address-sanitizerflag. This will make PHP crash immediately when the buffer overflow occurs. There are also quite a few non-bundled extensions. Can any of them be disabled to see if the error still occurs?gnif commentedon Oct 3, 2023
Unfortunately this is a production system, as such rebuilding with ASAN and disabling extensions are not possible here. I am also unable to reliably reproduce the crash in a controlled way as I can not generate enough traffic to trigger the bug.
I will disable JIT and report back if this resolves the issue.
gnif commentedon Oct 3, 2023
One other thing that should be noted, the state of the heap looks good, I don't see odd/strange values there that might indicate corruption that a buffer overflow would cause, however accessing element
11in thefree_slotresults in the fault, which indicates it references memory that has been released/freed. Perhaps a garbage collection bugiluuu1994 commentedon Oct 3, 2023
@gnif I suppose you're getting a segfault when dereferencing the pointer in
heap->free_slot[11]? That pointer could still have been corrupted I suppose. But it's hard to tell retrospectively. PHP doesn't have a traditional garbage collector. It's just a cycle collector, that then triggers a free in zmm. Other objects are freed with ref-counting (also through zmm).gnif commentedon Oct 3, 2023
Understood. FTR so far it seems that disabling opcache.jit has worked.
iluuu1994 commentedon Oct 3, 2023
That's good to know, thank you! Unfortunately, JIT bugs are very hard to track down. There's also only one person who fully understands the JITs ins and outs (the person who wrote it). The good news is that a new JIT is in the works that is easier to understand and maintain, and hopefully will lead to fewer and easier to fix bugs. https://wiki.php.net/rfc/jit-ir
gnif commentedon Oct 3, 2023
Sorry but it's not JIT related, we just had it occur several dozen times with it still disabled.
1 remaining item
iluuu1994 commentedon Oct 3, 2023
@gnif Is the stack trace more helpful in that case? (without the jit traces)
gnif commentedon Oct 3, 2023
Stack trace has not changed
gnif commentedon Oct 3, 2023
Correction sorry, yes it has, I was looking at the wrong terminal :)
Here is the zbacktrace too:
iluuu1994 commentedon Oct 3, 2023
@gnif Since the error now doesn't occur in
unserializeanymore it seems the error is unrelated. Unfortunately, this will be hard to diagnose without ASAN. Can you see whether this affects a particular endpoint in your app? If so, maybe it would be possible to trigger this in a dev environment. If that's not possible, then I unfortunately do not now how to proceed.ndossche commentedon Oct 3, 2023
Does this happen with opcache off too?
Also might be worth trying to set opcache.protect_memory=1 and see if it happens more regularly then.
gnif commentedon Oct 4, 2023
Thanks guys, I am trying to arrange a build with ASAN and will update this with what I find (if anything)
gnif commentedon Oct 4, 2023
ASAN build is running and faults are occurring as the FPM log reports that children are exiting with code 1, but there are no longer any core dumps, nor is there any output from ASAN even when configuring the
ASAN_OPTIONSwithlog-path.If I set
atexit=trueASAN does give me it's exit status when I terminate the process indicating I am looking in the right place for it's outputEdit: I am now rebuilding the modules with ASAN support also as it occurred to me that the fault may be there which is why ASAN is not catching it.
iluuu1994 commentedon Oct 4, 2023
@gnif Great! Thank you for your time investment! I'm guessing you already know, but ASAN will require the
USE_ZEND_ALLOC=0env var to properly catch buffer overflows (because ZMM puts them in shared pages that the application owns).gnif commentedon Oct 6, 2023
Unfortunately I can not reproduce the fault with this build as the PHP process ends up consuming so much RAM (90+GB) it gets terminated by the kernel's OOM killer.
iluuu1994 commentedon Oct 9, 2023
@gnif Hmm, that's unfortunate. 😕 What
./configureflags did you use? Also, does settingUSE_TRACKED_ALLOC=1(along withUSE_ZEND_ALLOC=0make a difference? ASAN does cause a slowdown of 2-3x but I don't know how much it is expected to affect memory, and how much memory is used without ASAN in your case.joehoyle commentedon Oct 25, 2023
I think I'm seeing the same issue as here, specifically with
unserializecausing a segfault inzend_mm_alloc_small. In my case, it seems thefree_slotarray has an invalid pointer, but of course the segfault doesn't occur until READ, at which point it's not clear what was responsible for writing the pointer in the first place:(See address
0xffffffff00000000).I've tried using
USE_ZEND_ALLOC=0andaddress-sanitizerbut that doesn't of course trigger the same code path and I can't reliably produce the segfaults / exits that way.kamil-tekiela commentedon Nov 8, 2026
It has been 3 years and we were unable to arrive at a reproducer with ASAN. I will close this issue as cannot reproduce. Since this issue has been raised many fixes landed including a new JIT. Please retest with the latest version to see if this is still happening.