Repository navigation
mod_php 8.1.26 crashes on apache graceful restart #12844
Description
Description
After updating from PHP 8.1.25 to 8.1.26 our apache24 servers started to crash from time-to-time on graceful restarts (e.g. after midnight log rotation).
Here is the core dump:
(lldb) target create "httpd" --core "httpd-86152.core"
Core file '/tmp/httpd-86152.core' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'httpd', stop reason = signal SIGSEGV
* frame #0: 0x0000000801673170 libphp.so`zend_optimizer_compact_literals(op_array=0x0000000000000000, ctx=<unavailable>) at compact_literals.c:520:16
frame #1: 0x00000008015ce1c6 libphp.so`zend_extension_startup(extension=0x0000000801150510) at zend_extensions.c:196:7
frame #2: 0x00000008015b0ba1 libphp.so`zend_llist_apply_with_del(l=0x0000000801754100, func=(libphp.so`zend_extension_startup at zend_extensions.c:193)) at zend_llist.c:171:7
frame #3: 0x00000008015ce1a8 libphp.so`zend_startup_extensions at zend_extensions.c:217:2 [artificial]
frame #4: 0x000000080154f8c7 libphp.so`php_module_startup(sf=<unavailable>, additional_modules=<unavailable>, num_additional_modules=1) at main.c:2263:2
frame #5: 0x00000008016ad385 libphp.so`php_apache2_startup(sapi_module=<unavailable>) at sapi_apache2.c:381:6
frame #6: 0x00000008016acbd6 libphp.so`php_apache_server_startup(pconf=0x0000000800907028, plog=<unavailable>, ptemp=<unavailable>, s=<unavailable>) at sapi_apache2.c:495:6
frame #7: 0x0000000000263d6f httpd`ap_run_post_config + 79
frame #8: 0x000000000025f4cc httpd`main + 2476
frame #9: 0x000000000025e910 httpd`_start + 256
PHP Version
PHP 8.1.26
Operating System
FreeBSD 12.4
Activity
ndossche commentedon Nov 30, 2023
So this never happened prior to version 8.1.26?
A couple of questions:
In any case it's very strange that this line:
php-src/Zend/zend_extensions.c
Line 196 in 3f57bd8
would end up in
zend_optimizer_compact_literals, unless part of the backtrace is missing...mariancerny commentedon Dec 1, 2023
Yes, that right, It never happened before version 8.1.26.
Sorry about the backtrace, it was wrong. It crashed at our portbuild server where I forgot to restart apache. So the crash is from old version of apache however the backtrace is done against new version of binaries.
Here is the correct backtrace:
It reliably crashes after
apachectl gracefulon the portbuild server.It also crashes when rotating logs using
newsyslogwhich is sending signal 30 (SIGUSR1) to apache httpd which I believe is also doing graceful restart.Here is the list of extensions:
I wanted find out which extension is causing the crash by disabling some of the extensions.
I don't know which extension is calling directly
libzstd.so. I don't build all the packages with debug symbols (only apache and php).I tried disabling
fileinfo(the only one in/usr/local/lib/php/20210902that is positive forgrep zstd), didn't help.I tried disabling
zlib, didn't help.I tried disabling all, no crash.
I tried disable all except
ext-10-opcache.iniandext-18-session.ini. Now the crash was different:I tried to disable
ext-10-opcache.iniand enable all the others, it doesn't crash.Unanswered questions:
apachectl graceful. However it does not crash reliably on our production servers - most of the time it does not crash onapachectl graceful, but some times it does. However I do not have crashdump on the production servers (I don't know why yet).I will investigate further. E.g. I will try to get crash dump on the production servers.
mariancerny commentedon Dec 1, 2023
The crash on production servers is different (which I thought it might be, because
apachectl gracefuldoes not reproduce the crash reliably -- it crashes only sometimes):ndossche commentedon Dec 1, 2023
It seems to call to bogus pointers. If I had to take a guess (but I haven't confirmed this) I think the only commit between 8.1.25 and 8.1.26 that could cause this is a8c6c61. This touches module startup code which is what I see in your backtraces.
If possible, could you create a custom PHP build with that commit reverted and check if there are issues?
I'll try to find some time soon to try and reproduce this myself.
mariancerny commentedon Dec 3, 2023
TLDR: It's not related to changes in php 8.1.26. Seems to be related to changes in FreeBSD package for 8.1.26.
I have tried to (manually) revert commit a8c6c61, but it (apachectl graceful) crashed anyway.
I have tried to (manually) revert all the commits from 8.1.26 (to confirm I am building it correctly), but it crashed anyway.
There have been some other changes in the FreeBSD port in 8.1.26 (https://www.freshports.org/commit.php?category=lang&port=php81&files=yes&message_id=06be0d5391f475771145a7a9d68d34b0ad5a9b85):
I have tried to change the Makefile for the FreeBSD port to build 8.1.25 instead of 8.1.26 and it crashed! (So the crash is not related to changes in php 8.1.26.)
I have tried to revert changes related to "Switch to system pcre", and it didn't crash! I removed those (FreeBSD) changes:
(and also used pkg-plist from the FreeBSD port for version 8.1.25, which re-adds include/php/ext/pcre/pcre2lib/ header files, otherwise some php extensions failed to build.)
So reverting
--with-external-pcrefixes the crash on our portbuild server. I will try to apply this "fix" on our production servers and see if it fixes the problem there as well. I will then report this problem to FreeBSD bug tracking system.However, I find it strange that changing the pcre library is causing this crash, because the backtrace from the core dump does not seem to be related to pcre.
ndossche commentedon Dec 3, 2023
Thanks for the investigation!
That's bizarre and seems like the problem is with whatever FreeBSD is doing.
If I had to take a guess, one possibility is that a conflicting version of pcre2 is used. e.g. when one (transitive) library dependency of an extension depends on pcre2 and php also uses pcre2 for ext/pcre, this can cause problems when different versions are used because of conflicting symbols.
Using the bundled pcre2 would fix this because we prefix some symbol names with the name php to avoid that exact issue.
mariancerny commentedon Dec 4, 2023
The fix to FreeBSD port reverting "Switch to system pcre" seems to have fixed also the random (sporadic) crash on reload on our production servers as well. No crash so far.
I have reported the bug to FreeBSD project:
Bug 275522 - www/mod_php81: Apache crashes on reload after switch to system pcre
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275522
2 remaining items
ndossche commentedon Dec 4, 2023
To clarify: this is only a problem if mixed versions are used of the same library because the symbols create a conflict. This doesn't happen on stock PHP. This could happen for example when there are other modules in Apache or 3rd party extensions in PHP that load a different pcre version (e.g. bundled library) where the names clash.
Also I'm not sure this is actually the problem the user is experiencing, this is just based on prior experience and the traces I saw.
Note that we do backport critical and security fixes to the bundled pcre library. So although the version is seemingly not updated we do backport fixes.
I don't know the full history of this, but AFAIK this is the preferred way.
I'm also reading something in the FreeBSD issue report comments about other Apache issues. Especially this is an interesting remark: "From my experience, Apache2 recently crashes almost on any module including its own.".
5u623l20 commentedon Dec 5, 2023
Thanks for your detailed answers. I will look into reverting to bundled pcre for php81 but will still stay with system pcre for php82 and php83. I don't want to introduce this into php80 as we are removing this on end of the year.
I am not into apache and cannot comment on that but so far the issue started after we switched to ASLR on default installs. And specially with php opcache module and mod_php for apache httpd. I hope one day php will remove the entire support for mod_php as this does not make any sense after the introduction of php_fpm. One another interesting result has been building httpd with gcc instead of llvm/clang.
mariancerny commentedon Dec 5, 2023
Thanks to your additional info I now think this bug is "the same" bug that has been seen with apache on FreeBSD with ASLR. I am using FreeBSD 12.4 and ASLR is enabled by default only since FreeBSD 13.2. So I guess there is a pointer bug somewhere in apache or php that is causing the crash. When ASLR is not used, the pointer points to some valid memory. However when ASLR is used, the memory becomes invalid. Similar thing probably happens with external pcre (php code image is smaller in memory) and gcc (it compiles the code differently so the pointer might point to some valid memory, unless it's llvm bug, which I guess it's much less probable).
The core dumps I have seen crash when going from libphp.so to some other place, so I guess the bug is in PHP. I would suspect the list that is accessed by zend_llist_apply_with_del to be causing the problem. If the crash is reproducible, somebody with source code knowledge of php and apache should be able to track it down.
gioman commentedon Apr 1, 2024
Observing the same on Ubuntu Server 22.04
ndossche commentedon Apr 1, 2024
@gioman do you have a reproducer? Otherwise it's near impossible to help.
gioman commentedon Apr 1, 2024
@nielsdos thanks for the reply. My observation until now (a deeper analysis may be needed):
issuing graceful restart (manually or by letting logrotate do its thing) causes error.log to be flooded with
until a
systemctl restart apache2is issued.Once libapache2-mod-php81 is disabled the above does not happen anymore.
ndossche commentedon Apr 1, 2024
If you're able to get a backtrace from gdb for example that would be helpful, as there is still too little information do know what's going on.
realFlowControl commentedon Dec 3, 2025
IDK if this is news to anyone, but
mod_phpis special in one regard: a graceful restart does re-use the process. It goes through theMSHUTDOWNphase and afterwards (in the same process) goes throughMINITagain.mod_phpis to my knowledge the only SAPI doing this, so any pointer that is left dangling afterMSHUTDOWNmay bite you in that case.yuuki-yano commentedon Aug 4, 2026
We've reproduced this crash across 7 FreeBSD servers in production. All servers use mod_php + OPcache, with Apache being reloaded (
apachectl graceful) via certbot'srenew_hook/deploy_hookset toservice apache24 reload.Environment (representative of all 7 servers)
--with-external-pcre=/usr/local), PCRE 10.47kern.elf64.aslr.enable=1), httpd binary hasnoaslrunsetTrigger
certbot's renewal hook invokes
service apache24 reload(graceful restart) after a successful certificate renewal. This is a very common/default-ish pattern for certbot + Apache setups.Observed behavior
Apache's error log shows the graceful restart being requested, and then goes completely silent — no further log lines at all — until Apache is manually or automatically restarted hours later (in one case, over 8 hours).
The corresponding kernel log confirms the master process died from SIGSEGV at exactly the timestamp of the graceful restart:
Because
reloadonly signals an already-running process, once the master process dies mid-restart, there is no automatic recovery — the server stays down silently until someone notices and manually restarts it.Backtrace (gdb)
This matches the frame pattern reported earlier in this issue by @mariancerny (
zend_llist_apply_with_del→php_module_startup→ap_run_post_config), rather than thezend_optimizer_compact_literalsvariant.Workaround
Changing the certbot renewal hook from
reloadtorestart(full process restart instead of graceful) across all 7 servers has resolved the issue — no further crashes since. This avoids the in-process MSHUTDOWN→MINIT re-initialization path entirely, consistent with @realFlowControl's explanation above about mod_php being the only SAPI that reuses the process across a graceful restart.Given this reproduces on FreeBSD 14.0 with PHP 8.3.x / Apache 2.4.67 (newer than the versions previously reported here), this doesn't appear to be fixed in current versions.
日本語訳（Japanese）
本番環境のFreeBSDサーバー7台でこのクラッシュを再現しました。全サーバーでmod_php + OPcacheを使用しており、certbotの
renew_hook/deploy_hookにservice apache24 reloadを設定して証明書更新後にApacheをreload(graceful restart)する構成です。環境(7台に共通する代表的な構成)
--with-external-pcre=/usr/local)、PCRE 10.47kern.elf64.aslr.enable=1)、httpdバイナリもnoaslrはunset(ASLR対象)トリガー
certbotのrenewal hookが、証明書更新成功後に
service apache24 reload(graceful restart)を実行します。これはcertbot + Apache構成において非常に一般的な(ほぼデフォルトに近い)設定パターンです。観測された挙動
Apacheのエラーログにはgraceful restartのリクエストが記録された後、完全に沈黙します。ログが一切出力されないまま、数時間後(あるケースでは8時間以上)に手動または自動でApacheが再起動されるまで、その状態が続きます。
対応するカーネルログでも、graceful restartが要求されたのと全く同じタイムスタンプで、masterプロセスがSIGSEGVで死んでいることが確認できます。
reloadは既に動いているプロセスへのシグナル送信でしかないため、再起動処理の途中でmasterプロセスが死んでしまうと、自動的な復旧手段が一切なくなります。誰かが気づいて手動で再起動するまで、サーバーは無言のまま停止し続けます。バックトレース(gdb)
このフレーム構成は、本Issue内で@mariancerny氏が以前報告していたパターン(
zend_llist_apply_with_del→php_module_startup→ap_run_post_config)と一致しており、zend_optimizer_compact_literals系の別パターンとは異なります。回避策
certbotのrenewal hookを
reloadからrestart(gracefulではなく完全なプロセス再起動)に変更したところ、7台全てで問題が解消し、それ以降クラッシュは発生していません。これにより、プロセス内でのMSHUTDOWN→MINITの再初期化処理そのものを回避できます。これは@realFlowControl氏が上記コメントで説明されている、「mod_phpはgraceful restart時にプロセスを再利用する唯一のSAPIである」という点とも整合しています。FreeBSD 14.0、PHP 8.3.x、Apache 2.4.67という、本Issueでこれまで報告されていたバージョンよりも新しい環境で再現していることから、現行バージョンでも未修正の問題であると考えられます。
kamil-tekiela commentedon Nov 8, 2026
Does this still happen with PHP 8.5 or PHP 8.6?
mariancerny commentedon Nov 8, 2026
I don't know. We have migrated from mod_php to PHP-FPM and nginx because of this problem.