Skip to content

Sporadic opcache crash with apache and php-cgi.exe #11549

Closed

Description

@sagentac

Description

We recently updated our Project from PHP 7.1.6 to 8.2. Since the switch we are sporadically experiencing opcache crashes.
With PHP 7.1.6 we did not experience any opcache issues. The server hardware and opcache configuration stayed the same.

In the windows event log we see the following:

Faulting application name: php-cgi.exe, version: 8.2.3.0, time stamp: 0x63eb5a8b
Faulting module name: php_opcache.dll, version: 8.2.3.0, time stamp: 0x62949a01
Exception code: 0xc0000005
Fault offset: 0x00000000000100b3
Faulting process id: 0x1e568
Faulting application start time: 0x01d9a3013fdeff1f
Faulting application path: D:\wap\php\php-8.2.3-nts-Win32-vs16-x64\php-cgi.exe
Faulting module path: D:\wap\php\php-8.2.3-nts-Win32-vs16-x64\ext\php_opcache.dll
Faulting package full name:
Faulting package-relative application ID:

The opcache config is loogking like this (php.ini):

[opcache]
opcache.enable=1
opcache.enable_cli=0
opcache.memory_consumption=128
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=4000
opcache.use_cwd=1
opcache.validate_timestamps=1
opcache.revalidate_path=1
opcache.save_comments=1
opcache.file_cache=D:\wap\php\opcache

I also tried PHP 8.2.7 with Apache VS17 and had the same problem.

Crash dump and analysis pdf:
php-cgi.exe_230623_.zip
opcache-crash-analysis_php-cgi.exe_230623_.pdf

PHP Version

8.2.3 NTS

Operating System

Windows Server 2019; Apache httpd 2.4.56-win64-VS16

Activity

  1. changed the title [-]Sporadically opcache crash with apache and php-cgi.exe[/-] [+]Sporadic opcache crash with apache and php-cgi.exe[/+] on Jun 28, 2023
  2. sagentac commented on Jun 29, 2023

    @sagentac
    Author

    When the service crashes we also get these error message in the php error logs:

    [29-Jun-2023 07:58:05 UTC] PHP Warning: include(D:\wap\htdocs\l�4��): Failed to open stream: No such file or directory in D:\wap\htdocs...

  3. ndossche commented on Jul 1, 2023

    @ndossche
    Member

    The crash happens in zend_jit_deactivate, so I'm assuming you're using the opcache JIT? If yes, does the problem go away when you disable JIT? And also, is it possible to use the latest 8.2.x release? I see you're on 8.2.3, so maybe the underling problem has been fixed in the meantime.

  4. sagentac commented on Jul 3, 2023

    @sagentac
    Author

    Thanks for your reply!

    Yes we were using the jit default (tracing).
    When disabling jit (off) we also experienced sproradic crashes (I dont have any dump file here).

    Regarding newer versions:
    I also tried PHP 8.2.7 with Apache VS17 and had the same problem.

    Maybe the issue is not related to the opcache specifically, but occurs more often with opcache.
    We just experienced another crash:

    Faulting application name: php-cgi.exe, version: 8.2.7.0, time stamp: 0x64805b65
    Faulting module name: WS2_32.dll, version: 10.0.17763.2028, time stamp: 0x62949a01
    Exception code: 0xc0000005
    Fault offset: 0x000000000000ffb9
    Faulting process id: 0x4fd28
    Faulting application start time: 0x01d9ad8227af359e
    Faulting application path: D:\wap\php\827-x64-VS16-NTS\php-cgi.exe
    Faulting module path: C:\Windows\System32\WS2_32.dll
    Report Id: e48a8845-ad4c-4c66-890d-f1afb64ebb5d
    Faulting package full name:
    Faulting package-relative application ID:

    This time it was WS2_32.dll.

    Could there be a problem with fcgid?
    The settings there are

    FcgidInitialEnv SystemRoot "C:/Windows"
    FcgidInitialEnv SystemDrive "C:"
    FcgidInitialEnv TEMP "D:/wap/tmp"
    FcgidInitialEnv TMP "D:/wap/tmp"
    FcgidInitialEnv windir "C:/WINDOWS"
    FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 10000 
    FcgidInitialEnv PHP_FCGI_CHILDREN 0
    
    FcgidIdleScanInterval 10
    FcgidIdleTimeout 300
    FcgidIOTimeout 500
    FcgidMinProcessesPerClass 0 
    FcgidMaxProcessesPerClass  9999 
    FcgidMaxRequestsPerProcess 10000
    FcgidProcessLifeTime 3600
    FcgidMaxRequestInMem 1 
    FcgidWin32PreventOrphans On
    FcgidMaxRequestLen 1000000000
    

    Also The apache error log says

    [Mon Jul 03 10:25:59.895957 2023] [fcgid:warn] [pid 256276:tid 5020] (OS 109)The pipe has been ended.  : [client 10.123.123.123:56738] mod_fcgid: get overlap result error, referer: https://my.domain.com/ntlmAuth/authnew.php
    [Mon Jul 03 10:25:59.895957 2023] [core:error] [pid 256276:tid 5020] [client 10.123.123.123:56738] End of script output before headers: index.php, referer: https://my.domain.com/ntlmAuth/authnew.php
    
  5. ndossche commented on Jul 3, 2023

    @ndossche
    Member

    It's difficult to say where the problem could be. The crash dumps aren't very informative because the crash likely comes from a bug that was triggered way earlier in execution. It looks like a corruption bug somewhere.
    How often does it crash? And does it always crash at the end of a request?
    It would also be helpful if we had a way to reproduce the issue on our systems.

  6. philipphaeusle commented on Jul 4, 2023

    @philipphaeusle

    With opcache.file_cache_only=1 the it crashes once a day (Some random dll, not opcache). Today it was icuuc71.dll.
    With opcache.file_cache_only=0, the opcache.dll continuously crashes for a few minutes (5-10), and then heals itself.
    I dont know if it crashes at the end of the request. With opcache.file_cache_only=0, the application gets unresponsive for few minutes where the error occurs and works normally again after that timeframe.

    Im afraid i dont know how to reproduce this issue since we cannot reproduce either on our machines.
    It randomly occurs on our production server which is under heavy load (about 3000 users).

    What also conserns me is that PHP is this message here.

  7. sagentac commented on Sep 27, 2023

    @sagentac
    Author

    We just updated to 8.2.10, but unfortunately it did not solve the crashes.

  8. ndossche commented on Sep 27, 2023

    @ndossche
    Member

    Without a reproducer it will be very hard to find the root cause and fix it. The problem with crash dumps is that they're taken long after it went wrong.

  9. sagentac commented on Sep 27, 2023

    @sagentac
    Author

    Thanks for your reply,
    sadly we also cannot reproduce on our local machines. Only our production server is affected.
    This issues seems related i think: #12010

  10. mvachette commented on Nov 13, 2023

    @mvachette

    We experiment a similar problem on some production server with PHP8.1. (Windows 2019 / IIS 10)

    Please find below the error messages collected, and opcache config

    Error happens only on production server, under load.

    We activate file cache fallback to reduce the error frequency (but error is still present sometime, and we loose performances)

    IIS 10.0 Detailed Error - 500.0 - Internal Server Error

    Detailed Error Information:
    Module: FastCgiModule
    Notification: ExecuteRequestHandler
    Handler: PHP81
    Error Code: 0xc0000005

    Windows log

    Faulting application name: php-cgi.exe, version: 8.1.22.0, time stamp: 0x64c93ded
    Faulting module name: php_opcache.dll, version: 8.1.22.0, time stamp: 0x64c94350
    Exception code: 0xc0000005
    Fault offset: 0x0000000000013e75
    Faulting process id: 0x9cc
    Faulting application start time: 0x01d9d3f4eac42752
    Faulting application path: c:\php81\php-cgi.exe

    opcache config

    opcache.enable=1
    opcache.enable_cli=0
    opcache.memory_consumption=512
    opcache.interned_strings_buffer=256
    opcache.max_accelerated_files=20000
    opcache.validate_timestamps=0
    opcache.revalidate_freq=120
    opcache.save_comments=1

  11. ndossche commented on Nov 13, 2023

    @ndossche
    Member

    There will be an opcache fix related to restart crashes in 8.1.26, which will be released on Nov 26th. Would be interesting to know if that fixes this issue.

  12. mvachette commented on Dec 21, 2023

    @mvachette

    The version 8.1.26 solve the opcache.dll crashes, but on some servers, we continue to face soma random crashes of php_cgi.dll.

    Here ares some error traces we have :

    Faulting application name: php-cgi.exe, version: 8.1.26.0, time stamp: 0x655dd195
    Faulting module name: php8.dll, version: 8.1.26.0, time stamp: 0x655dd5c0
    Exception code: 0xc0000005
    Fault offset: 0x0000000000433242
    Faulting process id: 0x2144
    Faulting application start time: 0x01da33298a792c77
    Faulting application path: c:\php81\php-cgi.exe
    Faulting module path: c:\php81\php8.dll
    Report Id: dab9fa50-79c1-4399-a8c7-f0beffb9b050
    Faulting package full name: 
    Faulting package-relative application ID: 
    
  13. kamil-tekiela commented on Nov 7, 2026

    @kamil-tekiela
    Member

    GH-8143 is fixed since PHP 8.1.26 and I assume that other crashes are unrelated and point to a different issue. However, without more information and a complete reproducer we cannot identify the root cause. Please open a new GH issue with a reproducer if you still encouter these problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions