Ongoing loopback challenge

[TonyGman]

Summary

We frequently see reports here in the Local forum and elsewhere about timeouts, curl issues, loopback errors, etc. What’s happening is that requests are queued, but only executed on exit of the calling code. So errors are reported from WP Site Health, wp-cron, WooCommerce, outbound REST queries, etc…

These common loopback issues seem related to PHP FPM on Windows. This inquiry is about getting a firm understanding of the problem, even if we don’t have a solution yet - though once we have the former we might immediately get the latter.

Environment

The issue is independent of any versions of this stack: Windows, Local, Apache, PHP. I’m running the latest Local over W10 with PHP 7.4.

I have not tried to create a new WAMP environment other than Local to compare differences. I do not want to modify the stack to avoid the problem - I want to understand and solve the problem.

Replication

In the WordPress root, add two files: loopclient.php and loopserver.php. In loopclient, use common curl_init, curl_setopt, and curl_exec functions to request “http://yoursite.local/loopserver.php” :

error_log('client before' . PHP_EOL, 3, 'loop.log');
$response = curl_exec($ch);
curl_close($ch);
error_log('client after' . PHP_EOL . $response . PHP_EOL, 3, 'loop.log');

In loopserver:

error_log(date('c') . ' server' . PHP_EOL, 3, 'loop.log');
echo "from server";

Verify that loopserver.php is available by opening it with a browser.

Browse to loopclient.php. We expect the log will show ‘client before’, ‘server’, ‘client after’, and then a response. We expect the browser will show “from server”.

That does not happen. It will show ‘client before’, there will be a pause for the timeout, then we see ‘client after’, no response to the log, and then ‘server’, and no response to the browser.

Replace the url with “http://google.com”. The response is now correct: ‘client before’, ‘client after’, …Google HTML.

Using Xdebug, you can follow execution to see curl_exec, then the exit from loopclient, followed by entry into loopserver. However, the desired behavior is curl_exec, processing of loopserver, and a response available to loopclient which then goes back to the browser.

Analysis

I think the request is getting into Apache because it is being processed even after the client side disconnects. It makes no sense that a cURL transaction would still be in client-side memory and be transmitted after curl_close().

If that’s correct, the next process that I believe would be executed would be the PHP handler. From site.conf.hbs:

    {{#if os.windows}}
    <FilesMatch \.php$>
        {{#unless shouldUseFcgid}}
        ProxyFCGIBackendType GENERIC
        SetHandler "proxy:fcgi://{{fastcgiWin32}}#"
        {{/unless}}
    </FilesMatch>
    {{else}}

The flag ‘shouldUseFcgid’ is only set for PHPv5 so that’s not an issue.
ProxyFCGIBackendType has two possible values, GENERIC and FPM. I’ve tried both with no change. But that parameter exists because in some cases environment variables can be processed differently.

• Might there be an environment variable, missing or with an unexpected value, something specific to Local, that causes PHP FPM to get blocked before processing a transaction?

• Might there be an option that is provided for us in php.ini.hbs, or not provided, that could affect the number of processes concurrently being served? Maybe a missing extension?

• WordPress itself doesn’t use file-based sessions anymore. Might there be a file lock on a session due to one of the [Session] options in php.ini?

This is not related to security. Note the query above is for http, not https, just to keep it simple and avoid confusion with TLS, certs, etc.

Specific code is not provided for curl_setopt because I don’t think any options work differently for this issue. The problem is not what is in the request. The problem is that the request isn’t being processed by the server until after the client side (prior server process) terminates.

The timing issue is not related to Xdebug. Turn it off and the logging is the same.

This is almost certainly not an issue with .htaccess. Removing .htaccess does not affect this behavior.

This isn’t a DNS issue. /etc/hosts has localhost pointing to 127.0.0.1. The Apache log shows connections for loopserver on fccgi:127.0.0.1:10002. (Again confirming that .htaccess isn’t blocking the loopback request.)

I tried “Listen 81” and connecting to “yoursite.local:81”. The connection worked the same as a loopback on 80.

I don’t know if there is a request header that is missing on a curl request from which is required by this Apache configuration. The solution might be as simple as adding a curl_setopt for loopbacks. However, from the command-line, “curl yoursite.local” works, and “curl yoursite.local/loopclient.php” fails to loopback.

Note that without .htaccess, rewrites, etc, that the problem is not in WordPress.

The issue is not related to a local firewall, which has been disabled for testing. It’s almost certainly not related to AV interference which has also been disabled.

I believe that internally, cURL processes all transactions asynchronously with threads. To force a check on this however, I wrote a test using curl_multi_exec() which processes URIs in parallel. With an array of three sites, site1, yoursite.local, and site3, #1 and #3 successfully returned data and this test instance of local still failed. Again, I really think this issue occurs after a successful connection is made into Apache, and that the issue is not with blocking outbound comms, or routing a socket request through Apache.

My questions

• Can someone at WPEngine verify my notes here?

• Is Apache over Local somehow limited to one thread or process? Or might there be a setting specifically to allow more than one “truly” concurrent transaction per client? Tips for Apache over Windows were not helpful here.

• Can we substitute something else for the PHP FPM extension? And might that help?

• Are all of the components that we’re running already “thread safe”? The pattern here indicates something is blocking the execution of multiple threads.

• Might this be specific to PHP v7 and not v8?

Ref the code above for site.conf.hbs. That led me to wonder how the template values are set, and that led to ApacheService.js, where we see this comment (the only comment in the entire file) among other details: “We can only support connecting to one CGI process on Windows right now with Apache due to issues with mod_proxy_balancer and mod_proxy_fcgi”.

• I understand that’s talking about CGI processes, not individual CGI processes that are multi-threaded. But does that confirm recognition of this or a related issue?

Conclusion

Questions about this problem (in this forum and elsewhere) are unresolved and eventually get locked. OK, we don’t have solutions. For now let’s just try to define the source problem. Then, later, we can work on getting options on the table to solve the problem.

Thanks!

[afragen]

Awesome work @TonyGman !!

I don’t have this issue, on Mac and using nginx options, but this is great info.