Wordpress Content Crawler plugin cannot fetch HTML page

Support
, ,
1

Issue Summary

  • I have used the WordPress Content Crawler plugin, this will open the target page for me to select HTML elements and crawl the content but when open the target page to select HTML element occur the error:
    URL: https://someurl.com/, Message: cURL error 60: SSL certificate problem: unable to get local issuer certificate (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)
  • I turn on SSL Trust

Troubleshooting Questions

  • Does this happen for all sites in Local, or just one in particular?
  • I just testing one particular site
  • Are you able to create a new, plain WordPress site in Local and access it in a Browser?
    Yes

Replication

Describe the steps that others can take to replicate this issue. If you have screenshots that can help clarify what is happening, please include them!

  • Create a new fresh site and install the plugin Wordpress Content Crawler, then go to Add new section and select Category tag, select Category Post URL Selector

System Details

  • Which version of Local is being used?
    Newest

  • What Operating System (OS) and OS version is being used?

    • _For example: win11
2

I haven’t seen that error before – specifically, the unable to get local issuer certificate part.

I’m guessing that more info would be in the logs. Can you provide a zip of the logs folder under the site’s folder? You should see logs next to app.

In addition to those PHP logs, can you please provide your Local Log? See this Community Forum post for instructions on how to do so:

3

Hi, thank you for your reply, here is two logs:

4

There doesn’t appear to be much info in the Local log, but the PHP error log has a number of errors that look like:

[08-Oct-2021 23:22:43 UTC] WPCC (info): Request error (Details: URL: https://www.google.com/, Message: cURL error 60: SSL certificate problem: unable to get local issuer certificate (see https://curl.haxx.se/libcurl/c/libcurl-errors.html))
(Exception: #0 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Handler\CurlFactory.php(155): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array)
#1 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Handler\CurlFactory.php(105): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
#2 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Handler\CurlHandler.php(43): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
#3 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Handler\Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#4 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Handler\Proxy.php(51): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#5 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\PrepareBodyMiddleware.php(37): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#6 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Middleware.php(35): GuzzleHttp\PrepareBodyMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#7 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\RedirectMiddleware.php(54): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#8 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Middleware.php(59): GuzzleHttp\RedirectMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#9 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\HandlerStack.php(71): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#10 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Client.php(351): GuzzleHttp\HandlerStack->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#11 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Client.php(162): GuzzleHttp\Client->transfer(Object(GuzzleHttp\Psr7\Request), Array)
#12 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\guzzlehttp\guzzle\src\Client.php(182): GuzzleHttp\Client->requestAsync('GET', Object(GuzzleHttp\Psr7\Uri), Array)
#13 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\fabpot\goutte\Goutte\Client.php(174): GuzzleHttp\Client->request('GET', 'https://www.goo...', Array)
#14 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\vendor\symfony\browser-kit\Client.php(404): Goutte\Client->doRequest(Object(Symfony\Component\BrowserKit\Request))
#15 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\Objects\Crawling\Bot\AbstractBot.php(548): Symfony\Component\BrowserKit\Client->request('GET', 'https://www.goo...')
#16 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\Objects\Crawling\Bot\AbstractBot.php(393): WPCCrawler\Objects\Crawling\Bot\AbstractBot->getResponseText('GET', 'https://www.goo...', false, 'https')
#17 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\Objects\Crawling\Bot\DummyBot.php(31): WPCCrawler\Objects\Crawling\Bot\AbstractBot->request('https://www.goo...', 'GET', Array)
#18 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\Test\Tests\SourceCodeTest.php(42): WPCCrawler\Objects\Crawling\Bot\DummyBot->request('https://www.goo...', 'GET', Array)
#19 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\Test\Base\AbstractTest.php(110): WPCCrawler\Test\Tests\SourceCodeTest->createResults(Object(WPCCrawler\Test\Data\TestData))
#20 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\Test\Test.php(153): WPCCrawler\Test\Base\AbstractTest->run()
#21 C:\Users\BonSuperman\Local Sites\test\app\public\wp-content\plugins\wp-content-crawler\app\Services\PostService.php(133): WPCCrawler\Test\Test::respondToTestRequest(Array)
#22 C:\Users\BonSuperman\Local Sites\test\app\public\wp-includes\class-wp-hook.php(303): WPCCrawler\Services\PostService->WPCCrawler\Services\{closure}('')
#23 C:\Users\BonSuperman\Local Sites\test\app\public\wp-includes\class-wp-hook.php(327): WP_Hook->apply_filters('', Array)
#24 C:\Users\BonSuperman\Local Sites\test\app\public\wp-includes\plugin.php(470): WP_Hook->do_action(Array)
#25 C:\Users\BonSuperman\Local Sites\test\app\public\wp-admin\admin-ajax.php(187): do_action('wp_ajax_wcc_tes...')
#26 {main})

I’ve never seen this specific error before, but here are some observations:

  • This plugin appears to be using Guzzle which is an HTTP client for PHP. This is significant because it means that the plugin is rolling its own HTTP requests as opposed to using the WordPress HTTP API.
  • Because this plugin isn’t using the WordPress HTTP API, it’s obtaining the SSL certificate from whatever the system’s SSL implementation is using.
  • The questions to answer are:
    • Where is that cert set?
    • How to update it to the latest version?

I don’t have the answers to those last two questions yet since I haven’t really dug that deeply into how Local configures SSL under Windows, but I have it on my list of things to investigate further!

For reference, I think the solution will be similar to this topic: Expired SSL certificate - #9 by ben.turner

However, that topic is focused on Mac, whereas this issue is specific to Windows.

5

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.