Cloudflare won't cache page via CURL request

Question

One.php makes two separate CURL requests to two.php. I want Cloudflare to cache the first response, and to serve that cache in the second response. But for some reason, Cloudflare is not caching either response. If I then visit two.php using my browser, it's still not cached. If I then refresh two.php in my browser, I finally receive a cache HIT. How do I trigger caching on Cloudflare via CURL requests?

Bare Bones Test:

./one.php

header("Cache-Control: no-store");
$url = "http://www.example.com/two.php";

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);

sleep(1);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);

./two.php

header( 'Cache-Control: public, max-age=10' );
print( time()."<br>\r\n" );

Extended Test:

I tried setting the User-Agent header and using Cloudflare's cfduid cookie. I tried issuing the CURL requests on and between two different websites on two different servers. I tried HTTP and HTTPS. I tried running one.php from direct.example.com (which bypasses cloudflare).

./one.php

header("Cache-Control: no-store");

$url = "http://www.example.com/target.php";
$request_headers = array(
  'Connection: keep-alive',
  'Upgrade-Insecure-Requests: 1',
  'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
  'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
  'Accept-Encoding: compressed',
  'Accept-Language: en-US,en;q=0.9',
);

// Fetch CFDUID Cookie
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
$response_headers = curl_exec($ch);
curl_close($ch);

// Extract Cookies
preg_match_all( '/^Set-Cookie:\s*(.+?);?\s*$/mi', $response_headers, $cookies);
$cookies = implode( ';', $cookies[1] );

// Setup First request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );

// Print First Response
echo( "First Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );

// Pause
sleep(1);

// Setup Second request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );

// Record CURL Request Info
curl_setopt($ch, CURLOPT_VERBOSE, true);
$verbose = fopen('php://temp', 'w+');
curl_setopt($ch, CURLOPT_STDERR, $verbose);

// Print Second Response
echo( "Second Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );

// Show CURL Request Info
rewind($verbose);
$verboseLog = stream_get_contents($verbose);
echo "CURL Verbose Info:<br>\r\n<pre>", htmlspecialchars($verboseLog), "</pre>\r\n";

./two.php

header("Cache-Control: public, max-age=10");
header( "set-cookie: test1=true; test2=true" );
echo( time() . "\r\n" );
var_dump( getallheaders() );

Extended Test Output (Censored):

First Response:
1536190748
array(25) {
  ["Content-Type"]=>
  string(0) ""
  ["Content-Length"]=>
  string(1) "0"
  ["X-Original-Url"]=>
  string(20) "/target.php"
  ["Was-Default-Hostname"]=>
  string(33) "*****.azurewebsites.net"
  ["X-Site-Deployment-Id"]=>
  string(15) "*****"
  ["Disguised-Host"]=>
  string(21) "www.*****.com"
  ["X-Arr-Log-Id"]=>
  string(36) "c157*****e003"
  ["Is-Service-Tunneled"]=>
  string(1) "0"
  ["Client-Ip"]=>
  string(21) "*.*.*.175:18968"
  ["X-Waws-Unencoded-Url"]=>
  string(20) "/target.php"
  ["Cf-Connecting-Ip"]=>
  string(12) "*.*.*.33"
  ["Upgrade-Insecure-Requests"]=>
  string(1) "1"
  ["Cf-Visitor"]=>
  string(17) "{"scheme":"http"}"
  ["X-Forwarded-Proto"]=>
  string(4) "http"
  ["Cf-Ray"]=>
  string(20) "455c*****-ATL"
  ["X-Forwarded-For"]=>
  string(35) "*.*.*.33, *.*.*.175:18968"
  ["Cf-Ipcountry"]=>
  string(2) "US"
  ["User-Agent"]=>
  string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
  ["Max-Forwards"]=>
  string(2) "10"
  ["Host"]=>
  string(21) "www.*****.com"
  ["Cookie"]=>
  string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
  ["Accept-Language"]=>
  string(14) "en-US,en;q=0.9"
  ["Accept-Encoding"]=>
  string(4) "gzip"
  ["Accept"]=>
  string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
  ["Connection"]=>
  string(10) "Keep-Alive"
}

Second Response:
1536190749
array(25) {
  ["Content-Type"]=>
  string(0) ""
  ["Content-Length"]=>
  string(1) "0"
  ["X-Original-Url"]=>
  string(20) "/target.php"
  ["Was-Default-Hostname"]=>
  string(33) "*****.azurewebsites.net"
  ["X-Site-Deployment-Id"]=>
  string(15) "*****"
  ["Disguised-Host"]=>
  string(21) "www.*****.com"
  ["X-Arr-Log-Id"]=>
  string(36) "588f*****3a02"
  ["Is-Service-Tunneled"]=>
  string(1) "0"
  ["Client-Ip"]=>
  string(21) "*.*.*.175:19820"
  ["X-Waws-Unencoded-Url"]=>
  string(20) "/target.php"
  ["Cf-Connecting-Ip"]=>
  string(12) "*.*.*.33"
  ["Upgrade-Insecure-Requests"]=>
  string(1) "1"
  ["Cf-Visitor"]=>
  string(17) "{"scheme":"http"}"
  ["X-Forwarded-Proto"]=>
  string(4) "http"
  ["Cf-Ray"]=>
  string(20) "455c8*****-ATL"
  ["X-Forwarded-For"]=>
  string(35) "*.*.*.33, *.*.*.175:19820"
  ["Cf-Ipcountry"]=>
  string(2) "US"
  ["User-Agent"]=>
  string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
  ["Max-Forwards"]=>
  string(2) "10"
  ["Host"]=>
  string(21) "www.*****.com"
  ["Cookie"]=>
  string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
  ["Accept-Language"]=>
  string(14) "en-US,en;q=0.9"
  ["Accept-Encoding"]=>
  string(4) "gzip"
  ["Accept"]=>
  string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
  ["Connection"]=>
  string(10) "Keep-Alive"
}

CURL Verbose Info:
* Hostname www.*****.com was found in DNS cache
*   Trying *.*.*.46...
* TCP_NODELAY set
* Connected to www.*****.com (*.*.*.46) port 80 (#0)
> GET /target.php HTTP/1.1
Host: www.*****.com
Cookie: __cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: compressed
Accept-Language: en-US,en;q=0.9

< HTTP/1.1 200 OK
< Date: Wed, 05 Sep 2018 23:39:10 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: public, max-age=10
< Vary: Accept-Encoding
< Set-Cookie: test1=true; test2=true
< X-Powered-By: PHP/5.6.37
< X-Powered-By: ASP.NET
< CF-Cache-Status: MISS
< Server: cloudflare
< CF-RAY: 455c*****-ATL
< 
* Connection #0 to host www.*****.com left intact

@illiterate did you figure this out in the end? I am having the same issue — Ryan NZ, Jun 02 '19 at 08:11
sorry wrong username :) I'm going to try this via puppeteer (google chrome headless extension), I feel like it should work — Ryan NZ, Jun 02 '19 at 14:09

score 2 · Answer 1 · answered Sep 03 '18 at 14:57

2

Requests to CloudFlare are geographically distributed. Chances are your server is hitting a different CloudFlare edge location than you are (there are 152 right now), so you don't benefit from its calls.

answered Sep 03 '18 at 14:57

ceejayoz

176,543
40
303
368

Good idea, but I don't believe that's the problem in this case. See my amended test above, which returns the same results. When uncommented, *one.php* echos *both* referrers. – skibulk Sep 03 '18 at 15:38
Just noting that my previous comment is obsolete. I have revised the example to demonstrate that two requests from the same server are not triggering the cache. I also started a bounty. – skibulk Sep 05 '18 at 23:49

Aefits · Answer 2 · 2018-09-07T09:51:47.617

Have a look at this github repo cloudflare-bypass. His code has a cache on/off parameter. Which I think might be helpful for you. See blow his code snippet.

<?php
require __DIR__ . '/../vendor/autoload.php';
use CloudflareBypass\RequestMethod\CFCurl;
$curl_cf_wrapper = new CFCurl(array(
    'cache'         => true,   // Caching now enabled by default; stores clearance tokens in Cache folder
    'max_retries'   => 5       // Max attempts to try and get CF clearance
));
// Get Example: 1
$ch = curl_init("http://thebot.net/");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36');
echo $curl_cf_wrapper->exec($ch); // Done! NOTE: HEAD requests not supported!
curl_close($ch);

That setting just saves the clearance token. It does not instruct Cloudflare to cache the request. https://github.com/KyranRana/cloudflare-bypass/issues/66#issuecomment-419484676 — skibulk, Sep 07 '18 at 16:10

score 1 · Accepted Answer · answered Jan 17 '20 at 02:34

Well, it's 2020 and it seems that the "bare bones" example in my question is now working. I'm guessing Cloudflare changed something on their end. One note: My "extended" example should not work because Cloudflare does not cache pages that set custom cookies! (test1=true in this case.)

Cloudflare won't cache page via CURL request

3 Answers3