Varnish not processing ESI includes

Question

I'm trying to setup Varnish to process ESI includes on a local environment.

I am running varnish in a virtual machine and the content is running on the host machine.

I have two files "index.html" and "test.html". These are both stored in a folder called "esi" in the docroot of an apache server.

index.html

<h1>It Works!</h1>
<esi:include src="test.html" />

test.html

<p>ESI HAS BEEN INCLUDED</p>

Varnish is running on the virtual machine on port 8000. So I access it here: http://192.168.56.101:8000/esi/

in /etc/varnish/default.vcl on the virtual machine I have added the followin config to the bottom of the file:

sub vcl_fetch {
   set beresp.do_esi = true; /* Do ESI processing               */
   set beresp.ttl = 24 h;    /* Sets the TTL on the HTML above  */
}

With the idea that it should process ESI on ALL requests (Dont care if its bad practice just trying to get this thing to work :))

The result when I load http://192.168.56.101:8000/esi/ is:

<h1>It Works!</h1>
<esi:include src="test.html" />

ie. the ESI is shown in the markup, it is not being processed.

I have checked the Varnish log, however there are no errors in there and nothing related to ESIs.

Can anyone see what I am doing wrong here? Let me know if more information is needed.. thanks

Are you using Varnish > 3.0 ? Because you're using the new syntax... — ivy, Apr 10 '12 at 16:57

score 2 · Answer 1 · answered Aug 21 '14 at 08:49

For ESI works (varnish 3.x), the first char must be a "<" so simply add HTML structure

Here my test :

index.php

<html>
<head>
    <title></title>
</head>
<body>
<?php

    $now = new \DateTime('now');
    echo "hello world from index.php ".$now->format('Y-m-d H:i:s');
?>

<br/>

<esi:include src="/date.php"/>

<br/>

<esi:remove>
    ESI NOT AVAILABLE
</esi:remove>

<br/>

<!--esi
ESI AVAILABLE !!

-->
</body>
</html>

date.php

<?php
$now = new \DateTime('now');
echo "hello world from date.php ".$now->format('Y-m-d H:i:s');

Output :

hello world from index.php 2014-08-21 10:45:29
hello world from date.php 2014-08-21 10:46:35

score 1 · Accepted Answer · edited Jun 09 '12 at 09:45

If your esi include src is "test.html" then varnish will be sending that request to your default backend, which is 127.0.0.1. I believe you need to configure a second backend for your remote server. Something like this:

backend default {
    .host = "127.0.0.1";
    .port = "8000";
}

backend hostmachine {
    .host = "50.18.104.129"; # Enter your IP address here
    .port = "80";
}

Then in your sub vcl_recv you need to redirect traffic that has /esi/ in the URL to the remote server.

sub vcl_recv {
      if (req.url ~ "^/esi/") {
            set req.backend = hostmachine;
            set req.http.host = "www.correctdomainname.com";
      } else {
            set req.backend = default;
      }
}

I'm working on the same thing right now so give it a try and let me know if it works for you.

How is this or the other answer an answer? OP (and me to) wants to include something from the same server and even though `sub vcl_fetch` contains `set beresp.do_esi = true;` varnish does not bother to make the include request to the backend and just dumps the esi tag in the response. This shows up high in google but I fail to see how this or the other answer is actually an answer. — HMR, Jun 24 '14 at 06:20

score 1 · Answer 3 · answered Apr 25 '14 at 19:26

Varnish only implemented a small subset of ESI. As of 2.1 three ESI statements:

    esi:include
    esi:remove
    <!--esi ...-->

Content substitution based on variables and cookies is not implemented but is on the roadmap. Varnish will not process ESI instructions in HTML comments. For ESI to work you need to activate ESI processing in VCL, like this:

sub vcl_fetch {
if (req.url == "/index.html") {
   set beresp.do_esi = true; /* Do ESI processing               */
   set beresp.ttl = 24 h;    /* Sets the TTL on the HTML above  */
} elseif (req.url == "/test.html") {
   set beresp.ttl = 1m;      /* Sets a one minute TTL on        */
                             /*  the included object            */
}

}

score 0 · Answer 4 · answered Jan 04 '19 at 10:20

Make sure your config is read by varnish. If you are running varnish in a docker container you might have to rebuild it.

You can change your config to something ridiculous and see if it catches it. For instance change the backend to w3.org.

If this still gives you the same result, you config isn't used.

Varnish not processing ESI includes

4 Answers4

Linked