0

Intro -- the scenario

I am relatively new to linux server configuration, and with Microsoft Azure cloud services.

I have a VM sep up with a stack consisting of Nginx, Varnish, PHP7-FPM, MariaDB, FAIL2BAN, on top of Ubuntu 16.04.

Wordpress Multisite is installed. As far as I can tell, everything is configured correctly in the Nginx and Varnish .conf files, and in the Wordpress installation.

Port 80 and 443 are both open. The primary domaind and the networked site domains all resolve to the IP of the VM.

All of this was deployed using debops-wordpress (an extension of the Ansible project).

The issue

The issue is that the primary site works fine, but all networked sites result in an ERR_EMPTY_RESPONSE error. As far as I can determine, no logs are generated by those requests. For instance, varnishlog will report all the expected varnish activity (lots) when I request the primary domain. It generates no activity at all when I request a networked site domain. I conclude from this that the request never even makes it through to varnish.

Log files

When I make an http request for a networked site (a Wordpress multisite), a basic entry is written to the Varnish access log. (see below). But there is no entry in the Nginx log. And no entries in the Varnish or Nginx error logs.

::ffff:122.57.253.24 - - [17/Jul/2017:01:58:35 +0000] "GET /favicon.ico HTTP/1.1" 204 0 "http://www.sewalunafoundation.nz/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"

So the domain request is at least getting through to the stack, but it aborts with an empty response error.

I was thinking that perhaps I'd overlooked something in the Azure side of things, that is preventing requests to any of the networked site domains from actually reaching the server stack. Although I am not sure quite how, since the necessary endpoints (80 and 443) are open, and working fine for the primary domain, and now that I see the secondary (WP networked) domains do reach varnish at least enough for it to create a log entry of the fact, this has me thinking it's unlikely network related.

Configuration

Varnish

server {

        listen [::]:80 default_server ipv6only=off;
        server_name www.primarydomain.nz;
        server_name primarydomain.nz;
        server_name www.2nd-domain.org.uk;
        server_name 2nd-domain.org.uk;
        server_name www.3rd-domain.org.za;
        server_name 3rd-domain.org.za;
        server_name 4th-domain.ca;
        server_name www.4th-domain.ca;

        include snippets/acme-challenge.conf;

        keepalive_timeout 60;

        access_log /var/log/nginx/varnish.www.primarydomain.nz_access.log;
        error_log /var/log/nginx/varnish.www.primarydomain.nz_error.log;
        client_max_body_size  32M;

        if ($host != $server_name) {
            return 444;
       }

        location = /nginx_status {
                stub_status on;
                access_log on;
                allow 127.0.0.1/32;
                allow ::1/128;
                allow 10.0.0.8;
                deny all;
        }

        include /etc/nginx/sites-default.d/*.conf;

        location / {
                proxy_set_header Host              $host;
                proxy_set_header X-Real-IP         $remote_addr;
                proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Forwarded-Port  80;
                proxy_pass http://varnish;
        }
}

Nginx

server {

        listen 8080 default_server ipv6only=off;

        server_name www.primarydomain.nz;
            server_name primarydomain.nz;
            server_name www.2nd-domain.org.uk;
            server_name 2nd-domain.org.uk;
            server_name www.3rd-domain.org.za;
            server_name 3rd-domain.org.za;
            server_name 4th-domain.ca;
            server_name www.4th-domain.ca;

        root /var/www/www.primarydomain.nz;

        keepalive_timeout 60;

        access_log /var/log/nginx/www.primarydomain.nz_access.log;
        error_log /var/log/nginx/www.primarydomain.nz_error.log;
        index index.html index.htm index.php;

        client_max_body_size  32M;
        gzip_comp_level 5;
        gzip_min_length 256;
        gzip_proxied any;
        gzip_vary on;
        gzip_types application/atom+xml
          application/javascript
          application/json
          application/ld+json
          application/manifest+json
          application/rdf+xml
          application/rss+xml
          application/schema+json
          application/vnd.geo+json
          application/vnd.ms-fontobject
          application/x-font-ttf
          application/x-javascript
          application/x-web-app-manifest+json
          application/xhtml+xml
          application/xml
          font/eot
          font/opentype
          image/bmp
          image/svg+xml
          image/vnd.microsoft.icon
          image/x-icon
          text/cache-manifest
          text/css
          text/javascript
          text/plain
          text/vcard
          text/vnd.rim.location.xloc
          text/vtt
          text/x-component
          text/x-cross-domain-policy
          text/xml;

        rewrite /wp-admin$ $scheme://$host$uri/ permanent;

        # Disallow access to hidden files and directories
        location ~ /\. {
                return 404;
        }

        location = /favicon.ico {
                try_files /favicon.ico =204;
                access_log off;
                log_not_found off;
        }

        location = /nginx_status {
                stub_status on;
                access_log off;
                allow 127.0.0.1/32;
                allow ::1/128;
                allow 10.0.0.8;
                deny all;
        }

        include /etc/nginx/sites-default.d/*.conf;

        location ~ /(\.|wp-config.php|readme.html|license.txt|wp-cli.local.yml|wp-cli.yml) {
                return 404;
        }

        location ~* ^.+\.(css|js|ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|rss|atom|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {
                access_log off;
                expires max;
                add_header Cache-Control "public";
        }

        location = /robots.txt {
                access_log off;
                log_not_found off;
                try_files /robots.txt /index.php;
        }

        location / {
                try_files $uri $uri/ /index.php?$args;
        }

        location ~ ^(?!.+\.php/)(?<script_name>.+\.php)$ {
                limit_except GET HEAD POST { deny all; }

                try_files $script_name =404;

                include fastcgi.conf;

                set $php_https off;

                if ($http_x_forwarded_proto = "https") {
                  set $php_https on;
                }

                fastcgi_param HTTPS $php_https;
                fastcgi_next_upstream error timeout invalid_header http_500;

                # Mitigate HTTPOXY attacks (https://httpoxy.org/)
                fastcgi_param HTTP_PROXY "";

                fastcgi_index index.php;
                fastcgi_pass php-wordpress;
        }

        location ~ ^(?<script_name>.+\.php)(?<path_info>/.*)$ {
                limit_except GET HEAD POST { deny all; }

                try_files $script_name =404;

                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root$script_name;
                fastcgi_param PATH_INFO $path_info;
                #fastcgi_param PATH_TRANSLATED $document_root$path_info;

                set $php_https off;

                if ($http_x_forwarded_proto = "https") {
                  set $php_https on;
                }

                fastcgi_param HTTPS $php_https;
                fastcgi_next_upstream error timeout invalid_header http_500;

                # Mitigate HTTPOXY attacks (https://httpoxy.org/)
                fastcgi_param HTTP_PROXY "";

                fastcgi_index index.php;
                fastcgi_pass php-wordpress;
        }
}

Content of /etc/default/varnish

(all comment lines removed)

START=yes
NFILES=131072
MEMLOCK=82000
DAEMON_OPTS="-a :6081 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256m"

Content of /etc/varnish/default.vcl

(all comment lines removed)

vcl 4.0;
import std;
import directors;
backend wordpress {
  .host = "127.0.0.1";
  .port = "8080";
  .max_connections = 300;
  .first_byte_timeout     = 300s;
  .connect_timeout        = 5s;
  .between_bytes_timeout  = 2s;
}
acl purge {
  "10.0.0.8";
  "127.0.0.1";
  "::1";
}
sub vcl_init {
  new vdir = directors.round_robin();
  vdir.add_backend(wordpress);
}
sub vcl_recv {
  set req.backend_hint = vdir.backend(); # send all traffic to the vdir director
  set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
  unset req.http.proxy;
  if (req.method == "PURGE") {
    if (!std.ip(req.http.X-Real-IP, "0.0.0.0") ~ purge) {
      return (synth(405, "This IP is not allowed to send PURGE requests."));
    }
    return (purge);
  }
  if (req.method != "GET" &&
      req.method != "HEAD" &&
      req.method != "PUT" &&
      req.method != "POST" &&
      req.method != "TRACE" &&
      req.method != "OPTIONS" &&
      req.method != "PATCH" &&
      req.method != "DELETE") {
    /* Non-RFC2616 or CONNECT which is weird. */
    return (pipe);
  }
  if (req.http.Upgrade ~ "(?i)websocket") {
    return (pipe);
  }
  if (req.method != "GET" && req.method != "HEAD") {
    return (pass);
  }
  if (req.url ~ "(\?|&)(utm_(campaign|medium|source|term)|gclid|cx|ie|cof|siteurl)=") {
    set req.url = regsuball(req.url, "&(utm_(campaign|medium|source|term)|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
    set req.url = regsuball(req.url, "\?(utm_(campaign|medium|source|term)|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
    set req.url = regsub(req.url, "\?&", "?");
    set req.url = regsub(req.url, "\?$", "");
  }
  if (req.url ~ "\#") {
    set req.url = regsub(req.url, "\#.*$", "");
  }
  if (req.url ~ "\?$") {
    set req.url = regsub(req.url, "\?$", "");
  }
  set req.http.Cookie = regsuball(req.http.Cookie, "wordpress-settings-[0-9]+?=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "wordpress-settings-time-[0-9]+?=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");
  if (req.http.cookie ~ "^\s*$") {
    unset req.http.cookie;
  }
  if (req.http.Cache-Control ~ "(?i)no-cache") {
    if (! (req.http.Via || req.http.User-Agent ~ "(?i)bot" || req.http.X-Purge)) {
      return(purge); # Couple this with restart in vcl_purge and X-Purge header to avoid loops
    }
  }
  if (req.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|rar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") {
    unset req.http.Cookie;
    return (hash);
  }
  if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
    unset req.http.Cookie;
    return (hash);
  }
  if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
    return (pass);
  }
  if (req.url ~ "wp-(admin|cron|login)" || req.url ~ "preview=true" || req.url ~ "xmlrpc.php") {
    return (pass);
  }
  set req.http.Surrogate-Capability = "key=ESI/1.0";
  if (req.http.Authorization) {
    return (pass);
  }
  return (hash);
}
sub vcl_pipe {
  if (req.http.upgrade) {
    set bereq.http.upgrade = req.http.upgrade;
  }
  return (pipe);
}
sub vcl_pass {
}
sub vcl_hash {
  hash_data(req.url);
  if (req.http.host) {
    hash_data(req.http.host);
  } else {
    hash_data(server.ip);
  }
  if (req.http.Cookie) {
    hash_data(req.http.Cookie);
  }
}
sub vcl_hit {
  if (obj.ttl >= 0s) {
    return (deliver);
  }
  if (!std.healthy(req.backend_hint) && (obj.ttl + obj.grace > 0s)) {
    return (deliver);
  } else {
    return (fetch);
  }
  if (std.healthy(req.backend_hint)) {
    if (obj.ttl + 10s > 0s) {
      return (deliver);
    } else {
      return(fetch);
    }
  } else {
      if (obj.ttl + obj.grace > 0s) {
      return (deliver);
    } else {
      return (fetch);
    }
  }
  return (fetch); # Dead code, keep as a safeguard
}
sub vcl_miss {
  return (fetch);
}
sub vcl_backend_response {
  if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
    unset beresp.http.Surrogate-Control;
    set beresp.do_esi = true;
  }
  if (bereq.url ~ "^[^?]*\.(bmp|bz2|css|doc|eot|flv|gif|gz|ico|jpeg|jpg|js|less|mp[34]|pdf|png|rar|rtf|swf|tar|tgz|txt|wav|woff|xml|zip|webm)(\?.*)?$") {
    unset beresp.http.set-cookie;
  }
  if (bereq.url ~ "^[^?]*\.(mp[34]|rar|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
    unset beresp.http.set-cookie;
    set beresp.do_stream = true;  # Check memory usage it'll grow in fetch_chunksize blocks (128k by default) if the backend doesn't send a Content-Length header, so only enable it for big objects
    set beresp.do_gzip = false;   # Don't try to compress it for storage
  }
  if (beresp.status == 301 || beresp.status == 302) {
    set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+", "");
  }
  if (beresp.http.Set-Cookie && bereq.url !~ "wp-(login|admin)") {
    unset beresp.http.Set-Cookie;
  }
  if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") {
    set beresp.ttl = 120s;
    set beresp.uncacheable = true; # Flags the response as hit-for-pass
    return (deliver);
  }
  if (beresp.status == 500 || beresp.status == 502 || beresp.status == 503 || beresp.status == 504) {
    return (abandon);
  }
  set beresp.grace = 6h;
  return (deliver);
}
sub vcl_deliver {
  if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed
    set resp.http.X-Cache = "HIT";
  } else {
    set resp.http.X-Cache = "MISS";
  }
  set resp.http.X-Cache-Hits = obj.hits;
  unset resp.http.X-Powered-By;
  unset resp.http.Server;
  unset resp.http.X-Drupal-Cache;
  unset resp.http.X-Varnish;
  unset resp.http.Via;
  unset resp.http.Link;
  unset resp.http.X-Generator;
  unset resp.http.X-Clacks-Overhead;
  return (deliver);
}
sub vcl_purge {
  if (req.method != "PURGE") {
    set req.http.X-Purge = "Yes";
    return(restart);
  }
}
sub vcl_synth {
  if (resp.status == 720) {
    set resp.http.Location = resp.reason;
    set resp.status = 301;
    return (deliver);
  } elseif (resp.status == 721) {
    set resp.http.Location = resp.reason;
    set resp.status = 302;
    return (deliver);
  }
  return (deliver);
}
sub vcl_fini {
  return (ok);
}

Result of netstat -ant|grep 80

$ netstat -ant|grep 80
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN
tcp        0      0 10.0.0.8:58972          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58912          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58888          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:44180          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:58948          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58984          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58828          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58998          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58840          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58900          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58924          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58960          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:59014          168.63.129.16:80        TIME_WAIT
tcp        1      0 10.0.0.8:58972          169.254.169.254:80      CLOSE_WAIT
tcp        0      0 10.0.0.8:58872          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58936          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58816          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58852          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:54180          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:58804          168.63.129.16:80        TIME_WAIT
tcp        0      0 10.0.0.8:58874          168.63.129.16:80        TIME_WAIT
tcp6       0      0 :::80                   :::*                    LISTEN

Result of netstat -ant|grep 443

tcp        0      0 10.0.0.8:54514          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54622          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54626          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44560          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54480          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:50576          13.66.176.26:443        TIME_WAIT
tcp        0      0 10.0.0.8:44576          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54660          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44528          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54504          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44636          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54502          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44478          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:50572          13.66.176.26:443        TIME_WAIT
tcp        0      0 10.0.0.8:44468          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54466          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44610          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54482          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54578          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44466          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:44672          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54646          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54454          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54662          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:50570          13.66.176.26:443        TIME_WAIT
tcp        0      0 10.0.0.8:54456          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54506          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44624          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:44598          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54636          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54492          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44526          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:50666          13.66.176.26:443        TIME_WAIT
tcp        0      0 10.0.0.8:50682          13.66.176.26:443        TIME_WAIT
tcp        0      0 10.0.0.8:54648          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44660          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:44492          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54548          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54470          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54650          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44538          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54658          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54610          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44540          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:44658          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54638          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44586          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54562          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54458          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44514          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54574          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44490          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:44562          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54634          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54600          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44504          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54614          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:50660          13.66.176.26:443        TIME_WAIT
tcp        0      0 10.0.0.8:54526          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54612          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54550          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:50662          13.66.176.26:443        TIME_WAIT
tcp        0      0 10.0.0.8:54490          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54598          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54590          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54602          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54552          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44480          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54468          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54518          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54588          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54528          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44502          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54576          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44622          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:44646          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54564          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44588          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:44612          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54530          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54624          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54478          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:54516          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44600          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:44670          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:44648          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54566          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:50668          13.66.176.26:443        TIME_WAIT
tcp        0      0 10.0.0.8:44574          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54494          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44516          52.239.148.164:443      TIME_WAIT
tcp        0      0 10.0.0.8:54586          13.77.184.72:443        TIME_WAIT
tcp        0      0 10.0.0.8:44634          52.239.148.164:443      TIME_WAIT

Public IP

The public IP of the server is 52.183.115.144

Questions

My questions are:

  1. How can I go about troubleshooting this issue to find the cause?
  2. Does anything come to mind about steps one must take for additional domains to access a VM on Azure?
  3. Do the above details give any indication as to what the cause of this issue might be?
inspirednz
  • 173
  • 1
  • 9
  • You have not posted your Varnish config (VCL). Can you please provide results for ```telnet example.com 80``` (for network websites) – Danila Vershinin Jul 17 '17 at 05:33
  • What is the result `netstat -ant|grep 80` `netstat -ant|grep 443`. You also could test port 80 and 443 with `curl`. – Shui shengbao Jul 17 '17 at 06:45
  • @DanielV. I've now included two varnish config files. I assume these are what you are referring to. Pls let me know if they are not, and if so where I they are located on Ubuntu environ. – inspirednz Jul 17 '17 at 09:25
  • @Walter-MSFT: Netstat results posted. So, I didn't mention in my post that curl gives the same result locally as I get from requesting the domain publicly. Namely, no response. `curl: (52) Empty reply from server` – inspirednz Jul 17 '17 at 09:30
  • @inspirednz Your 443 port is not listening. Do you check it? – Shui shengbao Jul 17 '17 at 09:32
  • Just to clarify what you are expecting to happen here.... You want all web traffic to hit varnish, which then passes it off to nginx – Drifter104 Jul 17 '17 at 10:00
  • @Walter-MSFT Honestly, I've not even given a moment of attention to post 443, other than setting up the endpoint for it in Azure. At this stage, my focus is on figuring out why all domains except the primary domain get no response from the server (on port 80). Once that's fixed, if there's something that needs to be done to have port 443 listened to, please advise what that is. – inspirednz Jul 17 '17 at 10:01
  • @Drifter104 Yes, as far as I know that's the way this is set up. When I request the primary domain that seems to be how it successfully works. When I request any other domain (on the multisite network) nothing at all happens, except one line in the varnish access log, as mentioned. – inspirednz Jul 17 '17 at 10:03
  • Ok where are you testing it from? Something on the same network/machine. Also.... Are you doing port forwarding in the azure firewall? 80---> 80 or 80 --> 6081 – Drifter104 Jul 17 '17 at 10:39
  • My primary test is from the public (i.e. my computer, over the Internet), since that's what it ultimately must be accessible to. I have also tested it from the server itself, and from another VM on the same resource group in Azure. Perhaps it's my ignorance, but I am having trouble understanding the purpose of your questions. For example, if the primary domain works fine, does that not suggest the ports are already open and fine, on the Azure side? E.g.2: If the primary domain works fine, naturally I am testing the other domains in the same way (over the Internet). Or am I missing something? – inspirednz Jul 17 '17 at 10:55
  • Never assume is basically the answer to the why behind some of my questions.... For example your varnish config is set to listen on 6081 but you talk about opening port 80, so I was trying to establish if you were port forwarding 80 to 6081 in the azure firewall. There is also an IPv6 question, if you weren't port forwarding 80 to 6081 then port 80 would be hiting nginx not varnish. Finally nginx had (has?) a strange ipv6 bug with multi domains... See here https://serverfault.com/questions/638367/do-you-need-separate-ipv4-and-ipv6-listen-directives-in-nginx – Drifter104 Jul 17 '17 at 11:07
  • Why I questioned the reason for such questions is that the site functions perfectly with the primary domain. That suggests to me that the port configuration of Varnish, etc., must be okay in this particular configuration, even if the reasons are not clear based on the config data provided. Therefore I have no port forwarding set up on Azure, and I question if that would be necessary, in light of the fact it already works on one domain. – inspirednz Jul 17 '17 at 11:47
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/62348/discussion-between-drifter104-and-inspirednz). – Drifter104 Jul 17 '17 at 11:51
  • Looks like a found a way to get this to work. Thank you @Drifter104 for your help in getting me to look in the right areas, and letting me know the right checks and tests to figure out the issue. – inspirednz Jul 19 '17 at 22:04
  • Thank you @Walter-MSFT for your help on this, and for going to the trouble of setting up a varnish server to test some things at your end. – inspirednz Jul 19 '17 at 22:05

1 Answers1

1

After many days on this, I've managed to figure out the solution.

This should be of use to anyone else using debops-wordpres to deploy Wordpress Multisite, with custom domains. I can't say whether or not the cloud environment makes any difference (Azure, as opposed to other options). The developer seemed to think the issues I faced were because he's not tested and tweaked it for Azure. But I don't really see how it makes any difference. An Ubuntu 16.04 VM is the same, where ever it might be hosted (as far as I know).

I've communicated this solution to the developer of the debops-wordpres project and he tells me that in effect what I have done is this:

it seems all you did is go to varnish directly vs through the nginx vhost.

For whatever reason, going to varnish directly was necessary in order to get the multi-side custom domains operational.

Essentially, what was required was to edit all configuration files that had any effect on the ports listened to by Nginx and Varnish. I took the following steps:

  1. Remove the second Nginx .conf file. Namely, /etc/nginx/sites-enabled/varnish.www.sewalunafoundation.nz.conf. You are just removing a symlink; the actual file remains in the /sites-available/ folder
  2. Edit /etc/nginx/sites-enabled/default so that the line listen [::]:80 default_server; is changed to listen 8080;
  3. Edit /etc/systemd/system/varnish.service. On the ExecStart setting, change -a *:6081 to -a *:80. The end result of that setting is:

ExecStart=/usr/sbin/varnishd -j unix,user=vcache -P /run/varnishd.pid -F -a *:80 -T localhost:6082 -t 120 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,1715.0

  1. Edit /etc/default/varnish, changing DAEMON_OPTS="-a :6081 \ to DAEMON_OPTS="-a :80 \

After making those changes, run the following commands:

sudo systemctl daemon-reload sudo service nginx restart sudo service varnish restart sudo service varnish status (check Varnish has loaded okay) sudo netstat -tulnp (check the right ports are being listened to)

The last command should produce the following:

Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1842/mysqld tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 1475/redis-server l tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1325/varnishd tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 1517/nginx -g daemo tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1412/sshd tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 2293/master tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 1325/varnishd tcp6 0 0 :::80 :::* LISTEN 1325/varnishd tcp6 0 0 :::22 :::* LISTEN 1412/sshd tcp6 0 0 :::25 :::* LISTEN 2293/master udp 0 0 0.0.0.0:68 0.0.0.0:* 1006/dhclient

You'll see Varnish is listening to tcp 80, and tcpv6 80. It's also listening to 6082. Nginx is listening to tcp 8080.

Your site should now load on the primary domain and on the multisite domains. You may need to restart the Azure VM. I did, but I can't say if it was necessary to get things work (after making the above changes).

SSL

I have been informed by the developer of debops-wordpress that SSL will not work under this modified configuration. That's unfortunate, and I am going to look into whether there's any way to resolve that. Even if it means dropping Varnish out for all SSL requests, I would do it, but just need to figure out how.

inspirednz
  • 173
  • 1
  • 9
  • I want to thank Walter-MSFT and drifter104 for their persistence is helping me look in the right areas whilst figuring out a solution for this. – inspirednz Jul 19 '17 at 22:03