force mediawiki squid cache to fill up with all pages

Question

For speeding up a MediaWiki site which has content that uses a lot of templates but otherwise pretty much has static content when the templates have done their jobs I'd like to setup a squid server see

https://www.mediawiki.org/wiki/Manual:PurgeList.php

and

https://www.mediawiki.org/wiki/Manual:Squid_caching

and then fill the squid server's cache "automatically" by using a script doing wget/curl calls that hit all pages of the Mediawiki. My expecation would be that after this procedure every single page is in the squid cache (if I make it big enough) and then each access would be done by squid.

How would i get this working? E.g.:

1. How do I check my configuration?
2. How would I find out how much memory is needed?
3. How could I check that the pages are in the squid3 cache?

What I tried so far

I started out by finding out how to install squid using:

• https://wiki.ubuntuusers.de/squid

and

• https://www.mediawiki.org/wiki/Manual:Squid_caching

I figured out my ip address xx.xxx.xxx.xxx (not disclosed here) via ifconfig eth0

in /etc/squid3/squid.conf I put

http port xx.xxx.xxx.xxx:80 transparent vhost defaultsite=XXXXXX
cache_peer 127.0.0.1 parent 80 3130 originserver 

acl manager proto cache_object
acl localhost src 127.0.0.1/32

# Allow access to the web ports
acl web_ports port 80
http_access allow web_ports

# Allow cachemgr access from localhost only for maintenance purposes
http_access allow manager localhost
http_access deny manager

# Allow cache purge requests from MediaWiki/localhost only
acl purge method PURGE
http_access allow purge localhost
http_access deny purge

# And finally deny all other access to this proxy
http_access deny all

Then I configured my apache2 server

# /etc/apache2/sites-enabled/000-default.conf   
Listen 127.0.0.1:80

I added

$wgUseSquid = true;
$wgSquidServers = array('xx.xxx.xxx.xxx');
$wgSquidServersNoPurge = array('127.0.0.1');

to my LocalSettings.php

Then I restarted apache2 and started squid3 with

service squid3 restart

and did a first access attempt with

wget --cache=off -r http://XXXXXX/mediawiki

the result is:

Resolving XXXXXXX (XXXXXXX)... xx.xxx.xxx.xxx
Connecting to XXXXXXX (XXXXXXX|xx.xxx.xx.xxx|:80... failed: Connection refused.

Answer 1

Assuming Apache 2.x.

While not Squid related, you can achieve this using just Apache modules. Have a look at mod_cache here: https://httpd.apache.org/docs/2.2/mod/mod_cache.html

You can simply add this to your Apache configuration and ask Apache to do disk caching of rendered content.

You need to ensure your content has appropriate cache expiry information in the resulting PHP response, MediaWiki should take care of this for you.

Adding such a cache layer may not have the desired outcome as this layer does not know if a page has changed, cache management is difficult here and should only be used for actually static content.

Ubuntu:

a2enmod cache cache_disk

Apache configuration:

CacheRoot /var/cache/apache2/mod_disk_cache
CacheEnable disk /

I would not recommend pre-filling your cache by accessing every page. This will only cause dormant (not frequently used) pages to take up valuable space / memory. If you still wish to do this, you may look at wget:

Description from: http://www.linuxjournal.com/content/downloading-entire-web-site-wget
$ wget \
     --recursive \
     --no-clobber \
     --page-requisites \
     --html-extension \
     --convert-links \
     --restrict-file-names=windows \
     --domains website.org \
     --no-parent \
         www.website.org/tutorials/html/

This command downloads the Web site www.website.org/tutorials/html/.

The options are:

    --recursive: download the entire Web site.

    --domains website.org: don't follow links outside website.org.

    --no-parent: don't follow links outside the directory tutorials/html/.

    --page-requisites: get all the elements that compose the page (images, CSS and so on).

    --html-extension: save files with the .html extension.

    --convert-links: convert links so that they work locally, off-line.

    --restrict-file-names=windows: modify filenames so that they will work in Windows as well.

    --no-clobber: don't overwrite any existing files (used in case the download is interrupted and
    resumed).

A better option: Memcached

MediaWiki also supports the use of Memcached as a very fast in-memory caching service for data and templates only. This is not as brutal as a website-wide cache like Squid or Apache mod_cache. MediaWiki will manage Memcached so that any changes are immediately reflected in the cache store, meaning your content will always be valid.

Please see the installation instructions at MediaWiki here: https://www.mediawiki.org/wiki/Memcached

My recommendation is not to use Apache mod_cache or Squid for this task, and instead to install Memcached and configure MediaWiki to use it.