23

We've a list of 3000 301 redirects. We need assistance on

  1. What would the best place to put these? It seems putting these 3000 lines inside vhost in httpd.conf would be a mess.
  2. What are recommended ways to handle thousands of urls?
  3. How much is it going to affect page loading speed and apache server load ?

Thanks.

Blackbam
  • 113
  • 7
ucker
  • 245
  • 2
  • 5

2 Answers2

32

You can use Include directive in httpd.conf to be able to maintain redirects in another file. But it would not be very efficient, as every request would need to be checked against a lot of regular expressions. Also a server restart would be required after every change in the file.

A better way for so many redirects would be to use RewriteMap directive of type dbm to declare a map from URI's to redirects. This way it will be efficient, as dbm lookups are very fast, and after a change in the map you would not need to restart a server, as httpd checks for map file modification time.

A rewrite rules would look like this (tested on my Fedora 16 computer):

RewriteEngine On
RewriteMap redirects dbm=db:/etc/httpd/conf/redirects.db
RewriteCond ${redirects:$1} !=""
RewriteRule ^(.*)$ ${redirects:$1} [redirect=permanent,last]

And dbm map would be created from text map /etc/httpd/conf/redirects.txt looking like this:

/foo http://serverfault.com/
/bar/lorem/ipsum/ http://stackoverflow.com/

using a command 

httxt2dbm -f db -i /etc/httpd/conf/redirects.txt -o /etc/httpd/conf/redirects.db
Tometzky
  • 2,679
  • 4
  • 26
  • 32
  • Thanks for the answer. I tried it but getting errors: my one line redirects.txt file content (space separated values) http://www.mydomain.com/experiment http://www.mydomain.com/new_page It gave me parse error on line 'RewriteCond ${redirects:$1} != ""' . Syntax error on line 314 of /usr/local/apache/conf/httpd.conf: RewriteCond: bad flag delimiters I commented that out and then apache started but visiting http://www.mydomain.com/experiment (or any other url from mydomain) gave me error "request is being redirected in a way it could not be completed". – ucker Aug 05 '12 at 07:17
  • Apache is not re-starting with above snippet. Could you please help? I'm not able to see any error related to this in any of the log files. – ucker Aug 05 '12 at 12:42
  • I said it was untested - I expected that it would need some tweaking. I thought it would be enough, sorry. Httpd did not start as there can not be a space between `!=` and `""` - I've corrected my example to a working and tested version now. – Tometzky Aug 05 '12 at 21:25
  • One more comment: you'd need to use a different approach if your redirects need to depend on query strings (everything after `?` in URL), but [it is also possible](http://statichtml.com/2010/mod-rewrite-baseon-on-query-string.html). As it is now it would just add query string to redirect, for example `http://yourserver.com/foo?q=bar` would redirect to `http://serverfault.com/?q=bar`. – Tometzky Aug 05 '12 at 21:41
  • Thanks for the update. `RewriteMap redirects dbm=db:/etc/httpd/conf/redirects.db` gave error `RewriteMap dbm type db is invalid` . But when I used `RewriteMap redirects txt:/etc/httpd/conf/redirects.txt` it worked. I still want to go with hashed way like you explained instead of plain text. But I could not find much about the dbm error. Thanks for your help. – ucker Aug 06 '12 at 09:13
  • `httxt2dbm` command with no arguments will show you available dbm formats. In my system it was SBDM and DB. Make sure that `-f` option passed for `httxt2dbm` while generating a map (for example `-f db`) is the same as `RewriteMap` map type (for example `dbm=db:`). – Tometzky Aug 06 '12 at 12:29
  • In the end sdbm option worked beautifully. – ucker Aug 15 '12 at 03:47
  • Wow it works well, perfect ! (just used it with Apache Apache/2.4.7 (Ubuntu)) – COil Sep 29 '16 at 09:50
2

I'd write a script that generates a new VHOST with just the links from the list.

Lucas Kauffman
  • 16,880
  • 9
  • 58
  • 93