Tokyo Tyrant ulog / update log management

Question

I'm testing Tokyo Tyrant in a master-master setup and have found the ulog grows out of control and locks up the disk.

At first I found the -ulim option useful and limited the logfile size, however it simply rolls over to a new log, leaving the old ones to clutter up the partition.

I suppose I'll write a shell script that will delete ulogs older than X, once I find out how far back Tokyo Tyrant needs in the update log in order to failover.

Does anyone have any experience with this Tokyo Tyrant? Do you have a feel (acknowledging that every install is different based on what is being stored) for the optimal ulog size vs how far back a Tokyo Tyrant instance needs to look in the ulog to assume master status?

Thanks, nathan

Answer 1

FYI, I've written a ulog management script that takes the replication delay into account:

http://conigliaro.org/2010/04/28/tokyo-tyrant-update-log-ulog-management/

Answer 2

Just to follow up, below is from Mikio Hirabayashi's (TT developer) reply to a similarly worded e-mail:

If you can confirm only one slave, which is another part of dual master, accesses the master server, please query the delay time to the slave by the command "tcrmgr inform -st ..." and you can determine which file can be removed.

Running that command will allow you to see how far back a slave is behind a master. Once you know that you can spend some time finding the right ulog turnover size and haw many ulog files back you can trash and feel safe. Probably best to do it under a load that simulates a heavy day on your Tokyo Tyrant key/value databases etc..

I shamelessly ripped off a script from stackoverflow:

> #!/bin/bash
> 
> # Deletes all but the newest 5 files to keep Tokyo Tyrant ulogs from
> killing the disk.
> logdir='/path/to/ttserver/ulog/' 
> mydir=`ls -t $logdir` it=1
> 
> for file in $mydir
>     do
>         if [ $it -gt 5 ]
>         then
>             echo file $it will be deleted: $logdir$file
>             #rm -rf $file
>         fi
>         it=$((it+1))
>     done

@kubanskamac's answer was correct in abstract, but Mikio gives the command to start optimization.

Answer 3

Disclaimer: this is the first time I've even heard about Tokyo Tyrant. I just see some familiar patterns looking at the docs.

In the transactional systems (e.g. databases) I know, attention is paid to two types of unexpected events:

• power failure - when you lose for some reason contents of RAM cache, but after restart you are still able to access disk files
• disk failure - you are no longer able to access data from your disk files; files are lost or contain only garbage; you have to restore from backup

Each log passes usually through three stages of existence:

1. Active log that has been just created with very fresh transactions; the data it contains have not yet been saved to data files; you absolutely need this to restore consistency of data files in case of power failure
2. After data are committed (written to database files) the log is now called committed log; data are on disk in two different formats/places - in this log and in database file; you do not need this log in case of power failure, but you need it in case of disk failure - after restoring database file (from previous week?), you can re-apply all transactions from consecutive commited+active logs in the same chronological sequence to arrive at fresh copy of data; if you have lost latest logs, you've lost recent transactions, but at least you still have pretty fresh and completely consistent database
3. After the database file is backed up (consistently copied to safe place), your log becomes archive log. You do not need it to protect against either failure.

I have no idea how to figure out which of your ulogs are committed by Tokyo Tyrant. But maybe this general outline will help.