0

We have been experiencing a very strange anomaly, with one specific user with our UW-IMAP server. We have about 75 users using the server, and one particular user, who is in about the middle as far as used storage keeps having issues with slow speed.

Most of our users all use Thunderbird 2, or Thunderbird 3. Mostly 2, because of the performance issues we have had with 3. This user was on 3, and I downgraded him to 2. The performance has gotten better, but according to the imapd processes on the server, his username is using the most CPU % and CPU time.

I've already done all the usual T/S'ing: Started profile from scratch, compacted folders, re-indexed, newer faster computer, etc.. Still, this users' imapd process is always using the most CPU on the server. For troubleshooting, we setup another user which has more usage, folders, etc.. than he does, but we don't see the users process taking up most of the CPU with the imapd process.

So, it almost sounds like a particular email may be the culprit, but how can we find it, if thats the problem?

This has been going on for a while, and he is a management person, so his patience is about to end. Does anyone have any ideas?

bgarlock
  • 129
  • 1
  • 8

5 Answers5

3

In the past, I've found uw-imap can be very brutal on system performance if there is a lot of emails in individual mail folders. What is the user's mail count like? What about mailbox sizes? Have you tried moving some of the messages around? Have you tried another imap client to eliminate it being an issue there? Do the users have login access to the mail server? ie, can they alter content in $hOME using ftp, or ssh? Another issue I've spotted (at least a long time ago) was that uw-imap would scan all files in $HOME looking for mail folders.

Jon Angliss
  • 1,782
  • 10
  • 8
  • I agree that in addition to raw size of mail in MB, the number of individual messages in each mail file also plays a part. In other words, a single mail file of 50MB full of 10,000 messages will be slower than another 50MB file containing 2 messages. – jj33 Jan 14 '11 at 15:31
  • All great points. I have split up some of the larger number of email folders (actually, mbox is a single file as you know), but I have not moved them off the server - just to other mailboxes on the server. – bgarlock Jan 14 '11 at 17:07
  • Also, the users don't have SHELL or FTP access to their $HOME. This particular user did have a fetchmail process from a POP3 account, so I turned that off. The fetchmail would run on a different server, with a NFS mount to his home folder, via procmail. I did turn that fetchmail process off today to see if that has anything to do with it. The NFS is a 'soft' mount, which I think is the preferred mount type when an imap server is concerned. – bgarlock Jan 14 '11 at 17:10
1

If you use a different client like "mutt" for testing, the problem may be easier to spot; at least in which phase of the protocol things get stuck (but then I'd assume the Thunderbird also has a debugging switch). I tried finding how to switch debugging output on in wu-imap, but couldn't find it in a hurry.

Volker Stolz
  • 416
  • 2
  • 10
1

Converting to the MIX format can easily decrease the load by 10x or more. It's a little unnerving at first as the format stores mail in folders that appear empty (filenames beginning with a dot are used). There are a few files with meta data and indexing info. These are extremely small and are accessed a lot. The rest is the actual mail which is split up into bite size pieces and accessed only when that content is read. For indexing purposes the system is only looking at very small files and retrieving any particular piece of email requires radically less RAM and disk access time.

Email stores of 10 GB start being as fast as traditional email stores of 10MB and mail size limits are huge.

UW-IMAP can use MIX, MBX and traditional format at the same time (I'd skip MBX, it has a 2GB limit and is not as fast as MIX) so you needn't convert all at once and you can keep the primary mail spool traditional if you like, using MIX in ~/INBOX and still get 99% of the benefit. The mailutil tool can convert mailbox formats at will.

One of the best parts is converting any one user speeds things up significantly for them and a little bit for everyone else until finally EVERYONE is going a LOT faster. This happens suddenly when mail clients stop adding more and more connections to try to speed up. One or two connections become enough and the load radically decreases.

Usage is transparent (except for speedup) to any clients that use imap or pop for access.

Really old mail programs that can only directly access email can't use it but pine users (for instance) can either use imap or switch to alpine, which is a direct descendent of pine.

It is my understanding that at least one other IMAP/POP daemon has something similar but I'm a UW-IMAP person.

I had to spend a lot less money on hardware thanks to this switch.

david
  • 26
  • 1
  • This is a great idea, and one that I considered a while ago. I'm trying to find some good info on the best way to perform this migration, so if you can offer any links that you think would be helpful, that would be great. – bgarlock Jan 29 '11 at 15:04
  • Wow, and I mean WOW! I did convert a few of my larger files, and did some tests, and after the conversion, it hardly registered a blip on the imapd process when I accessed these mail folders. Not to mention, we can slowly migrate people, instead of having to do this in one shot. Thanks so much for this!! – bgarlock Jan 29 '11 at 16:15
  • Thanks again, David. Conversions are going well. About the only thing I have had to hack is some procmailrc recipes so that mail can be delivered to a MIX format mailbox. Procmail doesn't do so well with that, so I use 'dmail' and all is well. – bgarlock Jan 31 '11 at 22:34
  • Just to continue information about the MIX format, I'll add that UW-IMAP was discontinued, and was forked to Panda IMAP: http://panda.com/imap/ – bgarlock Feb 01 '11 at 23:06
0

I don't know much about thunderbird, but I have found that the search indexing done by outlook can be brutal on a UW-IMAP install, is there something similar in thunderbird?

jj33
  • 11,178
  • 1
  • 37
  • 50
  • I took that out of the loop by going back to Thunderbird-3. TB 2 doesn't have indexing. Great point though! – bgarlock Jan 14 '11 at 17:05
0

So, what we ended up doing was downgrading him to Thunderbird 2. The performance is much, much better, and things are very snappy. One thing that we did notice, and I'll have to research this issue now, is that sometimes Thunderbird does not download the latest messages, once it has been running for a while. You have to shut it down, and start it back up. This has been confirmed with several users now, so I have not narrowed it down to a server issue or a Thunderbird issue.

On another note, here is a link to a very nice article on Thunderbird 2 versus Thunderbird 3 performance. Bottom line is that Thunderbird 3 is a pig:

http://perfprotector.blogspot.com/2010/08/thunderbird-3-is-it-really-all-yours.html

bgarlock
  • 129
  • 1
  • 8