1

Why is the Plone data.fs file so large, will it effect performance?

on our current site the file is over 8GB. I recently took over dev of this project on a legacy Plone 3 installation and there are huge performance bottle necks on the database.

My first impression is 8GB is huge...?

  • There are no large files stored.
  • Performance is likely the database as:
    1. There is a zeo cluster,
    2. Load balanced with apache
    3. 8 core vCPU server with 16GB ram
    4. Any cached page (apache layer) is lightning, others are incredibly slow

UPDATE

After the suggestions below and further investigation. Here are some stats from the server:

  • Memory Usage 15792/16045 <<--- ouch?

  • AVG CPU 300-400%, 8 CPUs, 5 x zope instances. Im assuming this is very high? as each instance uses a single CPU thread to handle a request? thats almost an average of 100% cpu usage accross the instances?

  • Disk IO is high at avg 8984.85 blocks/sec

So what does that point to? The apache log files are huge at 7GB... i will install logrotate. But surely those stats: High disk IO... point to DB issues? Will packing alleviate this? Is packing dangerous on such a high volume production site?

AndrewMcLagan
  • 13,459
  • 23
  • 91
  • 158
  • 4
    8GB Data.fs is not huge at all, especialy for a Plone 3 version (that not use plone.app.blob by default) but it depends on lot of information that you didn't provide (please, update your question). For example: how many object do you have in your site? Do you use a lot of large files? How can you say that the performance bottleneck is the database? – keul May 03 '16 at 13:13
  • updated, let me know if the information is still sparse. Just getting my head around Plone. – AndrewMcLagan May 03 '16 at 13:34
  • 1
    @AndrewMcLagan you can install https://pypi.python.org/pypi/Products.LongRequestLogger and see what's the real bottleneck within Zope/Plone – gforcada May 03 '16 at 14:17
  • 1
    Pack the Database, maybe it will shrink. – 1letter May 03 '16 at 14:18
  • 1
    You say that the performance problem is likely the db since you have load balancing for a zeo cluster. But what do your system stats look like? Are you IO-bound? CPU-bound? Into swap? Constantly having client servers killed by the OS because you're out of memory? – SteveM May 03 '16 at 15:23
  • 1
    You really should use sysstat too (maybe 2 minute interval), keep track of average IO wait. If `sar` returns IO waits that are small (less than 100ms, usually), and your IO usage is small, it is not the storage that is slowing you down. This can be a problem on virtual infrastructure (cloud, vmware), for the record, or anything hosted on SAN/NAS. – sdupton May 03 '16 at 17:31
  • My initial thoughts were with the db as its relatively unconventional in comparison to alternate CMS, frameworks and languages. I have sites with many times less resources, reserving hundreds of times more requests. The suggestions to investigate other bottle necks are great. If anyone has any other general suggestions? – AndrewMcLagan May 03 '16 at 22:36
  • updated the question with some stats – AndrewMcLagan May 04 '16 at 00:31
  • 1
    IOWait or blocking is your most important IO measure. If you're not blocking, you haven't really got a problem there. 3-400% CPU usage with 8 CPUs means you're fine there. Gross memory usage isn't a very userful # by itself. Use "free" to see how it's allocated. If you're into swap or getting processes killed, or have constant page swapping, you're memory stressed. – SteveM May 04 '16 at 01:03
  • will do that, also in the zeo logs there are many entries with the following: `Transaction blocked waiting for storage. Clients waiting: 1.` followed by `Blocked transaction restarted.` – AndrewMcLagan May 04 '16 at 01:17
  • when running `vmstat` its showing ALLOT of waiting processes. usually around 2-8. Never 0. and the swap usage is huge.... – AndrewMcLagan May 04 '16 at 01:45
  • 1
    Assuming a modern Linux or BSD system, substantial swap usage is a nightmare scenario. Get your system well-instrumented with something like Munin. Then, either add RAM or start trimming the memory use of your ZEO clients. Make one change at a time and watch the results. – SteveM May 04 '16 at 15:34

1 Answers1

6

It is highly unlikely that the ZODB itself or the size of your database is responsible for the bad performance. I would recommend to pack your database as a first step. If you haven't done that on a regular basis before, packing will drastically decrease the size of the ZODB:

https://plone.org/documentation/faq/how-do-i-pack-the-zodb

tisto
  • 1,337
  • 7
  • 13
  • 1
    Also, packing on nightly cron job is usually good idea: http://docs.plone.org/manage/deploying/packing.html -- Use `zeopack` script for ZEO or `zodbpack` script for RelStorage. My sites use 7 days kept pack history, run every night. – sdupton May 03 '16 at 17:30
  • My initial thoughts were with the db as its relatively unconventional in comparison to alternate CMS, frameworks and languages. I have sites with many times less resources, reserving hundreds of times more requests. The suggestions to investigate other bottle necks are great. If anyone has any other general suggestions? – AndrewMcLagan May 03 '16 at 22:36
  • 1
    Size of the DB may just mean that it hasn't been packed. Or that you're keeping lots of old versions of content items. ZODB performance does not in particular decline with size of Data.fs. – SteveM May 04 '16 at 00:58
  • 1
    This is not related in anyway with the size of the database file: I have similar servers running perfectly well with databases double that size before packing; you need to monitor your server and install some caching; that's it. – hvelarde May 06 '16 at 01:42