Rendering large collections of articles to PDF fails in MediaWiki with mwlib

Question

I have installed the Mediawiki Collection Extension and mwlib to render articles (or collections of articles) to PDF. This works very well for single articles and collections with up to 20 articles.

When I render larger collections, the percentage counter in the parsing page (which counts to a 100% when rendering succeeds) is stuck at 1%.

Looking at the mwrender.log I see an Error 32 - Pipe Broken error. Searching the internet reveals that Error 32 can be caused by the receiving process (the part after the pipe) crashing or not responding.

From here it is hard to proceed. Where should I look for more clues? Could it be the connection to the MySQL server that dies?

The whole applicance is running on a Turnkey Linux Mediawiki VM.

score 0 · Answer 1 · answered Dec 08 '10 at 15:47

0

I'm using PDF Export Extension and it works with more than 20 articles. Maybe try that?

answered Dec 08 '10 at 15:47

Adrian Archer

2,323
1
17
20

I use that too, in conjunction with the Collection extension. Do you also succeed with very large collections (70+)? – Docbuilder Dec 08 '10 at 20:03
I think the Collection Extension actually uses the PDF Writer Extension. – Adrian Archer Dec 08 '10 at 20:36
I was too quick. I do not use PDF Export, I tried it two years ago and found the output quality not good enough and the resulting PDF hard to style. I could try the 2.4 version. – Docbuilder Dec 08 '10 at 20:37
Still, the collection extension is used on Wikipedia. It _should_ be able to handle large collections. – Docbuilder Dec 08 '10 at 20:38
Yea you would think it should work. I don't use the Collection Extension on my wiki – Adrian Archer Dec 08 '10 at 20:51

score 0 · Answer 2 · answered Dec 10 '10 at 08:03

I figured out the problem myself.

Mw-render spawns a parallel request for every article in a collection. This means that for a collection of 50 pages, 50 simultaneous requests are made. Apache could handle this, but not the MySQL db of MediaWiki.

You can limit the amount of threads that mw-render spawns with the --num-threads=NUM option. I couldn't find where mw-serve calls mw-render, so I just limited the maximum amount of threads (workers) Apache could spawn to 10.

mw-render automatically repeats requests for articles if the first ones fail, so this approach worked.

I rendered a PDF with 185 articles within 4 minutes, the resulting PDF had 300+ pages.

Rendering large collections of articles to PDF fails in MediaWiki with mwlib

2 Answers2