Problem with Rails, Webrick or PostgreSQL? Malformed HTML generated after DB query

Question

I am running Rails 3 on Ubuntu with PostgreSQL 8.4 and WEBrick.

It has been working fine, but when displaying a large table (hundreds of rows), something is going wrong. The browser shows that the page is still loading, but it looks like the result of the query is getting truncated or Rails is breaking. There are no errors in the log or console.

The 8th line below is malformed HTML. It occurs around the 140th row of the table. But it continues to display more rows after the ones below.

<tr class="from_hc"> 
  <td class="date_and_time">Jul 13, 2011 11:00 AM</td> 
  <td class="name">Kim Orange</td> 
  <td>PHYSICAL_ACTIVITY</td> 
  <td>text text</td> 
  <td>ok</td> 
  <td></td> 
  <td><a href="/messag/a></td>
</tr>
<tr class="from_hc"> 
  <td class="date_and_time">Jul 13, 2011 11:00 AM</td> 
  <td class="name">Tom White</td> 
  <td>PHYSICAL_ACTIVITY</td> 
  <td>text text</td> 
  <td>ok</td> 
  <td></td> 
  <td><a href="/messages/260/edit">Edit</a></td> 
</tr>

Is it a problem with the number of rows returned? Is there a config that needs to be set in Rails, PostgreSQL or WEBrick? Could it be an issue with SSL? What could cause this?

The same code and table works fine on Heroku, so I wonder if it is a WEBrick config issue...

I increased the "shared_memory" for Postgres, but it did not help.

Thanks!!!

Answer 1

You are most probably hitting response time time outs or memory/server resource limits.

You should never really return huge amounts of data in any app never mind a web app as the data becomes totally meaningless. If this is for a view then use pagination so only a small amount of data is processed at one time (People are just not capable of making sense of that number of records)

If you need this for some statistical analysis then run a background task (rake task maybe?) to gather the data, analyse it and output a simple single record with all the stats calculated

If this is some kind of feed or export function then again a background job in a rake task that will produce a final xml document file stored as an asset that can be served up by your app at an appropriate time is the best solution

for reasons other than avoiding having a user sitting and staring at a monitor for ages waiting you are also in danger of tying up your server and response times for other visitors will drop dramatically.

This is REALLY bad practice. It costs money and time. Both you (bandwidth and server resources), your visitors have to wait longer (Time is money) and you will get a larger hosting bill if on a cloud pay per mb host so I really urge you to change your approach.

As a rule of thumb, do what google do and serve up 10 results per page (How many surfers ever get past page 10 on a search result?)

Answer 2

Have you tried changing out web servers and using mongrel or thin? You should also try changing versions of Ruby and Rails.

I'd say it is completely possible you're hitting a bug in some part of the stack, and you need to rule that out as much as possible.

Try opening up a console and running @model.to_param on the affected record. Does it look different than other records? Does it have special characters?

Answer 3

It turns out that there was non UTF-8 (UTF-16?) characters in a few rows of the database. Specifically, those smart quotes, apostrophe and ellipsis that Microsoft Word creates. A user had cut and pasted from Word and into a web form.

I wonder if there is a way to force UTF-8 in PostgreSQL to protect against this in the future...