Reading large strings from database without fragmenting large object heap

Question

I have data database containing some rather large strings, each of which holds a serialized hierarchical data collection (data is stored as strings rather than as a binary stream to allow interaction with VB6). From what I can tell, any database query that returns a string over 85,000 bytes will immediately throw that string onto the Large Object Heap. If the strings are immediately going to be split into smaller pieces, and thus the large objects will be short-lived, is there any way to avoid having those objects go on the Large Object Heap and remain there uselessly until the next LOH collection? I keep reading that LOH objects should be reused, but I have no idea how I would go about doing that in this context.

EDIT--I'm using the SqlClient object with a DataReader.

score 2 · Accepted Answer · answered Jun 05 '11 at 16:33

2

You can't reuse strings, because they are immutable.

What you can do is either stream the data from the database (example for ADO), which means you can avoid LOH completely.

Or you can load your column into a char array (or a byte array), which can be reused, as long as it is big enough.

This all depends on what database provider or ORM are you using, though.

answered Jun 05 '11 at 16:33

svick

236,525
50
385
514

I'm using the SqlClient object. – supercat Jun 06 '11 at 02:06
Then the article I linked directly applies to you. – svick Jun 06 '11 at 02:10
Indeed, it looks like exactly what I'm after. Thanks. – supercat Jun 06 '11 at 03:13

score 1 · Answer 2 · answered Jun 05 '11 at 17:09

1

Whereas it's true that LOH fragmentation can be a problem, it's not guaranteed to be a problem. I have long-running programs that allocates tens of millions of short-lived large objects over a 24-hour period, and it's never had a problem with LOH heap fragmentation.

It's something that you should be aware of, but it's apparently not as common a problem as some might have you believe. I suggest that you let the program run as it is unless you see some indication that the LOH is an issue.

answered Jun 05 '11 at 17:09

Jim Mischel

131,090
20
188
351

At minimum, having large items go into the LOH would imply that they won't be collected until a level-2 collection, would it not? My program (a specialized type of server) seems to be bogging down when multiple clients connect at once, and excess L2 heap allocation would seem like it might be something that could cause poor performance. – supercat Jun 06 '11 at 02:10
@supercat: yes, it *might* be something that *could* cause poor performance. My experience has been that it's better (faster and more effective) to go ahead and code it up the simple way. Then if there is a performance problem, look for an alternative. But your mileage may vary. – Jim Mischel Jun 06 '11 at 15:02

Reading large strings from database without fragmenting large object heap

2 Answers2