I have a database that contains 250,000 records. I am using a DataReader to loop the records and export to a file. Just looping the records with a DataReader and no WHERE conditions is taking approx 22 minutes. I am only selecting two columns (the id and a nvarchar(max) column with about 1000 characters in it).
Does 22 minutes sound correct for SQL Server Express? Would the 1GB of RAM or 1CPU have an impact on this?
22 minutes sounds way too long for a single basic (non-aggregating) SELECT against 250K records (even 22 seconds sounds awfully long for that to me).
To say why, it would help if you could post some code and your schema definition. Do you have any triggers configured?
With 1K characters in each record (2KB), 250K records (500MB) should fit within SQL Express' 1GB limit, so memory shouldn't be an issue for that query alone.
Possible causes of the performance problems you're seeing include:
Update: I did a quick test. On my machine, reading 250K 2KB rows with a SqlDataReader takes under 1 second.
First, create test table with 256K rows (this only took about 30 seconds):
CREATE TABLE dbo.data (num int PRIMARY KEY, val nvarchar(max))
GO
DECLARE @txt nvarchar(max)
SET @txt = N'put 1000 characters here....'
INSERT dbo.data VALUES (1, @txt);
GO
INSERT dbo.data
SELECT num + (SELECT COUNT(*) FROM dbo.data), val FROM dbo.data
GO 18
Test web page to read data and display the statistics:
using System;
using System.Collections;
using System.Data.SqlClient;
using System.Text;
public partial class pages_default
{
protected override void OnLoad(EventArgs e)
{
base.OnLoad(e);
using (SqlConnection conn = new SqlConnection(DAL.ConnectionString))
{
using (SqlCommand cmd = new SqlCommand("SELECT num, val FROM dbo.data", conn))
{
conn.Open();
conn.StatisticsEnabled = true;
using (SqlDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
{
}
}
StringBuilder result = new StringBuilder();
IDictionary stats = conn.RetrieveStatistics();
foreach (string key in stats.Keys)
{
result.Append(key);
result.Append(" = ");
result.Append(stats[key]);
result.Append("<br/>");
}
this.info.Text = result.ToString();
}
}
}
}
Results (ExecutionTime in milliseconds):
IduRows = 0
Prepares = 0
PreparedExecs = 0
ConnectionTime = 930
SelectCount = 1
Transactions = 0
BytesSent = 88
NetworkServerTime = 0
SumResultSets = 1
BuffersReceived = 66324
BytesReceived = 530586745
UnpreparedExecs = 1
ServerRoundtrips = 1
IduCount = 0
BuffersSent = 1
ExecutionTime = 893
SelectRows = 262144
CursorOpens = 0
I repeated the test with SQL Enterprise and SQL Express, with similar results.
Capturing the "val" element from each row increased ExecutionTime to 4093 ms (string val = (string)reader["val"];). Using DataTable.Load(reader) took about 4600 ms.
Running the same query in SSMS took about 8 seconds to capture all 256K rows.
Your results from running exec sp_spaceused myTable provide a potential hint:
rows = 255,000
reserved = 1994320 KB
data = 1911088 KB
index_size = 82752 KB
unused 480KB
The important thing to note here is reserved = 1994320 KB meaning your table is some 1866 MB, when reading fields that are not indexed (since NVARCHAR(MAX) can not be indexed) SQL Server must read the entire row into memory before restricting the columns. Hence you're easily running past the 1GB RAM limit.
As a simple test delete the last (or first) 150k rows and try the query again see what performance you get.
A few questions:
id field or something else)?`nvarchar(max) field?In the best scenario for you your PK is id and also a clustered index and you either have no order by or you are order by id:
Assuming your varchar(max) field is named comments:
SELECT id, comments
FROM myTable
ORDER BY id
This will work fine but it will require you to read all the rows into memory (but it will only do one parse over the table), since comments is VARCHAR(MAX) and cannot be indexed and table is 2GB SQL will then have to load the table into memory in parts.
Likely what is happening is you have something like this:
SELECT id, comments
FROM myTable
ORDER BY comment_date
Where comment_date is an additional field that is not indexed. The behaviour in this case would be that SQL would be unable to actually sort the rows all in memory and it will end up having to page the table in and out of memory several times likely causing the problem you are seeing.
A simple solution in this case is to add an index to comment_date.
But suppose that is not possible as you only have read access to the database, then another solution is make a local temp table of the data you want using the following:
DECLARE @T TABLE
(
id BIGINT,
comments NVARCHAR(MAX),
comment_date date
)
INSERT INTO @T SELECT id, comments, comment_date FROM myTable
SELECT id, comments
FROM @T
ORDER BY comment_date
If this doesn't help then additional information is required, can you please post your actual query along with your entire table definition and what the index is.
Beyond all of this run the following after you restore backups to rebuild indexes and statistics, you could just be suffering from corrupted statistics (which happens when you backup a fragmented database and then restore it to a new instance):
EXEC [sp_MSforeachtable] @command1="RAISERROR('UPDATE STATISTICS(''?'') ...',10,1) WITH NOWAIT UPDATE STATISTICS ? "
EXEC [sp_MSforeachtable] @command1="RAISERROR('DBCC DBREINDEX(''?'') ...',10,1) WITH NOWAIT DBCC DBREINDEX('?')"
EXEC [sp_MSforeachtable] @command1="RAISERROR('UPDATE STATISTICS(''?'') ...',10,1) WITH NOWAIT UPDATE STATISTICS ? "
