Converting XLS to PDF: Getting "Data index must be a valid index in the field" exception during GetBytes()

Question

I have an application that converts files into PDF. It first saves a blob from MySQL into a temp file and then it converts that temp file into a PDF. I'm getting this "Data index must be a valid index in the field" exception error at the GetBytes() only when I try and convert an XLS file. Other file types (BMP, XLSX, DOC, DOCX, etc.) all convert.

private WriteBlobToTempFileResult WriteBlobToTempFile(int id, string fileType)
{
    Logger.Log(string.Format("Inside WriteBlobToTempFile() id: {0} fileType: {1}", id, fileType));
    WriteBlobToTempFileResult res = new WriteBlobToTempFileResult //return object
    {
        PrimaryKey = id
    }; 
    FileStream fs;                          // Writes the BLOB to a file 
    BinaryWriter bw;                        // Streams the BLOB to the FileStream object.
    int bufferSize = 100;                   // Size of the BLOB buffer.
    byte[] outbyte = new byte[bufferSize];  // The BLOB byte[] buffer to be filled by GetBytes.
    long retval;                            // The bytes returned from GetBytes.
    long startIndex = 0;                    // The starting position in the BLOB output.                        
    string connectionString = ConfigurationManager.AppSettings["MySQLConnectionString"]; //connection string from app.config
    string path = ConfigurationManager.AppSettings["fileDirectory"]; //get directory from App.Config
    
    try
    {
        MySqlConnection conn = new MySqlConnection(connectionString);
        conn.Open();
        //Determine records to convert, retrieve Primary Key and file type
        string sql = "SELECT FILE_DATA from " + TableName + " WHERE PK_TSP_DOCS_ID = @id";
        MySqlCommand cmd = new MySqlCommand(sql, conn);
        cmd.Parameters.AddWithValue("@id", id);
        MySqlDataReader rdr = cmd.ExecuteReader(CommandBehavior.SequentialAccess);                                                
        while (rdr.Read())
        {   
            // Create a file to hold the output.
            fs = new FileStream(path + @"\" + id + "." + fileType, FileMode.OpenOrCreate, FileAccess.Write);
            bw = new BinaryWriter(fs);

            // Reset the starting byte for the new BLOB.
            startIndex = 0;

            // Read the bytes into outbyte[] and retain the number of bytes returned.
            retval = rdr.GetBytes(rdr.GetOrdinal("FILE_DATA"), startIndex, outbyte, 0, bufferSize);

            // Continue reading and writing while there are bytes beyond the size of the buffer.
            while (retval == bufferSize)
            {
                bw.Write(outbyte);
                bw.Flush();

                // Reposition the start index to the end of the last buffer and fill the buffer.
                startIndex += bufferSize;
                // *****IT FAILS AT THE LINE BELOW*****
                retval = rdr.GetBytes(rdr.GetOrdinal("FILE_DATA"), startIndex, outbyte, 0, bufferSize);
                // *****IT FAILS AT THE LINE ABOVE*****
            }
            // Write the remaining buffer.
            bw.Write(outbyte, 0, (int)retval);
            bw.Flush();

            // Close the output file.
            bw.Close();
            fs.Close();
        }
        // Close the reader and the connection.
        rdr.Close();
        conn.Close();
        res.FullPath = path + @"\" + id + "." + fileType;
    }
    catch (Exception ex)
    {
        res.Error = true;
        res.ErrorMessage = string.Format("Failed to write temporary file for record id: {0} of file type: {1}", id.ToString(), fileType);
        res.InternalErrorMessage = ex.Message; //string.Format("Caught Exception in WriteBlobToTempPDF(). Stack Trace: {0}", ex.StackTrace);
    }                                    
    return res;
}

I just wonder if the problem is that `Lenght == N * bufferSize` ... then `N * bufferSize` start index is not valid ... — Selvin, Oct 02 '20 at 14:53
use `var length = rdr.GetBytes(index, 0, null, 0,0);` to get length ... then add `startIndex < length` in while — Selvin, Oct 02 '20 at 15:01
off-topic comment: buffer size should be bigger ... use 4KB as 100 B is really small — Selvin, Oct 02 '20 at 15:05
@Selvin Changing the buffer size from 100 to 4000 did the trick. Do I still need to add the code above? — E. Choi, Oct 02 '20 at 15:11
*Changing the buffer size from 100 to 4000 did the trick.* No ... it is not enough ... now you will have problem when `file length % 4000 == 0` use code from 2nd comment ... *Do I still need to add the code above?* **YES** — Selvin, Oct 02 '20 at 15:12
@Selvin Where do I put `var length = rdr.GetBytes(index, 0, null, 0,0);` and `startIndex < length` exactly? — E. Choi, Oct 02 '20 at 15:16
`var length...` before `while` ... and change while to `while(retval == bufferSize && startIndex < length)` I'm not sure .. but think that even `while(startIndex < length)` should do the thing ... yeah also you may add `var index = rdr.GetOrdinal("FILE_DATA");` and use it instead calling `rdr.GetOrdinal("FILE_DATA")` multiple times — Selvin, Oct 02 '20 at 15:18

score 0 · Answer 1 · answered Oct 02 '20 at 18:26

This is a bug in Oracle's MySQL Connector/NET. You can see at https://mysql-net.github.io/AdoNetResults/#GetBytes_reads_nothing_at_end_of_buffer that MySql.Data throws an IndexOutOfRangeException when trying to read 0 bytes at the end of the buffer, but no other ADO.NET provider does.

You should switch to MySqlConnector (disclaimer: I'm a contributor) and use the MySqlDataReader.GetStream() and Stream.CopyTo method to simplify your code:

MySqlDataReader rdr = cmd.ExecuteReader(CommandBehavior.SequentialAccess);                                                
while (rdr.Read())
{   
    // Create a file to hold the output.
    using (var fs = new FileStream(path + @"\" + id + "." + fileType, FileMode.OpenOrCreate, FileAccess.Write))
    {
        // Open a Stream from the data reader
        using (var stream = rdr.GetStream(rdr.GetOrdinal("FILE_DATA"))
        {
            // Copy the data
            stream.CopyTo(fs);
        }
    }
}

Converting XLS to PDF: Getting "Data index must be a valid index in the field" exception during GetBytes()

1 Answers1