Panic during sql data warehouse bulkcopy

Question

I'm writing the data to Azure SQL Datawarehouse using the go-mssql driver.

I'm getting a panic thrown at random (at least I haven't been able to reliably replicate this issue) when using the bulkcopy functionality to write some data.

The error is

panic: runtime error: slice bounds out of range

goroutine 56 [running]:
github.com/denisenkom/go-mssqldb.(*tdsBuffer).Write(0xc420614800, 0xc420547d40, 0x2, 0x8, 0x0, 0x0, 0x0)
        /Users/thihara/workspace/golang/src/favmed.unfuddle.com/src/github.com/denisenkom/go-mssqldb/buf.go:93 +0x194
encoding/binary.Write(0xd2f320, 0xc420614800, 0xd3b500, 0xda4e70, 0x8ea680, 0xa2cfb4, 0x8, 0x0)
        /usr/local/go/src/encoding/binary/binary.go:354 +0x188
github.com/denisenkom/go-mssqldb.(*Bulk).Done(0xc420506600, 0xc420691550, 0xda4e70, 0x10)
        /Users/thihara/workspace/golang/src/favmed.unfuddle.com/src/github.com/denisenkom/go-mssqldb/bulkcopy.go:219 +0xae
github.com/denisenkom/go-mssqldb.(*copyin).Exec(0xc4204a6bc0, 0xda4e70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/thihara/workspace/golang/src/favmed.unfuddle.com/src/github.com/denisenkom/go-mssqldb/bulkcopy_sql.go:73 +0x1cb
database/sql.ctxDriverStmtExec(0xd38d60, 0xc42001a070, 0xd38fe0, 0xc4204a6bc0, 0xda4e70, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/database/sql/ctxutil.go:78 +0x176
database/sql.resultFromStatement(0xd38d60, 0xc42001a070, 0xd36720, 0xc4205e95f0, 0xc42099c580, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/database/sql/sql.go:2109 +0x184
database/sql.(*Stmt).ExecContext(0xc4202f87e0, 0xd38d60, 0xc42001a070, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/database/sql/sql.go:2085 +0x223
database/sql.(*Stmt).Exec(0xc4202f87e0, 0x0, 0x0, 0x0, 0xd352e0, 0xc420421180, 0x0, 0x0)
        /usr/local/go/src/database/sql/sql.go:2097 +0x65

Does anyone have any idea what's going on?

I thought perhaps this might be related to concurrency, but has ruled it out by running just a single go routine (initially it was 3 go routines).

For some reason this seem to be occurring for some batch inserts and not all.

It's not related to the batch size either. I had batches of 1500 rows go through while a batch of 1161 rows failed with this panic.

EDIT:

I've worked around this by batching my inserts manually into 1000s.

However this is not a very practical solution since the throughput will become very limited.

The code :

func (w Writer) Write(schema string, data []Datum) error {
    txn, err := w.connection.Begin()
    if err != nil {
        return err
    }

    fullTableName := fmt.Sprintf("[%s].[%s]", schema, TABLE_NAME)

    //Create a prepared statement, BulkOptions can be configured to handle the insert limits if necessary.
    stmt, err := txn.Prepare(mssql.CopyIn(fullTableName, mssql.BulkOptions{RowsPerBatch: 100000}, "unit_did", "chassis_did",
        "gateway_did", "location_code", "data_type", "value", "last_detected_timestamp", "sample_timestamp", "db_timestamp"))
    if err != nil {
        return err
    }
    defer stmt.Close()

    // Prepare the values to be inserted.
    for _, value := range data {
        _, err = stmt.Exec(value.UnitDID, value.ChassisDID, value.GatewayDID, value.LocationCode, value.DataType,
            value.Value, value.LastDetectedTimestamp, value.SampleTimestamp, value.DbTimestamp)

        // Fail the transaction if an error is encountered.
        if err != nil {
            txn.Rollback()
            return err
        }
    }

    // Execute the query against the database and fail the transaction if there's an error.
    _, err = stmt.Exec()
    if err != nil {
        txn.Rollback()
        return err
    }

    return txn.Commit()
}

A github issue has been raised for this with a test show casing the issue.

https://github.com/denisenkom/go-mssqldb/issues/370

Have you run the program with the race detector? Just because you're not using goroutines, doesn't mean there aren't any. Otherwise you need to provide an [mcve]. — JimB, May 03 '18 at 16:39
@JimB Yes, -race was used. Didn't detect anything. This is all the information I have. — Thihara, May 03 '18 at 17:01

score 1 · Answer 1 · answered May 03 '18 at 20:57

The panic provides useful information:

panic: runtime error: slice bounds out of range

and

goroutine 56 [running]:
github.com/denisenkom/go-mssqldb.(*tdsBuffer).Write(0xc420614800, 0xc420547d40, 0x2, 0x8, 0x0, 0x0, 0x0)
/Users/thihara/workspace/golang/src/favmed.unfuddle.com/src/github.com/denisenkom/go-mssqldb/buf.go:93 +0x194

So, the panicking line may well be here:

https://github.com/denisenkom/go-mssqldb/blob/master/buf.go#L93

There might be an off-by-one bug in looping through the buffer, causing the out of range error.

The project has been updated recently. I would just submit an issue on the project.

Panic during sql data warehouse bulkcopy

1 Answers1