1

So I have the following code that is supposed to batch insert values into Postgres:

config const batchsize = 10000;
proc bulkInsertSparseMatrix(con:Connection, aTable: string, fromField: string, toField: string, weightField: string, A) {
  const q = "INSERT INTO %s (%s, %s, %s) VALUES (%s, %s, %s);";
  var cur = con.cursor();
  var count = 0;
  var buffer: batchsize*(string, string, string, string, int, int, real);
  for ij in A.domain {
    buffer[count] = (aTable, fromField, toField, weightField, ij(1), ij(2), A(ij));
    count += 1;
    if count == batchsize {
      cur.execute(q, buffer);
      count = 0;
    }
  }
  cur.execute(q, buffer[0..#count]);
}

So essentially a tuple buffer of user defined length in which I nest the value tuples. The buffer's filling is tracked by the count and the entries are committed periodically with a final partial dump at the end for completeness. Nothing fancy. The reason for this nested tuple approach was that the CDO library's .execute() method is formatted for nested tuples and I roughly follow the suggestion in the answer to this question.

The problem is when running the following test:

config const DB_HOST: string = "localhost";
config const DB_USER: string = "postgres";
config const DB_NAME: string = "testing";
config const DB_PWD: string = "noether";
var con = PgConnectionFactory(host=DB_HOST, user=DB_USER, database=DB_NAME, passwd=DB_PWD);

config const batchsize: int = 3;

var fromField = "fromID",
    toField = "toID",
    wField = "w",
    wTable = "tester";

var X = Matrix(
   [1.0,0.0,1.0,0.0],
   [1.0,0.0,1.0,1.0]);

bulkInsertSparseMatrix(con, aTable=wTable, fromField=fromField, toField=toField, weightField=wField, A=X);

I get the following error:

bulk_insert-test.chpl:36: error: unresolved call 'bulkInsertSparseMatrix(Connection, aTable=string, fromField=string, toField=string, weightField=string, A=[domain(2,int(64),false)] real(64))'

I think it looks like a type error, but I'm not sure where I sinned. Some guidance or pointers to a better way would be appreciated!

Tshimanga
  • 845
  • 6
  • 16
  • 1
    I'm not yet sure what's going on with the 'unresolved call' error, but there are some problems in the bulkInsertSparseMatrix function. The 'buffer' variable is incorrectly declared to be a tuple with size 'batchsize'. Tuples' size must be known at compile-time. You can use 'config param' instead of 'config const'. Also, tuples cannot be sliced, so this is invalid: "buffer[0..#count]". – benharsh Jan 30 '18 at 19:45
  • 1
    What does `writeln(con.type:string);` give you, if inserted in place of the `bulkInsertSparseMatrix()` call? Or alternatively you could insert `compilerWarning(con.type:string);` which would cause the compiler to print the type of `con`. FWIW, I'm not sure this is a great SO question since it's unlikely to be relevant to other readers (as far as I can tell) -- maybe better as a GitHub issue on the Chapel repository? – Brad Jan 31 '18 at 00:50

0 Answers0