1

I'm writing an FDW for a non-SQL data source. Platform is Windows 10, C (MS Visual Studio), Postgresql 14. My FDW code is modeled after FDW example codes I have studied such as SQLite, JSON, CSV, File, DB2 and others. There is a common practice of storing metadata in a pg List as part of GetForeignPlan() and passing that via fdw_private. This list is then retrieved in BeginForeignScan() and made available for IterateForeignScan().

My question is how to share a large amount of metadata across the fdw_private mechanism? Using the pg List macros, I have been unable to store and retrieve more than 5 List cells. I tried passing a single List cell with a JSON string containing all of my metadata, but the string becomes corrupted along the way.

List* mdList = NIL;
char feJSON[MAX_FESTATE_JSON_SIZE];
/// ... some code to format a JSON string into the feJSON buffer.
mdList = list_make1(makeString(feJSON));
return mdList;

I have also used lappend() to try and extend the List by more than 5 cells, but the additional cells' values are not maintained across the callbacks...

#define serializeInt(x) makeConst(INT4OID, -1, InvalidOid, 4, Int32GetDatum((int32)(x)), false, true)

result = list_make5(makeInteger(feState->start), makeInteger(feState->rows), makeString(feState->ltName), makeString(feState->ftName), makeInteger(feState->myTable->npgcols));
result = lappend(result, serializeInt(feState->myTable->ncols));

There is a hint in the pg source plannodes.h suggesting the use of bytea (byte array?) as an alternative to the pg List structure, but I'm not finding any examples for that, so far.

I'm suspecting certain characters in JSON strings may be part of the issue, but I also found that...

#define MAX_FESTATE_JSON_SIZE 2048
List* serializeMetadata(...)
{
  List* mdList = NIL;
  /// ...stuff...
    char smokeTest[MAX_FESTATE_JSON_SIZE + 1];
    memset(smokeTest, 'X', MAX_FESTATE_JSON_SIZE);
    smokeTest[MAX_FESTATE_JSON_SIZE] = '\0';
  mdList = list_make1(makeString(smokeTest));
  return mdList;
}

...revealed some truncation of the List as a return value (but it's a pointer!?). So I'm not sure if casting a bytea* as a List* will help, but that's where I'm headed.

Suggestions are most welcome!

Kenigmatic
  • 448
  • 6
  • 16
  • A list should work fine. You don't describe what `serializeInt` does, but I hope it returns a `datum *`. – Laurenz Albe Oct 28 '22 at 05:21
  • @LaurenzAlbe, I updated the Q showing the definition of `serializeInt()`. I used it in the same way as the DB2 FDW. I'm wondering if the issue is build related since I can ^C ^V existing FDW codes and get broken results. – Kenigmatic Oct 28 '22 at 18:25
  • @LaurenzAlbe, I replaced `serializeInt()` with `makeInteger()` and the subsequent `lappend()` appears to work now. I'm going to experiment further with codes based on something other than the DB2 FDW. – Kenigmatic Oct 28 '22 at 18:41
  • Funny, that's a copy of my code. Both should work; but an `Integer` is different from a `Const`. Perhaps the error is in the code that extracts the values from the list, in that you cast it to something wrong. – Laurenz Albe Oct 29 '22 at 09:14

0 Answers0