libpqxx C Aggregate Extension returns wrong data?

Question

I am learning how to create C aggregate extensions and using libpqxx with C++ on the client side to process the data.

My toy aggregate extension has one argument of type bytea, and the state is also of type bytea. The following is the simplest example of my problem:

Server side:

PG_FUNCTION_INFO_V1( simple_func );

Datum simple_func( PG_FUNCTION_ARGS ){

    bytea *new_state   = (bytea *) palloc( 128 + VARHDRSZ );
    memset(new_state, 0, 128 + VARHDRSZ );
    SET_VARSIZE( new_state,128 + VARHDRSZ );

    PG_RETURN_BYTEA_P( new_state );

}

Client side:

std::basic_string< std::byte > buffer;

pqxx::connection c{"postgresql://user:simplepassword@localhost/contrib_regression"};
pqxx::work w(c);
c.prepare( "simple_func", "SELECT simple_func( $1 )  FROM table" );
pqxx::result r = w.exec_prepared( "simple_func", buffer );

for (auto row: r){ 
        cout << "  Result Size: " << row[ "simple_func" ].size() << endl;
        cout << "Raw Result Data: ";
        for( int jj=0; jj < row[ "simple_func" ].size(); jj++ ) printf( "%02" PRIx8,   (uint8_t) row[ "simple_func" ].c_str()[jj] )  ;
        cout << endl;
}

The result on the client side prints :

Result Size: 258
Raw Result Data: 5c783030303030303030303030303030...

Where the 30 pattern repeats until the end of the string and the printed string in hex is 512 bytes.

I expected to receive an array of length 128 bytes where every byte is set to zero. What am I doing wrong?

The libpqxx version is 7.2 and PostgreSQL 12 on Ubuntu 20.04.

Addendum

Installation of the extesion sql statement;

CREATE OR REPLACE FUNCTION agg_simple_func( state bytea, arg1 bytea)
RETURNS bytea
AS '$libdir/agg_simple_func'
LANGUAGE C IMMUTABLE STRICT;

CREATE OR REPLACE AGGREGATE simple_func( arg1 bytea)  
(
    sfunc = agg_simple_func,
    stype = bytea,
    initcond = "\xFFFF" 
);

score 1 · Accepted Answer · answered Oct 02 '20 at 18:38

1

The answer appears to be that the bytea type data on the client side must be retrieved as follows in the libpqxx library as of 7.0 (Not tested in earlier versions):

row[ "simple_func" ].as<std::basic_string<std::byte>>()

This retrieves the right bytea data without any conversions, string idiosyncrasies or unexpected behavior like I was seeing.

answered Oct 02 '20 at 18:38

ReverseFlowControl

766
6
30

Good that you solved the problem - I could only help with the server side. But you see that it makes sense to examine both sides independently if possible. – Laurenz Albe Oct 05 '20 at 04:43
@LaurenzAlbe Yep. Thanks. – ReverseFlowControl Oct 06 '20 at 04:24

score 0 · Answer 2 · answered Oct 02 '20 at 06:40

0

I recommend that you tackle these things one by one: first get the function to work, testing it with psql in interactive queries, then write the client code (or vice versa).

I can't speak about libpqxx, but I have to complain about your function: what you presented won't even compile, because you wrote DATUM in upper case and forgot headers and other important stuff.

This function will compile and run as you expect:

#include "postgres.h"
#include "fmgr.h"

PG_MODULE_MAGIC;

PG_FUNCTION_INFO_V1(simplest_func);

Datum simplest_func(PG_FUNCTION_ARGS) {
    bytea *new_state = (bytea *) palloc(128 + VARHDRSZ);
    memset(new_state, 0, 128 + VARHDRSZ);
    SET_VARSIZE(new_state, 128 + VARHDRSZ);

    PG_RETURN_BYTEA_P(new_state);
}

The memset will work that way, but the better and more idiomatic and robust way to set the value of a varlena is

    memset(VARDATA(new_state), 0, 128);

I have no idea, how you got your result, but since the code you presented doesn't compile, I don't know how your function really looks.

answered Oct 02 '20 at 06:40

Laurenz Albe

209,280
17
206
263

The capital datum was a typo on my part. I should have included the boiler plate stuff for the full extension, but then there is the psql part, etc...gotta stop somewhere. But, that is my function, I am testing that exact same code. I have removed everything else, reduced it to this example, and tested it, and I get the result I mentioned with libpqxx. I suspect the problem might be with libpqxx, but I don't know what is wrong. As for memset, I wanted to set it all to zero, including the header, so that I know what to expect on the client side. Perhaps, that was unnecessary. – ReverseFlowControl Oct 02 '20 at 06:48
Perhaps you should share the `CREATE FUNCTION` statement. – Laurenz Albe Oct 02 '20 at 06:54
I added the sql statement to creates the aggregate extension. – ReverseFlowControl Oct 02 '20 at 07:01
That would call a different function! It should be `CREATE FUNCTION ... AS '$libdir/agg_simple_func', 'simplest_func';`. – Laurenz Albe Oct 02 '20 at 08:35
No, I am pretty sure I got it right. Those are the only extensions I have defined, and I get no errors calling them. Plus, \da shows my aggregate functions and the one defined shows up. – ReverseFlowControl Oct 02 '20 at 08:39
The documentation says: "`AS obj_file, link_symbol`: The string `obj_file` is the name of the shared library file containing the compiled C function, and is interpreted as for the `LOAD` command. The string `link_symbol` is the function's link symbol, that is, the name of the function in the C language source code. **If the link symbol is omitted, it is assumed to be the same as the name of the SQL function being defined.**". But that's `simple_func` in your case. – Laurenz Albe Oct 02 '20 at 09:58
I fixed a typo, it should be "simple_func" everywhere, not "simplest_func". Also, I imagine that is in case obj_file and link_symbol are different, which is good to know :). In my case, they are both the same. – ReverseFlowControl Oct 02 '20 at 17:40

libpqxx C Aggregate Extension returns wrong data?

2 Answers2