2

I get ERROR: invalid byte sequence for encoding "UTF8": 0xe0 0x20 0x63 when i try to execute sql statement (insert into) of a std::vector into a bytea.

I'm using c++ (11), and pqxx postgresql client, and posgresql 9.6 . I need to write into db a byte stream (vector).

Very often my code works well, but now i faced with the following error:

ERROR: invalid byte sequence for encoding "UTF8": 0xe0 0x20 0x63

From pgAdmin4 i get:

show client_encoding; -> unicode

From c++, i fill this vector from std::stringstream using the following code:

//mOutputStream is std::stringstream correctly filled
mOutputStream.seekg(0, std::ios::end);
int sizeOS = mOutputStream.tellg();
mOutputStream.seekg(0, std::ios::beg);
mOutputStream.unsetf(std::ios::skipws);

std::streampos vecSize;
mOutputStream.seekg(0, std::ios::end);
vecSize = mOutputStream.tellg();
mOutputStream.seekg(0, std::ios::beg);

//std::vector<unsigned char>& arrayChar
arrayChar.reserve(vecSize);

arrayChar.insert(arrayChar.begin(),                  

std::istream_iterator<unsigned char>(mOutputStream),
std::istream_iterator<unsigned char>());

Then, using pqxx library, i create a sql statement:

sql = "Insert into public.\"MyFile\" (id,rawdata) values ($1,$2);";
/* Create a transactional object. */
pqxx::work W(C);
C.prepare("InsertRaw", sql);
...
void * bin_data = static_cast<void*>(arrayChar.data());  
size_t bin_size = arrayChar.size(); // ...and the size of the binary data

pqxx::binarystring blob(bin_data, bin_size);
//at this line i get the error of UTF8 encoding
pqxx::result r = W.prepared("InsertRaw")(idFile.c_str())(blob).exec();

W.commit();
C.disconnect();

This is the error i get:

ERROR: invalid byte sequence for encoding "UTF8": 0xe0 0x20 0x63

KiranMayee Maddi
  • 301
  • 1
  • 12
MrMartins
  • 93
  • 5

1 Answers1

0

pqxx::binarystring is for bytea data you're reading from the server.

For submitting binary data, the doc says:

To convert the other way, i.e. from a raw series of bytes to a string suitable for inclusion as bytea values in your SQL, use the transaction's esc_raw() functions.

Daniel Vérité
  • 58,074
  • 15
  • 129
  • 156
  • Hi Daniel, thank you for your suggestion. I tried with esc_raw function, but i get the same error. I'm wondering if issue should be solved from c++ side or postgresql side (i.e. assign to rawColumn in my table a specific encoding or changing rawColumn type from bytea to string, keeping in mind that data should be very huge ... sometimes 1 GB). Kind regards – MrMartins Jun 24 '19 at 07:44