How to suppress unicode characters in QString or convert to latin1

Question

Trying to insert some data from CSV to Firebird table in Qt. DB is in ASCII. When inserting some strings with non-ascii symbols get error:

Cannot transliterate character between character sets

Setting QSqlDatabase::setConnectOptions("ISC_DPB_LC_CTYPE=UTF8;") and converting column to UTF8 (CHARACTER SET UTF8) does not help - same error. Trying to suppress unicode characters with no luck as well:

    QTextCodec *codec = QTextCodec::codecForName("latin1");
    QByteArray encodedString = codec->fromUnicode(str);
    str = QString(encodedString);

QString::toLatin1 () does not suppress characters as well. What solution could be here?

Strange, `ALTER TABLE ALTER COLUMN TYPE VARCHAR(x) CHARACTER SET UTF8` does not change column character set, remains ASCII. — Aleksey Kontsevich, Dec 02 '15 at 07:19
`QString` is already in utf-16, your convertion will strip all non 7-bit ascii characters and it useless. What is your output data type from FireBird? Show code, how you fill your `str` first time. — Dmitry Sazonov, Dec 02 '15 at 08:25
Sorry, it is not clear for me, what you want to achieve? You need to strip all non-ascii chars? Are you looking for http://doc.qt.io/qt-5/qstring.html#toLatin1 ? — Dmitry Sazonov, Dec 02 '15 at 08:27
I have Qt 4, however it does not work - does not suppress characters, and `toAscii()` and `toLatin()` do not work as well. — Aleksey Kontsevich, Dec 02 '15 at 08:39
Query: `INSERT INTO RXNORM_INGREDIENTS (RXCUI, RXAUI, STR) VALUES ('151513', '2355753', 'CinaziГЁre')` Database Text: arithmetic exception, numeric overflow, or string truncation - Cannot transliterate character between character sets Driver Text: Unable to execute query — Aleksey Kontsevich, Dec 02 '15 at 08:49
Nothing special, just reads line from CSV, splits it and then assigns str = columns[14], that it. — Aleksey Kontsevich, Dec 02 '15 at 09:53
Why do you read them to `QString` but not to `QByteArray`? In what encoding is your CSV? Utf-8 / ascii / utf-16 / other? — Dmitry Sazonov, Dec 02 '15 at 10:52
How QByteArray may help me? `QString line = importFile.readLine(); QStringList columns = line.split("|", QString::KeepEmptyParts);` Think most file is latin1 and some chars are > 127 or Unicode. — Aleksey Kontsevich, Dec 02 '15 at 11:36
You should know exact encoding to parse your CSV. And `readLine` returns `QByteArray`, not `QString` - this is your first mistake. — Dmitry Sazonov, Dec 02 '15 at 11:49
If I know encoding and read to byte array what this changes? — Aleksey Kontsevich, Dec 02 '15 at 12:03

Aaron · Accepted Answer · 2015-12-02T11:28:29.787

1

This piece of code should do what you need:

QString h("Honkäüö?ß#asdfe");

unsigned char * data = (unsigned char*)h.data();

QString result;

for(int i = 0; h.size()*2; i+=2) {
   if(data[i] > 127) {
     result.append("?");
  } else {
     result.append(QChar(data[i]));
  }
}

Here is another, more robust, version:

QString h("Honkäüö?ß#asdfe");
QString result;

for(int i = 0; i < h.size(); ++i) {
   QChar qc = h.at(i);
   unsigned char c = *(unsigned char*)(&qc);
   if(c >= 127) {
     result.append("?");
   } else if (QChar(c).isPrint()) {
     result.append(QChar(c));
   }
}

QString result is just used to show what is extracted. You could copy the data[i] in a char array or a append to a QByteArray.

result is Honk?????????#asdfe

This works well for 16bit characters. 32bit characters result in additional '?'s or other characters.

edited Dec 02 '15 at 11:28

answered Dec 02 '15 at 09:19

Aaron

1,181
7
15

good idea, will try if not find better way. Will replace chars > 127 with ? or smth like this. Good to replace chars with German umlaut 'Ä' for example to smth meaningful like 'A'. – Aleksey Kontsevich Dec 02 '15 at 09:29
You shoud not cast utf-16 data to 8-bit, because utf-16 chars may be with variable length. – Dmitry Sazonov Dec 02 '15 at 09:51
True, I accounted for that fact in my last sentence. – Aaron Dec 02 '15 at 09:54
@SaZ it is not a risk, doesn't render this approach useless and surely doesn't justify a down vote. – Aaron Dec 02 '15 at 10:42
Seems above answer is the only way for now: Firebird does not allow to change column/table/DB codepage, only for newer created data. – Aleksey Kontsevich Dec 02 '15 at 11:45

score 0 · Answer 2 · answered Jul 01 '20 at 01:53

This code extract any unicode (emojis) from qstring (16 or 32 bits) and toReturn contain only characters from ASCII table ( unicode value less than 256 )

QString cleanQString(QString toClean) {

    QString toReturn="";

    for(int i=0;i<toClean.size();i++){
        if(toClean.at(i).unicode()<256){
            toReturn.append(toClean.at(i));
        }
    }

    return toReturn;


}

How to suppress unicode characters in QString or convert to latin1

2 Answers2