spring boot UTF8 encode

Question

im still a beginner with spring boot, here im trying to fetch the data from database by using JPA then post the result with RestTemplate and everything is okay but my problem with the Arabic language, it does not send it correctly, I have searched and I found a couple of solutions like :

set the below configuration in application.property

spring.http.encoding.charset=UTF-8
spring.http.encoding.enabled=true
spring.http.encoding.force=true

or :

String message = new String(object.getMessage().getBytes(),
        StandardCharsets.UTF_8);

but im still getting same problem, any idea about this ?

note : im using Oracle 11g, with encoding type: AR8ISO8859P6

public class Messenger {

    private static Messenger instance = new Messenger();
    private static final RestTemplate restMessengerTemplate;
    private static final HttpHeaders restMessengerHeader;
    private HttpEntity<VasGateway> smsEntity;
    private ResponseEntity<String> messengerResponseEntity;

    static {
        restMessengerTemplate = new RestTemplate();
        restMessengerHeader = new HttpHeaders();
        restMessengerHeader.setAccept(Arrays.asList(MediaType.APPLICATION_JSON));
        restMessengerHeader.add("Authorization",
                "Basic " + Base64.getEncoder().encodeToString(("user:pass").getBytes()));
    }

    private Messenger() {

    }

    public static Messenger getInstance() {
        return instance;
    }

    public String call(String messageId, String msisdn, String messageBody, String language) throws Exception {
        try {
            if (msisdn.startsWith("011") || msisdn.startsWith("944")) {
                smsEntity = new HttpEntity<VasGateway>(
                        new VasGateway(new Sendsms(new ArrayList<Message>(
                                Collections.singletonList(new Message(messageId, "Test", msisdn,
                                        messageBody, language.equalsIgnoreCase("EN") ? "0" : "2"))))),
                        restMessengerHeader);

                messengerResponseEntity = restMessengerTemplate.exchange("link",
                        HttpMethod.POST, smsEntity, String.class);

                if (messengerResponseEntity.getStatusCode() == HttpStatus.OK) {
                    return "\"0\"";
                } else {
                    return messengerResponseEntity.getStatusCode().toString();
                }
            }
        } catch (Exception e) {
            throw new Exception();
        }
        return "\"-1\"";
    }

}

@BenjaminEckardt oracle 11 g, encoding value : `AR8ISO8859P6` — , Jun 03 '20 at 14:22
@Kayaman here : https://stackoverflow.com/questions/39729307/how-to-handle-arabic-in-java?noredirect=1&lq=1 — , Jun 03 '20 at 14:24
@jdbcspring Does the configuration of your database connection take the `AR8ISO8859P6` encoding into account? — Benjamin Eckardt, Jun 03 '20 at 14:53
@Kayaman The data is correct and readable when I pull it out of the database but the problem only when i send it through POST HTTP request — , Jun 03 '20 at 16:03
@Kayaman unfortunately, me while trying, can you tell me how do i correct it, please — , Jun 03 '20 at 16:16
@Kayaman i have tried to send it as it is `messageBody` it looks like unreadable, also while execution i print it out local the text looks as needed , the after sending the post request — , Jun 03 '20 at 16:22
You don't seem to be setting an encoding in your request, so it's going to post with the default (`ISO-8859-1`). See if [this](https://stackoverflow.com/questions/29392422/how-can-i-tell-resttemplate-to-post-with-utf-8-encoding) will help. — Kayaman, Jun 03 '20 at 16:22
@Kayaman i have tried them but it seems i did something wrong, can you show me how please ? — , Jun 03 '20 at 16:35
@Kayaman also setup the `application.properties` as above will setup the http request as utf8 ? — , Jun 03 '20 at 16:49
I wouldn't count on it affecting `RestTemplate`, but it's a good idea to have it as `UTF8` anyway (unless you're producing widely in a different encoding). What was wrong with the `StringHttpMessageConverter`? How does the message look like when it's received? Can you give an example of a short message in arabic and what it looks like when badly encoded? — Kayaman, Jun 03 '20 at 17:03
@Kayaman , for example this text : `مرحبا` , to be valid it should be : `06450631062d06280627` , but i cant figure out what kind of encoding is this, can u help me please — , Jun 04 '20 at 11:54
That's `UTF-16`, not `UTF-8`. You really shouldn't be "figuring out" these things. It should be clearly specified which encoding is to be used. I really hope you're not doing something weird, like using `UTF-8` for English, and `UTF-16` for Arabic. — Kayaman, Jun 04 '20 at 12:11

score 0 · Accepted Answer · answered Jun 04 '20 at 12:52

What you're looking to get is the hex representation of the arabic String in UTF-16BE, as can be verified with the following code (hex printing uses Java 8 JAXB class, for other Java versions use some other way to print hex):

String arabic = "مرحبا";
System.out.println(DatatypeConverter.printHexBinary(arabic.getBytes(StandardCharsets.UTF_8)));
System.out.println(DatatypeConverter.printHexBinary(arabic.getBytes(StandardCharsets.UTF_16BE)));

Which prints out

D985D8B1D8ADD8A8D8A7    // UTF-8, not what you're looking for
06450631062D06280627    // UTF-16 Big Endian

Based on the question I linked you to, this should be enough to change the encoding for the request. Put it in your static block:

restMessengerTemplate.getMessageConverters()
    .add(0, new StringHttpMessageConverter(StandardCharsets.UTF_16BE));

However since you say that other requests have been fine, and they clearly haven't used that encoding, I have a bad feeling that you're trying to use multiple encodings depending on the language. There's really no good reason to do that (even though English uses less bytes in UTF-8 and Arabic uses less bytes in UTF-16 on average), so that would be a major design flaw. There really should be one encoding used for all communications.

However, if you end up having to use different encodings nonetheless, you'd better have two RestTemplates with different encodings.

man thank you so much, honestly, I have learned something great from you, really thank you its worked fine with me — , Jun 04 '20 at 13:24
@jdbcspring No problem. People get tripped up on encoding issues often, but the problem is that they don't really understand how it works, and they try random things (many of the "answers" on SO are not correct) hoping a complicated issue would just go away. Glad this solved it for you. — Kayaman, Jun 04 '20 at 13:27

spring boot UTF8 encode

1 Answers1