Azure Web Job bad encoding downloaded data from data lake store

Question

Currently, I'm just want to download files from data lake store and store data into my sql database but I have problem with strings that shoudl containt characters like (ę, ą, ć, ł) but it is replaced by (e,a,c,l). Currently I'm tried changing Culture Information and Encoding in Stream Reader but it doesn't give me any better result (still getting replaced characters in my string values). So is there any work around or any place where I can globally set encoding parameters for my app service and web jobs included in web app service?

Answer 1

The issue is not related to WebJob. We could read any special character from any place and write it to another place due to the read and write work at byte level.

strings that shoudl containt characters like (ę, ą, ć, ł) but it is replaced by (e,a,c,l).

What column type did you define for the special characters in your SQL Server? If the column type is char or varchar. It will lost data if you store special characters. Change the column type to nchar or nvarchar will solve this issue.

Here is the test from my side.

Step 1, Define a table using following SQL statement.

CREATE TABLE [dbo].[mytable]
(
    [id] INT NOT NULL PRIMARY KEY,
    [text1] varchar(50),
    [text2] nvarchar(50)
)

Step 2, Insert a row using following SQL statement.

insert into mytable (id, text1, text2) values(1, 'ę, ą, ć, ł', 'ę, ą, ć, ł')

Step 3, Query data from mytable using following SQL statement.

select * from dbo.mytable

Here is the result I got.

According to the result, the value of text1 was changed to 'e,a,c,l' due to the column type is varchar.