0

To read a file contents of .txt file I am using

List<String> linesList = await file.readAsLines(encoding: latin1);
return linesList;

Files with Encodng UTF-8 are working perfectly with this above code. But for Encoding UTF-16LE its returning a list with length double of the lines in the file but are all empty except first line. This first index contains ÿþ#

Rakesh Verma
  • 766
  • 6
  • 14
  • 2
    "Files with Encodng UTF-8 are working perfectly..." That seems very unlikely since you're reading the files using a Latin-1 encoding and not UTF-8. If you want to read UTF-16LE files this way, then you will need to provide (and possibly implement) a UTF-16LE [`Encoding`](https://api.dartlang.org/stable/2.5.0/dart-convert/Encoding-class.html). (Storing files as UTF-16 is a bad idea and should be avoided if possible.) – jamesdlin Sep 15 '19 at 17:55
  • There's a UTF16-LE decoder in the `utf` package. https://pub.dev/packages/utf – Richard Heap Sep 15 '19 at 18:29
  • @jamesdlin I am not creating this file. I am using an imported file from some software and then extracting data from this file and saving it on the server. – Rakesh Verma Sep 16 '19 at 03:35
  • @RichardHeap , I checked the utf package you mentioned in the comment. Its seems to be useful. I will look into this and will update. Thanks :) – Rakesh Verma Sep 16 '19 at 03:36
  • @RichardHeap Thanks a lot . You had mentioned the right flutter package.Using this package I had been able to parse the file from any utf format and showing it as as String in the app. Thanks a lot again.It means a lot :) – Rakesh Verma Sep 17 '19 at 14:47

3 Answers3

7

As package:utf is now abandoned (and therefore will never support null-safety), another way to read a UTF-16LE file as a String is to take advantage of Dart Strings using UTF-16 code units internally. You therefore can read the file, interpret the data as 16-bit (unsigned) integers, and then create a String using those as UTF-16 code units:

Basically:

var f = File("utf-16le.txt");
var bytes = f.readAsBytesSync();

// Note that this assumes that the system's native endianness is the same as
// the file's.
var utf16CodeUnits = bytes.buffer.asUint16List();

var s = String.fromCharCodes(utf16CodeUnits);

I also leave it as an exercise for the reader to deal with potential BOMs at the beginning of the file.

Also see https://github.com/dart-lang/convert/issues/30, which requests that the Dart SDK provide UTF-16 conversion functions.

jamesdlin
  • 81,374
  • 13
  • 159
  • 204
0

So the first credit goes to @Richard_Heap , who has commented on the above question. He mentioned a dart package that encodes and decodes UTF formats. I have been able to decode the txt files as expected in my Flutter app using this package.

First I am identifying the utf format type by these functions available from the package mentioned by @Richard_Heap

 List<int> bytes = await file.readAsBytes();

hasUtf16beBom(bytes)
hasUtf16Bom(bytes)}
hasUtf16leBom(bytes)
hasUtf32beBom(bytes)
hasUtf32Bom(bytes)
hasUtf32leBom(bytes)

There are different decoder & encoder functions in this package that can be used once the utf format is known using these above functions. Like I used

String decodedString = decodeUtf16le(bytes);
Rakesh Verma
  • 766
  • 6
  • 14
0

Check out the charset package. It is null-safe and supports both UTF-16BE and UTF-16LE according to documentation.

Usage:

import 'package:charset/charset.dart';

main() {
  // default
  print(utf16.decode([254, 255, 78, 10, 85, 132, 130, 229, 108, 52]));

  print(utf16.encode("上善若水"));

  // detect
  print(hasUtf16Bom([0xFE, 0xFF, 0x6C, 0x34]));

  // advance
  Utf16Encoder encoder = utf16.encoder as Utf16Encoder;
  print(encoder.encodeUtf16Be("上善若水", false));
  print(encoder.encodeUtf16Le("上善若水", true));
}
AnT
  • 801
  • 9
  • 9