Flutter/Dart : reading in text file is prefixed by weird characters

Question

I'm moving from Android Studio to flutter. In the Android Studio project , I write files in a JSON format using this code:

String json = gson.toJson(item);
FileOutputStream fos = null;
try {
  fos = context.openFileOutput("item"+ item.getUid(), Context.MODE_PRIVATE);
  ObjectOutputStream os = new ObjectOutputStream(fos);
  os.writeObject(json);
  os.close();
  fos.close();
  } catch (FileNotFoundException e) {
      e.printStackTrace();
  } catch (IOException e) {
      e.printStackTrace();
  }

I now want to load in these files in Flutter SDK using this code

Future<List<Item>> loadAllItems() async {
  try{
    var items = <Item>[];
    var filesDir = Directory("/data/user/0/com.company.app/files");

    for(var f in filesDir.listSync()){
      if(f is File){
        f.readAsString(encoding: latin1).then((jsonstr) => {
          items.add(parse(jsonstr))
        });
      }
    }
    return items;
  } catch(e){
    log(e);
  }
  return [];
}

But the problem is that the string jsonstr is now prefixed by weird characters. Here's how it is saved and loaded in at Android Studio

{"property1":"value1", ...}

and this is how it's read in in Flutter SDK

¬í t${"property1":"value1", ... }

I also tried using utf8 encoding but this raises an exception

Failed to decode data using encoding 'utf-8'

So how to get the normal JSON string without the weird symbols before it?

Aren't those 3 characters the prefix that tell `ObjectInputStream` the java type of the bytes that follow? They look like the stream magic bytes from https://docs.oracle.com/javase/8/docs/platform/serialization/spec/protocol.html. You will need to snip them off (they should always be the same - allowing for the fact that 2 bytes encode the string length) and then UTF-8 decode the rest. — Richard Heap, Sep 18 '19 at 18:15

score 1 · Accepted Answer · answered Sep 19 '19 at 03:03

Your title says "reading in text file...", but of course you aren't reading in a text file - you are reading in a Java object serialization, as created by ObjectOutputStream.writeObject(). So now you need to read and interpret that format.

As you've written a single string, the output will consist of the stream header, then a string identifier plus the length of the string, then the string.

Create a method to read the file and parse the header:

Future<String> readJavaObjectFile(File file) async {
  var bytes = await file.readAsBytes() as Uint8List;
  var data = bytes.buffer.asByteData();
  assert(data.getUint16(0) == 0xaced); // stream_magic
  assert(data.getUint16(2) == 5); // stream_version
  assert(bytes[4] == 0x74); // tc_string
  var length = data.getUint16(5); // length thereof
  return utf8.decode(bytes.sublist(7, 7 + length));
}

and call it from your loop:

Future<List<Item>> loadAllItems(Directory filesDir) async {
  var items = <Item>[];
  for (var f in await filesDir.list().where((e) => e is File).toList()) {
    items.add(parse(await readJavaObjectFile(f)));
  }
  return items;
}

or like this

Future<List<Item>> loadAllItems(Directory filesDir) async {
  var files = await filesDir.list().where((e) => e is File).toList();
  return await Future.wait(
      files.map<Future<Item>>((f) async => parse(await readJavaObjectFile(f))));
}

Note that there's a bug in your version. You only add members to items in the then which won't get executed until the future. It's simpler to get right with the await syntax.

score 0 · Answer 2 · answered Sep 18 '19 at 14:12

0

you also need write files with utf8 encoding in your java code:

ObjectOutputStream os = new ObjectOutputStream(fos, StandardCharsets.UTF_8));

answered Sep 18 '19 at 14:12

Sahandevs

1,120
9
25

Then what's the default encoding? And since this is already too late (app is already deployed to thousands of users and their data needs to be kept), what could be a solution to this problem? – Wouter Vandenputte Sep 18 '19 at 14:19
i did a quick search and found this: https://stackoverflow.com/questions/4390457/what-character-encoding-does-objectoutputstream-s-writeobject-method-use @WouterVandenputte – Sahandevs Sep 18 '19 at 14:25

Flutter/Dart : reading in text file is prefixed by weird characters

2 Answers2