android file.renameTo changes subdir names

Question

I'm having a very strange problem around renaming folders in android and then reading the file again. I could never reproduce this on any or our local test devices and no one I know had a good idea.

Open Question:

What can it be that changes the sub folder name when renaming a file with renameTo()?
What are some limitations of the file system, combined with Char encoding that I'm missing here? (I also didn't find anything where I can spec char encoding for the File class)

Problem

I have a folder called name/ with it's content being other files and folders. name/ can be anything as it's user generated folder names. We have this working across many languages incl. Arabic, Chinese and more. This problem happens on multiple devices.

I'm moving all the content from name/ to .old/name/. The code I use is

File[] dirs = mRootDir.listFiles(FileFilters.FilterDotFiles); // name/ will be in that list
mOldTmp.mkdir(); //create the .old folder
for(File f : dirs) { //move all dirs into .old
    File oldSubDir = new File(mOldTmp, f.getName());
    boolean renameSuccess = f.renameTo(oldSubDir); //true, as I otherwise would never run into the problem later in code.
}

The files get all moved. It seems that the folder names get messed up on some occasion. When I list the files from the .old folder later with :

for(File dir : mOldTmp.listFiles(FileFilters.FilterFiles)) {
    for(File f : dir.listFiles(FileFilters.FilterDotFiles)) {
    }
}

I get files that have the following path. This particular one is an arabic user. When trying to read the file with creating a new new InputSteam(new FileInputStream(f)) I get an IOException:

.old/ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃÂ¬ÃÂÃÂÃÂÃÂ¯ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ/1392446611086.mzvg2bn56i1rsqe4ko1_1280.jpeg
open failed: ENOENT (No such file or directory)

The user is complaining that files are not visible anymore. This makes sense as I can't read them, but that also means that the file has been available before the rename started.

Trying to reproduce it

Test1: When I adb push a name/ folder with Arabic characters and run my code, the renameTo() file operations and everything works as expected.

Test 2: I can create a name/ folder on my computer with the name ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃÂ¬ÃÂÃÂÃÂÃÂ¯ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ. This works fine on OSX 10.8.5. When I try to use adb push then I get the same error

$ adb push . /mnt/sdcard/debug
push: ./.old/ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃÂ¬ÃÂÃÂÃÂÃÂ¯ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ/.thumbs/1398208681403.2013-12-24 15.11.43.jpg -> /mnt/sdcard/debug/.old/ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃÂ¬ÃÂÃÂÃÂÃÂ¯ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ/.thumbs/1398208681403.2013-12-24 15.11.43.jpg
failed to copy './.old/ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃÂ¬ÃÂÃÂÃÂÃÂ¯ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ/.thumbs/1398208681403.2013-12-24 15.11.43.jpg' to '/mnt/sdcard/debug/.old/ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃÂ¬ÃÂÃÂÃÂÃÂ¯ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ/.thumbs/1398208681403.2013-12-24 15.11.43.jpg': No such file or directory

$ ls -l .old/
total 0
drwxr-xr-x  24 philipp  staff   816B Apr 25 11:54 Ã?Â?Ã?Â§Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â¬Ã?Â?Ã?Â?Ã?Â?Ã?Â¯ Ã?Â?Ã?Â§Ã?Â?Ã?Â?Ã?Â?Ã?Â£Ã?Â?Ã?Â³Ã?Â?Ã?Â§Ã?Â?Ã?Â³Ã?Â?Ã?Â

Showing the content of this dir does not work consistent:

$ ls -l .old/ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃÂ¬ÃÂÃÂÃÂÃÂ¯\ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ/
total 69880
-rw-r--r--  1 philipp  staff   571K Apr 22 17:13 1398208669809.2014-02-08 14.26.50.jpg
-rw-r--r--  1 philipp  staff   2.2M Apr 22 17:13 1398208670282.2014-02-08 14.22.11.jpg
-rw-r--r--  1 philipp  staff   2.1M Apr 22 17:13 1398208670990.2014-02-08 14.21.11.jpg
... more files

next run

$ ls -l .old/ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃaÂ¬ÃÂÃÂÃÂÃÂ¯\ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ/
ls: .old/ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃaÂ¬ÃÂÃÂÃÂÃÂ¯ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ/: File name too long

When I cd into ÃÂÃÂ§ÃÂÃÂÃÂÃÂÃÂÃaÂ¬ÃÂÃÂÃÂÃÂ¯ ÃÂÃÂ§ÃÂÃÂÃÂÃÂ£ÃÂÃÂ³ÃÂÃÂ§ÃÂÃÂ³ÃÂÃÂ/ I can always list the entire content.

Indeed this looks like a really ugly encoding problem. In that case I don't believe anything shown to me by the system than hex dumps, because each piece of software does its own "intelligent" interpretations of the byte sequence. Can you in provide a hex dump of the offending directory name? And I am not even sure what would be the best way to create it. Maybe `ls -1 .old|od -t x1`. I would not use `*` or let the shell expand the name with TAB, because that could already result in "intelligent" intepretation by bash. — Harald, Apr 26 '14 at 14:26
You may want to check the section on invalid byte sequence of UTF-8 on wikipedia: http://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences . Your folder name may in fact contain an invalid sequence which is handled differently in different pieces of code on android, Java, ext4, bash, ls and whatever else is touching the byte sequence. — Harald, Apr 26 '14 at 15:08
Let me look into what you wrote. I think I might have the same probelm also with none unicode things. Any idea on how the `file.list()` method works. Because the folders worked before renaming them. — philipp, Apr 28 '14 at 19:01

android file.renameTo changes subdir names

Open Question:

Problem

Trying to reproduce it

0 Answers0