Numerous Hidden Files show on SdFat library ls()

Question

My goal is to be able to navigate through a file system on an SD card and pick out certain file types and display them to the user. When I execute sd.ls(LS_R) on the SdFat library, it shows lots of hidden files. I can deal with most of them fine, but some of them give me headaches. Since the library uses the 8.3 naming convention, it truncates the file/folder names that are too long and replaces it with a "~". This is a problem because then, I cannot distinguish between files/folders that are visible and files/folders that are hidden. Are there any known ways to solve this issue?

Here's my code:

#include <SdFat.h>

const uint8_t chipSelect = 10;

SdFat sd;
SdFile file;

void setup() 
{
    Serial.begin(9600);
    while (!Serial) {} // wait for Leonardo
    delay(1000);

    if (!sd.begin(chipSelect, SPI_HALF_SPEED)) sd.initErrorHalt();

    sd.ls(LS_R);
    while(1);
}

void loop() {}

Here is my "visible" file system:

Folder1/
    test3.txt
    test4.txt
Folder2/
BearsOutside/
test1.txt
test2.txt
LongFilename.txt

And here is the output:

FOLDER1/
  TEST4.TXT
  TEST3.TXT
TEST2.TXT
~1.TRA
TEST1.TXT
TRASHE~1/
SPOTLI~1/
  STORE-V2/
    F8D581~1/
      PSID.DB
      TM~1.SNO
      TM~1.LIO
      LIO~1.CRE
      TMP.CAB
      CA~1.CRE
      INDEXS~1
      ~1.IND
      ~~2.IND
      ~~~3.IND
      ~~~~4.IND
      ~~~~~5.IND
      ~~~~~~34.IND
      ~~~~~~37.IND
      ~~~~~~40.IND
      ~~~~~~43.IND
      ~~~~~~46.IND
      ~~~~~~48.IND
      ~1.DIR
      LIVE~~~4.IND
      LIVE~~2.IND
      LIVE~~~3.IND
      LIVE~~~5.IND
      LIVE~~66.IND
      LIVE~~69.IND
      LIVE~~73.IND
      LIVE~1.SHA
      LIVE~~79.IND
      LIVE~1.DIR
      LIVE0D~1.SHA
      STORE.DB
      STOR~1.DB
      REVERS~1
      TMPSPO~1.STA
      PERMST~1
      STORE_~1
      JOURNA~1.LIV/
      JOURNA~2.LIV/
        RETIRE.3
      JOURNA~3.LIV/
        RETIRE.4
      JOURNA~4.LIV/
      JOURNA~1.ASS/
      JOURNA~2.ASS/
      JOURNA~1.HEA/
      JOURNA~1.MIG/
      JOURNA~2.MIG/
      JOURNA~1
      JOURNA~1.SCA/
        RETIRE.11
      REVERS~1.SHA
      ~1.SHA
      SHUTDO~1
      JOURNA~1.REP/
      CA~1.MOD
      LIVE~155.IND
      LIVE~158.IND
      0DIREC~1.SHA
      ~~~~~166.SHA
      LIVE~169.IND
      LIVE~172.IND
      LIVE~175.IND
      LIVE~178.IND
      LIVE~181.IND
      LIVE~184.IND
      LIVE~1.IND
      LIVE~190.IND
      LIVE~194.SHA
      STOR~1.UPD
      REVERS~1.UPD
      LIVE~202.IND
      TMPSPO~1.LOC
      LIVE~208.IND
      LIVE~211.IND
      LIVE~215.IND
      LIVE~218.SHA
      LIVE~~2.DIR
      LIVE1D~1.SHA
      LIVE~264.SHA
      LIVE~267.IND
      LIVE~270.IND
      LIVE~274.IND
      LIVE~277.IND
      LIVE~~~3.DIR
      LIVE~~2.SHA
      LIVE~~~3.SHA
      LIVE~~~4.SHA
      LIVE~~~5.SHA
      LIVE~296.SHA
      LIVE~300.SHA
      LIVE2D~1.SHA
      LIVE~308.SHA
      LIVE~327.IND
  STORE-V1/
    VOLUME~1.PLI
  VOLUME~1.PLI
FOLDER2/
BEARSO~1/
LONGFI~1.TXT

So my issue is, how do I distinguish between BEARSO~1/ [BearsOutside], which is not hidden, and SPOTLI~1/, which IS hidden?

score 1 · Answer 1 · answered Feb 12 '14 at 18:53

1

These files are created automatically by the Mac OSX system... Is it an option for you not to plug the SD card on any Mac? This would solve your problem... ;-)

You could also try to remove them with the Terminal on Mac.

answered Feb 12 '14 at 18:53

Mike

11
1

score 0 · Accepted Answer · answered Feb 03 '14 at 19:09

unfortunately some of the primitive functions are not directly available with the

SdFat sd;

object. Hence precluding the use of the "ls" function. The following code should accomplish what you want. It passes the sd objects current file to the pointer a directory entry object of pointer p. which then can have its attributes tested to see if they are hidden or other.

Note there is a method to test if a directory entry is long name. However, I do not believe the entry is caching the long name itself.

The below does compile. And 99.5 of it is from working code that I use. I have added the attribute and long file name detection, so it should work.

void ListFiles2(uint8_t flags) {
  // This code is just copied from SdFile.cpp in the SDFat library
  // and tweaked to print to the serial output in html!
  dir_t p;

  sd.vwd()->rewind();
  Serial.println();
  while (sd.vwd()->readDir(&p) > 0) {
    // done if past last used entry
    if (p.name[0] == DIR_NAME_FREE) break;

    // skip deleted entry and entries for . and  ..
    if (p.name[0] == DIR_NAME_DELETED || p.name[0] == '.') continue;

    // only list subdirectories and files
    if (!DIR_IS_FILE_OR_SUBDIR(&p)) continue;

    if ((p.attributes & DIR_ATT_HIDDEN) != DIR_ATT_HIDDEN) continue;

    // print any indent spaces
    Serial.print(F("  "));
    for (uint8_t i = 0; i < 11; i++) {
      if (p.name[i] == ' ') continue;
      if (i == 8) {
        Serial.print('.');
      }
      Serial.print((char)p.name[i]);
    }
    Serial.print(F(" "));

    // print file name with possible blank fill
    for (uint8_t i = 0; i < 11; i++) {
      if (p.name[i] == ' ') continue;
      if (i == 8) {
        Serial.print('.');
      }
      Serial.print((char)p.name[i]);
      if (DIR_IS_LONG_NAME(&p)) {
        Serial.print(F(" long fn"));
      }
    }

    if (DIR_IS_SUBDIR(&p)) {
      Serial.print('/');
    }

    // print modify date/time if requested
    if (flags & LS_DATE) {
      sd.vwd()->printFatDate(p.lastWriteDate);
      Serial.print(' ');
      sd.vwd()->printFatTime(p.lastWriteTime);
    }
    // print size if requested
    if (!DIR_IS_SUBDIR(&p) && (flags & LS_SIZE)) {
      Serial.print(' ');
      Serial.print(p.fileSize);
    }
    Serial.println();
  }
  Serial.println();
}

score 0 · Answer 3 · answered Sep 04 '20 at 20:38

This is a very old question, but it comes up high in google search so I'll provide some updated information:

As of the time of this writing (Fall 2020) a major upgrade to SdFat exists. SdFat-2.0 has been available for several years and even though it continues to be labeled as beta.8 (and is referred to in the ReadMe as an "early beta") there are relatively few known issues and it is in production use at many sites.

I was initially hesitant to adopt it, but it provides so much needed new functionality that eventually I gritted my teeth and took it on. It has turned out to be very easy to integrate and very stable.

SdFat-2.0 includes robust support for ExFat volumes, long filenames, and preallocation of file clusters with a mechanism to guarantee non-blocking writes to the SD card. This solves the long-standing problem with occasional blocking long write latency that used to interfere with data-logging on SD cards. For ExFat volumes, the data size calculation for final truncation of the preallocated file space is handled automatically.

SdFat-2.0 supports multiple cards even if they are running on different busses or different bus types. I have a device with three SD cards, one addressed by SDIO and two running on different SPI busses, and I can copy files around without difficulty.

AFAIK there are only two incompatibilities with the older arduino <sd.h> library on which so many other libraries take a dependency:

SDFat-2.0 uses a new syntax for begin(), passing a config object that tells how your SD card is wired and how it should be addressed.

SdFat-2.0 doesn't support the older file.name() function because that was set up for 8.3 filenames. Instead you get the long filename by passing a const char* that points to somewhere the filename can be stored. You need the full filename in order to open a file, but if you are OK to stick with 8.3 filenames then you could just create a wrapper that implements a simple parameterless begin() along with the old name() function. That would make everything completely compatible with sd.h and avoid any #ifdef patching of libraries.

That's really the only compatibility issues I've found, and I haven't personally encountered any bugs. You can find the github project here: github.com/greiman/SdFat-beta

Numerous Hidden Files show on SdFat library ls()

3 Answers3