0

(This project would probably on-topic on the SuperUser or Electronics sites, but the focus of this question is programmable software for accomplishing it.)

I have an embedded system which produces data at about 100KB/s which needs to be stored and later processed on a Windows PC. Serial streaming works, but internal storage has the obvious advantage of not requiring a laptop and cabling present in the room where data collection occurs.

My circuit board has an SD card socket (SPI connected), and I had previously tried writing files into a FAT filesystem. But this turned out to be a horrible bottleneck, not least of the reasons for which is that, in order to support multiple files of arbitrary length growing at arbitrary times, file access in FAT requires reading and writing multiple layers of metadata -- the allocation table itself and directories -- in addition to the actual content, and these cannot fit entirely in microcontroller RAM, so the number of card accesses is magnified. Worse, some tasks such as finding a new free block have linear complexity in the card size.

Another strike against FAT is that it's quite prone to data corruption in case of power loss without prior clean unmount, and my system doesn't have any user interface which could initiate an unmount. I'm aware of techniques including power-failing interrupts and capacitance to keep the processor and SD card going for a few extra milliseconds, but since those don't help with hot removal anyway and my existing circuit doesn't have them, I'm trying to achieve a crash-only design for storage.

So I've decided to skip the filesystem and just use the card as a raw sequence of blocks. However, I don't want my users being invited to reformat cards containing data

enter image description here

So, I want to partition the cards and place a small FAT partition on the front which OSes will recognize, with an explanatory test file. Users need to be able to replace cards, so I need to automate the partitioning and formatting process within my PC application.

Raw access can be achieved by calling CreateFile("\\\\.\\PhysicalDisk3") and similar paths, and this can be done on removable disks without requiring admin rights. Still, this is code I'd rather not have to write myself. And the OS would still have to be told to rescan volumes after the change.

Are there APIs either built into Windows or part of the third-party open source framework for partitioning SD cards without requiring direct bitwise access to partition tables? If not, what should I do to have Windows rescan volumes after I change partitioning? I much prefer methods that work on XP and newer.

Ben Voigt
  • 277,958
  • 43
  • 419
  • 720
  • Q: What kinds of filesystems does your embedded OS support? Otherwise, Q: How comfortable do you feel getting up close and personal with the Windows DDK (realizing you might already be extremely proficient with it)? Q: Have you considered using FAT32, but overlaying your own error detection/error correction? Q: What *is* your embedded OS? – paulsm4 Nov 03 '13 at 19:52
  • There is no embedded OS. There's a total of 160KB RAM on the embedded system, and that has to handle other functions besides storage. My understanding is that FAT32 and exFAT are at least as bad as plain FAT (FAT12, FAT16) in terms of memory usage. I'm happy to use the DDK for user-mode development, (the DDK headers are packaged for use with a command-line build process even for user-mode, that I can deal with), but I'm not going to write kernel-mode code for something that can be accomplished in user-mode. – Ben Voigt Nov 03 '13 at 20:00
  • One more suggestion: take a look at the source for Win32DiskManager. You might be able to something similar for your project: http://sourceforge.net/projects/win32diskimager/ – paulsm4 Nov 03 '13 at 20:19
  • @paulsm4: I've used several image tools before, including that one. They simply copy a partition table stored in the image, they don't actually interact with it, and so you need to have an pre-existing image that matches your disk's size and (logical) geometry. – Ben Voigt Nov 03 '13 at 20:21
  • Hi - I wasn't suggesting using the tool itself, just the source. It sounded like the way Win32DiskImager (pardon my typo above) interacted with the SD card might be similar to the way you want to interact with your SD card, and you might be able to use some of the same Win32 APIs. I'm prepared to be wrong: Just a thought. IMHO... – paulsm4 Nov 03 '13 at 20:39
  • @paulsm4: I checked the source... it uses the same `CreateFile` on a physical disk that I mentioned in the question... and I don't see any code to trigger a volume rescan. – Ben Voigt Nov 03 '13 at 20:41

2 Answers2

1

Your problem seems to be nearly exactly the same I currently have to solve in one of our own projects.

I would recommend to leave the fat32 filesystem unchanged on the card.
But then you create one or more big container files, with continuous blocks, or a precached allocation table.

This can even be done with the embedded system itself.
elm-chan's FatFs

Then you can access all blocks inside the container native, without any further access to the fat, but the card can be handled by any OS and even a normal user can copy the files.

Smaple function for allocating a contiguous container:
Allocate a contiguous area to the file
Then you can access the blocks without the filesystem library, only for the first access to the container to determine where it starts.

jeb
  • 78,592
  • 17
  • 171
  • 225
  • I'm already using a filesystem library ([efsl](http://sourceforge.net/projects/efsl/)), but it's causing my issues. It uses the call stack for its state, making it incompatible with interrupt-driven designs, it requires a large amount of memory as block cache, reads and writes metadata often, etc. I've seen systems that use a large container file like you suggest, but ensuring that the container is perfectly contiguous requires low-level access and direct adjustment of filesystem structures anyway; if I have to do that, I'd rather just manage the whole partition myself. – Ben Voigt Nov 04 '13 at 16:02
0

It looks as if Windows uses the IOCTL mechanism for partition management. In particular, the IOCTL_DISK_SET_DRIVE_LAYOUT_EX control code looks very useful.

I'll be investigating further and then update this answer.

Ben Voigt
  • 277,958
  • 43
  • 419
  • 720