FAT: wrong file times

[MatejHorvat]

I created a text file on a FAT partition today (August 12, 2014) at 2:06 PM on hrev47617. Its time of modification was set to August 4, 2014, 2:32 AM. Modifying the file a few minutes later did not change it.

Making a copy of an existing file gave the duplicate the same modification time as the original. Modifying the duplicate did not change it there either.

My timezone is that of Slovenia. The time is synchronized with NTP.

[Jim906]

I'll see if I can fix this or at least gather some information about the problem and post it here.

[waddlesplash]

Honestly the FAT driver needs a major overhaul in general. Perhaps a major rewrite wouldn't even be a bad idea, with code borrowed from or referenced from FreeBSD or elsewhere.

[MatejHorvat]

Sure, as long as it gets fixed. Though theirs isn't perfect either:
​https://bugs.freebsd.org/bugzilla/buglist.cgi?bug_status=__open__&order=Importance&query_format=advanced&short_desc=msdosfs&short_desc_type=allwordssubstr

Microsoft has released the source code of their FAT driver under MS-PL, which I think is MIT-compatible. Even if it's too hard to adapt, it may be useful as a reference:
​https://github.com/Microsoft/Windows-driver-samples/tree/main/filesys/fastfat

[Jim906]

I will see if I can adapt one of these drivers. I'm sure I would learn a lot from the process. It would take me a while though.

It would nice if I could get the Microsoft driver to work. Microsoft does have online documentation that explains many of the Windows types/functions that would need to be substituted for, to compensate for absent Windows headers, so it might be possible.

[Jim906]

I am still a long way from submitting any major updates to the fat driver.

In the meantime, this is a suggested patch to fix the failure to update the time of modification: ​https://review.haiku-os.org/c/haiku/+/5798.

I could not reproduce the other problem from the bug report: the incorrect modification time being set when a file is created. MatejHorvat, does this problem still occur for you when using a more recent version of Haiku?

I will continue to study the FreeBSD and MS drivers to see if parts of either could be ported to Haiku.

[Jim906]

I haven't forgotten about this and am working on adapting the FreeBSD FAT driver.

I focused on that over the Microsoft driver because the MS-PL does not apply to the Windows interface that the driver is built around. If using a large portion of the Microsoft driver code, I think it would be hard to avoid basically replicating some non-MS-PL Windows headers for compatibility.

[waddlesplash]

It's worth noting that the Haiku bootloader also has a FAT implementation with partially implemented write support (just enough to create the "SYSLOG00" files), maybe you can use this also as a basis, in tandem with the FreeBSD driver sources.

[bbjimmy]

Interestingly, I copied foles from a usb mounted sd card to a usb stick, both reported 1887 for the creation date year. After mounting the usb stick in Windows, it showed the corect date. I then mounted the same usb stick in haiku hrev56838 x86_64 running in a VirtualBox on Windows 10. The file dates showed properly! To get the foile dates proper on my bare metal install, I zipped the contents of the usb stick in the virtual Haiku install, moved the stick to my main install, and copied the zip file to my BFS partition to unzip it, now I have all the file creation dates reporting properly on my main bfs partition.

• note I had to use my sneaker network ( manually moving media from one computer to another) as neither FTP, NETFS nor PoorMan could seem to stay connected long enoughh to transfer the zip file from the virtual machine Haiku install.

Maybe the date issue is really a usb driver issue and not a FAT issue?

[waddlesplash]

If the problem was really with the USB driver, a lot more would be corrupted on the FAT partition than just the dates. Something else must be different between these environments causing the FAT driver to mishandle the dates.

But as noted above, the FAT driver really needs a major overhaul...

[bbjimmy]

The FAT systm was used to read the SD card and used to write to the USB stick. When Windows was used, either natively or with haiku in a Virtual box, The files showed the proper dateon the usb stick. Only when haiku is used natively do the file dates show badly.

[Jim906]

I should have an updated FAT driver (a port of the FreeBSD driver) ready to submit soon, which should fix the reported problem with file dates. At this point the updated driver can perform basic functions but it needs more testing and debugging.

[waddlesplash]

Please retest after hrev57811.

[MatejHorvat]

Booting hrev57820

[MatejHorvat]

Tried to boot hrev57820. See screenshot. No accessible partition at that time was an (unencrypted) FAT partition, yet apparently one of them looks enough like one that the driver tried to divide by 0.

Side question: is there an easy way to compile and replace just the FAT driver in an existing Haiku installation? The 32 GB limit really limits its usefulness for me. I've used the FreeBSD driver mostly successfully with huge partitions (up to almost 2 TB), so I don't see a reason why the port would behave differently in that regard.

[waddlesplash]

It looks like sectorsPerCluster in that method isn't zero-checked.

[Jim906]

I will submit a patch to return early with an error code if sectorsPerCluster is zero. This will also serve as a 4th check of the filesystem type. I won't be able to submit this patch today but will try to submit it by Monday. I apologize for the delay.

The 32 GB limit was added because I had not tested any partitions > 32 GB in Haiku and was not expecting users to be interested in partitions of that size. I will try to test some larger partitions next week and then submit a patch to remove or extend the limit.

[Jim906]

Patch submitted (​https://review.haiku-os.org/c/haiku/+/7840).

I will also revisit _dosfs_initialize(). I don't think it is properly clearing the FAT signature when a partition is reformatted from FAT to another filesystem. This might explain the presence of a volume that is not FAT but can get past the signature check in check_bootsector().

[waddlesplash]

Patch merged in hrev57824.

[Jim906]

MatejHorvat, would extending the limit from 32 GB to 128 GB be sufficient for your uses? You mentioned using partitions of almost 2 TB but I don't know if that is something you want to do routinely or just an example from your past experience.

[bipolar]

Wikipedia list the following as max volume sizes for different FATs:

    FAT12: 32 MB (256 MB for 64 KB clusters)
    FAT16: 2 GB (4 GB for 64 KB clusters)
    FAT32: 2 TB (16 TB for 4 KB sectors)

Why artificially limit it below that? If you're affraid of data loss... allow mounting it read-only at least?

In any case, looking forward to be able to mount a ~90 GB partition :-).

Just in case... I tested the new FAT driver (latest nightlies) and 2 FAT32 partitions (same disk) appeared as "no name" on Tracker's mount menu (same on DriveSetup), but when mounted the one I could (the smaller one), it appeard it had a cabinet Icon labeled "winxp" (still mounted as "/no name"). Original partition label is "WinXP". Second one hit current 32 GB limitation.

---

PS: BTW Jim906... thanks a bunch for working on this! Much appreciated!

[Jim906]

Yes, the artificial 32 GB limit was meant as a safeguard in case of overflow of something like partition offset or sector number, if at some point the code added for the port stores a value in an inappropriate variable type. As you suggested, I could force the partition to mount read-only instead of disallowing access completely. I will see about changing that in the next patch I submit, and will look into the partition label issues you reported.

[nephele]

The 32 GB limit was added because I had not tested any partitions > 32 GB in Haiku and was not expecting users to be interested in partitions of that size. I will try to test some larger partitions next week and then submit a patch to remove or extend the limit.

fat32 is a pretty "safe" option if you want to transfer files. for my wii U modding I was going to get a bigger sdcard than that, and am now quite suprised about this limit. I think limiting it to what the filesystem itself allows makes sense, In any case.

[pulkomandy]

FAT on larger cards becomes inefficient. It achieves larger volume size by increasing cluster size, which means the smallest possible allocation is 16, 32 or even 64K (it is not possible to have files smaller than that). And also the file allocation table itself is starting to use a large part of the disk space.

That's why larger SD cards now come formatted as exFAT by default, it's a filesystem designed to fix these limitations. But, we don't have write support for that yet.

So, support up to 2TB should be fine (larger than that requires also increasing the sector size to 4K instead of 512 bytes, in addition to increasing the cluster size).

[Jim906]

After doing some more testing on partitions up to 256 GB, I submitted a patch to increase the artificial limit to 256 GB, and to mount read-only above that instead of refusing to mount the partition: ​https://review.haiku-os.org/c/haiku/+/7911.

[Jim906]

Replying to bipolar:

Just in case... I tested the new FAT driver (latest nightlies) and 2 FAT32 partitions (same disk) appeared as "no name" on Tracker's mount menu (same on DriveSetup), but when mounted the one I could (the smaller one), it appeard it had a cabinet Icon labeled "winxp" (still mounted as "/no name"). Original partition label is "WinXP". Second one hit current 32 GB limitation.

This patch (​https://review.haiku-os.org/c/haiku/+/8115) is intended to fix this, so that the the partition will be consistently listed as "WinXP" on the Tracker mount menu, in DriveSetup, and in the cabinet icon.

[Jim906]

I submitted a patch (​https://review.haiku-os.org/c/haiku/+/8362) to support write access to partitions of sizes up to 2 TB.

[waddlesplash]

The patch above was merged. MatejHorvat, can you retest?