RAMFS fails to fail opening a file as a directory

[dovsienko]

Haiku hrev57709

This bug manifested after I configured TMPDIR to the RAM filesystem in order to speed up compiling. It is easy to reproduce using /bin/install as follows.

Here the destination file is on BFS and invoking /bin/install more than once with the same arguments does not cause an error, which is the expected behaviour:

~> /bin/install -c -m 644 tcpdump.1 /tmp/tcpdump.1
~> /bin/install -c -m 644 tcpdump.1 /tmp/tcpdump.1
~> ls -l /tmp/tcpdump.1
-rw-r--r-- 1 user root 65794 Apr  24 13:09 /tmp/tcpdump.1

Here the destination file is on RAMFS and invoking /bin/install more than once with the same arguments fails because /bin/install unexpectedly tries to interpret the existing destination file as a directory:

~> /bin/install -c -m 644 tcpdump.1 /boot/system/var/shared_memory/tcpdump.1
~> /bin/install -c -m 644 tcpdump.1 /boot/system/var/shared_memory/tcpdump.1
install: cannot stat '/boot/system/var/shared_memory/tcpdump.1/tcpdump.1': Not a directory
~> ls -l /boot/system/var/shared_memory/tcpdump.1
-rw-r--r-- 1 user root 65794 Apr  24 13:09 /boot/system/var/shared_memory/tcpdump.1

Running the above commands using strace makes it clear that BFS rejects an attempt to open the existing file as a directory, so /bin/install correctly sees the destination as an existing file:

[ 59761] open(0xffffffff, "/tmp/tcpdump.1", O_DIRECTORY, 0x0) = 0x80006005 Not a directory (1311 us)
[ 59761] read_stat(0xffffffff, "tcpdump.1", true, 0x7f5c75511890, 0x80) = 0x0 No error (1043 us)
[ 59761] read_stat(0xffffffff, "/tmp/tcpdump.1", false, 0x7f5c75511910, 0x80) = 0x0 No error (918 us)
[ 59761] unlink(0xffffffff, "/tmp/tcpdump.1") = 0x0 No error (1228 us)

However, on RAMFS the call to open() erroneously "succeeds" and returns a file descriptor, which causes the spurious failure later:

[ 59765] open(0xffffffff, "/boot/system/var/shared_memory/tcpdump.1", O_DIRECTORY, 0x0) = 0x3 (914 us)
[ 59765] read_stat(0xffffffff, "tcpdump.1", true, 0x7f116eb5b560, 0x80) = 0x0 No error (845 us)
[ 59765] read_stat(0x3, "tcpdump.1", false, 0x7f116eb5b5e0, 0x80) = 0x80006005 Not a directory (962 us)

[korli]

​https://review.haiku-os.org/c/haiku/+/7629

[waddlesplash]

Fix merged in hrev57711.

[dovsienko]

Thank you!