ISO9660 can not read CDs bigger than 4.7G

[jahaiku]

I have tried to access a DVD which is bigger than 4.7G. If I try to access a file on it it only say's that the file can not be read. There is no error in log. So you can not access DoubleLayer DVDs or BluRays.

[anevilyak]

Are you certain it's iso9660 and not UDF that's used there?

[jahaiku]

Replying to anevilyak:

Are you certain it's iso9660 and not UDF that's used there?

Yes, I have generated it with xorriso on linux.

[diver]

hrev43980 gcc2 iso9660 module with trace enabled

[diver]

Try to replace your iso9660 with the attached one and see if there is anything in syslog.

[diver]

hrev43980 gcc4 iso9660 module with trace enabled

[jahaiku]

Replying to diver:

Try to replace your iso9660 with the attached one and see if there is anything in syslog.

I have own gcc4 build. How do I enable Trace in my sources for the iso9660?

[anevilyak]

Replying to jahaiku:

I have own gcc4 build. How do I enable Trace in my sources for the iso9660?

​http://cgit.haiku-os.org/haiku/tree/src/add-ons/kernel/file_systems/iso9660/iso9660.cpp#n30

​http://cgit.haiku-os.org/haiku/tree/src/add-ons/kernel/file_systems/iso9660/kernel_interface.cpp#n45

• uncomment those lines and rebuild.

[jahaiku]

Hmmm, I now generated a big ISO with 7.8G (Haiku-Sources). Tracker can show the Struktur, but if I less a testfile in Terminal it do NOT contain what it should contain. But I do NOT get a syslog when reading this file.

[anevilyak]

Out of curiosity, do the same problems occur on e.g. alpha 3? Just to see if it's a new regression or not.

[jahaiku]

Replying to anevilyak:

Out of curiosity, do the same problems occur on e.g. alpha 3? Just to see if it's a new regression or not.

Yes, it is older than alpha3. I remember that I have tried it years ago with a DoubleLayer-DVD and got wrong data or error box, when trying to play a MP3 from that disk.

[scdbackup]

I am the developer of xorriso.

You may let it report block addresses of data files, which should match the LBN from line 848 of iso9660.cpp:

         TRACE(("InitNode - data start LBN is %d\n",
                (int)node->startLBN[FS_DATA_FORMAT]));

The xorriso command would be

  xorriso -indev ...ISO-image-path... \
          -find / -sort_lba -exec report_lba -- \
  | less

It will print lines like

Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
...
File data lba:  0 ,     1907 ,       11 ,    21781 , '/md5sum.txt'
File data lba:  0 ,     1918 ,       71 ,   144786 , '/README.mirrors.html'
File data lba:  0 ,     1989 ,       39 ,    78036 , '/README.mirrors.txt'
...

The column "Startlba" should match the "LBN" values from the trace.

On a first glimpse, iso9660.cpp looks as if it was restricted to data files with a single data extent. This would restrict the size of a data file to 4 GiB-1. But this does not restrict the overall size of the image.

[jahaiku]

I have tested some more and it looks like all file data which is written after the first 4.7G results in data garbage when reading in haiku. File size looks ok, but data not.

[scdbackup]

4.7 GB is a strange threshhold for an error. Is there a module which does not believe in larger DVDs and silently fiddles with the addresses ?

Function iterative_io_get_vecs_hook() in iso9660/kernel_interface.cpp looks as if it computes the start byte of the data of a file. Its computation would yield the correct byte address in the image if parameter "offset" is 0, and type "off_t" has 64 bits.

Maybe one should trace some info there. E.g. the values of

offset, node->startLBN[FS_DATA_FORMAT]

and the resulting value of

vecs->offset

[jahaiku]

Replying to scdbackup:

I am the developer of xorriso.

You may let it report block addresses of data files, which should match the LBN from line 848 of iso9660.cpp:

         TRACE(("InitNode - data start LBN is %d\n",
                (int)node->startLBN[FS_DATA_FORMAT]));

The xorriso command would be

  xorriso -indev ...ISO-image-path... \
          -find / -sort_lba -exec report_lba -- \
  | less

It will print lines like

Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
...
File data lba:  0 ,     1907 ,       11 ,    21781 , '/md5sum.txt'
File data lba:  0 ,     1918 ,       71 ,   144786 , '/README.mirrors.html'
File data lba:  0 ,     1989 ,       39 ,    78036 , '/README.mirrors.txt'
...

The column "Startlba" should match the "LBN" values from the trace.

On a first glimpse, iso9660.cpp looks as if it was restricted to data files with a single data extent. This would restrict the size of a data file to 4 GiB-1. But this does not restrict the overall size of the image.

I have checked this and the LBN-Value from the TRACE and the value (LBA) I got from xorriso is the same.

[scdbackup]

Replying to jahaiku:

I have checked this and the LBN-Value from the TRACE and the value (LBA) I got from xorriso is the same. [[BR]

Well, the block addresses of the file content should be ok then.

So iterative_io_get_vecs_hook() in iso9660/kernel_interface.cpp would be the next place to put a TRACE(). Let's see whether the correct block adresses appear there and to what byte addresses they get converted.

I see another usage of startLBN which could lead to data file content. It is in function fs_read_pages() in iso9660/kernel_interface.cpp :

fileVec.offset = pos + node->startLBN[FS_DATA_FORMAT]
                * volume->logicalBlkSize[FS_DATA_FORMAT];

Again, pos , node->startLBN[FS_DATA_FORMAT], and the resulting fileVec.offset would be of interest.

Do all files show wrong content or does this happen only with those which have block numbers above 2097151 (= 4 GiB - 2 KiB) ?

[jahaiku]

Do all files show wrong content or does this happen only with those which have block numbers above 2097151 (= 4 GiB - 2 KiB) ?

Yes, looks like all files above block 2097151 are corrupted. From xorriso:

snip
File data lba:  0 ,   703998 ,   524288 , 1073741824 , '/OSb/bigFile1'
File data lba:  0 ,  1228286 ,   524288 , 1073741824 , '/OSb/bigFile2'
File data lba:  0 ,  1752574 ,   524288 , 1073741824 , '/OSb/bigFile3'
File data lba:  0 ,  2276862 ,   524288 , 1073741824 , '/OSb/bigFile4'
snip

Checking MD5s inside haiku results in wrong checksums beginning from bigFile4 and all following files.

[jahaiku]

Replying to jahaiku:

Do all files show wrong content or does this happen only with those which have block numbers above 2097151 (= 4 GiB - 2 KiB) ?

Yes, looks like all files above block 2097151 are corrupted. From xorriso:

snip
File data lba:  0 ,   703998 ,   524288 , 1073741824 , '/OSb/bigFile1'
File data lba:  0 ,  1228286 ,   524288 , 1073741824 , '/OSb/bigFile2'
File data lba:  0 ,  1752574 ,   524288 , 1073741824 , '/OSb/bigFile3'
File data lba:  0 ,  2276862 ,   524288 , 1073741824 , '/OSb/bigFile4'
snip

Checking MD5s inside haiku results in wrong checksums beginning from bigFile4 and all following files.

I have now added these lines into the requested places:

   TRACE(("JA - %d %d %d %d \n", (int)offset, (int)node->startLBN[FS_DATA_FORMAT], (int)node->volume->logicalBlkSize[FS_DATA_FORMAT], (int)vecs->offset ));
and
   TRACE(("JA - %d %d %d %d \n", (int)pos, (int)node->startLBN[FS_DATA_FORMAT], (int)volume->logicalBlkSize[FS_DATA_FORMAT], (int)fileVec.offset ));

Now I have called md5sum for bigFile4 and I get this:

snip
2012-04-14 13:22:10 KERN: JA - 0 2276862 2048 368046080 
2012-04-14 13:22:10 KERN: JA - 32768 2276862 2048 368078848 
2012-04-14 13:22:10 KERN: JA - 65536 2276862 2048 368111616 
2012-04-14 13:22:10 KERN: JA - 98304 2276862 2048 368144384 
2012-04-14 13:22:10 KERN: JA - 131072 2276862 2048 368177152 
2012-04-14 13:22:10 KERN: JA - 163840 2276862 2048 368209920 
2012-04-14 13:22:10 KERN: JA - 196608 2276862 2048 368242688 
snip

For the file after bigFile4 with xorriso info:

2801150 | 1 | 234

I get:

2012-04-14 13:42:38 KERN: JA - 0 2801150 2048 1441787904 

[scdbackup]

Replying to jahaiku:

   TRACE(("JA - %d %d %d %d \n", (int)offset, (int)node->startLBN[FS_DATA_FORMAT], (int)node->volume->logicalBlkSize[FS_DATA_FORMAT], (int)vecs->offset ));
and
   TRACE(("JA - %d %d %d %d \n", (int)pos, (int)node->startLBN[FS_DATA_FORMAT], (int)volume->logicalBlkSize[FS_DATA_FORMAT], (int)fileVec.offset ));

We need more than 32 bit for printing the byte result.Further it would be good if both TRACE were distinguishable. E.g.

TRACE(("JA hook  - %d %d %d %.f \n", (int)offset, (int)node->startLBN[FS_DATA_FORMAT], (int)node->volume->logicalBlkSize[FS_DATA_FORMAT], (double)vecs->offset ));
TRACE(("JA pages - %d %d %d %.f \n", (int)pos, (int)node->startLBN[FS_DATA_FORMAT], (int)volume->logicalBlkSize[FS_DATA_FORMAT], (double)fileVec.offset ));

2012-04-14 13:22:10 KERN: JA - 0 2276862 2048 368046080

The result should be 4,663,013,376 rather than 368,046,080. This is now of course an artefact by the %d formatter. But probably it is an omen about what happens to the correct result in the further course of processing.

I understand from ​http://api.haiku-os.org/SupportDefs_8h.html that off_t is 64 bit. The receiver object uses off_t : ​http://api.haiku-os.org/structfile__io__vec.html

But if i do on my amd64 GNU/Linux box:

#include <stdio.h>
#include <unistd.h>
#include <stdint.h>

int main()
{
 off_t result, pos = 0;
 uint32_t lba = 2276862, bs = 2049;

 result = pos + lba * bs;
 printf("%.f\n", (double) result);

 result = pos + ((off_t) lba) * ((off_t) bs);
 printf("%.f\n", (double) result);
}

I get this output:

370322942
4665290238

So i expect to see 370 million even with TRACE formatting for large numbers.

If so then, equip in iso9660/kernel_interface.cpp the occurences of

node->startLBN[FS_DATA_FORMAT] * node->volume->logicalBlkSize[FS_DATA_FORMAT]

with a generous amount of (off_t) casts and brackets.

Then try whether it works better.

[scdbackup]

With the correct bs = 2048 rather than 2049, the numbers of my test program are

368046080
4663013376

[jahaiku]

Replying to scdbackup:

With the correct bs = 2048 rather than 2049, the numbers of my test program are

368046080
4663013376

I have changed the (int) to (double) and I get the same:

2012-04-15 10:08:06 KERN: JA hook - 0 2276862 2048 368046080

Next I try to add some extra casts...

[jahaiku]

Yes, that was it. This patch fixes it. Now I get correct md5's.

[axeld]

Looks good, although it's enough to cast one of them to off_t to make the result 64 bits. Thanks!

[axeld]

Applied in hrev44032, thanks again! Please use git format-patch next time, though, so that we can properly attribute you, and also have the commit message ready :-)