Opened 8 years ago

Closed 8 years ago

#8460 closed bug (fixed)

ISO9660 can not read CDs bigger than 4.7G

Reported by: jahaiku Owned by: axeld
Priority: normal Milestone: R1
Component: File Systems/ISO 9660 Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: yes Platform: All

Description

I have tried to access a DVD which is bigger than 4.7G. If I try to access a file on it it only say's that the file can not be read. There is no error in log. So you can not access DoubleLayer DVDs or BluRays.

Attachments (3)

iso9660.zip (13.8 KB ) - added by diver 8 years ago.
hrev43980 gcc2 iso9660 module with trace enabled
iso9660_gcc4.zip (13.4 KB ) - added by diver 8 years ago.
hrev43980 gcc4 iso9660 module with trace enabled
iso9660_fix.patch (1.1 KB ) - added by jahaiku 8 years ago.

Download all attachments as: .zip

Change History (26)

comment:1 by anevilyak, 8 years ago

Version: R1/alpha3R1/Development

comment:2 by anevilyak, 8 years ago

Are you certain it's iso9660 and not UDF that's used there?

in reply to:  2 comment:3 by jahaiku, 8 years ago

Replying to anevilyak:

Are you certain it's iso9660 and not UDF that's used there?

Yes, I have generated it with xorriso on linux.

by diver, 8 years ago

Attachment: iso9660.zip added

hrev43980 gcc2 iso9660 module with trace enabled

comment:4 by diver, 8 years ago

Try to replace your iso9660 with the attached one and see if there is anything in syslog.

Last edited 8 years ago by diver (previous) (diff)

by diver, 8 years ago

Attachment: iso9660_gcc4.zip added

hrev43980 gcc4 iso9660 module with trace enabled

in reply to:  4 ; comment:5 by jahaiku, 8 years ago

Replying to diver:

Try to replace your iso9660 with the attached one and see if there is anything in syslog.

I have own gcc4 build. How do I enable Trace in my sources for the iso9660?

in reply to:  5 comment:6 by anevilyak, 8 years ago

Last edited 8 years ago by anevilyak (previous) (diff)

comment:7 by jahaiku, 8 years ago

Hmmm, I now generated a big ISO with 7.8G (Haiku-Sources). Tracker can show the Struktur, but if I less a testfile in Terminal it do NOT contain what it should contain. But I do NOT get a syslog when reading this file.

comment:8 by anevilyak, 8 years ago

Out of curiosity, do the same problems occur on e.g. alpha 3? Just to see if it's a new regression or not.

in reply to:  8 comment:9 by jahaiku, 8 years ago

Replying to anevilyak:

Out of curiosity, do the same problems occur on e.g. alpha 3? Just to see if it's a new regression or not.

Yes, it is older than alpha3. I remember that I have tried it years ago with a DoubleLayer-DVD and got wrong data or error box, when trying to play a MP3 from that disk.

comment:10 by scdbackup, 8 years ago

I am the developer of xorriso.

You may let it report block addresses of data files, which should match the LBN from line 848 of iso9660.cpp:

         TRACE(("InitNode - data start LBN is %d\n",
                (int)node->startLBN[FS_DATA_FORMAT]));

The xorriso command would be

  xorriso -indev ...ISO-image-path... \
          -find / -sort_lba -exec report_lba -- \
  | less

It will print lines like

Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
...
File data lba:  0 ,     1907 ,       11 ,    21781 , '/md5sum.txt'
File data lba:  0 ,     1918 ,       71 ,   144786 , '/README.mirrors.html'
File data lba:  0 ,     1989 ,       39 ,    78036 , '/README.mirrors.txt'
...

The column "Startlba" should match the "LBN" values from the trace.

On a first glimpse, iso9660.cpp looks as if it was restricted to data files with a single data extent. This would restrict the size of a data file to 4 GiB-1. But this does not restrict the overall size of the image.

comment:11 by jahaiku, 8 years ago

I have tested some more and it looks like all file data which is written after the first 4.7G results in data garbage when reading in haiku. File size looks ok, but data not.

comment:12 by scdbackup, 8 years ago

4.7 GB is a strange threshhold for an error. Is there a module which does not believe in larger DVDs and silently fiddles with the addresses ?

Function iterative_io_get_vecs_hook() in iso9660/kernel_interface.cpp looks as if it computes the start byte of the data of a file. Its computation would yield the correct byte address in the image if parameter "offset" is 0, and type "off_t" has 64 bits.

Maybe one should trace some info there. E.g. the values of

offset, node->startLBN[FS_DATA_FORMAT]

and the resulting value of

vecs->offset

in reply to:  10 ; comment:13 by jahaiku, 8 years ago

Replying to scdbackup:

I am the developer of xorriso.

You may let it report block addresses of data files, which should match the LBN from line 848 of iso9660.cpp:

         TRACE(("InitNode - data start LBN is %d\n",
                (int)node->startLBN[FS_DATA_FORMAT]));

The xorriso command would be

  xorriso -indev ...ISO-image-path... \
          -find / -sort_lba -exec report_lba -- \
  | less

It will print lines like

Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
...
File data lba:  0 ,     1907 ,       11 ,    21781 , '/md5sum.txt'
File data lba:  0 ,     1918 ,       71 ,   144786 , '/README.mirrors.html'
File data lba:  0 ,     1989 ,       39 ,    78036 , '/README.mirrors.txt'
...

The column "Startlba" should match the "LBN" values from the trace.

On a first glimpse, iso9660.cpp looks as if it was restricted to data files with a single data extent. This would restrict the size of a data file to 4 GiB-1. But this does not restrict the overall size of the image.


I have checked this and the LBN-Value from the TRACE and the value (LBA) I got from xorriso is the same.

Last edited 8 years ago by jahaiku (previous) (diff)

in reply to:  13 ; comment:14 by scdbackup, 8 years ago

Replying to jahaiku:

I have checked this and the LBN-Value from the TRACE and the value (LBA) I got from xorriso is the same. [[BR]


Well, the block addresses of the file content should be ok then.

So iterative_io_get_vecs_hook() in iso9660/kernel_interface.cpp would be the next place to put a TRACE(). Let's see whether the correct block adresses appear there and to what byte addresses they get converted.

I see another usage of startLBN which could lead to data file content. It is in function fs_read_pages() in iso9660/kernel_interface.cpp :

fileVec.offset = pos + node->startLBN[FS_DATA_FORMAT]
                * volume->logicalBlkSize[FS_DATA_FORMAT];

Again, pos , node->startLBN[FS_DATA_FORMAT], and the resulting fileVec.offset would be of interest.

Do all files show wrong content or does this happen only with those which have block numbers above 2097151 (= 4 GiB - 2 KiB) ?

Last edited 8 years ago by scdbackup (previous) (diff)

in reply to:  14 ; comment:15 by jahaiku, 8 years ago

Do all files show wrong content or does this happen only with those which have block numbers above 2097151 (= 4 GiB - 2 KiB) ?

Yes, looks like all files above block 2097151 are corrupted. From xorriso:

snip
File data lba:  0 ,   703998 ,   524288 , 1073741824 , '/OSb/bigFile1'
File data lba:  0 ,  1228286 ,   524288 , 1073741824 , '/OSb/bigFile2'
File data lba:  0 ,  1752574 ,   524288 , 1073741824 , '/OSb/bigFile3'
File data lba:  0 ,  2276862 ,   524288 , 1073741824 , '/OSb/bigFile4'
snip

Checking MD5s inside haiku results in wrong checksums beginning from bigFile4 and all following files.

I have now added these lines into the requested places:

   TRACE(("JA - %d %d %d %d \n", (int)offset, (int)node->startLBN[FS_DATA_FORMAT], (int)node->volume->logicalBlkSize[FS_DATA_FORMAT], (int)vecs->offset ));
and
   TRACE(("JA - %d %d %d %d \n", (int)pos, (int)node->startLBN[FS_DATA_FORMAT], (int)volume->logicalBlkSize[FS_DATA_FORMAT], (int)fileVec.offset ));

Now I have called md5sum for bigFile4 and I get this:

snip
2012-04-14 13:22:10 KERN: JA - 0 2276862 2048 368046080 
2012-04-14 13:22:10 KERN: JA - 32768 2276862 2048 368078848 
2012-04-14 13:22:10 KERN: JA - 65536 2276862 2048 368111616 
2012-04-14 13:22:10 KERN: JA - 98304 2276862 2048 368144384 
2012-04-14 13:22:10 KERN: JA - 131072 2276862 2048 368177152 
2012-04-14 13:22:10 KERN: JA - 163840 2276862 2048 368209920 
2012-04-14 13:22:10 KERN: JA - 196608 2276862 2048 368242688 
snip
Last edited 8 years ago by jahaiku (previous) (diff)

in reply to:  15 ; comment:16 by jahaiku, 8 years ago

Replying to jahaiku:

Do all files show wrong content or does this happen only with those which have block numbers above 2097151 (= 4 GiB - 2 KiB) ?

Yes, looks like all files above block 2097151 are corrupted. From xorriso:

snip
File data lba:  0 ,   703998 ,   524288 , 1073741824 , '/OSb/bigFile1'
File data lba:  0 ,  1228286 ,   524288 , 1073741824 , '/OSb/bigFile2'
File data lba:  0 ,  1752574 ,   524288 , 1073741824 , '/OSb/bigFile3'
File data lba:  0 ,  2276862 ,   524288 , 1073741824 , '/OSb/bigFile4'
snip

Checking MD5s inside haiku results in wrong checksums beginning from bigFile4 and all following files.

I have now added these lines into the requested places:

   TRACE(("JA - %d %d %d %d \n", (int)offset, (int)node->startLBN[FS_DATA_FORMAT], (int)node->volume->logicalBlkSize[FS_DATA_FORMAT], (int)vecs->offset ));
and
   TRACE(("JA - %d %d %d %d \n", (int)pos, (int)node->startLBN[FS_DATA_FORMAT], (int)volume->logicalBlkSize[FS_DATA_FORMAT], (int)fileVec.offset ));

Now I have called md5sum for bigFile4 and I get this:

snip
2012-04-14 13:22:10 KERN: JA - 0 2276862 2048 368046080 
2012-04-14 13:22:10 KERN: JA - 32768 2276862 2048 368078848 
2012-04-14 13:22:10 KERN: JA - 65536 2276862 2048 368111616 
2012-04-14 13:22:10 KERN: JA - 98304 2276862 2048 368144384 
2012-04-14 13:22:10 KERN: JA - 131072 2276862 2048 368177152 
2012-04-14 13:22:10 KERN: JA - 163840 2276862 2048 368209920 
2012-04-14 13:22:10 KERN: JA - 196608 2276862 2048 368242688 
snip

For the file after bigFile4 with xorriso info:

2801150 | 1 | 234

I get:

2012-04-14 13:42:38 KERN: JA - 0 2801150 2048 1441787904 

in reply to:  16 comment:17 by scdbackup, 8 years ago

Replying to jahaiku:

   TRACE(("JA - %d %d %d %d \n", (int)offset, (int)node->startLBN[FS_DATA_FORMAT], (int)node->volume->logicalBlkSize[FS_DATA_FORMAT], (int)vecs->offset ));
and
   TRACE(("JA - %d %d %d %d \n", (int)pos, (int)node->startLBN[FS_DATA_FORMAT], (int)volume->logicalBlkSize[FS_DATA_FORMAT], (int)fileVec.offset ));

We need more than 32 bit for printing the byte result.Further it would be good if both TRACE were distinguishable. E.g.

TRACE(("JA hook  - %d %d %d %.f \n", (int)offset, (int)node->startLBN[FS_DATA_FORMAT], (int)node->volume->logicalBlkSize[FS_DATA_FORMAT], (double)vecs->offset ));
TRACE(("JA pages - %d %d %d %.f \n", (int)pos, (int)node->startLBN[FS_DATA_FORMAT], (int)volume->logicalBlkSize[FS_DATA_FORMAT], (double)fileVec.offset ));

2012-04-14 13:22:10 KERN: JA - 0 2276862 2048 368046080

The result should be 4,663,013,376 rather than 368,046,080. This is now of course an artefact by the %d formatter. But probably it is an omen about what happens to the correct result in the further course of processing.

I understand from http://api.haiku-os.org/SupportDefs_8h.html that off_t is 64 bit. The receiver object uses off_t : http://api.haiku-os.org/structfile__io__vec.html

But if i do on my amd64 GNU/Linux box:

#include <stdio.h>
#include <unistd.h>
#include <stdint.h>

int main()
{
 off_t result, pos = 0;
 uint32_t lba = 2276862, bs = 2049;

 result = pos + lba * bs;
 printf("%.f\n", (double) result);

 result = pos + ((off_t) lba) * ((off_t) bs);
 printf("%.f\n", (double) result);
}

I get this output:

370322942
4665290238

So i expect to see 370 million even with TRACE formatting for large numbers.

If so then, equip in iso9660/kernel_interface.cpp the occurences of

node->startLBN[FS_DATA_FORMAT] * node->volume->logicalBlkSize[FS_DATA_FORMAT]

with a generous amount of (off_t) casts and brackets.

Then try whether it works better.

comment:18 by scdbackup, 8 years ago

With the correct bs = 2048 rather than 2049, the numbers of my test program are

368046080
4663013376

in reply to:  18 comment:19 by jahaiku, 8 years ago

Replying to scdbackup:

With the correct bs = 2048 rather than 2049, the numbers of my test program are

368046080
4663013376

I have changed the (int) to (double) and I get the same:

2012-04-15 10:08:06 KERN: JA hook - 0 2276862 2048 368046080

Next I try to add some extra casts...

comment:20 by jahaiku, 8 years ago

Yes, that was it. This patch fixes it. Now I get correct md5's.

by jahaiku, 8 years ago

Attachment: iso9660_fix.patch added

comment:21 by jahaiku, 8 years ago

Has a Patch: set

comment:22 by axeld, 8 years ago

Owner: changed from nobody to axeld
Status: newin-progress

Looks good, although it's enough to cast one of them to off_t to make the result 64 bits. Thanks!

comment:23 by axeld, 8 years ago

Resolution: fixed
Status: in-progressclosed

Applied in hrev44032, thanks again! Please use git format-patch next time, though, so that we can properly attribute you, and also have the commit message ready :-)

Note: See TracTickets for help on using tickets.