add iotop utility

[Prasad]

iotop is a simple top like utility for monitoring IOs. For utility to work, kernel must do IO accounting for each process. The patch adds struct io_accounting in the kernel account for IOs performed by each process.

The accounting for IO is performed in various version of read and write system calls.

The userland part (iotop command) uses team_info mechanism to get the accounting information in the userland. The utility provides user with lots of options like

• monitoring processes of particular user
• monitoring processes based on process ids
• only showing processes doing actual IOs
• configuration refreshing delay

iotop also allows user to sort the output on various parameters dynamically while running.

The utility also checks the visible terminal co-ordinates and updates the output accordingly.

Here is a sample output of the iotop:
~> ./iotop

Total Disk Reads:    2.00 B/s | Total Disk Writes:    0.00 B/s

    TID     USER     READ BW    WRITE BW      IO %   Command
    114     user    2.00 B/s    0.00 B/s  100.00 %   /boot/system/servers/power_daemon
     48     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/registrar
     55     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/debug_server
     56     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/net_server
     57     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/app_server
     72     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/syslog_daemon
     84     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/input_server
     94     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/mount_server
    106     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/Tracker
    107     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/Deskbar
    108     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/media_server
    109     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/midi_server
    110     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/print_server
    112     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/cddb_daemon
    113     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/notification_server
    143     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/media_addon_server
    193     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/apps/Terminal
    197     user    0.00 B/s    0.00 B/s    0.00 %   /bin/bash -l
      1     user    0.00 B/s    0.00 B/s    0.00 %   kernel_team
    212     user    0.00 B/s    0.00 B/s    0.00 %   ./iotop
-----------------------------------------------------------------------
User: all
Processes: all
Sorted according to 'Percentage of total IOs'.  Reverse Sorting: false
Refreshing every 3 seconds.

~> ./iotop -o

Total Disk Reads:    2.00 B/s | Total Disk Writes:    0.00 B/s

    TID     USER     READ BW    WRITE BW      IO %   Command
    114     user    2.00 B/s    0.00 B/s  100.00 %   /boot/system/servers/power_daemon
-----------------------------------------------------------------------
User: all
Processes: all
Only showing processes doing IOs.
Sorted according to 'Percentage of total IOs'.  Reverse Sorting: false
Refreshing every 3 seconds.

[Prasad]

Here is a sample output of the iotop:

~> ./iotop

Total Disk Reads:    2.00 B/s | Total Disk Writes:    0.00 B/s

    TID     USER     READ BW    WRITE BW      IO %   Command
    114     user    2.00 B/s    0.00 B/s  100.00 %   /boot/system/servers/power_daemon
     48     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/registrar
     55     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/debug_server
     56     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/net_server
     57     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/app_server
     72     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/syslog_daemon
     84     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/input_server
     94     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/mount_server
    106     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/Tracker
    107     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/Deskbar
    108     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/media_server
    109     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/midi_server
    110     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/print_server
    112     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/cddb_daemon
    113     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/notification_server
    143     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/media_addon_server
    193     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/apps/Terminal
    197     user    0.00 B/s    0.00 B/s    0.00 %   /bin/bash -l
      1     user    0.00 B/s    0.00 B/s    0.00 %   kernel_team
    212     user    0.00 B/s    0.00 B/s    0.00 %   ./iotop
-----------------------------------------------------------------------
User: all
Processes: all
Sorted according to 'Percentage of total IOs'.  Reverse Sorting: false
Refreshing every 3 seconds.

~> ./iotop -o

Total Disk Reads:    2.00 B/s | Total Disk Writes:    0.00 B/s

    TID     USER     READ BW    WRITE BW      IO %   Command
    114     user    2.00 B/s    0.00 B/s  100.00 %   /boot/system/servers/power_daemon
-----------------------------------------------------------------------
User: all
Processes: all
Only showing processes doing IOs.
Sorted according to 'Percentage of total IOs'.  Reverse Sorting: false
Refreshing every 3 seconds.

[bonefish]

Just a quick note: The accounting should really be done in the I/O scheduler. There it can not only be done per device, it is also possible to measure how much time the individual requests take. Otherwise I don't see how the I/O percentage number can be meaningful.

[Prasad]

Replying to bonefish:

Just a quick note: The accounting should really be done in the I/O scheduler. There it can not only be done per device, it is also possible to measure how much time the individual requests take. Otherwise I don't see how the I/O percentage number can be meaningful.

Accounting for IOs can be performed at various levels, for example: read, write system calls or, as you pointed out, when a an IO is submitted to the device.

IMHO, accounting stats at all the levels are interesting. Since this is the first version of the patch, I choose to do accounting at system call level. Further plan is to enhance it and account while IOs are submitted on to device.

[luroh]

Just an observation, this doesn't seem to build with gcc2, hrev44657:

Cc generated/objects/haiku/x86/release/bin/iotop.o 
src/bin/iotop.c: In function `compare_ios':
src/bin/iotop.c:482: warning: `rc' might be used uninitialized in this function
src/bin/iotop.c: In function `gather':
src/bin/iotop.c:581: parse error before `new_process'
src/bin/iotop.c:591: `new_process' undeclared (first use in this function)
src/bin/iotop.c:591: (Each undeclared identifier is reported only once
src/bin/iotop.c:591: for each function it appears in.)

[luroh]

Another small tip is that you can exclude your changes to build/jam/HaikuImage. Getting something into the default Haiku image is a two-stage rocket; first you iterate a patch until it gets accepted into the master branch, then follows a suggestion and discussion on the haiku-development mailinglist about whether to include it in the default image or to keep it in the source tree for the time being.

[anevilyak]

Replying to Prasad:

Accounting for IOs can be performed at various levels, for example: read, write system calls or, as you pointed out, when a an IO is submitted to the device.

The syscalls can't really report accurate/meaningful stats though, as they're completely unaware of a) any ancillary operations that are an indirect result of their particular request, and b) how many operations a given request actually resulted in. To illustrate, consider the case of creating/deleting files. The syscall layer has absolutely no idea as to how much I/O such an operation actually results in since that completely depends on what housekeeping metadata operations the underlying filesystem had to do. Likewise, writing to a file might well result in the filesystem having to allocate more blocks for it, which also incurs extra operations outside of the requested write. Furthermore, if the I/O scheduler batches reads/writes into a single call, the syscall interface wouldn't know that either. It also wouldn't be aware if the underlying filesystem/device is in fact a RAID or other form of distributed storage that resulted in duplicate I/O operations being dispatched to multiple devices. Given all these caveats, accounting via the syscall interface presents at best an incomplete picture, and at worst an outright misleading one, which renders it pretty much useless unless your only goal is to print some numbers.

[axeld]

In any case, thanks for the patch!

As pointed out already, there are a number of issues I'd like to point out:

• You must not enlarge the team_info structure -- this breaks binary compatibility. I'm afraid you'll have to find some other mechanism to pass on the data to userland.
• As others pointed out already, the syscall read/write don't measure I/O at all. They just measure how many bytes are read/written to that interface. For example, an application writing lots of data to the Terminal doesn't need to do any I/O, but will give a high ranking in your application. Same for an application that reads from /dev/zero. The data doesn't really deliver any useful information as far as I can see.
• How do you measure 100% I/O load?
• In C++ there is no need to denote an empty argument list with 'void'.
• Please use C++ style comments instead of C style comments.
• In Team: io_acc is not a good name. Why not just use io_accounting again? You could also instead use a general stats structure.
• The iotop application sports lots of small coding style violations, please consult ​https://www.haiku-os.org/development/coding-guidelines
• In fill_team_info() you change, but this should not be part of the same change:

  -    //info->uid = 
  -    //info->gid = 
  +    info->uid = team->effective_uid; 
  +    info->gid = team->effective_gid;
  

[Prasad]

Replying to luroh:

Another small tip is that you can exclude your changes to build/jam/HaikuImage. Getting something into the default Haiku image is a two-stage rocket; first you iterate a patch until it gets accepted into the master branch, then follows a suggestion and discussion on the haiku-development mailinglist about whether to include it in the default image or to keep it in the source tree for the time being.

Thanks luroh you have been very helpful.

[Prasad]

Replying to axeld:

In any case, thanks for the patch!

Thanks a lot for your comments.

As pointed out already, there are a number of issues I'd like to point out:

• You must not enlarge the team_info structure -- this breaks binary compatibility. I'm afraid you'll have to find some other mechanism to pass on the data to userland.

IMHO then, I will have to pass the complete io_accouting structure using the same mechanism as team_info. Would it be okay?

• As others pointed out already, the syscall read/write don't measure I/O at all. They just measure how many bytes are read/written to that interface. For example, an application writing lots of data to the Terminal doesn't need to do any I/O, but will give a high ranking in your application. Same for an application that reads from /dev/zero. The data doesn't really deliver any useful information as far as I can see.

I see the point you and others are making, I will work on accounting for real numbers. However, I still think it is useful to keep the system call information, which could be useful for debugging. We might add yet another utility which can print this debugging information on the terminal.

• How do you measure 100% I/O load?

100% is all the accumulated IOs in specific time frame.

• In C++ there is no need to denote an empty argument list with 'void'.

OK

• Please use C++ style comments instead of C style comments.

I think, I referred to other C files like ps.c and top.c, I guess they are using the C style comment. I followed the code there for formating.

• In Team: io_acc is not a good name. Why not just use io_accounting again? You could also instead use a general stats structure.

I will have a look. Thanks!

• The iotop application sports lots of small coding style violations, please consult ​https://www.haiku-os.org/development/coding-guidelines

Yes, Thanks!!!!!

• In fill_team_info() you change, but this should not be part of the same change:

  -    //info->uid = 
  -    //info->gid = 
  +    info->uid = team->effective_uid; 
  +    info->gid = team->effective_gid;
  

I had opened a BUG for this, however no one is looking into it (https://dev.haiku-os.org/ticket/8995)

[Prasad]

Replying to anevilyak:

Replying to Prasad:

Accounting for IOs can be performed at various levels, for example: read, write system calls or, as you pointed out, when a an IO is submitted to the device.

The syscalls can't really report accurate/meaningful stats though, as they're completely unaware of a) any ancillary operations that are an indirect result of their particular request, and b) how many operations a given request actually resulted in. To illustrate, consider the case of creating/deleting files. The syscall layer has absolutely no idea as to how much I/O such an operation actually results in since that completely depends on what housekeeping metadata operations the underlying filesystem had to do. Likewise, writing to a file might well result in the filesystem having to allocate more blocks for it, which also incurs extra operations outside of the requested write. Furthermore, if the I/O scheduler batches reads/writes into a single call, the syscall interface wouldn't know that either. It also wouldn't be aware if the underlying filesystem/device is in fact a RAID or other form of distributed storage that resulted in duplicate I/O operations being dispatched to multiple devices. Given all these caveats, accounting via the syscall interface presents at best an incomplete picture, and at worst an outright misleading one, which renders it pretty much useless unless your only goal is to print some numbers.

I understood the point you want to make, I will do necessary changes. I want to capture the IOs at process level and not at the device level, so that I can monitor the process hogging the devices.

Since this is the very first project I am doing with Haiku, I don't have enough information of Haiku source at the moment. I will send mail on the list if I get stuck.

[waddlesplash]

So ... it appears Linux iotop also does accounting at the syscall level, essentially (there is even a note that "these numbers include TTY operations", etc.) With that, it seems that this patch is actually not so far off of what people expect for an "iotop" operation. So -- axeld, bonefish, anevilyak: I'm not sure your comments are relevant here? (Not to mention there is no system-wide I/O scheduler.)

Perhaps a "devtop" command to show I/O to the devfs by thread may be more in line with what you are expecting. But the general idea behind this patch seems sound; it just needs to be cleaned up and then hooks added into the kernel read/write functions to track those too.

[diver]

0001-iotop-Add-an-iotop-utility.patch should probably be moved to Gerrit.

[X512]

I uploaded patch to Gerrit: ​https://review.haiku-os.org/c/haiku/+/3634. Compiles fine, but not tested yet.