Opened 7 years ago

Last modified 3 years ago

#9019 new enhancement

add iotop utility

Reported by: Prasad Owned by: nobody
Priority: normal Milestone: Unscheduled
Component: Applications/Command Line Tools Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

iotop is a simple top like utility for monitoring IOs. For utility to work, kernel must do IO accounting for each process. The patch adds struct io_accounting in the kernel account for IOs performed by each process.

The accounting for IO is performed in various version of read and write system calls.

The userland part (iotop command) uses team_info mechanism to get the accounting information in the userland. The utility provides user with lots of options like

  • monitoring processes of particular user
  • monitoring processes based on process ids
  • only showing processes doing actual IOs
  • configuration refreshing delay

iotop also allows user to sort the output on various parameters dynamically while running.

The utility also checks the visible terminal co-ordinates and updates the output accordingly.

Here is a sample output of the iotop: ~> ./iotop

Total Disk Reads: 2.00 B/s | Total Disk Writes: 0.00 B/s

TID USER READ BW WRITE BW IO % Command 114 user 2.00 B/s 0.00 B/s 100.00 % /boot/system/servers/power_daemon

48 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/registrar 55 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/debug_server 56 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/net_server 57 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/app_server 72 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/syslog_daemon 84 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/input_server 94 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/mount_server

106 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/Tracker 107 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/Deskbar 108 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/media_server 109 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/midi_server 110 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/print_server 112 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/cddb_daemon 113 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/notification_server 143 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/servers/media_addon_server 193 user 0.00 B/s 0.00 B/s 0.00 % /boot/system/apps/Terminal 197 user 0.00 B/s 0.00 B/s 0.00 % /bin/bash -l

1 user 0.00 B/s 0.00 B/s 0.00 % kernel_team

212 user 0.00 B/s 0.00 B/s 0.00 % ./iotop


User: all Processes: all Sorted according to 'Percentage of total IOs'. Reverse Sorting: false Refreshing every 3 seconds.

~> ./iotop -o

Total Disk Reads: 2.00 B/s | Total Disk Writes: 0.00 B/s

TID USER READ BW WRITE BW IO % Command 114 user 2.00 B/s 0.00 B/s 100.00 % /boot/system/servers/power_daemon


User: all Processes: all Only showing processes doing IOs. Sorted according to 'Percentage of total IOs'. Reverse Sorting: false Refreshing every 3 seconds.

Attachments (1)

0001-iotop-Add-an-iotop-utility.patch (24.3 KB) - added by Prasad 7 years ago.

Download all attachments as: .zip

Change History (15)

comment:1 Changed 7 years ago by Prasad

Has a Patch: set

comment:2 Changed 7 years ago by Prasad

Here is a sample output of the iotop:

~> ./iotop

Total Disk Reads:    2.00 B/s | Total Disk Writes:    0.00 B/s

    TID     USER     READ BW    WRITE BW      IO %   Command
    114     user    2.00 B/s    0.00 B/s  100.00 %   /boot/system/servers/power_daemon
     48     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/registrar
     55     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/debug_server
     56     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/net_server
     57     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/app_server
     72     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/syslog_daemon
     84     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/input_server
     94     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/mount_server
    106     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/Tracker
    107     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/Deskbar
    108     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/media_server
    109     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/midi_server
    110     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/print_server
    112     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/cddb_daemon
    113     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/notification_server
    143     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/servers/media_addon_server
    193     user    0.00 B/s    0.00 B/s    0.00 %   /boot/system/apps/Terminal
    197     user    0.00 B/s    0.00 B/s    0.00 %   /bin/bash -l
      1     user    0.00 B/s    0.00 B/s    0.00 %   kernel_team
    212     user    0.00 B/s    0.00 B/s    0.00 %   ./iotop
-----------------------------------------------------------------------
User: all
Processes: all
Sorted according to 'Percentage of total IOs'.  Reverse Sorting: false
Refreshing every 3 seconds.

~> ./iotop -o

Total Disk Reads:    2.00 B/s | Total Disk Writes:    0.00 B/s

    TID     USER     READ BW    WRITE BW      IO %   Command
    114     user    2.00 B/s    0.00 B/s  100.00 %   /boot/system/servers/power_daemon
-----------------------------------------------------------------------
User: all
Processes: all
Only showing processes doing IOs.
Sorted according to 'Percentage of total IOs'.  Reverse Sorting: false
Refreshing every 3 seconds.

Changed 7 years ago by Prasad

comment:3 Changed 7 years ago by bonefish

Just a quick note: The accounting should really be done in the I/O scheduler. There it can not only be done per device, it is also possible to measure how much time the individual requests take. Otherwise I don't see how the I/O percentage number can be meaningful.

comment:4 Changed 7 years ago by Prasad

Last edited 7 years ago by Prasad (previous) (diff)

comment:5 in reply to:  3 ; Changed 7 years ago by Prasad

Replying to bonefish:

Just a quick note: The accounting should really be done in the I/O scheduler. There it can not only be done per device, it is also possible to measure how much time the individual requests take. Otherwise I don't see how the I/O percentage number can be meaningful.

Accounting for IOs can be performed at various levels, for example: read, write system calls or, as you pointed out, when a an IO is submitted to the device.

IMHO, accounting stats at all the levels are interesting. Since this is the first version of the patch, I choose to do accounting at system call level. Further plan is to enhance it and account while IOs are submitted on to device.

comment:6 Changed 7 years ago by luroh

Just an observation, this doesn't seem to build with gcc2, hrev44657:

Cc generated/objects/haiku/x86/release/bin/iotop.o 
src/bin/iotop.c: In function `compare_ios':
src/bin/iotop.c:482: warning: `rc' might be used uninitialized in this function
src/bin/iotop.c: In function `gather':
src/bin/iotop.c:581: parse error before `new_process'
src/bin/iotop.c:591: `new_process' undeclared (first use in this function)
src/bin/iotop.c:591: (Each undeclared identifier is reported only once
src/bin/iotop.c:591: for each function it appears in.)

comment:7 Changed 7 years ago by luroh

Another small tip is that you can exclude your changes to build/jam/HaikuImage. Getting something into the default Haiku image is a two-stage rocket; first you iterate a patch until it gets accepted into the master branch, then follows a suggestion and discussion on the haiku-development mailinglist about whether to include it in the default image or to keep it in the source tree for the time being.

comment:8 in reply to:  5 ; Changed 7 years ago by anevilyak

Replying to Prasad:

Accounting for IOs can be performed at various levels, for example: read, write system calls or, as you pointed out, when a an IO is submitted to the device.

The syscalls can't really report accurate/meaningful stats though, as they're completely unaware of a) any ancillary operations that are an indirect result of their particular request, and b) how many operations a given request actually resulted in. To illustrate, consider the case of creating/deleting files. The syscall layer has absolutely no idea as to how much I/O such an operation actually results in since that completely depends on what housekeeping metadata operations the underlying filesystem had to do. Likewise, writing to a file might well result in the filesystem having to allocate more blocks for it, which also incurs extra operations outside of the requested write. Furthermore, if the I/O scheduler batches reads/writes into a single call, the syscall interface wouldn't know that either. It also wouldn't be aware if the underlying filesystem/device is in fact a RAID or other form of distributed storage that resulted in duplicate I/O operations being dispatched to multiple devices. Given all these caveats, accounting via the syscall interface presents at best an incomplete picture, and at worst an outright misleading one, which renders it pretty much useless unless your only goal is to print some numbers.

comment:9 Changed 7 years ago by axeld

In any case, thanks for the patch!

As pointed out already, there are a number of issues I'd like to point out:

  • You must not enlarge the team_info structure -- this breaks binary compatibility. I'm afraid you'll have to find some other mechanism to pass on the data to userland.
  • As others pointed out already, the syscall read/write don't measure I/O at all. They just measure how many bytes are read/written to that interface. For example, an application writing lots of data to the Terminal doesn't need to do any I/O, but will give a high ranking in your application. Same for an application that reads from /dev/zero. The data doesn't really deliver any useful information as far as I can see.
  • How do you measure 100% I/O load?
  • In C++ there is no need to denote an empty argument list with 'void'.
  • Please use C++ style comments instead of C style comments.
  • In Team: io_acc is not a good name. Why not just use io_accounting again? You could also instead use a general stats structure.
  • The iotop application sports lots of small coding style violations, please consult https://www.haiku-os.org/development/coding-guidelines
  • In fill_team_info() you change, but this should not be part of the same change:
    -    //info->uid = 
    -    //info->gid = 
    +    info->uid = team->effective_uid; 
    +    info->gid = team->effective_gid;
    

comment:10 in reply to:  7 Changed 7 years ago by Prasad

Replying to luroh:

Another small tip is that you can exclude your changes to build/jam/HaikuImage. Getting something into the default Haiku image is a two-stage rocket; first you iterate a patch until it gets accepted into the master branch, then follows a suggestion and discussion on the haiku-development mailinglist about whether to include it in the default image or to keep it in the source tree for the time being.

Thanks luroh you have been very helpful.

comment:11 in reply to:  9 Changed 7 years ago by Prasad

Replying to axeld:

In any case, thanks for the patch!

Thanks a lot for your comments.

As pointed out already, there are a number of issues I'd like to point out:

  • You must not enlarge the team_info structure -- this breaks binary compatibility. I'm afraid you'll have to find some other mechanism to pass on the data to userland.

IMHO then, I will have to pass the complete io_accouting structure using the same mechanism as team_info. Would it be okay?

  • As others pointed out already, the syscall read/write don't measure I/O at all. They just measure how many bytes are read/written to that interface. For example, an application writing lots of data to the Terminal doesn't need to do any I/O, but will give a high ranking in your application. Same for an application that reads from /dev/zero. The data doesn't really deliver any useful information as far as I can see.

I see the point you and others are making, I will work on accounting for real numbers. However, I still think it is useful to keep the system call information, which could be useful for debugging. We might add yet another utility which can print this debugging information on the terminal.

  • How do you measure 100% I/O load?

100% is all the accumulated IOs in specific time frame.

  • In C++ there is no need to denote an empty argument list with 'void'.

OK

  • Please use C++ style comments instead of C style comments.

I think, I referred to other C files like ps.c and top.c, I guess they are using the C style comment. I followed the code there for formating.

  • In Team: io_acc is not a good name. Why not just use io_accounting again? You could also instead use a general stats structure.

I will have a look. Thanks!

Yes, Thanks!!!!!

  • In fill_team_info() you change, but this should not be part of the same change:
    -    //info->uid = 
    -    //info->gid = 
    +    info->uid = team->effective_uid; 
    +    info->gid = team->effective_gid;
    

I had opened a BUG for this, however no one is looking into it (https://dev.haiku-os.org/ticket/8995)

comment:12 in reply to:  8 Changed 7 years ago by Prasad

Replying to anevilyak:

Replying to Prasad:

Accounting for IOs can be performed at various levels, for example: read, write system calls or, as you pointed out, when a an IO is submitted to the device.

The syscalls can't really report accurate/meaningful stats though, as they're completely unaware of a) any ancillary operations that are an indirect result of their particular request, and b) how many operations a given request actually resulted in. To illustrate, consider the case of creating/deleting files. The syscall layer has absolutely no idea as to how much I/O such an operation actually results in since that completely depends on what housekeeping metadata operations the underlying filesystem had to do. Likewise, writing to a file might well result in the filesystem having to allocate more blocks for it, which also incurs extra operations outside of the requested write. Furthermore, if the I/O scheduler batches reads/writes into a single call, the syscall interface wouldn't know that either. It also wouldn't be aware if the underlying filesystem/device is in fact a RAID or other form of distributed storage that resulted in duplicate I/O operations being dispatched to multiple devices. Given all these caveats, accounting via the syscall interface presents at best an incomplete picture, and at worst an outright misleading one, which renders it pretty much useless unless your only goal is to print some numbers.

I understood the point you want to make, I will do necessary changes. I want to capture the IOs at process level and not at the device level, so that I can monitor the process hogging the devices.

Since this is the very first project I am doing with Haiku, I don't have enough information of Haiku source at the moment. I will send mail on the list if I get stuck.

comment:13 Changed 4 years ago by pulkomandy

Milestone: R1Unscheduled

comment:14 Changed 3 years ago by pulkomandy

Has a Patch: unset
Note: See TracTickets for help on using tickets.