-------------------
输入输出(I/O)瓶颈
-------------------
1. iostat
iostat命令用于统计CPU的使用情况及tty设备、硬盘和CD-ROM的I/0量。
运行命令：
iostat 5 10
将显示10次统计结果，下面为其中一次的数据：
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 102.2 0.2 0.7 99.0 0.1
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk0 0.4 33.8 2.6 0 172
hdisk1 0.0 2.4 0.6 0 12
cd0 0.0 0.0 0.0 0 0
其中
%iowait 等待本地 I/O 时CPU 空闲时间的百分比
%idle 未等待本地 I/O 时CPU 空闲时间的百分比
当没有需使用CPU的进程但至少有一个进程在等待I/O 时， CPU时间属性标为iowait。
如果iowait 时间的百分比很高，sample示该磁盘输入输出(I/O)是导致系统运行速度缓慢的主要原因。
%tm_act 硬盘繁忙的百分比
注: tm_act的值很高，表明硬盘存在I/O瓶颈。
当%tm_act(硬盘繁忙时间)很高时，可能会感觉到系统运行速度在减慢。有些系统上某个硬盘的%tm_act值为60%或更高时，系统性能就会受到影响。
两点建议：
1. 观察繁忙与空闲的硬盘，将数据从忙的盘移至相对空闲的的盘上，这会在一定程度上减轻由此引起的I/O 瓶颈。
2. 依照上面"内存瓶颈"中介绍的方法检查内存换页的频繁程度。大量的换页操作会增加I/O负载。
2. filemon
filemon 命令用于查看哪些文件/逻辑卷/硬盘处于繁忙状态。在系统I/O 处于繁忙状态时运行下面的命令：
#filemon -u -O all -o /tmp/fmon.out; sleep 30; trcstop
30秒钟后会生成记录跟踪信息的文件 /tmp/fmon.out。从该文件中可以找出：
1. 最为繁忙的虚拟内存段(segment)、逻辑卷和物理卷；
2. 对页空间(paging space)的读写次数，确认硬盘I/O 是由于应用程序操作还是频繁的换页操作；
3. 最为活跃的文件或逻辑卷，如果它们存放在繁忙的物理卷上，可以考虑将数据移至相对空闲的硬盘上，
这样有助于提高系统性能。最为繁忙的段的报sample里列示出最繁忙的文件所对应的文件系统和节点。
文件系统的安装点(mount point)及文件的i节点(inode)可与命令ncheck一起使用，来找出相对应的文件。
这个报sample可用来判断该I/O操作是针对文件系统、JFS Log 还是系统页空间的。
通过检查字段"reads"和"read sequences"的值，可以判断该操作是顺序读取还是随机读取。
当 "read sequences"接近"reads"时，对该文件的访问则以随机读取方式居多。
附filemon 的输出示例:
P650A:/#filemon -u -O all -o /tmp/fmon.out; sleep 30; trcstop
Mon May 16 15:51:49 2011
System: AIX 5.3 Node: P650A Machine: 00C3EE9E4C00
Run trcstop command to signal end of trace.
P650A:/#[filemon: Reporting started]
[filemon: Reporting completed]
[filemon: 30.013 secs in measured interval]
P650A:/#more /tmp/fmon.out
Mon May 16 15:51:49 2011
System: AIX 5.3 Node: P650A Machine: 00C3EE9E4C00
Cpu utilization: 53.9%
Cpu allocation: 97.1%
9185662 events were lost. Reported data may have inconsistencies or errors.
Most Active Files
------------------------------------------------------------------------
#MBs #opns #rds #wrs file volume:inode
------------------------------------------------------------------------
168.4 0 43111 0 pid=1511880_fd=0
0.8 3 204 0 unix /dev/hd2:37196
0.7 0 47 0 pid=405822_fd=9
0.4 0 25 0 pid=364696_fd=15
0.2 0 158 0 pid=7827780_fd=5
0.1 0 210 0 pid=1_fd=0
0.1 0 7 0 pid=381286_fd=13
0.1 11 22 0 ksh.cat /dev/hd2:66100
0.1 0 5 0 pid=364696_fd=16
0.1 0 41 1 pid=7852356_fd=13
0.1 0 4 0 pid=294986_fd=15
0.1 0 4 0 pid=405822_fd=10
0.1 0 15 0 pid=7885144_fd=0
0.0 0 20 3 pid=7876884_fd=13
0.0 6 7 0 vfs /dev/hd4:930
0.0 0 1 0 pid=294986_fd=16
0.0 3 3 0 ps.cat /dev/hd2:66341
0.0 1 2 0 cmdtrace.cat /dev/hd2:65708
0.0 0 2 0 pid=0_fd=0
0.0 3 1 1 dsmscoutd.lst
Most Active Segments
------------------------------------------------------------------------
#MBs #rpgs #wpgs segid segtype volume:inode
------------------------------------------------------------------------
0.1 0 28 170536 client
0.1 0 28 90588 client
0.1 0 15 30562 client
0.0 0 10 b056a client
0.0 2 0 1705b6 working
0.0 1 0 e05af working
0.0 1 0 d05ac working
0.0 0 1 c280d client
Most Active Logical Volumes
------------------------------------------------------------------------
util #rblk #wblk KB/s volume description
------------------------------------------------------------------------
0.79 0 648 10.8 /dev/oradatalv /oradata
0.51 0 88 1.5 /dev/hd8 jfs2log
0.00 32 0 0.5 /dev/hd6 paging
0.00 0 8 0.1 /dev/hd1 /home
Most Active Physical Volumes
------------------------------------------------------------------------
util #rblk #wblk KB/s volume description
------------------------------------------------------------------------
0.79 24 608 10.5 /dev/hdisk0 N/A
0.17 8 352 6.0 /dev/hdisk1 N/A
------------------------------------------------------------------------
Detailed File Stats
------------------------------------------------------------------------
FILE: pid=1511880_fd=0
opens: 0
total bytes xfrd: 176582656
reads: 43111 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.009 min 0.003 max 36.139 sdev 0.355
FILE: /unix volume: /dev/hd2 inode: 37196
opens: 3
total bytes xfrd: 835584
reads: 204 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.006 min 0.003 max 0.008 sdev 0.001
lseeks: 375
FILE: pid=405822_fd=9
opens: 0
total bytes xfrd: 770048
reads: 47 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.010 min 0.010 max 0.010 sdev 0.000
FILE: pid=364696_fd=15
opens: 0
total bytes xfrd: 409600
reads: 25 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.012 min 0.012 max 0.012 sdev 0.000
FILE: pid=7827780_fd=5
opens: 0
total bytes xfrd: 161792
reads: 158 (158 errs)
read sizes (bytes): avg 1024.0 min 1024 max 1024 sdev 0.0
read times (msec): avg 0.003 min 0.002 max 0.006 sdev 0.001
FILE: pid=1_fd=0
opens: 0
total bytes xfrd: 136080
reads: 210 (0 errs)
read sizes (bytes): avg 648.0 min 648 max 648 sdev 0.0
read times (msec): avg 0.003 min 0.002 max 0.009 sdev 0.001
lseeks: 418
FILE: pid=381286_fd=13
opens: 0
total bytes xfrd: 114688
reads: 7 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.013 min 0.013 max 0.013 sdev 0.000
FILE: /usr/lib/nls/msg/en_US/ksh.cat volume: /dev/hd2 inode: 66100
opens: 11
total bytes xfrd: 90112
reads: 22 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.008 min 0.004 max 0.024 sdev 0.006
lseeks: 70
FILE: pid=364696_fd=16
opens: 0
total bytes xfrd: 81920
reads: 5 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.011 min 0.011 max 0.011 sdev 0.000
FILE: pid=7852356_fd=13
opens: 0
total bytes xfrd: 71272
reads: 41 (0 errs)
read sizes (bytes): avg 1734.6 min 221 max 2064 sdev 541.6
read times (msec): avg 0.042 min 0.003 max 0.271 sdev 0.065
writes: 1 (0 errs)
write sizes (bytes): avg 152.0 min 152 max 152 sdev 0.0
write times (msec): avg 0.017 min 0.017 max 0.017 sdev 0.000
FILE: pid=294986_fd=15
opens: 0
total bytes xfrd: 65536
reads: 4 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.009 min 0.009 max 0.009 sdev 0.000
FILE: pid=405822_fd=10
opens: 0
total bytes xfrd: 65536
reads: 4 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.007 min 0.007 max 0.007 sdev 0.000
FILE: pid=7885144_fd=0
opens: 0
total bytes xfrd: 61440
reads: 15 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.004 min 0.003 max 0.006 sdev 0.001
FILE: pid=7876884_fd=13
opens: 0
total bytes xfrd: 35230
reads: 20 (0 errs)
read sizes (bytes): avg 1738.7 min 551 max 2064 sdev 555.2
read times (msec): avg 1157.846 min 0.004 max 19000.089 sdev 4258.232
writes: 3 (0 errs)
write sizes (bytes): avg 152.0 min 152 max 152 sdev 0.0
write times (msec): avg 0.017 min 0.014 max 0.022 sdev 0.004
FILE: /etc/vfs volume: /dev/hd4 inode: 930
opens: 6
total bytes xfrd: 28672
reads: 7 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.005 min 0.002 max 0.008 sdev 0.002
FILE: pid=294986_fd=16
opens: 0
total bytes xfrd: 16384
reads: 1 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.007 min 0.007 max 0.007 sdev 0.000
FILE: /usr/lib/nls/msg/en_US/ps.cat volume: /dev/hd2 inode: 66341
opens: 3
total bytes xfrd: 12288
reads: 3 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.010 min 0.009 max 0.012 sdev 0.001
lseeks: 12
FILE: /usr/lib/nls/msg/en_US/cmdtrace.cat volume: /dev/hd2 inode: 65708
opens: 1
total bytes xfrd: 8192
reads: 2 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.007 min 0.006 max 0.007 sdev 0.000
lseeks: 8
FILE: pid=0_fd=0
opens: 0
total bytes xfrd: 8192
reads: 2 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 86.036 min 79.036 max 93.037 sdev 7.000
lseeks: 10
FILE: /tmp/dsmscoutd.lst
opens: 3
total bytes xfrd: 7757
reads: 1 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.004 min 0.004 max 0.004 sdev 0.000
writes: 1 (0 errs)
write sizes (bytes): avg 3661.0 min 3661 max 3661 sdev 0.0
write times (msec): avg 0.010 min 0.010 max 0.010 sdev 0.000
------------------------------------------------------------------------
Detailed VM Segment Stats (4096 byte pages)
------------------------------------------------------------------------
SEGMENT: 170536 segtype: client
segment flags: clnt
writes: 28 (0 errs)
write times (msec): avg 512.575 min 4.273 max 3025.921 sdev 1124.006
write sequences: 7
write seq. lengths: avg 4.0 min 4 max 4 sdev 0.0
SEGMENT: 90588 segtype: client
segment flags: clnt
writes: 28 (0 errs)
write times (msec): avg 511.387 min 3.031 max 3024.705 sdev 1123.995
write sequences: 7
write seq. lengths: avg 4.0 min 4 max 4 sdev 0.0
SEGMENT: 30562 segtype: client
segment flags: clnt
writes: 15 (0 errs)
write times (msec): avg 8.398 min 8.385 max 8.408 sdev 0.007
write sequences: 1
write seq. lengths: avg 15.0 min 15 max 15 sdev 0.0
SEGMENT: b056a segtype: client
segment flags: clnt
writes: 10 (0 errs)
write times (msec): avg 7.167 min 3.844 max 8.103 sdev 1.259
write sequences: 3
write seq. lengths: avg 3.3 min 1 max 5 sdev 1.7
SEGMENT: 1705b6 segtype: working
segment flags: work
reads: 2 (0 errs)
read times (msec): avg 5.557 min 5.021 max 6.094 sdev 0.536
read sequences: 1
read seq. lengths: avg 2.0 min 2 max 2 sdev 0.0
SEGMENT: e05af segtype: working
segment flags: work
reads: 1 (0 errs)
read times (msec): avg 6.231 min 6.231 max 6.231 sdev 0.000
read sequences: 1
read seq. lengths: avg 1.0 min 1 max 1 sdev 0.0
SEGMENT: d05ac segtype: working
segment flags: work
reads: 1 (0 errs)
read times (msec): avg 6.172 min 6.172 max 6.172 sdev 0.000
read sequences: 1
read seq. lengths: avg 1.0 min 1 max 1 sdev 0.0
SEGMENT: c280d segtype: client
segment flags: clnt
writes: 1 (0 errs)
write times (msec): avg 8.514 min 8.514 max 8.514 sdev 0.000
write sequences: 1
write seq. lengths: avg 1.0 min 1 max 1 sdev 0.0
------------------------------------------------------------------------
Detailed Logical Volume Stats (512 byte blocks)
------------------------------------------------------------------------
VOLUME: /dev/oradatalv description: /oradata
writes: 18 (0 errs)
write sizes (blks): avg 36.0 min 8 max 120 sdev 21.2
write times (msec): avg 8.575 min 3.017 max 14.165 sdev 3.649
write sequences: 18
write seq. lengths: avg 36.0 min 8 max 120 sdev 21.2
seeks: 18 (100.0%)
seek dist (blks): init 63840,
avg 83666530.4 min 23712 max 178591480 sdev 88716020.7
time to next req(msec): avg 1523.929 min 0.002 max 11804.280 sdev 2894.606
throughput: 10.8 KB/sec
utilization: 0.79
VOLUME: /dev/hd8 description: jfs2log
writes: 11 (0 errs)
write sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
write times (msec): avg 1014.631 min 2.864 max 10091.925 sdev 3025.765
write sequences: 9
write seq. lengths: avg 9.8 min 8 max 24 sdev 5.0
seeks: 9 (81.8%)
seek dist (blks): init 494280,
avg 8.0 min 8 max 8 sdev 0.0
time to next req(msec): avg 2265.352 min 5.804 max 10045.696 sdev 3589.596
throughput: 1.5 KB/sec
utilization: 0.51
VOLUME: /dev/hd6 description: paging
reads: 4 (0 errs)
read sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
read times (msec): avg 5.875 min 5.017 max 6.228 sdev 0.498
read sequences: 4
read seq. lengths: avg 8.0 min 8 max 8 sdev 0.0
seeks: 4 (100.0%)
seek dist (blks): init 2389536,
avg 5036200.0 min 1713808 max 11610392 sdev 4648745.1
time to next req(msec): avg 1762.830 min 4.373 max 7010.239 sdev 3029.610
throughput: 0.5 KB/sec
utilization: 0.00
VOLUME: /dev/hd1 description: /home
writes: 1 (0 errs)
write sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
write times (msec): avg 8.500 min 8.500 max 8.500 sdev 0.000
write sequences: 1
write seq. lengths: avg 8.0 min 8 max 8 sdev 0.0
seeks: 1 (100.0%)
seek dist (blks): init 957512
time to next req(msec): avg 24910.354 min 24910.354 max 24910.354 sdev 0.000
throughput: 0.1 KB/sec
utilization: 0.00
------------------------------------------------------------------------
Detailed Physical Volume Stats (512 byte blocks)
------------------------------------------------------------------------
VOLUME: /dev/hdisk0 description: N/A
reads: 3 (0 errs)
read sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
read times (msec): avg 5.757 min 5.000 max 6.213 sdev 0.539
read sequences: 3
read seq. lengths: avg 8.0 min 8 max 8 sdev 0.0
writes: 34 (0 errs)
write sizes (blks): avg 17.9 min 8 max 32 sdev 11.8
write times (msec): avg 9.867 min 1.498 max 49.412 sdev 9.138
write sequences: 33
write seq. lengths: avg 18.4 min 8 max 32 sdev 11.7
seeks: 36 (97.3%)
seek dist (blks): init 117212960,
avg 10718459.2 min 8 max 52186200 sdev 18350570.0
seek dist (%tot blks):init 40.87664,
avg 3.73794 min 0.00000 max 18.19932 sdev 6.39955
time to next req(msec): avg 945.888 min 0.003 max 10045.698 sdev 2098.774
throughput: 10.5 KB/sec
utilization: 0.79
VOLUME: /dev/hdisk1 description: N/A
reads: 1 (0 errs)
read sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
read times (msec): avg 6.149 min 6.149 max 6.149 sdev 0.000
read sequences: 1
read seq. lengths: avg 8.0 min 8 max 8 sdev 0.0
writes: 23 (0 errs)
write sizes (blks): avg 15.3 min 8 max 120 sdev 23.7
write times (msec): avg 7.579 min 1.715 max 46.186 sdev 9.665
write sequences: 23
write seq. lengths: avg 15.3 min 8 max 120 sdev 23.7
seeks: 24 (100.0%)
seek dist (blks): init 23402504,
avg 36368105.7 min 8 max 108692392 sdev 48254032.5
seek dist (%tot blks):init 8.16135,
avg 12.68295 min 0.00000 max 37.90520 sdev 16.82803
time to next req(msec): avg 1465.817 min 5.801 max 9765.621 sdev 2781.132
throughput: 6.0 KB/sec
utilization: 0.17
P650A:/#
本文转自zylhsy 51CTO博客，原文链接：http://blog.51cto.com/yunlongzheng/566519，如需转载请自行联系原作者
