一套Solaris上的9.2.0.7系统，实例意外终止，告警日志中出现以下记录:
提交SR后，MOS认为可能是swap空间不足导致，建议检查系统日志:"Please check your OS logs for any errors during this time.Also please check swap space. This error is sometimes recorded when swap space becomes full"。 系统日志显示在Sep 2出现过/tmp文件系统空间耗尽，swap空间不足的问题:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
Thu Sep 2 02:15:41 2010
Errors
in
file /u01/app/oracle/admin/preg063/bdump/preg063_smon_11391.trc:
ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CB3BF90] [] []
Thu Sep 2 02:15:48 2010
Errors
in
file /u01/app/oracle/admin/preg063/bdump/preg063_pmon_11379.trc:
ORA-00474: SMON process terminated
with
error
Thu Sep 2 02:15:48 2010
PMON: terminating instance due
to
error 474
Wed Sep 1 15:04:20 2010
Errors
in
file /u01/app/oracle/admin/preg063/udump/preg063_ora_1772.trc:
ORA-07445: exception encountered: core dump [FFFFFFFF7F400980] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CA78000] [] []
Wed Sep 1 15:06:24 2010
Errors
in
file /u01/app/oracle/admin/preg063/udump/preg063_ora_5316.trc:
ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C843DC8] [] []
Wed Sep 1 15:06:24 2010
Errors
in
file /u01/app/oracle/admin/preg063/udump/preg063_ora_5207.trc:
ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C887A70] [] []
Wed Sep 1 15:06:24 2010
Errors
in
file /u01/app/oracle/admin/preg063/udump/preg063_ora_28532.trc:
ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C9B7FE8] [] []
Wed Sep 1 15:06:31 2010
Errors
in
file /u01/app/oracle/admin/preg063/udump/preg063_ora_4392.trc:
ORA-07445: exception encountered: core dump [0000000100624600] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CA80000] [] []
Wed Sep 1 15:06:33 2010
Errors
in
file /u01/app/oracle/admin/preg063/udump/preg063_ora_3748.trc:
ORA-07445: exception encountered: core dump [FFFFFFFF7F400980] [SIGBUS] [Object specific hardware error]
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
|
trace日志:
Dump file /u01/app/oracle/admin/preg063/bdump/preg063_smon_11391.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With
the Partitioning, OLAP
and
Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /u01/app/oracle/product/920preg063
System
name
: SunOS
2010-09-02 02:15:41.917
*** SESSION ID:(7.1) 2010-09-02 02:15:33.090
Exception signal: 10 (SIGBUS), code: 3 (Object specific hardware error), addr: 0xffffffff7cb3bf90, PC: [0x101e05500, 0000000101E05500]
*** 2010-09-02 02:15:41.936
ksedmp: internal
or
fatal error
ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CB3BF90] [] []
Current
SQL statement
for
this session:
update
sys.col_usage$
set
equality_preds = equality_preds + decode(bitand(:flag,1),0,0,1), equijoin_preds = equijoin_preds + decode(bitand(:flag,2),0,0,1), nonequijoin_preds = nonequijoin_preds + decode(bitand(:flag,4),0,0,1), range_preds = range_preds + decode(bitand(:flag,8),0,0,1), like_preds = like_preds + decode(bitand(:flag,16),0,0,1), null_preds = null_preds + decode(bitand(:flag,32),0,0,1),
timestamp
= :
time
where
obj# = :objn
and
intcol# = :coln
----- Call Stack Trace -----
calling call entry argument
values
in
hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+328 CALL ksedst() 00000000B ? 000000000 ?
000000000 ? 103357D68 ?
00000003E ?
FFFFFFFF7FFE3EE8 ?
ssexhd()+676 CALL ksedmp() 000103788 ? 103788000 ?
103788C68 ? 10378D000 ?
000102C00 ? 000000000 ?
sigacthandler()+44 PTR_CALL 0000000000000000 000103790 ?
FFFFFFFF7FFEAF80 ?
103790000 ? 10378DE30 ?
000000000 ? 103790DB0 ?
kghbshrt()+96 PTR_CALL 0000000000000000 00000000A ?
FFFFFFFF7FFEAF80 ?
FFFFFFFF7FFEACA0 ?
000000000 ? 00000BF78 ?
0B38F0000 ?
kghalo()+612 CALL kghbshrt() 103788EC8 ? 000000000 ?
10378C2E0 ?
FFFFFFFF7CB30018 ?
000000458 ? 080000000 ?
kghgex()+336 CALL kghalo() 000000000 ?
FFFFFFFF7CB30018 ?
103788EC8 ? 000001000 ?
FFFFFFFF7CB3E120 ?
C0B38F000000C3D1 ?
kghalf()+316 CALL kghgex() 000007C00 ? 000000440 ?
000000228 ? 103788EC8 ?
000007FFF ? 000000000 ?
kdb4chk()+216 PTR_CALL 0000000000000000 103788EC8 ? 000000178 ?
FFFFFFFF7CB3E180 ?
FFFFFFFF7CB3E150 ?
000000000 ? 102DA35D0 ?
kd4chk()+88 CALL kdb4chk() 3F5484014 ? 1007E7780 ?
000000000 ? 000000000 ?
1007E76C0 ? 000000000 ?
kcbchk()+244 PTR_CALL 0000000000000000 3F5484000 ? 000000000 ?
000002000 ? 000000000 ?
1007E76C0 ? 1007E7720 ?
kcoapl()+1288 CALL kcbchk() 3F5484000 ? 0001007E7 ?
1007E7000 ? 101BA4440 ?
000100400 ? 0001007E7 ?
kcbapl()+96 CALL kcoapl() 000000002 ? 000000000 ?
000000001 ? 3F5484000 ?
10388DA08 ? 3F5486000 ?
kcrfwr()+1716 CALL kcbapl() 10388D990 ? 3F5FA6200 ?
FFFFFFFF7FFEC3B0 ?
00010378E ? 000000004 ?
000000004 ?
kcbchg1()+2056 CALL kcrfwr() FFFFFFFF7FFEC638 ?
40E36C218 ? 10378F5E8 ?
FFFFFFFF7FFEC3B0 ?
10378F690 ? 000000002 ?
ktuchg()+1044 CALL kcbchg1() 000000000 ?
FFFFFFFF7FFECE08 ?
000000000 ? 10378B530 ?
000000000 ?
FFFFFFFF7FFEC730 ?
ktbchg2()+132
Dump file /u01/app/oracle/admin/preg063/udump/preg063_ora_10110.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With
the Partitioning, OLAP
and
Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /u01/app/oracle/product/920preg063
System
name
: SunOS
2010-09-01 00:29:31.655
ksedmp: internal
or
fatal error
ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CAAC000] [] []
Current
SQL statement
for
this session:
select
lxOid,lxVal
from
lxString_c83b5454
where
lxOid
in
(:va,:vb,:vc,:vd,:ve,:vf,:vg,:vh,:vi,:vj,:vk,:vl,:vm,:vn,:vo,:vp,:vq,:vr,:vs,:vt,:vu,:vv,:vw,:vx,:vy,:vz,:v26,:v27,:v28,:v29,:v30,:v31,:v32,:v33,:v34,:v35,:v36,:v37,:v38,:v39,:v40,:v41,:v42,:v43,:v44,:v45,:v46,:v47,:v48,:v49,:v50,:v51,:v52,:v53,:v54,:v55,:v56,:v57,:v58,:v59)
and
lxType=:v60
----- Call Stack Trace -----
calling call entry argument
values
in
hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+328 CALL ksedst() 00000000B ? 000000000 ?
000000000 ? 103357D68 ?
00000003E ?
FFFFFFFF7FFF3B98 ?
ssexhd()+676 CALL ksedmp() 000103788 ? 103788000 ?
103788C68 ? 10378D000 ?
000102C00 ? 000000000 ?
sigacthandler()+44 PTR_CALL 0000000000000000 000103790 ?
FFFFFFFF7FFFAC30 ?
103790000 ? 10378DE30 ?
000000000 ? 103790DB0 ?
kghalf()+436 PTR_CALL 0000000000000000 00000000A ?
FFFFFFFF7FFFAC30 ?
FFFFFFFF7FFFA950 ?
103788EC8 ? 0B38F0000 ?
B38F0000000000 ?
qerhjManageListElem CALL kghalf() 103788EC8 ?
ent()+116 10B38F0000000001 ?
FFFFFFFF7CC73418 ?
FFFFFFFF7CC73418 ?
FFFFFFFF7CAABFF8 ?
1034CC568 ?
qerhjadf()+248 CALL qerhjManageListElem FFFFFFFF7CBA2568 ?
ent() FFFFFFFF7CC733A8 ?
000000002 ? 000000049 ?
FFFFFFFF7CA41E9A ?
0000000AF ?
qerhjBuildHashTable CALL qerhjadf() FFFFFFFF7FFFB0F8 ?
()+4236 FFFFFFFF7C9D2DB0 ?
000000013 ?
FFFFFFFF7C9D2D90 ?
FFFFFFFF7CA41E9A ?
000000001 ?
qerhjFetch()+772 CALL qerhjBuildHashTable 000000000 ? 1038698D8 ?
() 000000040 ?
FFFFFFFF7C9B16A8 ?
000000001 ?
FFFFFFFF7CBA2630 ?
qervwFetch()+148 PTR_CALL 0000000000000000 43C80CBB0 ? 101624E80 ?
FFFFFFFF7FFFB338 ?
00000001E ? 000000008 ?
00000005F ?
kpofrws()+124 PTR_CALL 0000000000000000 43C80CB58 ? 100FEF620 ?
FFFFFFFF7FFFB940 ?
00000001E ? 000101400 ?
000101624 ?
opifch2()+1704 CALL kpofrws() 000000004 ? 000000004 ?
100FEF620 ?
FFFFFFFF7FFFB940 ?
102EC9160 ? 000000010 ?
opiall0()+3900 CALL opifch2() 10101B000 ? 000000006 ?
|
|
1
2
3
4
5
6
|
Aug 15 02:41:25 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system
full
, swap
space
limit exceeded
Aug 15 02:41:25 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system
full
, swap
space
limit exceeded
Aug 15 02:48:06 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system
full
, swap
space
limit exceeded
Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system
full
, swap
space
limit exceeded
Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system
full
, swap
space
limit exceeded
Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system
full
, swap
space
limit exceeded
|
该内部错误通过增大swap最后解决了，that's great!
本文转自maclean_007 51CTO博客，原文链接:http://blog.51cto.com/maclean/1277651
