Toad World® Forums

Toad 10 - DBA is convinced that Toad 10 is causing db crashes


#1

I have a DBA who is convinced that Toad 10 is causing database crashes. He points to the error messages and that the trace logs show that Toad was causing the problem. We have 30+ Oracle databases and about 25 developers converted to Toad 10. I have posted the error below. Has anyone else had this error. Or can anyone help proving that Toad 10 is not responcible?
thanks
Ian Jackson

SQL*Plus: Release 9.0.1.4.0 - Production on Fri Mar 5 09:01:04 2010

© Copyright 2001 Oracle Corporation. All rights reserved.

ERROR:
ORA-00600: internal error code, arguments: [kqrfrpo], [0xFFFFFEF905F71428],
[11], [], [], [], [], []

Error accessing PRODUCT_USER_PROFILE
Warning: Product user profile information not loaded!
You may need to run PUPBLD.SQL as SYSTEM
ERROR:
ORA-00600: internal error code, arguments: [kqrfrpo], [0xFFFFFEF905F71428],
[11], [], [], [], [], []

Error accessing package DBMS_APPLICATION_INFO

Connected to:
Oracle9i Enterprise Edition Release 9.2.0.4.0 - Production
With the Partitioning option
JServer Release 9.2.0.4.0 - Production


#2

Have they trapped the SQL Toad executed that caused the 600? Toad is probably
exposing a database issue and is being shot as the messenger…

Your message below indicates that you can replicate via SQL*Plus? If
that’s the case, why do they think it’s Toad?


#3

ORA-600 errors are always server side problems – yes they can be exposed
by an application. So maybe a new version of an app issues a legit query that
causes the database to die. ORA-600 is a very special Oracle error message
number – as it’s a scenario that’s not yet been defined a
permanent error number and message. So the parameters you see in brackets with
the Ora-600 are usually special code that Oracle second level support can
troubleshoot from. For example there are some dozen or two database bugs in
10.2.0.1 that TOAD can expose and that is fixed by patches such as 10.2.0.4 and
10.2.0.4. That does not make it a toad bug – but rather your database has
a bug that someone has exposed and that the database vendor supplies a fix to.
In ora-600 case – almost universally these are bugs that require oracle
support (often second level) and a patch or patchset. As for the other errors
like product_user_profile – seems to me that your dba needs to set some
privs/grants and maybe run some scripts as SYS to properly setup the database.
This message for example is due to a SQL*Plus admin script that was not run.
That has nothing to do with any app like toad.


#4

Is the product_user_profile table and synonym OK? Have they tried running
pupbld.sql as the error message suggests? What happens if the users try to
connect using just SQL Plus (not when it is called from Toad)? If they have more
than one Oracle home, be sure to use the same one that Toad is using.


#5

Ian,

I have a DBA who is convinced that Toad 10 is causing
database crashes. He points to the error messages and that
the trace logs show that Toad was causing the problem.

With all due respect,

go around to your DBA and slap him! :wink:

have 30+ Oracle databases and about 25 developers converted
to Toad 10. I have posted the error below. Has anyone else
had this error. Or can anyone help proving that Toad 10 is
not responsible?
thanks

ERROR:
ORA-00600: internal error code, arguments: [kqrfrpo],
[0xFFFFFEF905F71428],
[11], [], [], [], [], []

An ORA-600 is an internal Oracle error that must be reported to Oracle
Support. The fact that Toad may be causing the appearance of said crash
is coincidental as a quick test in SQL*Plus would most likeley show the
same problem.

Looking up the cause of the above error shows :

Bug# 3835429 See Note:3835429.8
OERI[kqrfrpo] / DB hang after killing a user process
Fixed: 9.2.0.7

And Note 3835429.8 has this to say:

Description

If a user process is killed at an inopportune time then an
ORA-600 [kqrfrpo] may be signalled and the "row cache latch" may
be held indefinitely leading to a database wide hang.

Affects versions

HTH

Cheers,
Norm. [TeamT}

PS. Only kidding about slapping the DBA, however, he should know what
to do with an ORA 600 error.

N.

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else. We have checked this email and its attachments for viruses. But you should still check any attachment before opening it. We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes. If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.

If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk


#6

go around to your DBA and slap him! :wink:

THANKS for saying the obvious. I as a Quest person have to try to be a little
more PC (just a little). But you’re spot on – this is a no brainer.


#7

I like to think of ORA-600 as the Oracle equivalent of an access violation.


#8

go around to your DBA and slap him! :wink:

What we’re all thinking but too chicken to just come out and say J


#9

Hey Ian,

SQL*Plus: Release 9.0.1.4.0 - Production on Fri Mar 5 09:01:04 2010

[snip]

Connected to:
Oracle9i Enterprise Edition Release 9.2.0.4.0 - Production

The version mismatch between client and server may be worth noting. As
others have mentioned, the ORA-600 is a DB server issue, but you’re using a
client version older than the server, which is rarely a good thing.

My $.02,

Rich – [TeamT]

Disclaimer: I didn’t do it! It was the one-armed man!


#10

Thanks for the replies … I will pass them on. When we find out what is happening, I will post it.

Ian Jackson


#11

Hi Bert,

go around to your DBA and slap him! :wink:

THANKS for saying the obvious. I as a Quest person have to
try to be a little more PC (just a little). But you’re spot
on - this is a no brainer.

I thought, seeing it was Friday, that we could use a little levity! But
it is obvious to any DBA that ORA-600 is “report to support”.

As for PC, sorry, no can do. I call a spayed a cat with no b-lls! :wink:

Cheers,
Norm. [TeamVeryUnPC]

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else. We have checked this email and its attachments for viruses. But you should still check any attachment before opening it. We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes. If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.

If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk


#12

Hi Jeff,

go around to your DBA and slap him! :wink:

What we’re all thinking but too chicken to just come out and say J

I should be safe enough, I’m (hopefully) in a different Time Zone!
:wink:

Cheers,
Norm. [TeamT]

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else. We have checked this email and its attachments for viruses. But you should still check any attachment before opening it. We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes. If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.

If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk


#13

And you’re a customer who helps other customers as opposed to us guys who
work for all of you (i.e. you pay our bills) J


#14

So there we have it.
The DBA should be slapped, with all due respect, or maybe not so much respect, and TOAD at worst is the bearer of bad news.

There is that little question of what SQL was executing at the time of the ORA-00600 error. (Thank you HillbillyToad.)

*** 2010-03-05 09:46:33.508
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [kqrfrpo], [0xFFFFFEF905F71428], [11

No current SQL statement being executed.

Other trace files show various pieces of SQL were being executed.
At some point, every SQL statement generated ORA-00600 [kqrfrpo]

Hmmmm.

Were there some errors preceding the [kqrfrpo] errors?

4-MAR-2010 09:40:52.78:
Errors in file Nodename_DEV7_FG_ORACLE_031.TRC;:
ORA-00600: internal error code, arguments: [12333], [0], [0], [0], [], [], [],

What SQL was running when that trace file was created?

*** 2010-03-04 09:47:27.414
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [12333], [0], [0], [0], [], [], [],

Current SQL statement for this session:
Select CREATED, LAST_DDL_TIME, OBJECT_ID, STATUS, TIMESTAMP
FROM SYS.ALL_OBJECTS
WHERE object_name = :nm
AND object_type = :t
AND owner = :o

.
.
.
O/S info: user: USERNAME, term: PC###, ospid: 1724:2828, machine: DOMAIN\WS
program:
application name: TOAD 10.1.1.8, hash value=3248960949
last wait for ‘SQL*Net message from client’ blocking sess=0x0 seq=149 wait_
driver id=28444553, #bytes=1, =0

Then there was that mismatch between client software and database.
Now that was an interesting thought. (Thank you, Rich Jesse.)

Interestingly enough, that ties in with ORA-00600: [12333]

For more details, see Metalink Note: ORA-600 [12333] “Fatal Two-Task Protocol Violation” [ID 35928.1]

Silly DBA to accuse TOAD of anything.

How long has TOAD been in use in your environment? Ten years or more?
How long has the DBA used TOAD? Since just before Quest purchased it.

How frequently has this happened in the last two months? At least six times.
Has there been any pattern to the ORA-00600 errors?
Install TOAD 10.x - fails, maybe during the installation?
Try to run it with an old client -> errors. -> see [12333] in alert.log
Tracefile with a “select __ from sys.all_objects where object_name = :nm
and object_type = :t and owner = :o;”
Upgrade the client.
Start TOAD 10 - all is well until the database crashes sometime later with [kqrfrpo].

The Oracle version is very suspect.
Who would be using a 9.2.0.4 database when 11gR2 is out?
There couldn’t possibly be any business case for running on obsolete software.

By your replies, I take it that no one else has seen this type of error or the precursor
[12333].

That would likely take a specific combination of Client, Database and TOAD.
So maybe it is only the client, not TOAD 10. Apparently though, TOAD 9 did not have any such issue.

Have a great weekend, everyone.
Regards,
Nelson Petersen