We use a couple of products to dynamically feed data to our 10.2.0.5 database and it seemed the performance degraded  over time (many days). We did a 10046 session trace and found we lost a lot of time between the dbms_sql.parse of some dynamic SQL and the bind phase. We found reference in Metalink to Bug 10269717 (DBMS_SQL.PARSE leaks session heap memory in 10.2.0.5).  Although the bug discusses a memory leak, we found that the performance also degrades over time.

We applied the patch for 10269717 and the PGA memory leak was resolved but more importantly the performance remained constant. I just checked the 10.2.0.5 known issues document and this does now list the bug, but it still only references the leak and not the performance implications.

My site introduced a “parallel scheduler” which allows us to break some of our business transactions into parallel jobs. It simply manages the running of some time critical  business tasks in parallel but takes full control of the business rules and co-ordinates that all the tasks are complete, verified and handles the rules if parts fail to complete. The program runs itself as a continually running dbms_scheduler job and then schedules the worker threads (within various rules) then keeps careful track of the completion status of each part.

The Problem

One day our scheduler program failed with ORA-4030 (“out of process memory when trying to allocate %s bytes (%s,%s)”). We re-started the job, but further investigation showed that the job process which had previously been running our scheduler (which was still attached)  although now showing a low “session pga memory” had at some point had “session pga memory max” of 4GB. I set up a monitor to collect the “session pga memory” of the re-started job and left it collecting for a few days. When I plotted the PGA memory data we could clearly see the PGA memory appeared to grow during busy periods and not at all at off peak times but importantly never reduced. I sent the memory usage graph to a colleague and after a short while, he sent me back a graph which looked 100% the same as mine……except his graph had a totally different scale and was not memory. The graph he sent me was actually the total number of tasks our scheduler processes was asked to run in the same time period. So we knew we appeared to “leak” a fixed amount of PGA every time we ran a scheduled job.

Investigation

After some further investigation (dumping “session pga memory” after every call on a test system) we found the problem was not the submission of the job, but after the job had completed, we formally make a call to dbms_scheduler.drop_job and this call “leaks” approximately 21k every call. NOTE: I’ve used the term “leak” a lot and strictly speaking this is not a leak. Oracle knows about all about the memory and when your plsql package completes all the PGA memory is returned. The problem is Oracle does not free the memory during the execution of the main plsql procedure.

Bug

We have raised a new bug with Oracle:  9957867 – EXECUTING DBMS_SCHEDULER.DROP_JOB REPEATEDLY IN A PLSQL PROCEDURE LEAKS MEMORY which is still under investigation by Oracle Support. I’ll post an update if I get anymore information.

Work Around

This problem will really only cause problems if you schedule and then explicitly drop (dbms_scheduler.drop_job)  a lot of jobs in the same plsql loop without completing the procedure.  The dbms scheduler will by default create jobs with auto_drop=true so again this is only a problem if you create jobs with auto_drop=false because you want to take full control. We have a number of options which my site will consider (Change our code and use auto_drop=true, move the drop_job to another scheduled task which does restart from time to time or finally restart our scheduler task from time to time.

The conditions required to cause the issue

My site has a number of daemon style jobs running permanently on the database loading data into a message table. The daemon is managed by dbms_job and is re-started when the database is first started and can run for many days, weeks (or in some cases months) before the job is ever stopped. We only need to keep the messages for a short time, so we have another daemon job whose role is to delete the messages from the table as soon as the expiry time is reached. In one example we only need to retain the data for a few minutes (after which time we no longer need it) and we also wanted to keep the table as small as possible so it remained cached in the buffer cache (helped by a KEEP pool). When we developed the code, we expected the message table to remain at a fairly constant size of 50 – 100MB in size. What we found was the table continued to grow at a consistent rate to many gigabytes in size until we stopped the test. If we copied the table (eg using “create table <new table> as select * from <current table>”) the new table would again be 50MB so most of the space in the table was empty “deleted” space. The INSERT statements were never re-using the space made free by the delete statement (run in another session).

The problem

What we found was the INSERT statement could not reuse the space which was freed by the other session (which was deleting space). We played around with a number of different configurations and what we found was when we created our table in a Manually Managed Tablespace, the application worked perfectly and the table remained at a constant <100MB size for many hours. We spoke to a contact within Oracle and it seems that there was some optimization added to INSERT statements to reduce the time lost in re-evaluating the ASSM free space map. If the session remains connected and does not reparse the INSERT statement, Oracle will continue to use a static (stale) space map.

Not a problem in most applications

In most cases, this “bug” (Still not sure if I should call it a “bug” or “feature”) does not cause any problem because when lots of sessions are running, connecting, disconnecting and re-parsing over the life of your application they would be able to pick up a fresh “space map” and be able to re-use any “deleted” available blocks. It may account for a small overhead in the size of the tables and indexes but suspect you would never really notice it. Worth remembering that this is caused by optimization to make INSERT statements faster.

My site

My site has seen a couple of variations of the problem.  Firstly the table we expected to be very small  (example above), but the other was a table we expected to be large because the table held 7 days data and we had a nightly job which deleted data older than 7 days but, it seems we were having the same issue. We expected the table to be large so never noticed it was twice the size it should have been. As with the other example, the larger table was loaded using a single dedicated job which stayed running for weeks at a time. Only when the application was restarted, would the spaced freed from previous delete operations now be available for reuse. The issue causes the  High Water Mark of the table (suspect indexes too) to grow to the largest amount of time which the INSERT daemon runs without being restarted. Once the daemon is restart, it would could continue to re-use freed blocks for many weeks without the table/indexes growing at all until such a time that all the remaining blocks are filled and then the object would start to grow again (ignoring newly created free blocks)

Solution:

The work-around my site uses is when we know we are loading tables by deamons which will constantly run and that same table will be deleted at a fixed rate (by other session(s)), we just place the table and related indexes in MANUALLY MANAGED tablespaces (“SEGMENT SPACE MANAGEMENT MANUAL”)

The 10.2.0.5 patchset has also back-ported the same direct read feature which is new to 11g although I don’t know if the rules are the same as 11g. The site where I work makes significant use of KEEP pools and also has spent some time investigating aspects relating to serial direct IO vs. buffer cache IO. 

 I want to use this blog entry to explore a number of related issues but also demonstrate that the 11g bug Jonathan identified seems to also exists in 10.2.0.5 patchset (and 11gR2). This blog item will cover:

• Brief reference relating to the 11g “adaptive serial direct path read”
• The 10.2.0.5 implementation and how to switch it on
• 10.2.0.5 demonstration showing the relative difference for different types of IO vs read from cache
• 10.2.0.5 demonstration which shows the KEEP pool bug also exists (but not by default)
• Some real life comparison figures of disk reads via “direct path read” and via “buffer cache” to show why the ”adaptive serial direct path read” feature is worth exploring in more detail.   

11g adaptive direct path reads

There are already some good blog notes out there (follow the links on Jonathans and Alex Fatkulin links at the top)  but just wanted to say a couple of things.

• it’s ON by default, but can be switched off with database event (Event 10949 at level 1)
• Patch for  8897574 is now available for 11.1.0.7 (Linux x86-64) to resolve the issue with KEEP pools. However, there is a clash of objects on the latest 11.1.0.7 PSU so a merge patch would be required.
• I have also tested the bug on 11gR2 and the problem also still exists but no 11gR2 patch is available yet
• **UPDATE** 8/10/2010 Please read note at end of this page blog item**
• More information on Article ID 1081553.1

10.2.0.5

I am going to give a simple demo of using the serial direct read feature implemented in 10.2.0.5 but also use the example to highlight some of the pros and cons. I also hope to show that the same 11g KEEP pool bug exists in 10.2.0.5 and may also look at raising a back port request to get this addressed at some stage.

The 10.2.0.5 version of the Serial Direct Path Read features is NOT ENABLED by default meaning there should not be any issues (including the KEEP pool) unless someone explicitly decided to enable it. The feature is enabled by the use of Event (10358 is set to level 4).

NOTE: The Event to disable the feature in 11g is NOT the same event/level to enable it in 10.2.0.5

Details of my test database

I have installed 10.2.0.5 and created a default database (using dbca) on a test Linux (x86-64) server using Netapp nfs mounted storage. The hardware is not setup for high end performance but it is able to demonstrate the relative difference of various IO. I have set filesystemio_options  to “setall” to ensure we can use asynchronous IO (when required) and also direct IO (also avoids IO from the file system cache which skew the results). I have manually set a db_cache_size of 3GB and db_keep_cache_size  of 1GB. I have loaded a table of approximate 775MB which I will use to test.  The table is larger than 10% of the buffer cache, but can fit in both the default cache and the keep pool. fyi, it’s a 8k block size.

There is more information on enabling the feature in Metalink note: 8332368.8. Bug 8332368 – “Enable automatic serial direct read under an event” is actually the code change which implements the new feature in 10.2.0.5.

About the test script

note: This test should only be run on 10.2.0.5 as it uses 10.2.0.5 specific database events (remove the “alter sessions set events” if running on any other version and that  will also effect the results. The script also contains a flush of the buffer cache so watch out!

The script is included as a link to reduce the blog size.  Included as a template for doing your own testing. 

Test against the DEFAULT pool using 10.2.0.5

Lets run my test script against the 775MB table which is held in the DEFAULT buffer pool and discuss some of the observations we can make. I will include the spool of the output.  I have not included the output of AWR which confirms use of buffered or direct IO at various stages because I want to concentrate on the relative response times.

SQL>
SQL> rem
SQL> rem Option to trace the test
SQL> rem alter session set events = '10046 trace name context forever, level 12';
SQL>
SQL>
SQL> rem ============================================================
SQL>
SQL> rem confirm parameters
SQL> rem
SQL>
SQL> show parameter db_cache_size

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_cache_size                        big integer 3008M
SQL> show parameter db_keep_cache_size

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_keep_cache_size                   big integer 1008M
SQL> show parameter filesystemio_options

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
filesystemio_options                 string      SETALL
SQL>
SQL> rem
SQL> rem Set the buffer pool of the table  (DEFAULT or KEEP)
SQL> rem
SQL> alter table testuser.TABLE_DATA storage (buffer_pool &ENTER_BUFFER_POOL);
Enter value for enter_buffer_pool: default
old   1: alter table testuser.TABLE_DATA storage (buffer_pool &ENTER_BUFFER_POOL)
new   1: alter table testuser.TABLE_DATA storage (buffer_pool default)

Table altered.

SQL>
SQL>
SQL> rem
SQL> rem show size of the test table and which buffer pool
SQL> rem
SQL> column segment_name format a30 truncate
SQL> column meg format 999999
SQL> select segment_name,
  2         bytes / (1024 * 1024 ) Meg, blocks ,buffer_pool
  3  from sys.dba_segments
  4  where segment_name = 'TABLE_DATA'
  5  /

SEGMENT_NAME                       MEG     BLOCKS BUFFER_
------------------------------ ------- ---------- -------
TABLE_DATA                         775      99200 DEFAULT

SQL>
SQL>
SQL> rem
SQL> rem lets flush the buffer cache to start fresh (or bounce the instance)
SQL> rem
SQL> rem WARNING: Do NOT run this script on any production systems
SQL> rem
SQL>
SQL> alter system flush buffer_cache;

System altered.

SQL>
SQL> rem
SQL> rem check which buffer pool the table is assigned too
SQL> rem
SQL>
SQL> select buffer_pool from dba_tables where table_name = 'TABLE_DATA';

BUFFER_
-------
DEFAULT

So far we have confirmed the setup of key database parameters, ensured our 775MB table is assigned to the DEFAULT buffer pool and also flushed the buffer cache so we can start from an empty and consistent point.
We will now go and enable the 10.2.0.5 direct IO feature and see what happens when we scan our table a couple of times.

SQL>
SQL> rem
SQL> rem lets scan the table using the new serial direct path read option
SQL> rem going to switch it on at session level but can be set global
SQL> rem Will run it twice to show a consistant read time (ie data never made the buffer cache
SQL> rem
SQL>
SQL> alter session set events = '10358 trace name context forever, level 4';

Session altered.

SQL> set timing on
SQL>
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:18.37
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:19.20
SQL>
SQL> set timing off
SQL>

So, you can see that with the direct read feature switched on, both the queries take around 20 seconds each. So the second query has not made use of any buffer cache.

Now.. Let’s go and switch OFF the new feature and then scan the data.

SQL>
SQL>
SQL> rem
SQL> rem Lets switch OFF the new 10.2.0.5 serial direct path read option
SQL> rem again, this is done at session level for purpose of this test
SQL>
SQL> alter session set events = '10358 trace name context off';

Session altered.

SQL>
SQL>
SQL> rem scan it twice and see what happens
SQL> set timing on
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:55.44
SQL>
SQL> rem this second read is VERY MUCH FASTERbecause the data is already in the buffer cache
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:00.54
SQL>

Ok, now we see the first query takes 55 seconds via the buffer cache (rather than 20 seconds direct). So I hope you can see that when you actually do need to scan large volumes of data direct is much faster. BUT hold on… now look at the second query. This only takes less than 1 second because it was read from the buffer cache. So if my query was usually found in the buffer cache, then the direct IO feature may actually be very bad.

Now my data is already in the buffer cache, what happens if I ask to do a direct path read?

SQL> set timing off
SQL>
SQL> rem
SQL> rem One last test, lets now switch on the serial direct read event, BUT the data is already in the buffer cache.
SQL> rem
SQL> rem
SQL>
SQL> alter session set events = '10358 trace name context forever, level 4';

Session altered.

SQL>
SQL> set timing on
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:00.56
SQL> set timing  off
SQL>
SQL>
SQL> rem ============================================================
SQL> rem option to switch off trace
SQL> rem alter session set events = '10046 trace name context off';
SQL>
SQL> set timing off
SQL> spool off

The test shows that Oracle will read blocks from the buffer cache before deciding it need to read the data using the direct path option (Assuming your blocks could get in the cache in the first place)

Test against the KEEP pool using 10.2.0.5

I hope you will see that the table is assigned to the KEEP POOL but the query results are 100% as the test run above. Since I have told Oracle this is an important object which should be protected in the KEEP pool, it should really read it via the buffer cache (slowly) but all the subsequent reads should be out of the cache in sub 1 second response. Because the data is read in directly, EVERY query now takes around 15 seconds and not .5 seconds.

SQL>
SQL> rem
SQL> rem Option to trace the test
SQL> rem alter session set events = '10046 trace name context forever, level 12';
SQL>
SQL>
SQL> rem ============================================================
SQL>
SQL> rem confirm parameters
SQL> rem
SQL>
SQL> show parameter db_cache_size

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_cache_size                        big integer 3008M
SQL> show parameter db_keep_cache_size

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_keep_cache_size                   big integer 1008M
SQL> show parameter filesystemio_options

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
filesystemio_options                 string      SETALL
SQL>
SQL> rem
SQL> rem Set the buffer pool of the table  (DEFAULT or KEEP)
SQL> rem
SQL> alter table testuser.TABLE_DATA storage (buffer_pool &ENTER_BUFFER_POOL);
Enter value for enter_buffer_pool: keep
old   1: alter table testuser.TABLE_DATA storage (buffer_pool &ENTER_BUFFER_POOL)
new   1: alter table testuser.TABLE_DATA storage (buffer_pool keep)

Table altered.

SQL>
SQL>
SQL> rem
SQL> rem show size of the test table and which buffer pool
SQL> rem
SQL> column segment_name format a30 truncate
SQL> column meg format 999999
SQL> select segment_name,
  2         bytes / (1024 * 1024 ) Meg, blocks ,buffer_pool
  3  from sys.dba_segments
  4  where segment_name = 'TABLE_DATA'
  5  /

SEGMENT_NAME                       MEG     BLOCKS BUFFER_
------------------------------ ------- ---------- -------
TABLE_DATA                         775      99200 KEEP

SQL>
SQL>
SQL> rem
SQL> rem lets flush the buffer cache to start fresh (or bounce the instance)
SQL> rem
SQL> rem WARNING: Do NOT run this script on any production systems
SQL> rem
SQL>
SQL> alter system flush buffer_cache;

System altered.

SQL>
SQL> rem
SQL> rem check which buffer pool the table is assigned too
SQL> rem
SQL>
SQL> select buffer_pool from dba_tables where table_name = 'TABLE_DATA';

BUFFER_
-------
KEEP

SQL>
SQL> rem
SQL> rem lets scan the table using the new serial direct path read option
SQL> rem going to switch it on at session level but can be set global
SQL> rem Will run it twice to show a consistant read time (ie data never made the buffer cache
SQL> rem
SQL>
SQL> alter session set events = '10358 trace name context forever, level 4';

Session altered.

SQL> set timing on
SQL>
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:17.89
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:17.24
SQL>
SQL> set timing off
SQL>
SQL>
SQL>
SQL> rem
SQL> rem Lets switch OFF the new 10.2.0.5 serial direct path read option
SQL> rem again, this is done at session level for purpose of this test
SQL>
SQL> alter session set events = '10358 trace name context off';

Session altered.

SQL>
SQL>
SQL> rem scan it twice and see what happens
SQL> set timing on
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:56.03
SQL>
SQL> rem this second read is VERY MUCH FASTERbecause the data is already in the buffer cache
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:00.64
SQL>
SQL> set timing off
SQL>
SQL> rem
SQL> rem One last test, lets now switch on the serial direct read event, BUT the data is already in the buffer cache.
SQL> rem
SQL> rem
SQL>
SQL> alter session set events = '10358 trace name context forever, level 4';

Session altered.

SQL>
SQL> set timing on
SQL> select /*+FULL(c)*/ count(*) from testuser.TABLE_DATA c;

  COUNT(*)
----------
   1958939

Elapsed: 00:00:00.58
SQL> set timing  off
SQL>
SQL>
SQL> rem ============================================================
SQL> rem option to switch off trace
SQL> rem alter session set events = '10046 trace name context off';
SQL>
SQL> set timing off
SQL> spool off

I tested (using a modified version of the script) the patch for 8897574 on 11.1.0.7 and could show with the patch applied and my table assigned to the KEEP POOL, the first execution of the query was read into the buffer cache (and slow) but all further requests were sub 1 second. I don’t yet fully understand the rules which Oracle uses in 10.2.0.5 or 11g to make the decision for direct read, but the key aspect of success will be how well Oracle manages to understand your data usage…. Get it wrong and you may well find “Full Scans” taking very much longer (with higher disk IO) but if they get it right it could have huge improvements in IO response times. The “adaptive serial direct path read” needs to come into play for very large object scans that would be unlikely found in the buffer cache (e.g. very large Data Warehouse objects).

Real Life Numbers

The final part I wanted to give a few real life examples of why queries using the “direct path options” can be so much faster but also reiterate why it can also cause issues. The following are some numbers taken from a live 10.2.0.3 RAC based Data Warehouse (running ASM). It is possible to experiment with direct path reads on older versions of Oracle by either the “_serial_direct_read” session parameter or using parallel query. The “_serial_direct_read” internal session parameter can be used to test running a serial query using direct path read but it can be difficult to use because it is effected by the parsed open cursors (i.e. if someone has already parsed a statement without it, it will not be used, also if someone parses a cursor with it set other sessions using the same sql statement will be direct). But “_serial_direct_read” can be useful for testing. The other option is to run sql statements using parallel query. The following numbers are taken from Full Table Scanning running COUNT(*) on a 100GB table (with 1MB multi block read set). 100GB is larger than the cache in the storage system and also the database instance. We also verified that at the time of the query, none of the data was cached in any node of the RAC cluster.

query by                                             rate
------------------------------               --------------
via buffer cache                              33 MBytes/sec
_serial_direct_read                          300 MBytes/sec
Parallel degress 2 (direct io)               339 MBytes/sec

Things we observed.

• It was not possible to ever make reads via the buffer cache to use async IO
• We lost some time when reading via buffer cache in RAC verifying the blocks were not on any other node, but not enough to account for such a poor rate.
• We suspect that +300MB/sec is achieved with direct path reads, because firstly it does seem to use async IO, secondly we suspect the IO rate is sufficiently high that it can drive the read-a-head caching in our storage system (eg: read blocks 10,11,12,.. so start fetching 13,14,15 before we ask for it)
• Since _serial_direct_read is not supported (and difficult to use), we tested using very low degrees parallel. Because parallel query is implemented to use “direct path reads”, we could see huge increases in through-put just by using 2 degrees parallel.
• The key point I am making about these numbers is the huge difference we have seen on our system between buffer cache and direct reads.
• I may well write a longer blog on these results and include some of the supporting tkprof/trace data at a later time.

One final comment I would like to make and hope someone else may be able to help (Someone good at Hints or within Oracle Development). What I would really like to have seen is a HINT which allows sql statements to indicate preference of “buffer cache” or “direct path” for full scans. That way you could code sql statement for direct path knowing how your data is expected to be used. It may well be possible that people may need to get the KEEP POOL patch applied and use this as the means to help Oracle make the correct decision.

It would be interesting to hear peoples experiences of using the “adaptive serial direct reads” feature in production 11g environments and how good a job Oracle does (or not) at choosing which blocks are cached vs read direct.

update 8-oct-2010

The latest 11.2.0.2 patchset (10098816)  now includes the fix for Bug 8897574. So large objects assigned to the KEEP pool will always do buffered IO and never direct IO (regardless of the size of the object scan).   

A number of years ago, I wrote a SQL Trace File formatter when I needed to process a large amount of trace files for a complex project we were working on. The formatter is not intended to replace the really good tools that are out there, but I like reading the detail which appears in a raw trace file (but with some additional help). I also wanted to structure the trace file so it could be processed by other scripts separately. This is by no means written to a commercial standard, but I thought people may find it useful and also provide interesting insite on how to process trace files.  I worked with James Morle in a previous company to build a benchmark based on Oracle trace files using some software he had written in his book (Scaling Oracle 8i, the  paper Brewing Benchmarks and the later software simora ). The benchmark software contained an awk script to process traces files and extract the entry point database calls and convert them to a tcl scripting language to then drive the benchmark. I took (with permission) the conversion script and modified it so it generated a Oracle trace file but with extra information.

Reading a large Oracle Trace file can be difficult (particularly if the application is well written to avoid unnecessary parsing of sql). Usage of a “prepared statement cache” and later versions of JDBC drivers  mean that a trace file may show the sql text during the first parse at the top of the file, but never show any sql text ever again.

Example raw trace file

note: 10g version of trace file without bind tracing

EXEC #9:c=295955,e=289036,p=0,cr=7760,cu=18338,mis=0,r=1,dep=2,og=1,tim=1180634170580983
EXEC #29:c=0,e=125,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=2,tim=1180634170581711
FETCH #29:c=0,e=44,p=0,cr=1,cu=0,mis=0,r=0,dep=2,og=2,tim=1180634170581781
XCTEND rlbk=0, rd_only=1
EXEC #31:c=0,e=25,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=0,tim=1180634170581989
EXEC #53:c=0,e=136,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170589160
FETCH #53:c=33995,e=33271,p=0,cr=7029,cu=0,mis=0,r=700,dep=3,og=1,tim=1180634170622566
EXEC #44:c=0,e=389,p=0,cr=0,cu=4,mis=0,r=1,dep=3,og=1,tim=1180634170623588
XCTEND rlbk=0, rd_only=0
EXEC #45:c=0,e=111,p=0,cr=0,cu=1,mis=0,r=0,dep=3,og=0,tim=1180634170623747
EXEC #54:c=0,e=71,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170623859
FETCH #54:c=1000,e=279,p=0,cr=50,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170624189
EXEC #55:c=0,e=28,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170624287
FETCH #55:c=0,e=12,p=0,cr=1,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170624330
EXEC #57:c=0,e=65,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170700148
=====================
PARSING IN CURSOR #63 len=129 dep=4 uid=0 oct=6 lid=0 tim=1180634170702781 hv=2635489469 ad='22daed20'
update seq$ set increment$=:2,minvalue=:3,maxvalue=:4,cycle#=:5,order$=:6,cache=:7,highwater=:8,audit$=:9,flags=:10 where obj#=:1
END OF STMT
PARSE #63:c=0,e=31,p=0,cr=0,cu=0,mis=0,r=0,dep=4,og=4,tim=1180634170702778
EXEC #63:c=0,e=293,p=0,cr=1,cu=2,mis=0,r=1,dep=4,og=4,tim=1180634170703175
STAT #63 id=1 cnt=1 pid=0 pos=1 obj=0 op='UPDATE  SEQ$ (cr=1 pr=0 pw=0 time=239 us)'
STAT #63 id=2 cnt=1 pid=1 pos=1 obj=102 op='INDEX UNIQUE SCAN I_SEQ1 (cr=1 pr=0 pw=0 time=23 us)'
FETCH #57:c=4999,e=4311,p=0,cr=6,cu=3,mis=0,r=1358,dep=3,og=1,tim=1180634170704540
WAIT #58: nam='log file switch completion' ela= 228006 p1=0 p2=0 p3=0 obj#=-1 tim=1180634170950223
WAIT #58: nam='log file switch completion' ela= 244847 p1=0 p2=0 p3=0 obj#=-1 tim=1180634171195162
WAIT #58: nam='buffer busy waits' ela= 1689 file#=35 block#=20489 class#=155 obj#=0 tim=1180634171210594
EXEC #58:c=90985,e=566666,p=0,cr=149,cu=15399,mis=0,r=1358,dep=3,og=1,tim=1180634171272924
EXEC #44:c=1000,e=266,p=0,cr=0,cu=4,mis=0,r=1,dep=3,og=1,tim=1180634171273760
XCTEND rlbk=0, rd_only=0
EXEC #45:c=0,e=316,p=0,cr=0,cu=1,mis=0,r=0,dep=3,og=0,tim=1180634171274111
EXEC #59:c=1000,e=1087,p=0,cr=26,cu=26,mis=0,r=24,dep=3,og=1,tim=1180634171275272
EXEC #60:c=40994,e=40592,p=0,cr=40,cu=3207,mis=0,r=1056,dep=3,og=1,tim=1180634171319080
EXEC #61:c=0,e=69,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634171321161
FETCH #61:c=0,e=37,p=0,cr=0,cu=0,mis=0,r=1,dep=3,og=1,tim=1180634171321254
EXEC #62:c=7999,e=7531,p=0,cr=529,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634171328838
EXEC #46:c=1000,e=309,p=0,cr=1,cu=1,mis=0,r=1,dep=3,og=1,tim=1180634171349850
EXEC #47:c=2999,e=3049,p=0,cr=31,cu=237,mis=0,r=63,dep=3,og=1,tim=1180634171353333
EXEC #48:c=0,e=274,p=0,cr=0,cu=3,mis=0,r=1,dep=3,og=4,tim=1180634171354545
XCTEND rlbk=0, rd_only=0
WAIT #56: nam='latch: enqueue hash chains' ela= 230 address=35343054720 number=19 tries=1 obj#=-1 tim=1180634171360412
WAIT #56: nam='latch: enqueue hash chains' ela= 37255 address=35343054720 number=19 tries=2 obj#=-1 tim=1180634171397762
EXEC #56:c=5999,e=43121,p=0,cr=0,cu=1,mis=0,r=0,dep=3,og=0,tim=1180634171398190
EXEC #9:c=308953,e=816696,p=0,cr=7865,cu=18908,mis=0,r=1,dep=2,og=1,tim=1180634171398856
EXEC #50:c=0,e=288,p=0,cr=1,cu=2,mis=0,r=1,dep=2,og=1,tim=1180634171399378
XCTEND rlbk=0, rd_only=0
EXEC #52:c=1000,e=87,p=0,cr=0,cu=1,mis=0,r=0,dep=2,og=0,tim=1180634171399503

When you look at a large trace file (like above), you have to try to remember which cursor number relates to which sql text. It’s possible for the same cursor number to be reused for different sql statements.  It can be difficult to compare several trace files generated by different sessions because the cursor numbers can vary (One session may call extra internal recursive sql and use the cursor number which was used by something else in another sessions trace file.

My trace file formatter attempts to deal with these issues with the following features. Note: Some or all the features can be switched on by a set of configuration parameters in the top of the script.

Summary of Features

I will give a brief summary of the  features before exploring them in more detail. Some of the features work better than others. trace_by_hash.awk has the following options:

• conversion of the cursor numbers into the sql hash_value.   eg “EXEC #59:” becomes “EXEC #3096109005:”. Because all references to the cursor number is now the fixed sql_hash_value, it is now possible to write scripts against a trace file to profile specific sql statements and also to profile many trace files with the same processing.
• Repeat the sql parse text on every execution (EXEC) call. eg, now possible to jump in at any part of the trace file and see the sql_text for the nearest execution (EXEC). The way this is done is to repeat the exact “PARSING IN CURSOR” information on every call. The reason for repeating the full “PARSING IN CURSOR” text is that it is now possible to cut out short parts of the trace file and have it run against utilities such as tkprof (Oracle’s trace profiler). This allows profiling of short parts of a large trace output.
• Indent trace output to reflect the recursive sql depth. eg, when reading the trace of a call to a stored plsql, which in turn executes sql, which in turn executes recursive SYS sql, the trace file will be indented making it easier to read.
• Add timestamp to trace file.  The script will add an approximate human readable time to the trace file output. There are a couple of ways to do this but I am only going to talk about the default method. The scripts see’s what time the trace file was started in the header, and assumes the “tim=” value on the first sql statement is that same time and then works out all time from that baseline. This method works reasonably well, but is possible for trace file to be started and then a delay before next sql statement. May be possible to improve this in the future. The other option involved getting the application to execute a special SQL statement which includes the system time so the formatter can find it and set the base time and system time at consistent point.
• Add delta time to every line. The script can also add a delta time between sql/wait statements. This can be important to spot delays which are not in sql but either the client or time spent in pure plsql. I do have a version of the script which attempts to break down this time further into time spent in plsql and sql but I only had limited success so not included it.
• Allow processing of bind variables. If the trace was made to include bind variables, the script will generate a single line for each execution which includes the sql_hash_value and all the bind data. This makes it possible to profile all the values for a specific query.
• profile sql dependency (note: very badly implemented but kind of works). This allows a file to be generated which shows for example which plsql procedure was called and which sql statements it calls. These can be then used in profiling sql for a specific procedure call.  **Note:  The code is left in for this but the output needs a lot of mucking about by hand** probably not worth trying!.

Using the above, we were able to generate additional scripts which could then profile specific parts of the application (both time in sql statements and procedures but also the time between sql statements (including inside large plsql procedures) and also give a summary of rows processed (r=) for key sql statements. 

Examples

Following example shows how we can profile specific sql statements. In this example I will format the trace file so all cursor numbers are converted to the sql_hash_value, then I can look for a specific sql statement by hash number (I can then run this on many trace files regardless of cursor numbers used)

$ ./trace_by_hash.awk abc1_ora_15772.trc | grep "EXEC #1711438125" | more
EXEC #1711438125:c=277957,e=278731,p=0,cr=7900,cu=18934,mis=0,r=1,dep=2,og=1,tim=1180634162359512
EXEC #1711438125:c=293955,e=289873,p=0,cr=7945,cu=20453,mis=0,r=1,dep=2,og=1,tim=1180634162650576
EXEC #1711438125:c=283957,e=301283,p=0,cr=7808,cu=19294,mis=0,r=1,dep=2,og=1,tim=1180634162954735
EXEC #1711438125:c=307953,e=325653,p=0,cr=8077,cu=20290,mis=0,r=1,dep=2,og=1,tim=1180634163281707
EXEC #1711438125:c=298955,e=293236,p=0,cr=7981,cu=20165,mis=0,r=1,dep=2,og=1,tim=1180634163576579
EXEC #1711438125:c=291956,e=285030,p=0,cr=7893,cu=19470,mis=0,r=1,dep=2,og=1,tim=1180634163862972
EXEC #1711438125:c=287957,e=289269,p=0,cr=7842,cu=18255,mis=0,r=1,dep=2,og=1,tim=1180634164153867
EXEC #1711438125:c=301953,e=306938,p=0,cr=7851,cu=19901,mis=0,r=1,dep=2,og=1,tim=1180634164462182

I can now write a simple awk filter to find examples of the elapsed time (e=) taking a long time. In this example, sql_hash_value 1711438125 is a plsql stored procedure so I will find any examples longer than half a second (500ms)

$ ./trace_by_hash.awk abc1_ora_15772.trc | grep "EXEC #1711438125" | gawk -F, '{e=$2;x=sub("e=","",e);if (e + 0 > 500000) {print $0;}}'
EXEC #1711438125:c=308953,e=816696,p=0,cr=7865,cu=18908,mis=0,r=1,dep=2,og=1,tim=1180634171398856
EXEC #1711438125:c=347948,e=507123,p=0,cr=8391,cu=19861,mis=0,r=1,dep=2,og=1,tim=1180634172341323
EXEC #1711438125:c=338948,e=523424,p=0,cr=7963,cu=19401,mis=0,r=1,dep=2,og=1,tim=1180634185387239
EXEC #1711438125:c=323951,e=597571,p=0,cr=8003,cu=18819,mis=0,r=1,dep=2,og=1,tim=1180634217021013
EXEC #1711438125:c=334949,e=563515,p=0,cr=8016,cu=19676,mis=0,r=1,dep=2,og=1,tim=1180634218655660
EXEC #1711438125:c=302954,e=543647,p=0,cr=7883,cu=19719,mis=0,r=1,dep=2,og=1,tim=1180634280115132
EXEC #1711438125:c=290955,e=519560,p=1,cr=7982,cu=18639,mis=0,r=1,dep=2,og=1,tim=1180634346928803

I can now take a specific tim= value and search for this in the fully formatted trace file.  If we look up the plsql call for the 816ms example above (first slow example) we can jump into the trace file and see how that 816ms elapsed time is accounted for. I have switched on all the formatting options here.

Things to remember:

• This example sql is faked so don’t worry if the sql application does not make sense.
• None of the sql text would  appear in the middle of a real raw trace file making it difficult to read and understand.
• Cursor numbers are converted to sql_hash_values making possible to profile by hash_value
• a time and delta of tim= values added (note: I only change the delta time when a new tim= value is changed, so you will see the same value repeated until a new tim= is found) 
• EXEC line only appears when a sql statement completes, so you need to find the EXEC you are interested in and read nested sql statements above that. This is helped by the indentation of the SQL depth.
• I have added a few comments in the following output (**COMMENT**)
• One final point, this is the formatted trace for the exact raw trace section shown at the top of this blog article.

./trace_by_hash.awk abc1_ora_15772.trc > abc1_ora_15772.txt
**comment: I have cut the output around the slow 816ms procedure call
**comment: ignore this first procedure call, this is the EXEC of the PREVIOUS plsql procedure
2008-04-23 16:49:50      689           PARSING IN CURSOR #1711438125 len=223 dep=2 uid=68 oct=47 lid=68 tim=1180634162359516 hv=1711438125 ad='233a05e8'
2008-04-23 16:49:50      689            BEGIN pkg_do_stuff.pr_process( :id,  :reload_true_false,  :dump_data, :call_immediately,  :error_id );  END;
2008-04-23 16:49:50      689           END OF STMT
2008-04-23 16:49:50      689           EXEC #1711438125:c=295955,e=289036,p=0,cr=7760,cu=18338,mis=0,r=1,dep=2,og=1,tim=1180634170580983
2008-04-23 16:49:50      728           =====================
2008-04-23 16:49:50      728           PARSING IN CURSOR #525534860 len=570 dep=2 uid=0 oct=3 lid=0 tim=1180634162360228 hv=525534860 ad='2344b810'
2008-04-23 16:49:50      728            select  /*+ FIRST_ROWS(1) */   tab.rowid, tab.msgid, tab.corrid, tab.priority, tab.delay,   tab.expiration, tab.retry_count, tab.exception_qschema,   tab.exception_queue, tab.chain_no, tab.local_order_no, tab.enq_time,   tab.time_manager_info, tab.state, tab.enq_tid, tab.step_no,   tab.sender_name, tab.sender_address, tab.sender_protocol,   tab.dequeue_msgid, tab.user_prop, tab.user_data  from "CODE"."QTAB_INFO" tab  where q_name = :1 and (state = :2  )  order by q_name, state, enq_time, step_no, chain_no, local_order_no for update skip locked
2008-04-23 16:49:50      728           END OF STMT
2008-04-23 16:49:50      728           EXEC #525534860:c=0,e=125,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=2,tim=1180634170581711
2008-04-23 16:49:50       70           FETCH #525534860:c=0,e=44,p=0,cr=1,cu=0,mis=0,r=0,dep=2,og=2,tim=1180634170581781
2008-04-23 16:49:50       70           XCTEND rlbk=0, rd_only=1
2008-04-23 16:49:50      208           =====================
2008-04-23 16:49:50      208           PARSING IN CURSOR #255718823 len=6 dep=2 uid=68 oct=44 lid=68 tim=1180634162360510 hv=255718823 ad='0'
2008-04-23 16:49:50      208            COMMIT
2008-04-23 16:49:50      208           END OF STMT
2008-04-23 16:49:50      208           EXEC #255718823:c=0,e=25,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=0,tim=1180634170581989
**comment: this is in fact the first sql statement in side the slow procedure, it is indented (dep=3)
**comment: The actual slow EXEC will be at dep=2 at the end of the output
2008-04-23 16:49:50     7171                =====================
2008-04-23 16:49:50     7171                PARSING IN CURSOR #1676635612 len=1083 dep=3 uid=68 oct=3 lid=68 tim=1180634162366901 hv=1676635612 ad='22b0c098'
2008-04-23 16:49:50     7171                 SELECT DATA1,DATA2,DATA3 FROM T_INPUT TBU
2008-04-23 16:49:50     7171                END OF STMT
2008-04-23 16:49:50     7171                EXEC #1676635612:c=0,e=136,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170589160
2008-04-23 16:49:50    33406                FETCH #1676635612:c=33995,e=33271,p=0,cr=7029,cu=0,mis=0,r=700,dep=3,og=1,tim=1180634170622566
2008-04-23 16:49:50     1022                =====================
2008-04-23 16:49:50     1022                PARSING IN CURSOR #776466503 len=196 dep=3 uid=68 oct=2 lid=68 tim=1180634162396345 hv=776466503 ad='24cc4f58'
2008-04-23 16:49:50     1022                 INSERT INTO T_LOG ( ID , TYPE , MESSAGE , DAEMON_ID ) VALUES ( LOG_ID.NEXTVAL , :B7 , :B6 , :B5 , :B4 , )
2008-04-23 16:49:50     1022                END OF STMT
2008-04-23 16:49:50     1022                EXEC #776466503:c=0,e=389,p=0,cr=0,cu=4,mis=0,r=1,dep=3,og=1,tim=1180634170623588
2008-04-23 16:49:50     1022                XCTEND rlbk=0, rd_only=0
2008-04-23 16:49:50      159                =====================
2008-04-23 16:49:50      159                PARSING IN CURSOR #255718823 len=6 dep=3 uid=68 oct=44 lid=68 tim=1180634162396510 hv=255718823 ad='0'
2008-04-23 16:49:50      159                 COMMIT
2008-04-23 16:49:50      159                END OF STMT
2008-04-23 16:49:50      159                EXEC #255718823:c=0,e=111,p=0,cr=0,cu=1,mis=0,r=0,dep=3,og=0,tim=1180634170623747
2008-04-23 16:49:50      112                =====================
2008-04-23 16:49:50      112                PARSING IN CURSOR #4115507142 len=1012 dep=3 uid=68 oct=3 lid=68 tim=1180634162396622 hv=4115507142 ad='24a69150'
2008-04-23 16:49:50      112                 SELECT DATA1 FROM T_QUEUE
2008-04-23 16:49:50      112                END OF STMT
2008-04-23 16:49:50      112                EXEC #4115507142:c=0,e=71,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170623859
2008-04-23 16:49:50      330                FETCH #4115507142:c=1000,e=279,p=0,cr=50,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170624189
2008-04-23 16:49:50       98                =====================
2008-04-23 16:49:50       98                PARSING IN CURSOR #2654270261 len=796 dep=3 uid=68 oct=3 lid=68 tim=1180634162396990 hv=2654270261 ad='22b0b5a0'
2008-04-23 16:49:50       98                 SELECT DATA1,DATA2 FROM T_QUEUE2
2008-04-23 16:49:50       98                END OF STMT
2008-04-23 16:49:50       98                EXEC #2654270261:c=0,e=28,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170624287
2008-04-23 16:49:50       43                FETCH #2654270261:c=0,e=12,p=0,cr=1,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170624330
2008-04-23 16:49:50    75818                =====================
2008-04-23 16:49:50    75818                PARSING IN CURSOR #3186096073 len=75 dep=3 uid=68 oct=3 lid=68 tim=1180634162467210 hv=3186096073 ad='22acd4f0'
2008-04-23 16:49:50    75818                 SELECT APP_ID.NEXTVAL FROM T_APP_NUMS WHERE ROWNUM <= :B1
2008-04-23 16:49:50    75818                END OF STMT
2008-04-23 16:49:50    75818                EXEC #3186096073:c=0,e=65,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634170700148
2008-04-23 16:49:50    75818                =====================
**comment: recursive oracle sys sql to update a system sequence table
2008-04-23 16:49:50    75818                     PARSING IN CURSOR #2635489469 len=129 dep=4 uid=0 oct=6 lid=0 tim=1180634170702781 hv=2635489469 ad='22daed20'
2008-04-23 16:49:50    75818                     update seq$ set increment$=:2,minvalue=:3,maxvalue=:4,cycle#=:5,order$=:6,cache=:7,highwater=:8,audit$=:9,flags=:10 where obj#=:1
2008-04-23 16:49:50    75818                     END OF STMT
2008-04-23 16:49:50     2630                     PARSE #2635489469:c=0,e=31,p=0,cr=0,cu=0,mis=0,r=0,dep=4,og=4,tim=1180634170702778
2008-04-23 16:49:50      397                     =====================
2008-04-23 16:49:50      397                     PARSING IN CURSOR #2635489469 len=129 dep=4 uid=0 oct=6 lid=0 tim=1180634170702781 hv=2635489469 ad='22daed20'
2008-04-23 16:49:50      397                      update seq$ set increment$=:2,minvalue=:3,maxvalue=:4,cycle#=:5,order$=:6,cache=:7,highwater=:8,audit$=:9,flags=:10 where obj#=:1
2008-04-23 16:49:50      397                     END OF STMT
2008-04-23 16:49:50      397                     EXEC #2635489469:c=0,e=293,p=0,cr=1,cu=2,mis=0,r=1,dep=4,og=4,tim=1180634170703175
2008-04-23 16:49:50      397                     STAT #2635489469 id=1 cnt=1 pid=0 pos=1 obj=0 op='UPDATE  SEQ$ (cr=1 pr=0 pw=0 time=239 us)'
2008-04-23 16:49:50      397                     STAT #2635489469 id=2 cnt=1 pid=1 pos=1 obj=102 op='INDEX UNIQUE SCAN I_SEQ1 (cr=1 pr=0 pw=0 time=23 us)'
2008-04-23 16:49:50     1365                     FETCH #3186096073:c=4999,e=4311,p=0,cr=6,cu=3,mis=0,r=1358,dep=3,og=1,tim=1180634170704540
**comment: notice we loose 400ms in two waits on the database
**comment: any difference between the delta times and the elapsed sql time (e=) is normally time in the plsql or client
2008-04-23 16:49:51   245683                     WAIT #2577162295: nam='log file switch completion' ela= 228006 p1=0 p2=0 p3=0 obj#=-1 tim=1180634170950223
2008-04-23 16:49:51   244939                     WAIT #2577162295: nam='log file switch completion' ela= 244847 p1=0 p2=0 p3=0 obj#=-1 tim=1180634171195162
2008-04-23 16:49:51    15432                     WAIT #2577162295: nam='buffer busy waits' ela= 1689 file#=35 block#=20489 class#=155 obj#=0 tim=1180634171210594
2008-04-23 16:49:51    62330                =====================
2008-04-23 16:49:51    62330                PARSING IN CURSOR #2577162295 len=208 dep=3 uid=68 oct=2 lid=68 tim=1180634162279786 hv=2577162295 ad='24a2ad68'
2008-04-23 16:49:51    62330                 INSERT INTO T_APP_OUTPUT VALUES (:B1 ,:B2 ,:B3 ,:B4 ,:B5 ,:B6 ,:B7 ,:B8 ,:B9 ,:B10 ,:B11 ,:B12 ,:B13 ,:B14 ,:B15 ,:B16 ,:B17 ,:B18 ,:B19 ,:B20 ,:B21 ,:B22 ,:B23 ,:B24 ,:B25 ,:B26 ,:B27 ,:B28 ,:B29 ,:B30 )
2008-04-23 16:49:51    62330                END OF STMT
**comment: notice this next EXEC is 500ms but includes the time lost in the nested recursive waits above
**comment: Wait events are also reported in trace after they complete, hence a short time in delta between
**comment: the WAITs tim= value and the completion of the EXEC sql statement.
2008-04-23 16:49:51    62330                EXEC #2577162295:c=90985,e=566666,p=0,cr=149,cu=15399,mis=0,r=1358,dep=3,og=1,tim=1180634171272924
2008-04-23 16:49:51      836                =====================
2008-04-23 16:49:51      836                PARSING IN CURSOR #776466503 len=196 dep=3 uid=68 oct=2 lid=68 tim=1180634162396345 hv=776466503 ad='24cc4f58'
2008-04-23 16:49:51      836                 INSERT INTO T_LOG ( ID , TYPE , MESSAGE , DAEMON_ID ) VALUES ( LOG_ID.NEXTVAL , :B7 , :B6 , :B5 , :B4 , )
2008-04-23 16:49:51      836                END OF STMT
2008-04-23 16:49:51      836                EXEC #776466503:c=1000,e=266,p=0,cr=0,cu=4,mis=0,r=1,dep=3,og=1,tim=1180634171273760
2008-04-23 16:49:51      836                XCTEND rlbk=0, rd_only=0
2008-04-23 16:49:51      351                =====================
2008-04-23 16:49:51      351                PARSING IN CURSOR #255718823 len=6 dep=3 uid=68 oct=44 lid=68 tim=1180634162396510 hv=255718823 ad='0'
2008-04-23 16:49:51      351                 COMMIT
2008-04-23 16:49:51      351                END OF STMT
2008-04-23 16:49:51      351                EXEC #255718823:c=0,e=316,p=0,cr=0,cu=1,mis=0,r=0,dep=3,og=0,tim=1180634171274111
2008-04-23 16:49:51     1161                =====================
2008-04-23 16:49:51     1161                PARSING IN CURSOR #3096109005 len=385 dep=3 uid=68 oct=6 lid=68 tim=1180634162281365 hv=3096109005 ad='24a2b1e8'
2008-04-23 16:49:51     1161                 UPDATE T_CONTROL SET TOTAL=:B1 WHERE ID = :B2 AND OTHER_ID = :B3
2008-04-23 16:49:51     1161                END OF STMT
2008-04-23 16:49:51     1161                EXEC #3096109005:c=1000,e=1087,p=0,cr=26,cu=26,mis=0,r=24,dep=3,og=1,tim=1180634171275272
2008-04-23 16:49:51    43808                =====================
2008-04-23 16:49:51    43808                PARSING IN CURSOR #2694978569 len=275 dep=3 uid=68 oct=6 lid=68 tim=1180634162321122 hv=2694978569 ad='24a2bbe8'
2008-04-23 16:49:51    43808                 UPDATE T_INPUT TBUF SET TBUF.STATUS = :B4 , TBUF.SIZE = :B3 WHERE TBUF.ROWID = :B5
2008-04-23 16:49:51    43808                END OF STMT
2008-04-23 16:49:51    43808                EXEC #2694978569:c=40994,e=40592,p=0,cr=40,cu=3207,mis=0,r=1056,dep=3,og=1,tim=1180634171319080
2008-04-23 16:49:51     2081                =====================
2008-04-23 16:49:51     2081                PARSING IN CURSOR #624481974 len=48 dep=3 uid=68 oct=3 lid=68 tim=1180634162322917 hv=624481974 ad='23326260'
2008-04-23 16:49:51     2081                 SELECT SEQ_NO.NEXTVAL FROM DUAL
2008-04-23 16:49:51     2081                END OF STMT
2008-04-23 16:49:51     2081                EXEC #624481974:c=0,e=69,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634171321161
2008-04-23 16:49:51       93                FETCH #624481974:c=0,e=37,p=0,cr=0,cu=0,mis=0,r=1,dep=3,og=1,tim=1180634171321254
2008-04-23 16:49:51     7584                =====================
2008-04-23 16:49:51     7584                PARSING IN CURSOR #562877356 len=135 dep=3 uid=68 oct=6 lid=68 tim=1180634162329662 hv=562877356 ad='24a2a8e8'
2008-04-23 16:49:51     7584                 UPDATE T_MONEY SET ESTIMATED = 'Y' , LOCK = :B3 WHERE ID = :B2 AND ESTIMATED = 'N'
2008-04-23 16:49:51     7584                END OF STMT
2008-04-23 16:49:51     7584                EXEC #562877356:c=7999,e=7531,p=0,cr=529,cu=0,mis=0,r=0,dep=3,og=1,tim=1180634171328838
2008-04-23 16:49:51    21012                =====================
2008-04-23 16:49:51    21012                PARSING IN CURSOR #945999453 len=122 dep=3 uid=68 oct=6 lid=68 tim=1180634162349722 hv=945999453 ad='2299fff0'
2008-04-23 16:49:51    21012                 UPDATE T_SEQ SET ID = ID + 1 WHERE NAME = :B1 RETURNING ID INTO :O0
2008-04-23 16:49:51    21012                END OF STMT
2008-04-23 16:49:51    21012                EXEC #945999453:c=1000,e=309,p=0,cr=1,cu=1,mis=0,r=1,dep=3,og=1,tim=1180634171349850
2008-04-23 16:49:51     3483                =====================
2008-04-23 16:49:51     3483                PARSING IN CURSOR #3080672287 len=63 dep=3 uid=68 oct=2 lid=68 tim=1180634162353483 hv=3080672287 ad='23326620'
2008-04-23 16:49:51     3483                 INSERT INTO T_OUTPUT VALUES ( :B4 , :B1 , :B2 , :B3 )
2008-04-23 16:49:51     3483                END OF STMT
2008-04-23 16:49:51     3483                EXEC #3080672287:c=2999,e=3049,p=0,cr=31,cu=237,mis=0,r=63,dep=3,og=1,tim=1180634171353333
2008-04-23 16:49:51     1212                =====================
2008-04-23 16:49:51     1212                PARSING IN CURSOR #840807921 len=496 dep=3 uid=0 oct=2 lid=0 tim=1180634162354581 hv=840807921 ad='248a25e8'
2008-04-23 16:49:51     1212                 insert into "CODE"."DATA_QTAB"  (q_name, msgid, corrid, priority, state, delay, expiration,   time_manager_info, local_order_no, chain_no, enq_time, step_no, enq_uid,   enq_tid, retry_count, exception_qschema, exception_queue, recipient_key,   dequeue_msgid, user_data, sender_name, sender_address, sender_protocol,   user_prop, cscn, dscn)   values (:1, :2, :3, :4, :5, :6, :7, :8, :9, :10, :11, :12, :13, :14, 0, :15,         :16, :17, :18, :19, :20, :21, :22, :23, :24, :25)
2008-04-23 16:49:51     1212                END OF STMT
2008-04-23 16:49:51     1212                EXEC #840807921:c=0,e=274,p=0,cr=0,cu=3,mis=0,r=1,dep=3,og=4,tim=1180634171354545
2008-04-23 16:49:51     1212                XCTEND rlbk=0, rd_only=0
2008-04-23 16:49:51     5867                WAIT #255718823: enqueue hash chains' ela= 230 address=35343054720 number=19 tries=1 obj#=-1 tim=1180634171360412
2008-04-23 16:49:51    37350                WAIT #255718823: enqueue hash chains' ela= 37255 address=35343054720 number=19 tries=2 obj#=-1 tim=1180634171397762
2008-04-23 16:49:51      428                =====================
2008-04-23 16:49:51      428                PARSING IN CURSOR #255718823 len=6 dep=3 uid=68 oct=44 lid=68 tim=1180634162358893 hv=255718823 ad='0'
2008-04-23 16:49:51      428                 COMMIT
2008-04-23 16:49:51      428                END OF STMT
2008-04-23 16:49:51      428                EXEC #255718823:c=5999,e=43121,p=0,cr=0,cu=1,mis=0,r=0,dep=3,og=0,tim=1180634171398190
**comment: notice the depth returns to depth=2: the next EXEC is the slow 816ms statements which the sql above relates too
2008-04-23 16:49:51      666           =====================
2008-04-23 16:49:51      666           PARSING IN CURSOR #1711438125 len=223 dep=2 uid=68 oct=47 lid=68 tim=1180634162359516 hv=1711438125 ad='233a05e8'
2008-04-23 16:49:51      666            BEGIN pkg_do_stuff.pr_process( :id,  :reload_true_false,  :dump_data, :call_immediately,  :error_id );  END;
2008-04-23 16:49:51      666           END OF STMT
2008-04-23 16:49:51      666           EXEC #1711438125:c=308953,e=816696,p=0,cr=7865,cu=18908,mis=0,r=1,dep=2,og=1,tim=1180634171398856
**comment: we continue with the next statement also at depth 2 (dep=2)
2008-04-23 16:49:51      522           =====================
2008-04-23 16:49:51      522           PARSING IN CURSOR #375712160 len=108 dep=2 uid=68 oct=6 lid=68 tim=1180634162651576 hv=375712160 ad='2355ada0'
2008-04-23 16:49:51      522            UPDATE ALIVE_COUNT DAC SET DAC.COUNT = NVL( DAC.COUNT, 0 ) + 1 WHERE DAC.ID = :B1
2008-04-23 16:49:51      522           END OF STMT
2008-04-23 16:49:51      522           EXEC #375712160:c=0,e=288,p=0,cr=1,cu=2,mis=0,r=1,dep=2,og=1,tim=1180634171399378
2008-04-23 16:49:51      522           XCTEND rlbk=0, rd_only=0
2008-04-23 16:49:51      125           =====================
2008-04-23 16:49:51      125           PARSING IN CURSOR #255718823 len=6 dep=2 uid=68 oct=44 lid=68 tim=1180634162651797 hv=255718823 ad='0'
2008-04-23 16:49:51      125            COMMIT
2008-04-23 16:49:51      125           END OF STMT
2008-04-23 16:49:51      125           EXEC #255718823:c=1000,e=87,p=0,cr=0,cu=1,mis=0,r=0,dep=2,og=0,tim=1180634171399503
**comment: trace file then continues with the rest of the trace file.

Processing bind variables

The final example is of processing Bind variables (none of the previous examples had bind tracing on
processing bind variables can be difficult because they are not tagged to a cursor in a way which makes it easy to process via grep

PARSING IN CURSOR #1 len=132 dep=1 uid=0 oct=3 lid=0 tim=1273586951204031 hv=4260389146 ad='27ab99f28' sqlid='cvn54b7yz0s8u'
select /*+ index(idl_ub1$ i_idl_ub11) +*/ piece#,length,piece from idl_ub1$ where obj#=:1 and part=:2 and version=:3 order by piece#
END OF STMT
PARSE #1:c=0,e=25,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=3246118364,tim=1273586951204030
BINDS #1:
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=2b0113708e70  bln=22  avl=04  flg=05
  value=44125
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=2b0113708e40  bln=24  avl=02  flg=05
  value=2
 Bind#2
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=2b0113708e10  bln=24  avl=06  flg=05
  value=184549376
EXEC #1:c=0,e=113,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=3246118364,tim=1273586951204205
FETCH #1:c=0,e=19,p=0,cr=4,cu=0,mis=0,r=1,dep=1,og=4,plh=3246118364,tim=1273586951204252
FETCH #1:c=0,e=8,p=0,cr=3,cu=0,mis=0,r=1,dep=1,og=4,plh=3246118364,tim=1273586951204282

Included in the output is  a summary line added for each bind which is referenced by the sql_hash_value. In this example, I have run sql tracing on the execution of a statspack report (spreport). I have then profiled the bind variables for one statement. You can see BINDDUMP# tag, a short part of the sql_text, the hash_value, Number of Bind variables (4) and then each value of the bind eg, (1, 4140534469, 1, “db block changes”)

$ ./trace_by_hash.awk 11201_spreport.trc | grep "BINDDUMP.*3685970426"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "db block changes"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "db block changes"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "user calls"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "user calls"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "user rollbacks"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "user rollbacks"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "user commits"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "user commits"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "redo size"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "redo size"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "physical reads"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "physical reads"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "physical reads direct"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "physical reads direct"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "physical reads direct (lob)"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "physical reads direct (lob)"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "physical reads cache"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "physical reads cache"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "physical writes"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "physical writes"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "parse count (hard)"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "parse count (hard)"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "parse count (total)"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "parse count (total)"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "session logical reads"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "session logical reads"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "recursive calls"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "recursive calls"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "redo log space requests"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "redo log space requests"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "redo entries"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "redo entries"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "logons cumulative"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "logons cumulative"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "parse time cpu"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "parse time cpu"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "parse time elapsed"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "parse time elapsed"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "CPU used by this session"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "CPU used by this session"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "execute count"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "execute count"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "logons current"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "logons current"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 1, 4140534469, 1, "opened cursors current"
BINDDUMP# , SELECT VALUE FROM STATS$SYSSTAT WHERE S, 3685970426, num_binds ,4, bind_data, 2, 4140534469, 1, "opened cursors current"

Link to “readme” file which also includes link to the script: trace_by_hash_readme
I hope people find the script useful for profiling SQL trace files and also a good source for information on how trace files work. Please remember that this script is not supported and I can’t guarantee does not have any bugs, but I have used it extensively (mostly against 10gR2 trace files).

Out of interest, you can sometimes look up the text for Oracle Events using the “oerr” command .  

$ oerr ora 10629
10629, 00000, "force online index build to backoff and retry DML lock upgrade"
// *Cause:
// *Action: set this event only under the supervision of Oracle development
// *Comment: Change the behaviour of an online index rebuild such that it
// will backoff and retry a failed DML lock upgrade.
// The event level is the number of retries the online index rebuild
// should wait. Level 1 means backoff and retry indefinitely. Any
//           other value less than 32 will be adjusted automatically to be 32.

 
There is more information on Meta link (note: 3566511.8). The comments of “This issue is fixed in” mean the new Event feature is included in the release (you still need to set it).

I’ve just had a very quick test of the Event on 11gR1/R2 but it’s not clear if the Event still works. The very important thing (to me) is the 11g versions no longer cause other unrelated DML to become stuck behind a long running active transaction.

• maximum connections in the pool set too low? - Can lead to the requests queuing to get a connection or running out all together during peak spikes (leading to  application failure).
• maximum connections set too high? - Can lead to “logon storms”. A sudden surge in concurrent user activity can cause a huge demand on new sessions to be created on the database server. Because the database is slowed by the surge, you often get a “feed back loop” effect. Slow database response means even more connection requests. “Connection Storms” can also be caused by small problems which start in the database or network (unexpected slow network pause or database wait event…. then a sudden logon storm hits the database host as increasing connections slow the database even more (seen this lots in the past)
• minimum connections in the pool set too low? – Can lead to the added cost of always having to restart new database sessions just at the critical time you need them.
• idle out time set too low? - Once you have taken the hit of creating a new session in the database, you let this session reach the idle time too soon and the session is destroyed… only to be needed again a few minutes later.
• idle out time set too high? Lets say you have suffered a minor “logon storm”, if the idle time set too high you could have all these extra sessions hanging around for a very long time (There have been a number of good presentations by the Oracle Real World Performance group talking about benefits of reducing connections on a database server)
• minimum and maximum the same value? There is a lot to be said for running with a fixed number of sessions supporting your average usage and your peaks (The trick is finding the correct number to support peaks)

A very simple session monitoring script

Before I actually talk about the script itself, I thought I would give  an example of why I wrote it in the first place and how it was used, then I will show the script and why we still use it today.

The problem

When I first joined my company, I could never get my head around the number of sessions on the database compared to the number of “ACTIVE” sessions seen in v$session. We seemed to have many more connections from the application servers then sessions doing work. Every once in a while the company would review the peak connection pool settings (per box) and adjust them. The kind of conversations we would have around the office were:

 ”We are using the maximum 30 connections per box, let’s make it 35 per box for growth”

The next year we would say…

“We are using all the 35 per box, let’s make it 40 per box…..  for growth”

All the time I kept thinking maybe v$session status=”ACTIVE” was mis-leading, or maybe a problem with the connection pools (We had 30 minute idle out time but we always ran at the maximum connections but very few were actually seen to be used.

Back to the script

client_info.sql

set trimspool on
set pagesize 1000
set linesize 190
column since format a20
column logon format a20
column event format a25 truncate
column machine format a10 truncate
column status format a10 truncate
column username format a10 truncate
column n format 9999.99

break on machine on report  skip 1

compute sum of sess_count on machine
compute sum of sess_count on report

set time on
spool client.txt append

select 'CFROM' tag,
  to_char(sysdate,'hh24:mi:ss') when,
        machine,
        event,
        seconds_in_wait,
        sql_id,
        prev_sql_id,
        count(*) sess_count,
  to_char(sysdate - (SECONDS_IN_WAIT / (60 * 60 * 24)),'dd-mon-yy hh24:mi:ss') since,
--      next two lines useful if trying to predict a concurrent spike.   1200 being 20 minutes
--      Left over from site specific issue but could be useful.
--      next is last active time + 20 minutes
--      n is a count down to next predicted spike
-- to_char(sysdate - ( (seconds_in_wait ) / (60 * 60 * 24)) + (1200 / (60 * 60 * 24)),'dd-mon-yy hh24:mi:ss') NEXT ,
-- ((sysdate - (sysdate - ( (seconds_in_wait ) / (60 * 60 * 24)) + (1200 / (60 * 60 * 24)) ) ) * (60 * 60 * 24) ) / 60 n,
        username,
        status,
        state
from v$session
group by machine,
         event,
         seconds_in_wait,
         sql_id,
         prev_sql_id,
         username,
         status,
         state
order by machine,
         username,
         event,
         seconds_in_wait,
         sql_id,
         prev_sql_id,
         status,
         state
/

One day I wrote the above script and all became very clear. Something sent  a very fast (150 micro seconds) sql statement concurrently from each application server to every session in the connection pool. The request rate could easily have been serviced by a handful of sessions, but because they came concurrently at the exact same time every session was used. 

note: This output has been faked, Not able to find real example of the original issue we had (few years back)
note: All host names and user names are also faked...... so I can keep my job!
Columns            Description
-----------------  --------------------------
TAG                tag used so can grep info from log file
WHEN current system time when data collected
MACHINE            host where the session originates
EVENT              Current database event
SECONDS_IN_WAIT    Seconds in Wait
SQL_ID             Current SQL statement
PREV_SQL_ID        Previous SQL statement last run on the session
SESS_COUNT         Total number of sessions within group (sql_id,event,seconds in wait etc)
SINCE What time the session has been idle since (sysdate - seconds wait)
USERNAME           User name of sessions
STATUS             Session status (Active or Inactive)
STATE              Session Wait State

TAG   WHEN     MACHINE    EVENT                     SECONDS_IN_WAIT SQL_ID        PREV_SQL_ID   SESS_COUNT SINCE                USERNAME   STATUS     STATE
----- -------- ---------- ------------------------- --------------- ------------- ------------- ---------- -------------------- ---------- ---------- -------------------
CFROM 13:55:17 abcabc01.i SQL*Net message from clie               0 2j7pff3tfuzzz 2j7pff3tfuzzz          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               qw4bv6jwup5ab          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               5tarshstnypzv          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               1               8zv7177vuc8dt          3 14-may-10 13:55:16   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               5               4616pfpak8akh          1 14-may-10 13:55:12   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              22               amsmuu1pp1w74          1 14-may-10 13:54:55   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie             185               3q4bv6jx8wup5         30 14-may-10 13:52:12   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie             433               f3tg1gz4zdadm          1 14-may-10 13:48:04   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie             873               cmh3vh4pjs7q7          1 14-may-10 13:40:44   WEBUSERABC INACTIVE   WAITING
               **********                                                                       ----------
               sum                                                                                      40
CFROM 13:55:17 abcabc02.i SQL*Net message from clie               0 zzz0ahx447fpr zzz0ahx447fpr          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               3q4bv6jx8wup5          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               5tarshstnypzv          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               f3tg1gz4zdadm          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               6               cmh3vh4pjs7q7          1 14-may-10 13:55:11   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               8               8zv7177vuc8dt          1 14-may-10 13:55:09   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              11               cmh3vh4pjs7q7          1 14-may-10 13:55:06   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              12               f3tg1gz4zdadm          1 14-may-10 13:55:05   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              80               8zv7177vuc8dt          1 14-may-10 13:53:57   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              80               f3tg1gz4zdadm          1 14-may-10 13:53:57   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie             138               3q4bv6jx8wup5         28 14-may-10 13:52:59   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie             695               cmh3vh4pjs7q7          1 14-may-10 13:43:42   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie            1394               cmh3vh4pjs7q7          1 14-may-10 13:32:03   WEBUSERABC INACTIVE   WAITING
               **********                                                                       ----------
               sum                                                                                      40
CFROM 13:55:17 abcabc03.i SQL*Net message from clie               0 zzz0ahx447fpr zzz0ahx447fpr          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               5tarshstnypzv          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               cmh3vh4pjs7q7          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               2               5tarshstnypzv          1 14-may-10 13:55:15   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               7               8zv7177vuc8dt          2 14-may-10 13:55:10   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              14               3q4bv6jx8wup5          1 14-may-10 13:55:03   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              14               4616pfpak8akh          1 14-may-10 13:55:03   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              61               8zv7177vuc8dt          1 14-may-10 13:54:16   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie             159               3q4bv6jx8wup5         31 14-may-10 13:52:38   WEBUSERABC INACTIVE   WAITING
               **********                                                                       ----------
               sum                                                                                      40
CFROM 13:55:17 abcabc04.i SQL*Net message from clie               0               3q4bv6jx8wup5         39 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               5tarshstnypzv          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
               **********                                                                       ----------
               sum                                                                                      40
CFROM 13:55:17 abcabc05.i SQL*Net message from clie               0 zzz0ahx447fpr zzz0ahx447fpr          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               33hsynd62ka4k          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               0               cmh3vh4pjs7q7          1 14-may-10 13:55:17   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie               1 zzz0ahx447fpr zzz0ahx447fpr          2 14-may-10 13:55:16   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              10               cmh3vh4pjs7q7          1 14-may-10 13:55:07   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie              31               8zv7177vuc8dt          1 14-may-10 13:54:46   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie             157               amsmuu1pp1w74          1 14-may-10 13:52:40   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie             318               3q4bv6jx8wup5         29 14-may-10 13:49:59   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie             874               cmh3vh4pjs7q7          2 14-may-10 13:40:43   WEBUSERABC INACTIVE   WAITING
CFROM 13:55:17            SQL*Net message from clie            1040               8zv7177vuc8dt          1 14-may-10 13:37:57   WEBUSERABC INACTIVE   WAITING
               **********                                                                       ----------
               sum                                                                                      40

Whenever I ran the script, I could see that we had spikes of sessions running the same sql statement in the same second on any given host. Because the statement groups by “seconds_in_wait” we might see that say 25 sessions all were active 8 minutes ago with the same statement and have since done no work (waiting on “SQL*Net message from client”). When we looked at all the application hosts, each would spike at a 20 minute interval (although each host had its own time at which it occurred).

My company use a number of different application caching techniques in our middle-tier application, one of these was a concurrent “read- ahead” cache, this would allow a read from cache to detect its data was soon to expire and asynchronously request a load of the caches in parallel. Our “read-ahead” code had been set to allow 50 threads to get data, but we only had 40 connections in the database connection pool (each box). We had a 30 minute keep alive on the connections, but every 20 minutes exactly, we would touch every connection and keep them alive. The solution was to reduce the number of threads on the “read-ahead” and not even change the connection pool configuration. This fix automatically reduced the number of connections on the database by around 1000.

There have been so many examples where the script has helped, we leave it collecting every 5 minutes to a log file on the more sensitive databases. It can help identify cases where:

• Sudden concurrent requests by one or more severs or applications. We can get clues in what was the last sql statement run, who by and when
• Statements taking a long time on specific hosts (Look for “ACTIVE” sessions for long time on same sql)
• Idle connections which never get returned to the connection pool correctly within the application hosts
• user behaviour driving sudden spikes in concurrent requests from specific hosts
• sessions which are getting stuck on database waits (eg db links due to networking issues, row locking etc) and not returning to the pool
• What time an application host last sent any requests to the database

The script attempts to provide a fast summary of the connections on the database, how long they have been idle, what they last did, which current wait events are used.  Although the script is very simple, I have found it a really good way to summarise what our hosts (using connection pools)  are doing.

A colleague wrote a test case to show the issue for Oracle support so they could identify a fix and we could request a patch:

Source of the script:
Package Header (has no body!)

CREATE OR REPLACE PACKAGE tst_pkg_array IS

   TYPE typ_rec_1 IS RECORD(
       attr1  NUMBER(12)
      ,attr2  NUMBER(12)
      ,attr3  NUMBER(12)
      ,attr4  NUMBER(12)
      ,attr5  NUMBER(12)
      ,attr6  VARCHAR2(100)
      ,attr7  VARCHAR2(100)
      ,attr8  VARCHAR2(100)
      ,attr9  VARCHAR2(100)
      ,attr10 VARCHAR2(100)
      ,attr11 DATE
      ,attr12 DATE
      ,attr13 DATE
      ,attr14 DATE
      ,attr15 DATE
      ,attr16 NUMBER(12)
      ,attr17 NUMBER(12)
      ,attr18 NUMBER(12)
      ,attr19 NUMBER(12)
      ,attr20 NUMBER(12)
      ,attr21 VARCHAR2(100)
      ,attr22 VARCHAR2(100)
      ,attr23 VARCHAR2(100)
      ,attr24 VARCHAR2(100)
      ,attr25 VARCHAR2(100));

   TYPE typ_tab_1 IS TABLE OF typ_rec_1 INDEX BY PLS_INTEGER;

   tab_test_simple typ_tab_1;

END tst_pkg_array;
/

Test script to generate timings

set timing on

spool run.log append

set time on

select * from v$version;

set serveroutput on size 100000

DECLARE

   --
   PROCEDURE pr_put_elements_into_array(i_num_elements IN NUMBER) IS

      l_rec_test tst_pkg_array.typ_rec_1;

      tab_test_local tst_pkg_array.typ_tab_1;

   BEGIN
      --
      -- start by clearing down
      tst_pkg_array.tab_test_simple.delete;
      tst_pkg_array.tab_test_simple := tab_test_local;
      --
      l_rec_test.attr1  := 10000000;
      l_rec_test.attr2  := 10000000;
      l_rec_test.attr3  := 10000000;
      l_rec_test.attr4  := 10000000;
      l_rec_test.attr5  := 10000000;
      l_rec_test.attr6  := 'ABCDEFGH';
      l_rec_test.attr7  := 'ABCDEFGH';
      l_rec_test.attr8  := 'ABCDEFGH';
      l_rec_test.attr9  := 'ABCDEFGH';
      l_rec_test.attr10 := 'ABCDEFGH';
      l_rec_test.attr11 := SYSDATE;
      l_rec_test.attr12 := SYSDATE;
      l_rec_test.attr13 := SYSDATE;
      l_rec_test.attr14 := SYSDATE;
      l_rec_test.attr15 := SYSDATE;
      l_rec_test.attr16 := 10000000;
      l_rec_test.attr17 := 10000000;
      l_rec_test.attr18 := 10000000;
      l_rec_test.attr19 := 10000000;
      l_rec_test.attr20 := 10000000;
      l_rec_test.attr21 := 'ABCDEFGH';
      l_rec_test.attr22 := 'ABCDEFGH';
      l_rec_test.attr23 := 'ABCDEFGH';
      l_rec_test.attr24 := 'ABCDEFGH';
      l_rec_test.attr25 := 'ABCDEFGH';
      --
      FOR n IN 1 .. i_num_elements LOOP
         tst_pkg_array.tab_test_simple(n) := l_rec_test;
--       Using the line below instead of the one above (ie using a local variable rather than a package state variable) resolves the issue on 11.1.0.7
--       To test that, comment out the line above, and "comment in" the line below
--         tab_test_local(n) := l_rec_test;
      END LOOP;

   END pr_put_elements_into_array;

   PROCEDURE pr_run_test(i_num_elements IN NUMBER) IS
      l_duration_secs NUMBER;
      l_start_time    NUMBER;

   BEGIN

      l_start_time := DBMS_UTILITY.get_time;
      pr_put_elements_into_array(i_num_elements => i_num_elements);
      l_duration_secs := (DBMS_UTILITY.get_time - l_start_time) / 100;
      dbms_output.put_line(i_num_elements || ' elements took ' || (l_duration_secs) || ' secs or ' ||
                           (l_duration_secs / (i_num_elements / 1000)) || ' secs per thousand');
   END pr_run_test;
BEGIN
   pr_run_test(i_num_elements => 1000);
   pr_run_test(i_num_elements => 10000);
   pr_run_test(i_num_elements => 100000);
   pr_run_test(i_num_elements => 200000);
   pr_run_test(i_num_elements => 300000);
END;
/

exit

Output on 11.1.0.7

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
PL/SQL Release 11.1.0.7.0 - Production
CORE    11.1.0.7.0      Production
TNS for Linux: Version 11.1.0.7.0 - Production
NLSRTL Version 11.1.0.7.0 - Production

Elapsed: 00:00:00.40
1000 elements took .01 secs or .01 secs per thousand
10000 elements took .09 secs or .009 secs per thousand
100000 elements took 6.1 secs or .061 secs per thousand
200000 elements took 21.65 secs or .10825 secs per thousand
300000 elements took 46.55 secs or .1551666666666666666666666666666666666667
secs per thousand

PL/SQL procedure successfully completed.

Elapsed: 00:01:14.40

The above example shows the script took over 1 minute to complete and you can see the result degreades exponentially (per 1000 requests) as more entries are added to the array.

Output on 10.2.0.4

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi
PL/SQL Release 10.2.0.4.0 - Production
CORE    10.2.0.4.0      Production
TNS for Linux: Version 10.2.0.4.0 - Production
NLSRTL Version 10.2.0.4.0 - Production

Elapsed: 00:00:00.02
1000 elements took .01 secs or .01 secs per thousand
10000 elements took .05 secs or .005 secs per thousand
100000 elements took .51 secs or .0051 secs per thousand
200000 elements took 1.02 secs or .0051 secs per thousand
300000 elements took 1.17 secs or .0039 secs per thousand

PL/SQL procedure successfully completed.

Elapsed: 00:00:02.89

Patch details:

Patch 7671793: EXCESSIVE MEMORY USAGE WHEN USING KGHU
patch is avaiable for Linux x86-64)

Output once patch was applied:

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
PL/SQL Release 11.1.0.7.0 - Production
CORE    11.1.0.7.0      Production
TNS for Linux: Version 11.1.0.7.0 - Production
NLSRTL Version 11.1.0.7.0 - Production

Elapsed: 00:00:00.06
1000 elements took 0 secs or 0 secs per thousand
10000 elements took .05 secs or .005 secs per thousand
100000 elements took .55 secs or .0055 secs per thousand
200000 elements took .88 secs or .0044 secs per thousand
300000 elements took 1.25 secs or .004166666666666666666666666666666666666667
secs per thousand

PL/SQL procedure successfully completed.

Elapsed: 00:00:02.84

You can now see with the patch applied, the above test takes less than 3 seconds to complete.

Although our application uses plsql arrays in a very extreme way, it may be worth reviewing this patch if your application has some unexplained slow down and is using large plsql arrays AND 11.1.0.7.

I have tested this issue on 11.1.0.7.3 (latest PSU patch 9352179) and the problem still exists. It’s not clear if this is a generic bug or one specific to our platform (Linux x86-64). Anoyingly I have found that the available patch ( 7671793) clashes with all the avaiable  11.1.0.7 psu patches, so a merge patch would be needed or you could try a non 11.1.0.7 version of Oracle.

I want to try and combine two subjects in this entry:

• How you can try and investigate Oracle commands to see what they do, where Oracle stores it’s dictionary data and how you can get more information about an Oracle error.
• Trying to be topical, I thought I would use the example of enabling the Oracle 11gR2 Editions feature on a complex schema to allow multi versioning of code and hot releases.

One of the first issues you may find when you try and enable the Oracle 11gr2 Editions feature on a complex schema is the following error:

SQL> alter user test_codea enable editions;
alter user test_codea enable editions
*
ERROR at line 1:
ORA-38819: user TEST_CODEA owns one or more objects whose type is editionable
and that have noneditioned dependent objects

This is a very simple test script to cause the error (run on 11.2.0.1.1)

set echo on

rem set up two demo users called test_codea and test_codeb
connect / as sysdba
drop user test_codea cascade;
drop user test_codeb cascade;

create user test_codea identified by test_codea;
create user test_codeb identified by test_codeb;
grant create session,resource to test_codea;
grant create session,resource to test_codeb;

connect test_codea/test_codea

create or replace package ca as
    procedure a;
end;
/
create or replace package body ca as
    procedure a is
    begin
         null;
    end;
end;
/
grant execute on ca to test_codeb;
connect test_codeb/test_codeb

create or replace package cb as
    procedure b;
end;
/
create or replace package body cb as
    procedure b is
    begin
         test_codea.ca.a;    -- This procedure makes a call back to the procedure test_codeA
    end;
end;
/

rem confirm the code works
execute cb.b;

connect / as sysdba

rem now try and enable the 11gR2 Editions feature on the test_codeA user

alter user test_codea enable editions;

So what does the error mean and what can you do about it:

ORA-38819: user TEST_CODEA owns one or more objects whose type is editionable
and that have noneditioned dependent objects

Well, Oracle is kindly telling you that one or more objects in the schema is dependent on something which is not enabled for Editions, or another non Editioned schema will depend on your objects….. OR on objects which can never support Editions feature.

Well, kind Oracle gives as a nice work around to the issue…… or do they:

SQL> alter user test_codea enable editions force;

User altered.

SQL> connect test_codeb/test_codeb
Connected.
SQL> execute cb.b;
BEGIN cb.b; END;

*
ERROR at line 1:
ORA-04063: package body "TEST_CODEB.CB" has errors
ORA-06508: PL/SQL: could not find program unit being called: "TEST_CODEB.CB"
ORA-06512: at line 1

 

Oracle allows us to FORCE a user to be enabled for Editions however this can be dangerous because it can really break a complete schema in a very big way.

In the case of my simple example, there is a very simple solution, just enable Editions on the test_userB first so that when we enable Editions on test_userA both can now be versioned under the same Edition.

SQL> alter user test_codea enable editions;
alter user test_codea enable editions
*
ERROR at line 1:
ORA-38819: user TEST_CODEA owns one or more objects whose type is editionable
and that have noneditioned dependent objects
SQL>
SQL>
SQL>
SQL> alter user test_codeb enable editions;

User altered.

SQL> alter user test_codea enable editions;

User altered.

SQL>  connect test_codeb/test_codeb
Connected.
SQL> execute cb.b;

PL/SQL procedure successfully completed.

 

BUT, the above example is very simple case, what happens when you have dependency on objects which cannot be Editioned (ever!). There are a number of cases where objects can not be editioned and these objects must be moved out of the current schema to a non editioned schema. Examples of these objects are things like “user defined indexes” which include a user written plsql function, restrictions exist for the objects relating to Advanced Queues.

So, back to the other point of this post, how can we find out which objects are causing us problems so that we can fix or move them to another schema. Back to the good old 10046 trace. Lets trace the command and find out how Oracle is deciding which objects are invalid and need fixing before Editions can be enabled.

SQL> alter session set events '10046 trace name context forever, level 1';

Session altered.

SQL>
SQL> alter user test_codea enable editions;
alter user test_codea enable editions
*
ERROR at line 1:
ORA-38819: user TEST_CODEA owns one or more objects whose type is editionable
and that have noneditioned dependent objects

So if we look at the trace file we find the following internal Oracle sql statement along with a bunch of other stuff:

PARSING IN CURSOR #2 len=487 dep=1 uid=0 oct=3 lid=0 tim=1273060065216252 hv=305533609 ad='271fb5cf8' sqlid='btgrjvh93c4p9'
select d.owner#, u.name, d.name, d.namespace, d.stime    from obj$ d, dependency$ dep, obj$ p, user$ u    where d.obj# = dep.d_obj# and p.obj# = dep.p_obj#      and d.remoteowner is null and p.owner# = :1      and d.owner# = u.user#      and p.type# in (4,5,7,8,9,10,11,12,13,14,22,87)      and ((u.type#             != 2        and            bitand(u.spare1, 16) = 0        and            u.user#             != p.owner#) or           (d.type# not in (4,5,7,8,9,10,11,12,13,14,22,87)))
END OF STMT
PARSE #2:c=1000,e=758,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1273060065216251
EXEC #2:c=10999,e=10924,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=1111485757,tim=1273060065227267
FETCH #2:c=35994,e=37823,p=0,cr=872,cu=0,mis=0,r=1,dep=1,og=4,plh=1111485757,tim=1273060065265189

In the above trace output, we can see the SQL statement which Oracle runs to test for invalid objects which prevent the user Editions feature being enabled (In fact in my simple case, I can see “r=1″ on the FETCH meaning my simple case found 1 row matching invalid objects.

So, I can take that SQL statement and run it to find all objects which would cause problems enabling Editions and can fixed the problems in advance of trying to enable the editions feature:

Cleaned up version of the sql statement and output (Note: avoid running statements you extract this way on production systems unless you are really sure they will not have any adverse effects!)

connect / as sysdba
set echo on

rem find the user_id for the schema

select user_id from sys.dba_users where username = '&USERNAME'
/

set linesize 132

rem provide the user_id to the following sql statement

select d.owner#,
       u.name,
       d.name,
       d.namespace,
       d.stime
from   obj$ d,
       dependency$ dep,
       obj$ p, user$ u
where d.obj# = dep.d_obj#
and   p.obj# = dep.p_obj#
and   d.remoteowner is null
and   p.owner# = &user_id
and   d.owner# = u.user#
and p.type# in (4,5,7,8,9,10,11,12,13,14,22,87)
and ((u.type#             != 2
     and            bitand(u.spare1, 16) = 0
     and            u.user#             != p.owner#)
   or
     (d.type# not in (4,5,7,8,9,10,11,12,13,14,22,87)
     )
    )
/

Output from the script

    OWNER# NAME                           NAME                            NAMESPACE STIME
---------- ------------------------------ ------------------------------ ---------- ---------
       258 TEST_CODEB                     CB                                      2 05-MAY-10

When we ran this on one of our complex databases, we had 35 objects listed in just one schema which were preventing us enabling Editions (a test system I must point out… we are not ready for live yet). Most were the simple case of needing to also enable editions on other dependent schemas, but we also had problems with user written functional indexes, many of our Advanced Queues. The AQ and functional indexed had to be moved out to non editionable schemas.

summary

I hope I have shown the value of tracing Oracle commands, how you  might be able to use the information. I also hope you can see you need to be careful using “alter user x enable editions force” and the internal sql statement can help identify objects which cause issues.