The Enterprise Manager AWR Warehouse is designed to hold performance data from multiple databases for long-term analysis. It promoses that it will save storage and improve performance on your production systems. In that it is indeed correct. However the warehouse itself does not seem to be performant when taking in multiple sources and retaining them long-term – 400 days in our case. Why 400 days is an obvious question that might be asked. Primarily because we are a Retail organisation and Easter is variable each year.
The AWR repository database is performing poorly during the insert stage of the upload process.
Just to quickly summarise the process:
- A dmp file is extracted on the source database and transferred across to the AWR server
- The dmp file is then imported into a temporary schema called AWR$XXXXXX (this loads quickly)
- This data is then inserted into the main AWR tables inside the SYS schema. Is is this stage that is slow.
In order to completely isolate the issue, we altered a parameter, so only one AWR file gets loaded at once, cutting any contention / locking issues out of the equation:
UPDATE DBSNMP.CAW_PROPERTIESSET PROPERTY_VALUE=1where property_name=’max_parallel_loads’;
We can now see 2 sql statements running taking up large amounts of i/o. For a 30 minute trace, these 2 take up 22 minutes, mainly i/o reads. These are:
SQL ID: 130dvvr5s8bgn Plan Hash: 1160622595select obj#, dataobj#, part#, hiboundlen, hiboundval, ts#, file#, block#,pctfree$, pctused$, initrans, maxtrans, flags, analyzetime, samplesize,rowcnt, blkcnt, empcnt, avgspc, chncnt, avgrln, length(bhiboundval),bhiboundvalfromtabpart$ where bo# = :1 order by part#call count cpu elapsed disk query current rows——- —— ——– ———- ———- ———- ———- ———-Parse 3 0.00 0.00 0 0 0 0Execute 3 0.00 0.00 0 0 0 0Fetch 5 85.94 681.98 71875 260313 817 131255——- —— ——– ———- ———- ———- ———- ———-total 11 85.94 681.99 71875 260313 817 131255Misses in library cache during parse: 1Misses in library cache during execute: 1Optimizer mode: CHOOSEParsing user id: SYS (recursive depth: 4)Number of plan statistics captured: 2Rows (1st) Rows (avg) Rows (max) Row Source Operation———- ———- ———- —————————————————43847 43776 43847 TABLE ACCESS BY INDEX ROWID TABPART$ (cr=13860 pr=2 pw=0 time=1689938 us cost=4779 size=2879167 card=15907)43847 43776 43847 INDEX RANGE SCAN I_TABPART_BOPART$ (cr=400 pr=0 pw=0 time=311605 us cost=145 size=0 card=15907)(object id 571)Elapsed times include waiting on following events:Event waited on Times Max. Wait Total Waited—————————————- Waited ———- ————db file sequential read 69961 0.22 591.85latch: shared pool 8 0.00 0.00latch: row cache objects 1 0.02 0.02********************************************************************************SQL ID: c3zymn7x3k6wy Plan Hash: 3446064519select obj#, dataobj#, part#, hiboundlen, hiboundval, flags, ts#, file#,block#, pctfree$, initrans, maxtrans, analyzetime, samplesize, rowcnt,blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, pctthres$,length(bhiboundval), bhiboundvalfromindpart$ where bo# = :1 order by part#call count cpu elapsed disk query current rows——- —— ——– ———- ———- ———- ———- ———-Parse 2 0.00 0.00 0 0 0 0Execute 2 0.02 0.02 0 0 0 0Fetch 5 65.12 578.58 62301 174812 0 87553——- —— ——– ———- ———- ———- ———- ———-total 9 65.14 578.61 62301 174812 0 87553Misses in library cache during parse: 1Misses in library cache during execute: 1Optimizer mode: CHOOSEParsing user id: SYS (recursive depth: 4)Number of plan statistics captured: 2Rows (1st) Rows (avg) Rows (max) Row Source Operation———- ———- ———- —————————————————43847 43776 43847 TABLE ACCESS BY INDEX ROWID INDPART$ (cr=13241 pr=762 pw=0 time=6313753 us cost=4024 size=2224953 card=12861)43847 43776 43847 INDEX RANGE SCAN I_INDPART_BOPART$ (cr=424 pr=421 pw=0 time=4089911 us cost=128 size=0 card=12861)(object id 576)Elapsed times include waiting on following events:Event waited on Times Max. Wait Total Waited—————————————- Waited ———- ————db file sequential read 78826 0.28 655.63latch: shared pool 16 0.00 0.00********************************************************************************
Our guess is that AWR during inserts, tries to create a new partition for each snap shot, and it’s querying its metadata during this operation.
We have an SR open with Oracle Support, with no joy so far. There doesn’t appear to be much information available is on the web, so it may be that not many sites use this for the amount of source databases that we do – we upload and retain data from over 80 source databases.
Any help / advice re the above sqlid’s would be appreciated.
