PCP in non-RAC instances - Pitfalls

It was a wonderful surprise when I stumbled upon Metalink Note ID: 743716.1, which details steps for configuring PCP in non-RAC instances.

This is very useful while using hardware or software clustering and not Oracle clustering (RAC).

We are using Veritas clustering (software clustering) for server failover. We were recently testing server failover and there are two things that I am unable to figure out as of now and which necessitates manual intervention rendering the whole failover mechanism manual:

1. How would the VNC server failover when using PCP with non-RAC?

Since, the hostname changes when the failover happens, the display variable would still be pointing to the failed server and not the failover server.

The work around to the above problem is obviously to reset the display variable and run autoconfig, which would mean that the failover mechanism is no longer automatic.

2. As soon as the failover happens, we need to make a manual entry in the Web Tier $TNS_ADMIN/tnsnames.ora to reflect the change in the concurrent processing server. This entry takes the form FNDFS_<FAILOVER_SERVER_NAME>.. Thankfully, this does NOT necessitate downtime of the instance.

Do post your comments/suggestions/ideas on the above 2 problems.

Issue while exporting /importing the oracle outlines from one database to other

On of the DBA complained that he had created 25 outlines in the test instance while tuning the bad sql's. Now he wanted to move the outlines to production.
One of the method to achieve this is to export the tables from the outl schema of the source (DEV) to target(PROD)
Both the export and import utilities executed without any issues.

25 rows were exported from DEV and then imported to source(PROD).

But when we count the dba_outlines table it showed 21 rows , 4 outlines were missing.


Below are the details

--------------------EXPORT COMMAND on DEV-------------------




exp / file=/tmp/outl2.dmp log=/tmp/outl2.log CONSTRAINTS=n TRIGGERS=n tables=outln.ol\$,outln.ol\$hints,outln.ol\$nodes grants=n

Export: Release 10.2.0.3.0 - Production on Mon Mar 23 18:26:03 2009



About to export specified tables via Conventional Path ...
Current user changed to OUTLN
. . exporting table OL$ 25 rows exported
. . exporting table OL$HINTS 488 rows exported
. . exporting table OL$NODES 733 rows exported
Export terminated successfully without warnings.




-------------------import COMMAND on PROD-------------------------

Import the outlines on the target

ibisdev-e2-zone02.east:PATCH4:/orahome $ imp / file=/tmp/outl2.dmp log=/tmp/outlBTCsingle.log CONSTRAINTS=n grants=n fromuser=outln touser=outln ignore=y commit=y

Import: Release 10.2.0.3.0 - Production on Mon Mar 23 18:28:26 2009


Export file created by EXPORT:V10.02.01 via conventional path
import done in US7ASCII character set and AL16UTF16 NCHAR character set
import server uses UTF8 character set (possible charset conversion)
export client uses UTF8 character set (possible charset conversion)
. importing OUTLN's objects into OUTLN
. . importing table "OL$" 25 rows imported
. . importing table "OL$HINTS" 488 rows imported
. . importing table "OL$NODES" 733 rows imported
Import terminated successfully without warnings.


---- below sql shows that there are only 21 outline on PROD -----

login to database

sqlplus "/ as sysdba"

select count(*) from dba_outlines;

Count(*)
------------------------------------------

21

------------------------------------------



Below you can see a difference in the header hint count .

If there is a hint ol$ hint count is less than ol$hints then the import utility will first import the data and then sync both the tables.
If the ol$ hint count is more than ol$hints then the data will be first imported and then deleted due to hintcount mismatch.

In other words

You need to ensure that you use a similar SQL statement to swap the counts, because if the hint count and the number of hints do not match, then one of the closing procedural steps in the import will
be to delete any plan which is internally inconsistent - i.e. where the actual count doesn't match the recorded count.


Only way to overcome this is to update the outln.ol$ hintcount to the value less than or equal to hintcount of outln.ol$hints

=================Below query on PROD shows that there are no hints mismatch between the tables==================

SQL> clear buffer
buffer cleared
SQL> 1 select a.OL_NAME,a.HINTCOUNT,count(b.OL_NAME)
2 from outln.ol$ a,
3 outln.ol$hints b
4 where a.OL_NAME = b.OL_NAME
5 group by a.OL_NAME,a.HINTCOUNT
6* having count(b.OL_NAME) <> a.HINTCOUNT SQL> SQL> SQL> SQL> SQL>
SP2-0734: unknown command beginning "6* having ..." - rest of line ignored.
SQL>
SQL> 6 having count(b.OL_NAME) <> a.HINTCOUNT
SQL> /

no rows selected

===============================Shows that there are only 21 outlines on PROD and has the hints in sync===========================================

SQL> l
1 select a.OL_NAME,a.HINTCOUNT,count(b.OL_NAME)
2 from outln.ol$ a,
3 outln.ol$hints b
4 where a.OL_NAME = b.OL_NAME
5 group by a.OL_NAME,a.HINTCOUNT
6* having count(b.OL_NAME) <> a.HINTCOUNT
SQL> del 6
SQL> /
CR_6788803_FNDWFBG 22 22
CR_6807075_CWP_05 9 9
CWP_ACCT_INFO_01 27 27
CR_6805769_PR_08 36 36
CR_6807045_CWP_01_OLD 17 17
CR_6805769_PR_02 17 17
CR_6788792_ARBARL 22 22
CR_6805769_PR_03 12 12
CR_6805769_PR_04 17 17
CR_6805769_PR_10 20 20
CR_6789052_FNDWFBG 22 22
CR_6794612_CWP_04 12 12
CR_6805769_PR_12 18 18
CR_6807240_AMSRFSEG 18 18
CR_6794623_CWP_07 12 12
CR_6797878_ITA_SETUP 25 25
CR_6805769_PR_06 12 12
CR_6807068_CWP_03 29 29
CR_6805769_PRICE_REPORT 17 17
CR_6805769_PR_07 12 12
CR_6809701_OE_EBFF 53 53

21 rows selected.





-----------------------------------COMMAND shows thate outlines that have hintcount difference on DEV---------------------------------------------

SQL> select a.OL_NAME,a.HINTCOUNT,count(b.OL_NAME)
from outln.ol$ a,
outln.ol$hints b
where a.OL_NAME = b.OL_NAME
group by a.OL_NAME,a.HINTCOUNT 2 3 4 5
6 having count(b.OL_NAME) <> a.HINTCOUNT
7 ;

OL_NAME HINTCOUNT COUNT(B.OL_NAME)
------------------------------ ---------- ----------------
CR_6807045_CWP_01 18 17
CR_6807062_CWP_02 23 22
CR_6807090_CWP_09 24 14
CR_6788803_FNDWFBG 22 24
CR_6789052_FNDWFBG 22 24
CR_6809701_OE_EBFF 53 54

6 rows selected.


--------------------------------------------------------------------------------------------------------
Still we can see only 4 outline from the above six are missing , it is because the ol$ hintcount is LESS than ol$hints in the .




===================SOLUTION=========================

----------perform the below steps on source(DEVELOPMENT)-----------


SQL>select hintcount
from outln.ol$
where
OL_NAME='CR_6807045_CWP_01';

HINTCOUNT
----------
18

SQL> update outln.ol$
set hintcount=17
where OL_NAME='CR_6807045_CWP_01';

1 row updated.

SQL> select hintcount from outln.ol$ where OL_NAME='CR_6807045_CWP_01';

HINTCOUNT
----------
17

SQL> commit;

Commit complete.

SQL> !
$

------------EXPORT COMMAND on DEV after making the hints on ol$ equal to ol$hints------------------------------------



$ exp / file=/tmp/outl2.dmp log=/tmp/outl2.log CONSTRAINTS=n TRIGGERS=n tables=outln.ol\$,outln.ol\$hints,outln.ol\$nodes grants=n

Export: Release 10.2.0.3.0 - Production on Mon Mar 23 18:26:03 2009

Copyright (c) 1982, 2005, Oracle. All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP
and Data Mining options
Export done in UTF8 character set and AL16UTF16 NCHAR character set
Note: grants on tables/views/sequences/roles will not be exported
Note: constraints on tables will not be exported

About to export specified tables via Conventional Path ...
Current user changed to OUTLN
. . exporting table OL$ 25 rows exported
. . exporting table OL$HINTS 488 rows exported
. . exporting table OL$NODES 733 rows exported
Export terminated successfully without warnings.


---------------------import COMMAND on PROD-----------------------



:/orahome $ imp / file=/tmp/outl2.dmp log=/tmp/outlBTCsingle.log CONSTRAINTS=n grants=n fromuser=outln touser=outln ignore=y commit=y

Import: Release 10.2.0.3.0 - Production on Mon Mar 23 18:28:26 2009

Copyright (c) 1982, 2005, Oracle. All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP and Data Mining options

Export file created by EXPORT:V10.02.01 via conventional path
import done in US7ASCII character set and AL16UTF16 NCHAR character set
import server uses UTF8 character set (possible charset conversion)
export client uses UTF8 character set (possible charset conversion)
. importing OUTLN's objects into OUTLN
. . importing table "OL$" 25 rows imported
. . importing table "OL$HINTS" 488 rows imported
. . importing table "OL$NODES" 733 rows imported
Import terminated successfully without warnings.



--------------------------VALIDATING THE FIX ---------------------

:/orahome $
me $ sqlplus

SQL*Plus: Release 10.2.0.3.0 - Production on Mon Mar 23 18:28:40 2009

Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

Enter user-name: / as sysdba

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP and Data Mining options

SQL> select count(*) from dba_outlines;

COUNT(*)
----------
22


This shows that this time import has imported 22 outline (21 + the one which was fixed)

You may have to perform the workaround on all the outlines that have a hints mismatch (ol$ hintcount > ol$hints hintcount)


About EXPORT and IMPORT functionality with outline as an example


1) check - select * from EXPACT$ where owner = 'OUTLN'; - Import runs this to cleanup

Some of these rows are cleared out at later point by the packages called in order to ensure the internal consistency of the metadata.

The packages called are outline.drop_collision, outline.drop_unrefd_hints, outline.drop_extras. These packages are called as an Import/Export post table action and the main purpose of this procedure is to clean up after an import. The drop of outlines may be because they are simply not required in the target database to obtain plan stability. The definitions for all of these procedures can be found in dbmsol.sql. Taking
a look at the usage notes, there would appear to be some inconsistency which gets cleared up by these functions during export/import. This could explain the missing rows.


The outln$ table has a 'hintcount' column.

You need to ensure that you use a similar SQL statement to swap the counts, because if the hint count and the number of hints do not match, then one of the closing procedural steps in the import will
be to delete any plan which is internally inconsistent - i.e. where the actual count doesn't match the recorded count.


You might try establishing a db_link between the two databases and thendo an update using a slect from the other table. This would bypass the export/import actions that are deleting your new outlines even as itadds them. Another possibility if db_links aren't workable is to import into a different user then use the update using a select from that users tables.

Thanks, Sandarsh