PATCH 9239090

Adworker fails with below error




ATTENTION: All workers either have failed or are waiting:




FAILED: file bomprg.ldt on worker 2.

FAILED: file jg12acp.ldt on worker 3.



Dumping LCT file /db2/oracle/EBS/DEV1/apps/apps_st/appl/fnd/12.0.0/patch/115/import/afcpprog.lct(120.2.12010000.2) into FND_SEED_STAGE_CONFIG


Dumping LDT file /db2/oracle/EBS/DEV1/apps/apps_st/appl/bom/12.0.0/patch/115/import/US/bomprg.ldt into FND_SEED_STAGE_ENTITY

A database error occurred:

ORA-24816: Expanded non LONG bind data supplied after actual LONG or LOB column

 
 
Solution: Set the env and run the program manually or


To implement the solution, please execute the following steps:




If the Patch is stopped please just change setup for environment variable NLS_LANG to proper value and restart adpatch (continue previous session).



For example:



If the database characterset is UTF8 set NLS_LANG to American_America.UTF8.



a. Execute 'export' as follows.



# export NLS_LANG=American_America.UTF8



b. Query value for NLS_LANG to verify it.



#echo $NLS_LANG (Unix)

>echo %NLS_LANG% (Windows)



OR



If adpatch is still running but hanging due to failed workers:



1. Stop adpatch.



Please use adctrl utility to stop adpatch.



The best way to do this is to shut down the workers manually, which

also causes the AD utility to quit in an orderly fashion.



Action

a. Select AD Controller(adctrl) Option 3 and enter all for the worker number. Each worker

stops once it completes or fails its current job.



b. Verify that no worker processes are running. Use a command similar to the one

in the following example. The command arguments may vary on different

platforms.

UNIX:

$ ps -a
grep adworker

Windows:

Invoke the Task Manager (Ctrl-Alt-Delete) to view processes.



c. When all workers have shut down, the manager and the AD utility quits.



2. Change NLS_LANG setup.



Please refer to the example above.



3. Start adpatch again, continue previous session.



Check if AutoPatch is progressing, if not use adctrl utility, option 2 to restart failing worker.



4. How to avoid similar issue in the future.



Verify NLS_LANG setup in the Autoconfig Context file.



Edit the context file using Oracle Applications Manager(OAM) or Context Editor and find value for NLS_LANG:



If the database characterset is UTF8 it should be:



American_America.UTF8

American_America.UTF8

American_America.UTF8



If NLS_LANG parameters have different values than expected please apply corrections. Then execute Autoconfig and make sure it was successful.



5. Test if the changes to NLS_LANG were successful.



Source the env file and query:



#echo $NLS_LANG (Unix)

>echo %NLS_LANG% (Windows)

runInstaller doesnot display ASM disks during 11gR2 grid (cluster) installation

During 11gR2 Grid installation the candidate disks for vote and ocr were not displayed in the list screen. The issue may be many but one of the reason for this behaviour is the permission .



The runinstaller uses /usr/sbin/oracleasm-discover to to display the files and not oracleasm and its behaviour mainly depends on the filepermissoins under /dev/oracleasm

EG: ( First we will use root user and then the actual user trying to install the software )


[oragrd@appsonline-rac dev]$ cd oracleasm
[oragrd@appsonline-rac oracleasm]$ cd disks
[oragrd@appsonline-rac disks]$ ls
SOAPRD_DATA01 SOAPRD_DATA02 SOAPRD_DATA03 SOAPRD_DATA04 SOAPRD_FRA01 SOAPRD_FRA02 SOAPRD_FRA03 SOAPRD_FRA04 SOAPRD_VOTE01 SOAPRD_VOTE02 SOAPRD_VOTE03

mailto:root@appsonline-rac
/usr/sbin/oracleasm
[root@appsonline-rac disks]# ls -l /usr/sbin/oracleasm*
-rwxr-xr-x 1 root root 16600 Mar 18 2009 /usr/sbin/oracleasm
-rwxr-xr-x 1 root root 10664 Mar 18 2009 /usr/sbin/oracleasm-discover
[root@appsonline-rac disks]# which oracleasm-discover
/usr/sbin/oracleasm-discover

Run the Discover Command
 [root@appsonline-rac disks]# oracleasm-discover
Using ASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.so
ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]
Discovered disk: ORCL:SOAPRD_DATA01 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_DATA02 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_DATA03 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_DATA04 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_FRA01 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_FRA02 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_FRA03 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_FRA04 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_VOTE01 [2096753 blocks (1073537536 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_VOTE02 [2096753 blocks (1073537536 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_VOTE03 [2096512 blocks (1073414144 bytes), maxio 512]

Now Try executing the same commands as user than will own the Cluster , in my example its oragrd


[root@appsonline-rac disks]# su - oragrd
[oragrd@appsonline-rac ~]$ /usr/sbin/oracleasm-discover
Using ASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.so
asm_version() failed with code 2

If this fails then try tracing the session to find what exactly is the error. The above error is misleading and has nothing to do with library.

[oragrd@appsonline-rac ~]$ strace /usr/sbin/oracleasm-discover >/tmp/11.log
execve("/usr/sbin/oracleasm-discover", ["/usr/sbin/oracleasm-discover"], [/* 19 vars */]) = 0
brk(0) = 0x1692f000
mmap(NULL, 4096, PROT_READ
PROT_WRITE, MAP_PRIVATE
MAP_ANONYMOUS, -1, 0) = 0x2b0dd6b39000
uname({sys="Linux", node="appsonline-rac.oracle.com", ...}) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3

fstat(3, {st_mode=S_IFREG
0644, st_size=104289, ...}) = 0
mmap(NULL, 104289, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b0dd6b3a000
close(3) = 0
open("/lib64/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\16 \0331\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG
0755, st_size=23360, ...}) = 0
mmap(NULL, 4096, PROT_READ
PROT_WRITE, MAP_PRIVATE
MAP_ANONYMOUS, -1, 0) = 0x2b0dd6b54000
map(0x311b200000, 2109696, PROT_READ
PROT_EXEC, MAP_PRIVATE
MAP_DENYWRITE, 3, 0) = 0x311b200000
mprotect(0x311b202000, 2097152, PROT_NONE) = 0
mmap(0x311b402000, 8192, PROT_READ
PROT_WRITE, MAP_PRIVATE
MAP_FIXED
MAP_DENYWRITE, 3, 0x2000) = 0x311b402000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\332\241\0321\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG
0755, st_size=1713088, ...}) = 0
mmap(0x311aa00000, 3494168, PROT_READ
PROT_EXEC, MAP_PRIVATE
MAP_DENYWRITE, 3, 0) = 0x311aa00000
mprotect(0x311ab4c000, 2097152, PROT_NONE) = 0
mmap(0x311ad4c000, 20480, PROT_READ
PROT_WRITE, MAP_PRIVATE
MAP_FIXED
MAP_DENYWRITE, 3, 0x14c000) = 0x311ad4c000
mmap(0x311ad51000, 16664, PROT_READ
PROT_WRITE, MAP_PRIVATE
MAP_FIXED
MAP_ANONYMOUS, -1, 0) = 0x311ad51000
close(3) = 0
mmap(NULL, 4096, PROT_READ
PROT_WRITE, MAP_PRIVATE
MAP_ANONYMOUS, -1, 0) = 0x2b0dd6b55000
arch_prctl(ARCH_SET_FS, 0x2b0dd6b55800) = 0
mprotect(0x311b402000, 4096, PROT_READ) = 0
mprotect(0x311ad4c000, 16384, PROT_READ) = 0
mprotect(0x311a81b000, 4096, PROT_READ) = 0
munmap(0x2b0dd6b3a000, 104289) = 0
brk(0) = 0x1692f000
brk(0x16950000) = 0x16950000
open("/opt/oracle/extapi/64/asm/orcl/1/libasm.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\21\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG
755, st_size=18176, ...}) = 0
mmap(NULL, 2113440, PROT_READ
PROT_EXEC, MAP_PRIVATE
MAP_DENYWRITE, 3, 0) = 0x2b0dd6b56000
mprotect(0x2b0dd6b5a000, 2093056, PROT_NONE) = 0
mmap(0x2b0dd6d59000, 4096, PROT_READ
PROT_WRITE, MAP_PRIVATE
MAP_FIXED
MAP_DENYWRITE, 3, 0x3000) = 0x2b0dd6d59000


close(3) = 0
fstat(1, {st_mode=S_IFREG
644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ
PROT_WRITE, MAP_PRIVATE
MAP_ANONYMOUS, -1, 0) = 0x2b0dd6d5a000
open("/dev/oracleasm/.query_version", O_RDWR) = -1 EACCES (Permission denied)
open("/dev/oracleasm/iid", O_RDONLY) = -1 EACCES (Permission denied)
write(2, "asm_version() failed with code 2"..., 33asm_version() failed with code 2
) = 33


write(1, "Using ASMLib from /opt/oracle/ex"..., 61) = 61
exit_group(2)
= ?


The above error shows its unable to read some file under /dev/oracleasm . So change the permissions as below.

[root@appsonline-rac oracleasm]# ls -lart



total 0
-rwxrwx--- 1 root root 0 Jul 12 13:17 .query_version
-rwxrwx--- 1 root root 0 Jul 12 13:17 .query_disk
drwxrwx--- 1 root root 0 Jul 12 13:17 iid
-rwxrwx--- 1 root root 0 Jul 12 13:17 .get_iid
drwxr-xr-x 1 root root 0 Jul 12 13:17 disks
-rwxrwx--- 1 root root 0 Jul 12 13:17 .check_iid
rwxr-xr-x 4 root root 0 Jul 12 13:17 .
rwxr-xr-x 15 root root 6860 Jul 13 14:51 ..


[root@appsonline-rac oracleasm]# chown oragrd:asmdba /dev/oracleasm/.query_version


[root@appsonline-rac oracleasm]# chown -R oragrd:asmdba iid
[root@appsonline-rac oracleasm]# chown -R oragrd:asmdba .q*
drwxr-xr-x 15 root root 6860 Jul 13 14:51 ..
[root@appsonline-rac oracleasm]# chown -R oragrd:asmdba .check_iid .get_iid
[root@appsonline-rac oracleasm]# ls -lart
total 0
-rwxrwx--- 1 oragrd asmdba 0 Jul 12 13:17 .query_version
-rwxrwx--- 1 oragrd asmdba 0 Jul 12 13:17 .query_disk
drwxrwx--- 1 oragrd asmdba 0 Jul 12 13:17 iid
-rwxrwx--- 1 oragrd asmdba 0 Jul 12 13:17 .get_iid
drwxr-xr-x 1 root root 0 Jul 12 13:17 disks
-rwxrwx--- 1 oragrd asmdba 0 Jul 12 13:17 .check_iid
drwxr-xr-x 4 root root 0 Jul 12 13:17 .
drwxr-xr-x 15 root root 6860 Jul 13 14:51 ..

Rerun the command

[root@appsonline-rac oracleasm]# su - oragrd
[oragrd@appsonline-rac ~]$ /usr/sbin/oracleasm-discover
Using ASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]
iscovered disk: ORCL:SOAPRD_DATA01 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_DATA02 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_DATA03 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_DATA04 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_FRA01 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_FRA02 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_FRA03 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_FRA04 [93803472 blocks (48027377664 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_VOTE01 [2096753 blocks (1073537536 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_VOTE02 [2096753 blocks (1073537536 bytes), maxio 512]
Discovered disk: ORCL:SOAPRD_VOTE03 [2096512 blocks (1073414144 bytes), maxio 512]

Once you are done either restart the runinstaller or Browse and manually enter the discovery path.


Thanks , Sandarsh Chavalmane

Disabling RAC on 11gR2

There is some difference in enabling and disabling RAC on 11gR2 from the previous releases

Before we disable we need to verify if the products are really enabled.


cd $ORACLE_HOME/rdbms/lib

ar -t libknlopt.a | grep -c kcsm.o


would return 0 if RAC is disabled or a number >0 if enabled

For example:

Environment where RAC is disabled:

[oradev1@xxx lib]$ ar -t libknlopt.a |grep -c kcsm.o
0


Environment where RAC is enabled:

[oragrd@xxx lib]$ ar -t libknlopt.a |grep -c kcsm.o
1



DISABLE RAC:

To disable rac at oracle binary level we need to execute the below command.

make -f ins_rdbms.mk rac_off

After this ,relink the oracle exe

cd $ORACLE_HOME
make -f rdbms/lib/ins_rdbms.mk ioracle


Similarly we can use below key words to disable other components in 11gR2

• RAT - kecwr.o
• OLS - kzlibac.o
• DV - kzvidv.o
• ASM - kfon.o
• OLAP - xsyeolap.o
• PART - kkpoban.o
• CTX - kciwcx.o

where {option} is => OPTION - {option on} / {option off }

• RAT - rat_on / rat_off
• OLS - lbac_on / lbac_off
• DV - dv_on / dv_off
• ASM - asm_on / asm_off
• OLAP - olap_on / olap_off
• PART - part_on / part_off
• CTX - ctx_on / ctx_off

Thanks ,
Sandarsh Chavalmane

11gR2 nodeapps vip issue

VIP was down as it got attached to a different device.

11gR2 RAC environment has the below issues.


When we run a installer or cluster verify you could see that nodeapps is not running .

Verifying the nodeapps you see that one of the component (vip) is down.


[oragrd@XXX ~]$ srvctl status nodeapps -n XXX

VIP xxx-vip is enabled
VIP xxx-vip is not running
Network is enabled
Network is running on node: xxx
GSD is enabled
GSD is running on node: xxx
ONS is enabled
ONS daemon is running on node: xxx
eONS is enabled
eONS daemon is running on node: xxx



Check the status on cluster registry


$ crs_stat -t |grep vip
obi.vip application ONLINE ONLINE afso...b101
ora....101.vip ora....t1.type ONLINE OFFLINE
ora....102.vip ora....t1.type ONLINE OFFLINE
ora.scan1.vip ora....ip.type ONLINE ONLINE afso...b102
ora.scan2.vip ora....ip.type ONLINE ONLINE afso...b101
ora.scan3.vip ora....ip.type ONLINE ONLINE afso...b101


This indicates VIP is in inconsistent state. Now check weather vip is available on the network
Below command shows that vip is attached to eth3 and it is like a physical IP .


ifconfig -a


eth3 Link encap:Ethernet HWaddr F4:CE:46:AF:49:44
inet addr:10.32.200.151 Bcast:10.32.200.255 Mask:255.255.255.0
inet6 addr: fe80::f6ce:46ff:feaf:4944/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6116467 errors:0 dropped:0 overruns:0 frame:0
TX packets:70229 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1504763071 (1.4 GiB) TX bytes:3056240 (2.9 MiB)
Interrupt:139



10.32.200.151 is our vip and its running. Ideally it should not be as crs_stat is showing its down.
When you try restarting , you see that VIP is already in use in the cluster log

===============================

2010-05-27 14:56:15.509: [UiServer][1520855360] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-5005: IP Address: 10.32.200.151 is already in use in the network]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.SOAIAT2.lsnr afsoqatdb102 1]
WAIT:
TextMessage[0]

===============================



Work Around to fix this issue:

[oragrd@XXX ~]$ srvctl stop nodeapps -n XXX

Bringdown the device to which vip is attached

Run below command as root.

ifdown eth3



Run if config and make sure that eth3 is not running or attached to any device .

Then start the nodeapps

[oragrd@XXX ~]$ srvctl start nodeapps -n XXX

Check the status.

[oragrd@XXX ~]$ srvctl status nodeapps -n XXX
-n option has been deprecated.
VIP XXX-vip is enabled
VIP XXX-vip is running on node: XXX
Network is enabled
Network is running on node: XXX
GSD is enabled
GSD is running on node: XXX
ONS is enabled
ONS daemon is running on node: XXX
eONS is enabled
eONS daemon is running on node: XXX

Check on the cluster:

[oragrd@XXX ~]$ crs_stat -t | grep vip
obi.vip application ONLINE ONLINE afso...b101
ora....101.vip ora....t1.type ONLINE ONLINE afso...b101
ora....102.vip ora....t1.type ONLINE ONLINE afso...b102
ora.scan1.vip ora....ip.type ONLINE ONLINE afso...b101
ora.scan2.vip ora....ip.type ONLINE ONLINE afso...b101
ora.scan3.vip ora....ip.type ONLINE ONLINE afso...b101
[oragrd@XXX ~]$

Now make sure vip does not get attached to any device . In our case it was getting attached to eth3 . We need to disable it permanently

cd /etc/sysconfig/network-scripts
vi ifcfg-eth3

remove
IPADDR=
NETMASK=
PEERDNS=yes




This should prevent the device from picking up the IP after reboot.

Thanks ,
Sandarsh Chavalmane

Hyperion Installation document

Hyperion High Level Installation Steps

Download PDF with Architecture Diagram. Hyperion Installation

Here I am explaining about Hyperion installation on a distributed environment (High Level). Since planning and reporting are expected to consume high resources we kept these services separated.

Hyperion Installation Steps.

On LinuxDBHyperion.appsonlinedba.com
1) Install Oracle Binaries (11gR2) . Since we planned to test cold failover we had installed the oracle binaries on cluster.
2) Create a dummy (blank) db using DBCA.
3) Install 11gR2 client.
4) Install Hyperion essbase. You need to deselect all the other components which are selected by default.


On LinuxappHyperion.appsonlinedba.com
5) Install Foundation(Shared Services) and Financial reporting . By default essbase and planning are selected , deselect them before proceeding with the installation

On WindowsHyperion.appsonlinedba.com
6) Installation Planning . Here you need to deselect other components which are checked by default.

Hyperion Configuration Steps

Create Schema:

We can install each service on different schema or same schema. To isolate each module we created separate schemas

Hss: stands for Hyperion shared services
Hbi: Hyperion Business Intelligence
heas: Hyperion essbase administration services’
hes: Hyperion essbase studio

Configure different services:
When you invoke the installer it asks you about which service to configure make sure you do it as below on each server. Also make sure you select the correct schema created above .

a) Configure Shared service ( LinuxappHyperion)
b) Configure essbase ( LinuxDBHyperion )
c) Configure Planning ( WindowsHyperion )
d) Configure Reporting ( LinuxappHyperion )

Thanks ,
Sandarsh Chavalmane

11gR2 RAC , GRID infrastructure ASM ,vote ,ocr bla bla.... and a big confusion before you implement these.

11gR2 RAC , GRID infrastructure ASM :

Since I did my first implementation, I was interested in sharing

1) Why is VOTE and OCR inside ASM?

Storing the OCR and vote disk inside ASM eliminates the need for third party cluster and volume managers and also helps eliminate the complexity of managing separate disk partitions for the OCR in oracle clusterware installations.
Better manageability. and also Oracle WANT to SELL RAC with ASM ,....

2) How does cluster start as vote disk and OCR are inside ASM???

In 11GR2, every node has a local registry called OLR. OLR is created during the installation and by default its located under $GRID_HOME/cdata/.olr
This file actually facilitates the oracle cluterware to startup process when OCR and vote are inside ASM.

During the startup OLR is referenced to determine the exact location of the vote disks. This will facilitate the node to join the cluster. After the initial phase ASM is started and the processes that require full OCR can start and the clusterware startup process completes.

3) SAN and NAS.

I always assumed EMC sells only SAN . But they too have NAS storage.Since we prefer to have block storage for oracle database we opted SAN for oracle software and NAS for usual NFS.

Also good that now there is a EM plugin avaliable to monitor EMC disks.


Thanks

Sandarsh Chavalmane