Wednesday, July 15, 2015

Exadata patching Bios boot order is incorrect

Links to this post

in April - 2015 PSU , when we upgrade database node upgrade and again do ./dbnodeupdate.sh -c.

we may get below warning. to encounter it.

Warning: Bios boot order is incorrect - the system may have booting issues in a next reboot

if we encounter warning then do

ubiosconfig list status

if status pending then execute below command.

ubiosconfig cancel config

ubiosconfig list status

check status will be OK from Pending.

Monday, May 18, 2015

Exadata X5 Hybrid Rack | OVM and Physical Database Nodes in One Rack.

Links to this post
For my own curiosity, I was creating OEDA for latest Exadata X5 half rack and found that we can have Physical compute nodes and OVM Compute nodes in same cluster & in Same Rack.



Wednesday, May 13, 2015

Change Exadata Write-Back Flash cache in Rolling Mode.

Links to this post
---Make sure you have both of this files into one directory.

 [root@dummyCNadm01 ~]# pwd  
 /root  
 rwxr--r-- 1 root root 141286 May 12 13:47 setWBFC.sh  
 -rwxr-xr-x 1 root root  683 May 12 13:47 wbfc_FLUSH.sh  

----From compute Node. Run Pre-check to see if cell servers are ready to flip over WBFC .

 [root@dummyCNadm01 ~]# ./setWBFC.sh -g cell_group -l /tmp -m WriteBack -o rolling -p  
 ./setWBFC.sh: Using log directory '/tmp'  
 ./setWBFC.sh: Log File '/tmp/setWBFC_85166_2015-05-12-13:50:03.log' created successfully  
 2015-05-12 13:50:03  
 Starting ./setWBFC.sh on dummyCNadm01  
 Version: 1.0.0.1.6.20140716  
 Command line options used:  
  -g cell_group  
  -o rolling  
  -m WriteBack  
  -p (Perform pre-req checks only)  
  -t 21600  
  -x 0  
 2015-05-12 13:50:03  
 Performing pre-req checks.....  
 2015-05-12 13:50:03  
 Creating baseline inventory for griddisks  
 2015-05-12 13:50:06  
 Creating baseline inventory for flashdisks  
 2015-05-12 13:50:08  
 Creating baseline inventory for flashsize  
 2015-05-12 13:50:10  
 dcli present and in PATH.            [PASSED]  
 2015-05-12 13:50:10  
 Checking cell nodes are valid storage servers...  
 2015-05-12 13:50:10  
 All cells are valid Exadata storage cells.  
 2015-05-12 13:50:10  
 Checking Exadata Storage Software Versions...  
 2015-05-12 13:50:15  
 Software versions of the following cells:  
 dummy01celadm01: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm02: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm03: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm04: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm05: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm06: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm07: 12.1.2.1.1.150316.2           [PASSED]  
 2015-05-12 13:50:15  
 Checking Grid Infrastructure Software Version...  
 2015-05-12 13:50:17  
 Grid Infrastructure version: 12.1.0.2.00     [PASSED]  
 2015-05-12 13:50:17  
 Checking for active ASM operations....  
 2015-05-12 13:50:18  
 Check for no active ASM operations:       [PASSED]  
 2015-05-12 13:50:18  
 Checking griddisk status across all cells....  
 2015-05-12 13:50:20  
 All griddisks across all cells have asmdeactivationoutcome = Yes  
 All griddisks across all cells are ONLINE  
 Griddisk checks:                 [PASSED]  
 2015-05-12 13:50:20  
 Checking flash cache status.....  
 2015-05-12 13:50:21  
 Flashcache status normal:            [PASSED]  
 2015-05-12 13:50:21  
 Checking that all FlashDisks are present...  
 2015-05-12 13:50:22  
 FlashDisk validation:              [PASSED]  
 2015-05-12 13:50:22  
 Checking current flash cache mode.....  
 2015-05-12 13:50:23  
 Flashcache not already in target mode:      [PASSED]  
 2015-05-12 13:50:23  
 All pre-req checks completed:          [PASSED]  
 2015-05-12 13:50:24  
 dummy01celadm01: flashcache size: 5.82122802734375T  
 dummy01celadm02: flashcache size: 5.82122802734375T  
 dummy01celadm03: flashcache size: 5.82122802734375T  
 dummy01celadm04: flashcache size: 5.82122802734375T  
 dummy01celadm05: flashcache size: 5.82122802734375T  
 dummy01celadm06: flashcache size: 5.82122802734375T  
 dummy01celadm07: flashcache size: 5.82122802734375T  
 There are 7 storage cells to process.  

---- Turn write back ON in rolling fashion .

 [root@dummyCNadm01 ~]# ./setWBFC.sh -g cell_group -l /tmp -m WriteBack -o rolling  
 2015-05-12 14:59:41  
 Starting ./setWBFC.sh on dummy01dbadm01  
 Version: 1.0.0.1.6.20140716  
 Command line options used:  
  -g cell_group  
  -o rolling  
  -m WriteBack  
  -t 21600  
  -x 0  
 2015-05-12 14:59:41  
 Performing pre-req checks.....  
 2015-05-12 14:59:41  
 Creating baseline inventory for griddisks  
 2015-05-12 14:59:42  
 Creating baseline inventory for flashdisks  
 2015-05-12 14:59:43  
 Creating baseline inventory for flashsize  
 2015-05-12 14:59:44  
 dcli present and in PATH.            [PASSED]  
 2015-05-12 14:59:44  
 Checking cell nodes are valid storage servers...  
 2015-05-12 14:59:44  
 All cells are valid Exadata storage cells.  
 2015-05-12 14:59:44  
 Checking Exadata Storage Software Versions...  
 2015-05-12 14:59:48  
 Software versions of the following cells:  
 dummy01celadm01: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm02: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm03: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm04: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm05: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm06: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm07: 12.1.2.1.1.150316.2           [PASSED]  
 2015-05-12 14:59:48  
 Checking Grid Infrastructure Software Version...  
 2015-05-12 14:59:51  
 Grid Infrastructure version: 12.1.0.2.00     [PASSED]  
 2015-05-12 14:59:51  
 Checking for active ASM operations....  
 2015-05-12 14:59:51  
 Check for no active ASM operations:       [PASSED]  
 2015-05-12 14:59:51  
 Checking griddisk status across all cells....  
 2015-05-12 14:59:54  
 All griddisks across all cells have asmdeactivationoutcome = Yes  
 All griddisks across all cells are ONLINE  
 Griddisk checks:                 [PASSED]  
 2015-05-12 14:59:54  
 Checking flash cache status.....  
 2015-05-12 14:59:55  
 Flashcache status normal:            [PASSED]  
 2015-05-12 14:59:55  
 Checking that all FlashDisks are present...  
 2015-05-12 14:59:57  
 FlashDisk validation:              [PASSED]  
 2015-05-12 14:59:57  
 Checking current flash cache mode.....  
 2015-05-12 14:59:58  
 Flashcache not already in target mode:      [PASSED]  
 2015-05-12 14:59:58  
 All pre-req checks completed:          [PASSED]  
 2015-05-12 14:59:59  
 dummy01celadm01: flashcache size: 5.82122802734375T  
 dummy01celadm02: flashcache size: 5.82122802734375T  
 dummy01celadm03: flashcache size: 5.82122802734375T  
 dummy01celadm04: flashcache size: 5.82122802734375T  
 dummy01celadm05: flashcache size: 5.82122802734375T  
 dummy01celadm06: flashcache size: 5.82122802734375T  
 dummy01celadm07: flashcache size: 5.82122802734375T  
 There are 7 storage cells to process.  
 2015-05-12 14:59:59  
 Changing flash cache to WriteBack ROLLING....  
 2015-05-12 14:59:59  
 STEP 0: Checking gridisk status on cell: dummy01celadm01  
 2015-05-12 15:00:01  
 STEP 0 completed successfully on cell: dummy01celadm01  
 2015-05-12 15:00:04  
 STEP 1: Dropping flashcache on cell: dummy01celadm01  
 2015-05-12 15:00:36  
 STEP 1: Completed sucessfully on cell: dummy01celadm01  
 2015-05-12 15:00:39  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:00:39  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:00:39  
 STEP 4: Set the flashCachMode on cell: dummy01celadm01  
 2015-05-12 15:00:39  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:00:40  
 STEP 4: Completed successfully on cell: dummy01celadm01  
 2015-05-12 15:00:43  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:00:43  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:00:43  
 STEP 7: Creating flashcache on cell: dummy01celadm01  
 2015-05-12 15:01:32  
 STEP 7: Completed sucessfully on cell: dummy01celadm01  
 2015-05-12 15:01:35  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm01  
 2015-05-12 15:01:36  
 STEP 8: Completed sucessfully on cell: dummy01celadm01  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:01:39  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:01:39  
 STEP 0: Checking gridisk status on cell: dummy01celadm02  
 2015-05-12 15:01:41  
 STEP 0 completed successfully on cell: dummy01celadm02  
 2015-05-12 15:01:44  
 STEP 1: Dropping flashcache on cell: dummy01celadm02  
 2015-05-12 15:02:19  
 STEP 1: Completed sucessfully on cell: dummy01celadm02  
 2015-05-12 15:02:22  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:02:22  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:02:22  
 STEP 4: Set the flashCachMode on cell: dummy01celadm02  
 2015-05-12 15:02:22  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:02:23  
 STEP 4: Completed successfully on cell: dummy01celadm02  
 2015-05-12 15:02:26  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:02:26  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:02:26  
 STEP 7: Creating flashcache on cell: dummy01celadm02  
 2015-05-12 15:03:21  
 STEP 7: Completed sucessfully on cell: dummy01celadm02  
 2015-05-12 15:03:24  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm02  
 2015-05-12 15:03:25  
 STEP 8: Completed sucessfully on cell: dummy01celadm02  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:03:28  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:03:28  
 STEP 0: Checking gridisk status on cell: dummy01celadm03  
 2015-05-12 15:03:29  
 STEP 0 completed successfully on cell: dummy01celadm03  
 2015-05-12 15:03:32  
 STEP 1: Dropping flashcache on cell: dummy01celadm03  
 2015-05-12 15:04:07  
 STEP 1: Completed sucessfully on cell: dummy01celadm03  
 2015-05-12 15:04:10  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:04:10  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:04:10  
 STEP 4: Set the flashCachMode on cell: dummy01celadm03  
 2015-05-12 15:04:10  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:04:10  
 STEP 4: Completed successfully on cell: dummy01celadm03  
 2015-05-12 15:04:13  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:04:13  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:04:13  
 STEP 7: Creating flashcache on cell: dummy01celadm03  
 2015-05-12 15:05:09  
 STEP 7: Completed sucessfully on cell: dummy01celadm03  
 2015-05-12 15:05:12  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm03  
 2015-05-12 15:05:12  
 STEP 8: Completed sucessfully on cell: dummy01celadm03  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:05:15  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:05:15  
 STEP 0: Checking gridisk status on cell: dummy01celadm04  
 2015-05-12 15:05:17  
 STEP 0 completed successfully on cell: dummy01celadm04  
 2015-05-12 15:05:20  
 STEP 1: Dropping flashcache on cell: dummy01celadm04  
 2015-05-12 15:05:53  
 STEP 1: Completed sucessfully on cell: dummy01celadm04  
 2015-05-12 15:05:56  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:05:56  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:05:56  
 STEP 4: Set the flashCachMode on cell: dummy01celadm04  
 2015-05-12 15:05:56  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:06:01  
 STEP 4: Completed successfully on cell: dummy01celadm04  
 2015-05-12 15:06:04  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:06:04  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:06:04  
 STEP 7: Creating flashcache on cell: dummy01celadm04  
 2015-05-12 15:06:47  
 STEP 7: Completed sucessfully on cell: dummy01celadm04  
 2015-05-12 15:06:50  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm04  
 2015-05-12 15:06:51  
 STEP 8: Completed sucessfully on cell: dummy01celadm04  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:06:54  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:06:54  
 STEP 0: Checking gridisk status on cell: dummy01celadm05  
 2015-05-12 15:06:55  
 STEP 0 completed successfully on cell: dummy01celadm05  
 2015-05-12 15:06:58  
 STEP 1: Dropping flashcache on cell: dummy01celadm05  
 2015-05-12 15:07:40  
 STEP 1: Completed sucessfully on cell: dummy01celadm05  
 2015-05-12 15:07:43  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:07:43  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:07:43  
 STEP 4: Set the flashCachMode on cell: dummy01celadm05  
 2015-05-12 15:07:43  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:07:43  
 STEP 4: Completed successfully on cell: dummy01celadm05  
 2015-05-12 15:07:46  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:07:46  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:07:46  
 STEP 7: Creating flashcache on cell: dummy01celadm05  
 2015-05-12 15:08:37  
 STEP 7: Completed sucessfully on cell: dummy01celadm05  
 2015-05-12 15:08:40  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm05  
 2015-05-12 15:08:41  
 STEP 8: Completed sucessfully on cell: dummy01celadm05  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:08:44  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:08:44  
 STEP 0: Checking gridisk status on cell: dummy01celadm06  
 2015-05-12 15:08:46  
 STEP 0 completed successfully on cell: dummy01celadm06  
 2015-05-12 15:08:49  
 STEP 1: Dropping flashcache on cell: dummy01celadm06  
 2015-05-12 15:09:23  
 STEP 1: Completed sucessfully on cell: dummy01celadm06  
 2015-05-12 15:09:26  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:09:26  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:09:26  
 STEP 4: Set the flashCachMode on cell: dummy01celadm06  
 2015-05-12 15:09:26  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:09:30  
 STEP 4: Completed successfully on cell: dummy01celadm06  
 2015-05-12 15:09:33  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:09:33  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:09:33  
 STEP 7: Creating flashcache on cell: dummy01celadm06  
 2015-05-12 15:10:19  
 STEP 7: Completed sucessfully on cell: dummy01celadm06  
 2015-05-12 15:10:22  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm06  
 2015-05-12 15:10:23  
 STEP 8: Completed sucessfully on cell: dummy01celadm06  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:10:26  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:10:26  
 STEP 0: Checking gridisk status on cell: dummy01celadm07  
 2015-05-12 15:10:28  
 STEP 0 completed successfully on cell: dummy01celadm07  
 2015-05-12 15:10:31  
 STEP 1: Dropping flashcache on cell: dummy01celadm07  
 2015-05-12 15:11:04  
 STEP 1: Completed sucessfully on cell: dummy01celadm07  
 2015-05-12 15:11:07  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:11:07  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:11:07  
 STEP 4: Set the flashCachMode on cell: dummy01celadm07  
 2015-05-12 15:11:07  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:11:13  
 STEP 4: Completed successfully on cell: dummy01celadm07  
 2015-05-12 15:11:16  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:11:16  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:11:16  
 STEP 7: Creating flashcache on cell: dummy01celadm07  
 2015-05-12 15:12:02  
 STEP 7: Completed sucessfully on cell: dummy01celadm07  
 2015-05-12 15:12:05  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm07  
 2015-05-12 15:12:06  
 STEP 8: Completed sucessfully on cell: dummy01celadm07  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:12:09  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:12:09  
 Validating inventory for griddisks  
 2015-05-12 15:12:11  
 Validation of griddisk:             [PASSED]  
 2015-05-12 15:12:11  
 Validating inventory for flashdisks  
 2015-05-12 15:12:13  
 Validation of flashdisk:             [PASSED]  
 2015-05-12 15:12:13  
 Validating inventory for flashsize  
 2015-05-12 15:12:14  
 Validation of flashsize:             [PASSED]  
 2015-05-12 15:12:14  
 Setting flash cache to WriteBack completed successfully.  

dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode"

Wednesday, May 6, 2015

cp preserving permissions for | Operation not supported | NFS

Links to this post
This issue was in Exalogic ZFS , when we had exported shares via NFS3 over to few Linux Guest VMs.

 [root@dummyserver uxyz]# cp -p /uxyz/log.txt /uxyz/test.log  
 cp: preserving permissions for `/uxyz/test.log': Operation not supported  

----As you can see + sign for ACL permissions

 [root@dummyserver ~]# cd /uxyz  
 [root@dummyserver uxyz]# ll  
 -rw-r--r--+ 1 appUSER dba 0 May 6 15:59 bkp.txt  
 drwxr-sr-x+ 3 appUSER dba 3 Apr 25 02:22 home  
 -rw-r--r--+ 1 appUSER dba 0 May 6 15:59 log.txt  
 -rw-r--r--+ 1 appUSER dba 0 May 6 15:59 test.txt  

/etc/fstab file entry looked like this .

 000.00.0.9:/export/share-2-uxyz     /uxyz      nfs rw,nointr,bg,hard,timeo=600,wsize=65536,rsize=65536,nfsvers=3,tcp  

--to overcome this issue. append "noacl" at end of line.

 000.00.0.9:/export/share-2-uxyz     /uxyz      nfs rw,nointr,bg,hard,timeo=600,wsize=65536,rsize=65536,nfsvers=3,tcp,noacl  

----dont try to mount -o noacl,remount /u11
----unmount properly

 [root@dummyserver ~]# umount /uxyz  
 [root@dummyserver ~]# mount /uxyz  

------Magic. .

 [appUSER@dummyserver ~]$ cp -p /uxyz/log.txt /uxyz/test.log  
 [appUSER@dummyserver ~]$ ll  

-----There is no + sign for ACL.

 [root@dummyserver uxyz]# ll  
 total 10  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 bkp.txt  
 drwxr-sr-x 3 appUSER dba 3 Apr 25 02:22 home  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 log.txt  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 test.log  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 test.txt  

---Reference.

Solaris 10 NFS client fails in ACL or permissions when copying files to Linux NFS server (Doc ID 1917349.1)
cp ACL Error Creating OSB Domain, 'Operation not supported' (Doc ID 1959764.1)

Tuesday, May 5, 2015

Exadata Local Read-only file system | End_request: I/O error

Links to this post
---On database Nodes we wont be able to write on / or any local file system.

 [root@dummyhostname01 ~]# df  
 Filesystem       1K-blocks    Used  Available Use% Mounted on  
 /dev/mapper/VGExaDb-LVDbSys1  
             30963708  22372264   7018580 77% /  
 tmpfs         264152064     4  264152060  1% /dev/shm  
 /dev/sda1         516040   40016   449812  9% /boot  
 /dev/mapper/VGExaDb-LVDbOra1  
             103212320  25806260  72163180 27% /u01  
 [root@dummyhostname01 oracle.cellos]# cd conf  
 [root@dummyhostname01 conf]# ls  
 ls: reading directory .: Input/output error  
 [root@dummyhostname01 log]# cd /u01  
 [root@dummyhostname01 u01]# ls  
 app lost+found  
 [root@dummyhostname01 u01]# ll  
 total 20  
 drwxr-xr-x 5 root oinstall 4096 Mar 23 19:11 app  
 drwx------ 2 root root   16384 Mar 10 19:26 lost+found  
 [root@dummyhostname01 u01]# touch test  
 touch: cannot touch `test': Read-only file system  


---On looking at tail -f /var/log/messeges

 May 5 02:18:24 dummyhostname01 adclient[21135]: INFO <fd:25 sudo(100222)> client.sudo Set credentials for user 'root': mapping misconfiguration. Passing user to next service module.  
 May 5 02:18:26 dummyhostname01 kernel: megaraid_sas: Iop2SysDoorbellIntfor scsi0  
 May 5 02:18:27 dummyhostname01 kernel: megasas: Found FW in FAULT state, will reset adapter scsi0.  
 May 5 02:18:27 dummyhostname01 kernel: megaraid_sas: resetting fusion adapter scsi0.  
 May 5 02:18:27 dummyhostname01 kernel: megaraid_sas: Reset not supported, killing adapter scsi0.  
 May 5 02:18:27 dummyhostname01 kernel: sd 0:2:0:0: [sda] Unhandled error code  
 May 5 02:18:27 dummyhostname01 kernel: sd 0:2:0:0: [sda] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK  
 May 5 02:18:27 dummyhostname01 kernel: sd 0:2:0:0: [sda] CDB: Write(10): 2a 00 16 b5 5d 40 00 00 08 00  
 May 5 02:18:27 dummyhostname01 kernel: blk_update_request: 4 callbacks suppressed  
 May 5 02:18:27 dummyhostname01 kernel: end_request: I/O error, dev sda, sector 380984640  
 May 5 02:18:27 dummyhostname01 kernel: Buffer I/O error on device dm-3, logical block 17606304  

---firmware / hardware diagnostic on Database Node.
---As you can not write on Local Filesystem you could write on NFS mounts if there is present on DBNODE.

 dmesg > /NFS_MOUNT/sundiag/dmesg.txt   
 ipmitool sunoem cli force 'show /SP/console/history' > /NFS_MOUNT/sundiag/console.out   
 /opt/MegaRAID/MegaCli/MegaCli64 -FwTermLog Dsply -a0 >/NFS_MOUNT/sundiag/fwterm.txt   
 [root@dummyhostname01 sundiag]# cat /NFS_MOUNT/sundiag/fwterm.txt  
 User specified controller is not present.  
 Failed to get CpController object.  
 Exit Code: 0x01  
 /opt/MegaRAID/MegaCli/MegaCli64 -AdpEventLog -GetEvents -f /NFS_MOUNT/sundiag/events.txt -a0   
 lspci -vvv > /NFS_MOUNT/sundiag/lspci.out  

Solution.

--Finally reboot / restart force fully from ILOM.

 -> stop -f /SYS  
 Are you sure you want to immediately stop /SYS (y/n)? y  
 Stopping /SYS immediately  

Friday, March 6, 2015

Find DML SQL statement literal values caused Replicate to Abend.

Links to this post
Generally in Goldengate replicat apply SQL using bind variable so when error occurs report file or error log will have SQL statement with bind
variable. This is hard to catch which literal values caused a problem at Target side. Below is example to see Which literal values caused a
problem.

Error log / Reporfile (Normal Mode)
2013-05-15 11:08:31  WARNING OGG-01004  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
Aborted grouped transaction on 'DUMMY_ADMIN_6.DUMMYEVENT', Database error 1403 (OCI Error ORA-01403: no data found, SQL ).

2013-05-15 11:08:31  WARNING OGG-01154  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
SQL error 1403 mapping DUMMY_ADMIN.DUMMYEVENT to DUMMY_ADMIN_6.DUMMYEVENT OCI Error ORA-01403: no data found, SQL .

2013-05-15 11:08:31  ERROR   OGG-01296  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
Error mapping from DUMMY_ADMIN.DUMMYEVENT to DUMMY_ADMIN_6.DUMMYEVENT.

Edit below parameters in replicat and start again.

NOBINARYCHARS
NODYNSQL
SHOWSYNTAX

Error log / Reporfile (Actual value)
2013-05-15 11:17:35  WARNING OGG-01004  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
Aborted grouped transaction on 'DUMMY_ADMIN_6.DUMMYEVENT', Database error 1403 (OCI Error ORA-01403: no data found, SQL ).
2013-05-15 11:17:35  WARNING OGG-01154  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  

SQL error 1403 mapping DUMMY_ADMIN.DUMMYEVENT to DUMMY_ADMIN_6.DUMMYEVENT OCI Error ORA-01403: no data found, SQL .
2013-05-15 11:17:35  ERROR   OGG-01296  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
Error mapping from DUMMY_ADMIN.DUMMYEVENT to DUMMY_ADMIN_6.DUMMYEVENT.

Wednesday, February 25, 2015

RS-700 Celloflsrv hang detected on Exadata 12.1.1.1

Links to this post
RS-700 [Celloflsrv hang detected. It will be terminated] [SYS_121111_140712] [] [] [] [] [] [] [] [] [] []
Server Model Oracle Corporation SUN SERVER X4-2L High Capacity
Release Version 12.1.1.1.1
Release Label OSS_12.1.1.1.1_LINUX.X64_140712

This is Bug 19132065 - Oracle Linux semtimedop() wakeups by timeout are lagging causing offload operations to fail (which may degrade performance) and errors similar to one or more of the following:
? ORA-700 [Offload issue job timed out]
? ORA-700 [Offload group not open]
? RS-700 [Celloflsrv hang detected. It will be terminated]

This bug affects related to 12.1.1.1. storage Version.
It is due to DB Node RCU delayed and cause Offload job to fail on Cellservices .
it affects database performance not availability.
Error ocure mostly when cellserv tried to do Read optimization.
reducing Delay in RCU is work around accross whole stack.

Step 1: Set rcu_delay for runtime

# echo 1 > /proc/sys/kernel/rcu_delay
Verify the setting
# cat /proc/sys/kernel/rcu_delay
1

Step 2: Set rcu_delay in /etc/sysctl.conf for proper setting upon reboot

Add "kernel.rcu_delay=1" to /etc/sysctl.conf

Step 3: Restart cellsrv on storage servers

CellCLI> alter cell restart services cellsrv;


This workaround is automatically applied in the following cases:
When a new system is deployed with Exadata 11.2.3.3.1 or 12.1.1.1.1 using OEDA Sep 2014 or later.
When storage servers are upgraded to 11.2.3.3.1 or 12.1.1.1.1 and the patchmgr plugins patch is properly staged before running patchmgr, as documented.
When database servers are upgraded to 11.2.3.3.1 or 12.1.1.1.1 using dbnodeupdate.sh v3.58 or later.

References https://www.kernel.org/doc/Documentation/RCU/whatisRCU.txt