marcel.vandewaters.nlMarcel's Oracle Database Blog

Blog Blog A blog about Oracle and Security Home About AYTS 2011 Downloads ODA Security Sourcecode Tools ← Older posts New Blog Posted on September 21, 2016 by marcel Hi all, as you have probably noticed, there hasn’t been a lot of activity on this blog for some time now. I have been working on a new blog on which I will post as of today. See http://blogs.vandewaters.nl/tech for more information. Best regards, Marcel Posted in Uncategorized | Leave a comment Oracle Database Appliance – Problem Replacing Shared Disk (2.4) Posted on May 23, 2013 by marcel As with all systems, a disk failure can happen and so we had a failing disk on one of our ODA’s. As mentioned in the ODA documentation, replacing a disk is easy and OAK does everything for you. Actually it is doing everything for you, from all OS related actions (like multipath configuration and partitioning) to dropping and adding the replaced disk in ASM. It all looked fine until we got a close look after OAK was done adding the disk. Replacing the disk introduced the following 2 problems: 1. The ASM alertlog keeps logging: WARNING: Read Failed. group:0 disk:1 AU:0 offset:0 size:4096 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_25561.trc: ORA-27061: waiting for async I/Os failed Linux-x86_64 Error: 5: Input/output error Additional information: -1 Additional information: 4096 and the /var/log/messages keeps logging messages like: May 23 22:14:58 emn-odadb-ts1-03 kernel: end_request: I/O error, dev dm-22, sector May 23 22:14:58 emn-odadb-ts1-03 kernel: Buffer I/O error on device dm-46, logical block 0 .... 2. The size of the second partition on the new disk (used by diskgroup RECO) is not the same as the size of all other (orginal) disks. Partition 2 on original disks are 80800 MB but the partition 2 on the new disk is just 75080 MB. Causes Cause problem 1: At first it looked like problem 1 (I/O errors in ASM alertlog and OS messages file) was because of the new disk also failing, but looking further the logfile /opt/oracle/oak/log/ ODA node name /oak/oakd.log showed the new disk got a new device name, was partitioned and correctly added to the RECO diskgroup. The I/O errors the alertlog and messages file are showing are about the OLD (physically remove) disk and it is pretty hard to read/write to a physically removed disk :-). The reason why the old disk device still exists, is because there are still processes (Grid Infra/ASM/Database) having open file descriptors to the old device so Linux is not able to remove the device. Use the following commands to get a list of process id’s for the processes that are still having open file descriptors to the device file of the removed disk: /sbin/fuser /dev/mapper/disknamep1 /sbin/fuser /dev/mapper/disknamep2 Example: /sbin/fuser /dev/mapper/HDD_E1_S05_992884312p2 /dev/mapper/HDD_E1_S05_992884312p2: 3254 3298 5196 ps -ef|grep 3298 grid 3298 1 0 2012 ? 00:00:40 asm_vbg0_+ASM1 Cause problem 2: The /sbin/parted command that is executed by OAK when the new disk is inserted, is called to create 2 partitions on the new disk with using an partition size for partition 2, starting at the next cylinder after where partition 1 ends and ending on the cylinder based on 99% of the disk. When deploying the ODA these partitions are defined exactly the same way, but it seems that due to a change in the parted utility or because the new disk is different (Vendor: HITACHI, Model: HUS1560SCSUN600G) than the original disk (Vendor: SEAGATE, Model: ST360057SSUN600G) the 99% results in different end cylinder number, resulting in a smaller partition (around 6 GB smaller). Solutions Solution problem 1: There is no real solution for getting rid of the old disk device and there is a generic ASM problem (MOS note 1485163.1: Disks of Dismounted Diskgroup Are Still Hold / Lock By Oracle Process) created, and still open, for this problem. It is not an ODA specific problem. The only way to get rid of the old device is to restart CRS stack per node, thus including all database instances running on that node. Solution problem 2: This problem is said to be fixed in ODA 2.6.0.0.0 where you have the oppertunity to reinitialize the disk using oakcli which will recreate the disk partitions with the correct size (and of course all OS and ASM related actions too). Posted in Database Appliance | Tagged 2.4.0.0.0 , ASM , bug , bundle patch , database appliance , grid infrastructure , LinkedIn , oak , ODA , oracle , oracle database appliance , troubleshooting | Leave a comment Oracle Database Appliance – /opt filling up Posted on May 23, 2013 by marcel Last time we had the /opt filesystem on a couple of ODA nodes filling up. It turned out that the OS Watcher archive directory structure ( /opt/oracle/oak/osw/archive ) contained lots of old files that should have been cleaned by OSWatcher. When OSWatcher is started, it will also start a script called OSWatcherFM.sh that is responsible for cleaning up old files. It turned out this OSWatcherFM.sh script was not running and so the archive directory structure was not cleaned up. Solution: The solution is simply to restart the OSWatcher which in turn will start the OSWatcherFM.sh script. Execute the following commands as user root : /opt/oracle/oak/osw/stopOSW.sh /opt/oracle/oak/osw/startOSW.sh 10 504 gzip Check if both OSWatcher.sh and OSWatcherFM.sh are running using: ps -ef|grep OSW oracle 10046 24783 0 10:23 pts/0 00:00:00 grep OSW root 12704 1 0 2012 ? 01:11:42 /usr/bin/ksh ./OSWatcher.sh 10 504 gzip root 12922 12704 0 2012 ? 00:16:48 /usr/bin/ksh ./OSWatcherFM.sh 504 Note: it can take a couple of seconds before the OSWatcherFM.sh script is started, so if it doesn’t show try again a couple of seconds later. The OSWachterFM.sh script will cleanup the old files after a couple of minutes, so after some minutes the /opt/oracle/oak/osw/archive directory structure will be cleaned up. Posted in Database Appliance | Tagged database appliance , LinkedIn , ODA , oracle , oracle database appliance , OSWatcher , troubleshooting | Leave a comment ORA-29283: invalid file operation using UTL_FILE.FREMOVE Posted on March 13, 2013 by marcel I’ve been breaking my head over a problem I had with the UTL_FILE.FREMOVE command trying to remove a file created by another (not oracle) process. On *nix command line, logged in as user oracle (same as database) I’m able to remove the file using the rm” command, but when using UTL_FILE.FREMOVE it comes back with ORA-29283. Sample of directory and file permission for this article: drwxrwsr-x oracle:oinstall /var/import Content of /var/import: rw-rr grid:oinstall something.tst The first thing I thought about, was that Oracle was looking at the privileges of the file (which shows that the group oinstall only is allowed to READ the file) before it would remove it, but the Oracle documentation clearly states that: The FREMOVE procedure does not verify privileges before deleting a file. The O/S verifies file and directory permissions. Because on the directory level oracle (through being part of the oinstall group) is allowed to remove files from the directory, as is also proved by the rm” command, and UTL_FILE.FREMOVE does not verify the permissions, UTL_FILE.FREMOVE should be able todo so too. So why is the command failing? Quick answer : Oracle actually does some sort of a permission check before removing the file! Using the truss” command on an Oracle session using UTL_FILE.FREMOVE showed me that Oracle is executing the following system calls when removing a file with UTL_FILE.FREMOVE: 1. Get file attributes (that does not include permissions) stat(/var/import/something.tst”, …………. lstat(/var/import/something.tst”, …………. 2. Open the file using file access modes for writing and appending open(/var/import/something.tst”, O_WRONGLY|O_APPEND) 3. Get information about the open file descriptor fstat(……..) 4. Close the open file descriptor close(……..) 5. Actually remove the file unlink(/var/import/something.tst”)...