Friday, 25 August 2017

Oracle Database 12c Release 2 (12.2) RAC On Oracle Linux 7 Using VirtualBox Part-1

This article describes the installation of Oracle Database 12c Release 2 (12.2 64-bit) RAC on Linux (Oracle Linux 7 64-bit) using VirtualBox (5.1) with no additional shared disk devices.

  • Introduction


One of the biggest obstacles preventing people from setting up test RAC environments is the requirement for shared storage. In a production environment, shared storage is often provided by a SAN or high-end NAS device, but both of these options are very expensive when all you want to do is get some experience installing and using RAC. A cheaper alternative is to use virtualization to fake the shared storage.

Using VirtualBox you can run multiple Virtual Machines (VMs) on a single server, allowing you to run both RAC nodes on a single machine. In addition, it allows you to set up shared virtual disks, overcoming the obstacle of expensive shared storage.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Before you launch into this installation, here are a few things to consider.
  1. The finished system includes the host operating system, two guest operating systems, two sets of Oracle Grid Infrastructure (Clusterware + ASM) and two Database instances all on a single server. As you can imagine, this requires a significant amount of disk space, CPU and memory.
  2. Following on from the last point, the VMs will each need at least 4G of RAM, preferably more if you don't want the VMs to swap like crazy. Don't assume you will be able to run this on a small PC or laptop. You won't.
  3. This procedure provides a bare bones installation to get the RAC working. There is no redundancy in the Grid Infrastructure installation or the ASM installation. To add this, simply create double the amount of shared disks and select the "Normal" redundancy option when it is offered. Of course, this will take more disk space.
  4. During the virtual disk creation, I always choose not to preallocate the disk space. This makes virtual disk access slower during the installation, but saves on wasted disk space. The shared disks must have their space preallocated.
  5. This is not, and should not be considered, a production-ready system. It's simply to allow you to get used to installing and using RAC.
  6. The Single Client Access Name (SCAN) should be defined in the DNS or GNS and round-robin between one of 3 addresses, which are on the same subnet as the public and virtual IPs. Prior to 11.2.0.2 it could be defined as a single IP address in the "/etc/hosts" file, which is wrong and will cause the cluster verification to fail, but it allowed you to complete the install without the presence of a DNS. This does not seem to work for 11.2.0.2 onward.
  7. The virtual machines can be limited to 2Gig of swap, which causes a prerequisite check failure, but doesn't prevent the installation working. If you want to avoid this, define 3+Gig of swap.
  8. This article uses the 64-bit versions of Oracle Linux and Oracle 12c Release 2.
  9. When doing this installation on my server, I split the virtual disks on to different physical disks ("/u02", "/u03", "/u04"). This is not necessary, but makes things run a bit faster.
This procedure should run successfully on a Linux and Windows host. Where applicable, I've included both the Linux and Windows commands to be run on the host, so pick the relevant ones.

 Although I have completed this installation with 4G VMs, it was chronically slow. I would suggest using as much memory as possible, without making the host OS swap.

  • Download Software


Download the following software.
Depending on your version of VirtualBox and Oracle Linux, there may be some slight variation in how the screen shots look.

  • VirtualBox Installation


First, install the VirtualBox software. On RHEL and its clones you do this with the following type of command as the root user. On windows, just run the ".exe".

# rpm -Uvh VirtualBox*.rpm

The package name will vary depending on the host distribution you are using. Once complete, VirtualBox is started from the menu.

  • VirtualBox Network Setup


We need to make sure a host-only network is configured and check/modify the IP range for that network. This will be the public network for our RAC installation.
  1. Start VirtualBox from the menu.
  2. Select the "File > Preferences" menu option.
  3. Click "Network" in the left pane and click the "Host-only Networks" tab.
  4. Click the "Adds new host-only network" button on the right size of the screen. Depending on the host OS, a network called "vboxnet0" or "VirtualBox Host-Only Ethernet Adapter" will be created.
Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Click the "Edits selected host-only network." button on the right size of the screen.
  • If you want to use a different subnet for your public addresses you can change the network details here. Just make sure the subnet you choose doesn't match any real subnets on your network. I've decided to stick with the default, which for me is "192.168.56.X".
Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications
  • Use the "OK" buttons to exit out of this screen.
  • Click the "NAT Networks" tab. If you don't have a NAT network defined, click the "+" button. You shouldn't need to configure this.
  • Click the "OK" button to close the preferences dialog.

  • Virtual Machine Setup


Now we must define the two virtual RAC nodes. We can save time by defining one VM, then cloning it when it is installed.

Start VirtualBox and click the "New" button on the toolbar. Enter the name "ol7-122-rac1", OS "Linux" and Version "Oracle (64 bit)", then click the "Next" button.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Enter "4096" as the base memory size, then click the "Next" button. Use more memory if you have enough physical memory on your machine as it will make the process much quicker!

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Accept the default option to create a new virtual hard disk by clicking the "Create" button.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Acccept the default hard drive file type by clicking the "Next" button.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Acccept the "Dynamically allocated" option by clicking the "Next" button.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Accept the default location and set the size to "50G", then click the "Create" button. If you can spread the virtual disks onto different physical disks, that will improve performance.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

The "ol7-122-rac1" VM will appear on the left hand pane. Scroll down the details on the right and click on the "Network" link.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Make sure "Adapter 1" is enabled, set to "NAT", then click on the "Adapter 2" tab.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Make sure "Adapter 2" is enabled, set to "Host-only Adapter", then click on the "Adapter 3" tab.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Make sure "Adapter 3" is enabled, set to "Internal Network", then click on the "System" section.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Move "Hard Disk" to the top of the boot order and uncheck the "Floppy" option, then click the "OK" button.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

The virtual machine is now configured so we can start the guest operating system installation.

  • Guest Operating System Installation


With the new VM highlighted, click the "Start" button on the toolbar. On the "Select start-up disk" screen, choose the relevant Oracle Linux ISO image and click the "Start" button.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

If a "Select start-up disk" screen doesn't appear, use the "Devices > Optical Drives > Choose disk image..." menu option to select the relevant ISO image, then restart the VM using the "Machine > Reset" menu option.

The resulting console window will contain the Oracle Linux boot screen.

Oracle Database 12c, Oracle Database Tutorials and Materials, Oracle Database Certifications

Continue through the Oracle Linux 7 installation as you would for a basic server. A general pictorial guide to the installation can be found here. More specifically, it should be a server installation with a minimum of 4G+ swap, firewall disabled, SELinux set to permissive and the following package groups installed:
  • Server with GUI
  • Hardware Monitoring Utilities
  • Large Systems Performance
  • Network file system client
  • Performance Tools
  • Compatibility Libraries
  • Development Tools
To be consistent with the rest of the article, the following information should be set during the installation.
  • hostname: ol7-122-rac1.localdomain
  • enp0s3 (eth0): DHCP (Connect Automatically)
  • enp0s8 (eth1): IP=192.168.56.101, Subnet=255.255.255.0, Gateway=192.168.56.1, DNS=192.168.56.1, Search=localdomain (Connect Automatically)
  • enp0s9 (eth2): IP=192.168.1.101, Subnet=255.255.255.0, Gateway=<blank>, DNS=<blank>, Search=<blank> (Connect Automatically)
You are free to change the IP addresses to suit your network, but remember to stay consistent with those adjustments throughout the rest of the article. Likewise, in this article I will refer to the network adapters as enp0s3, enp0s8 and enp0s9, In previous Linux versions they would have been eth0, eth1 and eth2 respectively.

  • Oracle Installation Prerequisites


Perform either the Automatic Setup or the Manual Setup to complete the basic prerequisites. The Additional Setup is required for all installations.

    • Automatic Setup

If you plan to use the "oracle-database-server-12cR2-preinstall" package to perform all your prerequisite setup, issue the following command.

# yum install oracle-database-server-12cR2-preinstall -y

It is probably worth doing a full update as well, but this is not strictly speaking necessary.

# yum update -y

    • Manual Setup

If you have not used the "oracle-database-server-12cR2-preinstall" package to perform all prerequisites, you will need to manually perform the following setup tasks.

Add the following lines to the "/etc/sysctl.conf" file, or in a file called "/etc/sysctl.d/98-oracle.conf".

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

Run the following command to change the current kernel parameters.

/sbin/sysctl -p

Add the following lines to a file called "/etc/security/limits.d/oracle-database-server-12cR2-preinstall.conf" file.

oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc    16384
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768
oracle   hard   memlock    134217728
oracle   soft   memlock    134217728

In addition to the basic OS installation, the following packages must be installed whilst logged in as the root user. This includes the 64-bit and 32-bit versions of some packages.

# From Public Yum or ULN
yum install binutils -y
yum install compat-libstdc++-33 -y
yum install compat-libstdc++-33.i686 -y
yum install gcc -y
yum install gcc-c++ -y
yum install glibc -y
yum install glibc.i686 -y
yum install glibc-devel -y
yum install glibc-devel.i686 -y
yum install ksh -y
yum install libgcc -y
yum install libgcc.i686 -y
yum install libstdc++ -y
yum install libstdc++.i686 -y
yum install libstdc++-devel -y
yum install libstdc++-devel.i686 -y
yum install libaio -y
yum install libaio.i686 -y
yum install libaio-devel -y
yum install libaio-devel.i686 -y
yum install libXext -y
yum install libXext.i686 -y
yum install libXtst -y
yum install libXtst.i686 -y
yum install libX11 -y
yum install libX11.i686 -y
yum install libXau -y
yum install libXau.i686 -y
yum install libxcb -y
yum install libxcb.i686 -y
yum install libXi -y
yum install libXi.i686 -y
yum install make -y
yum install sysstat -y
yum install unixODBC -y
yum install unixODBC-devel -y
yum install zlib-devel -y
yum install zlib-devel.i686 -y

Create the new groups and users.

groupadd -g 54321 oinstall
groupadd -g 54322 dba
groupadd -g 54323 oper
#groupadd -g 54324 backupdba
#groupadd -g 54325 dgdba
#groupadd -g 54326 kmdba
#groupadd -g 54327 asmdba
#groupadd -g 54328 asmoper
#groupadd -g 54329 asmadmin
#groupadd -g 54330 racdba

useradd -u 54321 -g oinstall -G dba,oper oracle
You could define the additional groups and assign them to the "oracle" users. The would allow you to assign the individual groups during the installation. For this installation I've just used the "dba" group.

groupadd -g 54324 backupdba
groupadd -g 54325 dgdba
groupadd -g 54326 kmdba
groupadd -g 54327 asmdba
groupadd -g 54328 asmoper
groupadd -g 54329 asmadmin
groupadd -g 54330 racdba

useradd -u 54321 -g oinstall -G dba,oper,backupdba,dgdba,kmdba,asmdba,asmoper,asmadmin,racdba oracle

    • Additional Setup

The following steps must be performed, whether you did the manual or automatic setup.

Perform the following steps whilst logged into the "ol7-122-rac1" virtual machine as the root user.

Set the password for the "oracle" user.

passwd oracle

Apart form the localhost address, the "/etc/hosts" file can be left blank, but I prefer to put the addresses in for reference.

127.0.0.1       localhost.localdomain   localhost
# Public
192.168.56.101   ol7-122-rac1.localdomain        ol7-122-rac1
192.168.56.102   ol7-122-rac2.localdomain        ol7-122-rac2
# Private
192.168.1.101   ol7-122-rac1-priv.localdomain   ol7-122-rac1-priv
192.168.1.102   ol7-122-rac2-priv.localdomain   ol7-122-rac2-priv
# Virtual
192.168.56.103   ol7-122-rac1-vip.localdomain    ol7-122-rac1-vip
192.168.56.104   ol7-122-rac2-vip.localdomain    ol7-122-rac2-vip
# SCAN
#192.168.56.105   ol7-122-scan.localdomain ol7-122-scan
#192.168.56.106   ol7-122-scan.localdomain ol7-122-scan
#192.168.56.107   ol7-122-scan.localdomain ol7-122-scan

The SCAN address is commented out of the hosts file because it must be resolved using a DNS, so it can round-robin between 3 addresses on the same subnet as the public IPs. The DNS can be configured on the host machine using BIND or Dnsmasq, which is much simpler. If you are using Dnsmasq, put the RAC-specific entries in the hosts machines "/etc/hosts" file, with the SCAN entries uncommented, and restart Dnsmasq.

Make sure the "/etc/resolv.conf" file includes a nameserver entry that points to the correct nameserver. Also, if the "domain" and "search" entries are both present, comment out one of them. For this installation my "/etc/resolv.conf" looked like this.

#domain localdomain
search localdomain
nameserver 192.168.56.1

The changes to the "resolv.conf" will be overwritten by the network manager, due to the presence of the NAT interface. For this reason, this interface should now be disabled on startup. You can enable it manually if you need to access the internet from the VMs. Edit the "/etc/sysconfig/network-scripts/ifcfg-enp0s3" (eth0) file, making the following change. This will take effect after the next restart.

ONBOOT=no

There is no need to do the restart now. You can just run the following command. Remember to amend the adapter name if yours are named differently.

# ifdown enp0s3
# #ifdown eth0

At this point, the networking for the first node should look something like the following. Notice that enp0s3 (eth0) has no associated IP address because it is disabled.

# ifconfig
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 08:00:27:f6:88:78  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.101  netmask 255.255.255.0  broadcast 192.168.56.255
        inet6 fe80::cf8d:317d:534:17d9  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:82:06:32  txqueuelen 1000  (Ethernet)
        RX packets 574  bytes 54444 (53.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 547  bytes 71219 (69.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.169.1.101  netmask 255.255.255.0  broadcast 192.169.1.255
        inet6 fe80::9a9a:f249:61d1:5447  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:2e:2c:cf  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 29  bytes 4250 (4.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 68  bytes 5780 (5.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 68  bytes 5780 (5.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:4a:12:2f  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

#
With this in place and the DNS configured the SCAN address is being resolved to all three IP addresses.

# nslookup ol7-122-scan
Server: 192.168.56.1
Address: 192.168.56.1#53

Name: ol7-122-scan.localdomain
Address: 192.168.56.105
Name: ol7-122-scan.localdomain
Address: 192.168.56.106
Name: ol7-122-scan.localdomain
Address: 192.168.56.107

#
Change the setting of SELinux to permissive by editing the "/etc/selinux/config" file, making sure the SELINUX flag is set as follows.

SELINUX=permissive

If you have the Linux firewall enabled, you will need to disable or configure it, as shown here or here. The following is an example of disabling the firewall.

# systemctl stop firewalld
# systemctl disable firewalld
Make sure NTP (Chrony on OL7/RHEL7) is enabled.

# systemctl enable chronyd.service
# systemctl start chronyd.service

Create the directories in which the Oracle software will be installed.

mkdir -p /u01/app/12.2.0.1/grid
mkdir -p /u01/app/oracle/product/12.2.0.1/db_1
chown -R oracle:oinstall /u01
chmod -R 775 /u01/

Log in as the "oracle" user and add the following lines at the end of the "/home/oracle/.bash_profile" file.

# Oracle Settings
export TMP=/tmp
export TMPDIR=$TMP

export ORACLE_HOSTNAME=ol7-122-rac1.localdomain
export ORACLE_UNQNAME=CDBRAC
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/12.2.0.1/grid
export DB_HOME=$ORACLE_BASE/product/12.2.0.1/db_1
export ORACLE_HOME=$DB_HOME
export ORACLE_SID=cdbrac1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'

Create a file called "/home/oracle/grid_env" with the following contents.

export ORACLE_SID=+ASM1
export ORACLE_HOME=$GRID_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

Create a file called "/home/oracle/db_env" with the following contents.

export ORACLE_SID=cdbrac1
export ORACLE_HOME=$DB_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

Once the "/home/oracle/.bash_profile" has been run, you will be able to switch between environments as follows.

$ grid_env
$ echo $ORACLE_HOME
/u01/app/12.2.0.1/grid
$ db_env
$ echo $ORACLE_HOME
/u01/app/oracle/product/12.2.0.1/db_1
$

We've made a lot of changes, so it's worth doing a reboot of the VM at this point to make sure all the changes have taken effect.

# shutdown -r now