Engineered Systems

Oracle database appliance v1 - rac in a box – some strings attached Fuad Arshad, Walgreens Company

ABSTRACT The Oracle Database Appliance is an engineered system that is targeted towards small and medium businesses, but also has all of the bells and whistles required for enterprise deployments. This paper will introduce the Oracle Database Appliance (ODA), and review the deployment of the ODA from a non sysadmin perspective. This paper includes the deployment and patching processes and the best practices built into the Oracle Database Appliance. The paper will review the DBA's role in managing the ODA

TARGET AUDIENCE This paper is intended mostly for Database Architects and people that deploy Oracle database technologies. This paper will review various techniques for deploying the Oracle Database Appliance (ODA) successfully as well as how to perform day to day activities. The various components all tied together provide an effective use case for companies that want to invest in Engineered Systems, and provide an entry point to understand the management and maintenance considerations for deploying an engineered system.

EXECUTIVE SUMMARY Attendees and readers of this paper will learn the following after reading this paper.

Oracle Database Appliance - Overview

Oracle Database Appliance - Predeployment

Oracle Database Appliance - Deployment

Oracle Database Appliance - Patching

Oracle Database Appliance - One Button Database Appliance

INTRODUCTION The Oracle Database Appliance (ODA) is an entry level engineered system that provides easy access to preconfigured Real Application Cluster (RAC) database hardware and software. The ODA is accessible to customers at an entry level cost with software licensing available starting from 2 cores to 24 cores. The ODA is intended to provide companies with an easy to deploy model that includes all of the hardware and software required to run a complete system into one 4u rack mountable unit.

INSIDE THE ORACLE DATABASE APPLIANCE The Oracle Database Appliance (ODA) is a 4u rack mountable unit which consists of two 2u Sun Fire X4370 M2 units known as Server units (SN). Oracle has provided MOS Note 1385831.1 which is kept fairly up-to-date with the ODA components and software versions. Below is a snapshot as of Dec 14, 2012 and this paper only focuses on V1 functionality.

1 108

Engineered Systems

Sun X4370M2 System Server Components (2 Server Nodes per X4370M2)

CPU 2x 6-core Intel Xeon X5675 3.07GHz

Memory 96 GB RAM (12 x DDR3-1333 8GB DIMMs)

Network2x 10GbE (SFP+) PCIe card

6x 1GbE PCIe card2x 1GbE (Intel 82571) on-board Integrated redundant cluster interconnect

Internal Storage2x 500GB SATA - for operating system

1x 4GB USB internal

RAID Controller

2x LSI SAS9211-8i SAS HBA

Shared Storage

20x 600GB - 3.5" SAS 15k RPM HDD (Seagate Cheetah) - for RDBMS DATA (any slot except the top row of disks)

4x 73GB - 3.5" SAS2 SSDs - for RDBMS REDO (slot in the top row of 4 disks)SSD from STEC (ZeusIOPS - multi-level cell (MLC) version with SAS interface)

Operating System

Oracle Enterprise Linux 5.5 (on ODA 2.1) 5.8 (on ODA 2.2) x86_64

 The two ODA Server Nodes are packaged in a 4u chassis that includes 2 server nodes as well as direct attached disk in the front. Below are front and back images of an ODA. looks like from the front and back

The bottom server is Node 0 and the top is Node 1.These will be referred to as oak1 and oak2 in the factory default configuration. It is very important to keep this picture handy to work with the data center installation team to ensure they understand how each port is used to cable the ODA to meet your requirements. Cabling depends on the data center security zone specs that your infrastructure defines. There might be sensitive and non-sensitive zones as well as various cabling zones. These requirements need to be considered as part of the installation process. Attached in the Appendix is a sample spreadsheet that outlines the cabling specs needed to complete an ODA Installation.

2 108

Engineered Systems

Callout Label Ethernet Bond Description

1       Power connectors

2 PCIe 1Eth 7, 6, 5, 4 (left to right)

bond1, bond2

Eth 4 and Eth 5 are configured as bond1. Eth 6 and Eth 7 are configured as bond 2. These ports are used for custom configurations or for separate backup, disaster recovery, and network management.

3 PCIe 0 Eth 8, Eth 9 xbond0Two 10-GbE ports. In 10-GbE systems, these are connected to the public network.

4 SerMgt     Serial connector to Oracle ILOM and system console.

5Net 0,Net

1Eth2, Eth3 bond0

Two 1-GbE connectors. In 1-GbE systems, these are connected to the public network.

6 NetMgt     Ethernet connection for Oracle ILOM

7USB and

Video    Used for connecting to system console.

8       Server Node 0

9       Server Node 1

3 108

Engineered Systems

 

As you can see from above, it, is very important to understand the various components. The complete details are available in the MOS note referenced above, as well as the ODA Install, Configuration and User’s Guide. This paper will only touch the basic pieces that are important to create the fundamental understanding about the Oracle Database Appliance. Additional references are available at the end of this paper.

PREDEPLOYMENT OF THE ORACLE DATABASE APPLIANCE

The Oracle Database Appliance requires a lot of pre-deployment tasks to be completed to prepare for installing the ODA. These pre-deployment steps are standard tasks for deploying a server in a data-center and we are going to walk thru them from a Database Administrator perspective. The Oracle Database Appliance Owners guide covers most of these details.

In order to get the environment ready, the site needs to meet the environmental & electrical guidelines that are explained in the Owner’s manual. The ODA weighs about 96 lbs., and utmost care must be taken while handling the hardware. Four-post racks are compatible. Two-post racks are not compatible. Horizontal and vertical pitch must conform to ANSI/EIA 310-D-1992 or IEC 60927 standards. An Oracle Database Appliance requires 4 rack units (4RU) of rack space. A rack mounting kit is shipped as part of the hardware.

Once the hardware is racked and stacked in the data-center floor, the appliance needs to cabled. If 10g network connections will be used, SFA Adapters will need to be included in the ODA order for attaching the 10g network cables to the appliance. There is a minimum requirement of 2 power connections and 4 network connections to the public network in order to access the machine. There are other networks available for use for various purposes (connecting to a NAS, ILOM or Backup Network). Once all cabling is confirmed, it is very important to create a pre-deployment checklist to ensure that all appropriate information about the system is captured.An example template can be seen below.

Needed information Your Information

System Name: i.e. sales TSODA1Database Deployment Type: RAC, RAC One or Standalone

RAC

Internal or External Backup via NFS device

External

Root Passwords: default for nodes is welcome1

Leave the default for now

Domain Name: Example.com.comDNS Server(s): min 1 Dns.example.comNTP Server(s): optional but recommended

Ntp.example.com

Node 1 Name: i.e. sales1 Tsoda11Node 1 Public IP AddressNode 1 VIP Name: i.e. sales1-vip Tsoda11-vipNode 1 Public Virtual IP AddressNode 2 Name: i.e. sales2 Tsoda12Node 2 Public IP Address

4 108

Engineered Systems

Node 2 VIP Name: i.e. sales2-vip Tsoda12-vipNode 2 Public Virtual IP AddressSCAN Name: sales-scan Tsoda1-scanSCAN IP Addresses: min 2Netmask:Gateway IP:ILOM Names: ie. sales1-ilom Tsoda11-ilomILOM IP Address:ILOM Names: i.e. sales2-ilom Tsoda12-ilomILOM IP Address: ILOM Passwords: default is changeme Leave the default for nowILOM Gateway:ILOM Netmask:Database Name: TsodadbDatabase class: very small, small, medium, large, very large

Small

(optional) Proxy Server Name: for Auto Service RequestMetalink Login Information: User ID/Pwd

Since each Oracle Database Appliance comes with the same configuration, it is very easy to provide a configuration spreadsheet that includes various configuration specs and networking information to record the right information from the network and data center teams . An example spreadsheet that can be used to streamline the process is attached below.

It is very important to develop a standard process for deploying the ODA. This includes defining roles and responsibilities for deploying and managing the ODA. It is very important to understand these organizational responsibilities prior to deployment. A lot of activities that are required to deploy the ODA require system access and organizational structures typically have role separation. The ODA deployment activity will need the assistance of all teams within an organization from infrastructure to system administration if the role separation methodology is used. Conversely the Database Administrator deploying the ODA needs to understand various additional processes in order to successfully deploy the appliance.

In the Scenario, if the Database team is responsible for deploying the Oracle Database Appliance, the following is a brief example list of activities that will need to be performed.

1. Determine hardware placement zones.

2. Submit a request to get network ip’s

3. Submit DNS and Firewall requests

4. Work with datacenter team to determine rack space.

5. Ensure cables and adapters are available

6. Read thru the ODA manual

7. Provide datacenter team cabling specs

5 108

Engineered Systems

8. Get the ODA connected to a KVM or access via Serial cable to ensure the hosts are powered on and ILOM is accessible.

9. Walk thru deployment activities.

DEPLOYMENT OF THE ORACLE DATABASE APPLIANCE In order to understand how to deploy an Oracle Database Appliance (ODA) in a datacenter, one needs to understand the appliance itself. As illustrated above, The ODA consists of 2 x86 servers that have direct attached SAS as well as SSD storage. The environment is fully redundant from a hardware perspective. Storage redundancy is provided by Oracle ASM and can be triple mirrored or double mirrored (version 2.4 and above) for the SAS disk. The SSD is designed and recommended to be only used for the on-line redo logs and is triple mirrored regardless of ODA version. Oracle ASM also ensures that all the extents are mapped appropriately among the available disk to ensure that data extents are balanced. Oracle RAC is used to ensure any hardware failures on one node will not impact the running of the system as a whole.

As we look at the appliance we see 1 host node and 1 Service Processor node per hardware entity. The service processor also includes an Integrated Lights Out Management (ILOM) console which is used to manage each appliance. Deploying the ODA starts with getting the ILOM ready and accessible to the network.

DEPLOYMENT OF THE ILOM

The ILOM, or the Integrated Lights Out Manager, is the hardware administration and management brains and is also referred to as the service processor. The ILOM is used for various purposes in the lifecycle of a host including, but not limited to, performing a bare metal restore of an ODA. An Oracle Database Appliance contains 2 ILOM’s, or one for each of the two ODA servers. Configuring the ILOM is step 1 in the process of getting an ODA up and running. There are multiple ways to get into the ILOM. If you have a KVM device you can connect the KVM to the ODA serial management port to do the ILOM configuration remotely. The configuration steps for setting up the ILOM directly via the serial port are detailed below.

ACCESSING THE ILOM VIA SERIAL PORT The ODA ILOM can be accessed via a Serial management (SerMgt) port labeled #4 in the picture above. MOS Note ID 1395445.1 explains the process of connecting to the ODA to configure the ILOM port.

The process requires direct physical access to the serial port , which means, this process will need to be run at the place where the Oracle Database Appliance is installed. Each ODA ships with an RJ45-serial convertor like the one pictured below.

Using an RJ-45 cable, you can connect the cable to the serial port on your laptop or Datacenter terminal cart and get into the ILOM using putty. In the case that you do not have access to a serial port, Amazon sells a usb to serial converter cable at http://amzn.com/B00425S1H8 . The

6 108

Engineered Systems

cable looks like below and requires prolific drivers installed on your laptop in order to use the cable. This cable will be used in conjunction to the rj45-serial converter to connect to the ODA ILOM.

Once the connection cable is connected to the laptop and ODA serial port, check the device manager to see which com port the USB device is connected to and specify the com port in your Putty connection. This doc will only talk about windows but the MOS note above includes the Linux configuration as well. Once the port is determined, you can open up putty and connect to the serial connection.

Once the connection is made you will see a login prompt. Once the ODA is cabled, in and power cords are attached, the ILOM is automatically powered on. You will need to use the root account and the default root password to connect to the ILOM.

7 108

Engineered Systems

.

An initial configuration of the ILOM should have no IP, and this can be validated by issuing the following command:

# show /SP/network

While the OAK deployment can IP the ILOM, it is always faster to configure the ILOM IP’s using the following method. This will allow for a faster deployment by using the ILOM to deploy the ODA. You will need the following information to be able to IP the ILOM for both ODA servers.

1. Public IP Address

2. Netmask

3. Gateway Address

The following command will assign a static IP address to the ILOM

# set /SP/network pendingipdiscovery=static pendingipaddress=1.2.2.65 pendingipnetmask=255.255.255.0 pendingipgateway=1.2.2.254 commitpending=true

After the IP is established you can also use this session to power on each of the database nodes remotely.

# start /SYS

Once this is executed you should be able to access the ILOM via ssh or an internet browser, as well as connect to the database node remotely. The following screen shows browser access to an ILOM.

This is the ILOM console and this confirms the successful configuration of the ILOM. Once the ILOM is successfully configured you are now ready to configure the Oracle Database appliance

8 108

Engineered Systems

database nodes. The ILOM provides a lot of functionality including fault management and remote access. The functions of the ILOM are beyond the scope of this document but below you can see a picture of some of the ILOM’s capabilities. We will be using the ILOM in the next section to remote control into the Database Node to configure the ODA.

DEPLOYMENT OF THE DATABASE NODE The setup of the ILOM is the first step in deploying an ODA and configuring the two database nodes. As part of the deployment process you will need to create the deployment configuration files or enter the configuration information during the deployment process. To validate your configuration, it is often wise to use the offline configurator that is available via the Oracle Technology Network (OTN) website http://bit.ly/VpakT8. The Offline Configurator allows an administrator to enter, and validate, the complete set of configuration information prior to starting the deployment. The offline configurator which is meant currently for Bare Metal installations allows for complete network validation prior to deployment.

The first step in the deployment is to access each of the database nodes through the ILOM, using SSH as follows:

The ILOM can also be used to access the database node by launching the remote console through a standard web browser. The host first needs to be powered on either remotely using the ILOM or through the power button on the front panel on the Oracle Database Appliance boxes.

9 108

Engineered Systems

Once you launch the remote console. you will be taken to the host.

After logging into the host, you will need to IP each of the two database hosts by running the following command using the root account:

# /opt/oracle/oak/bin/oakcli configure firstnet

This will allow for public access to the appliance and allow for completion of the deployment. Discussion about patching will be in a separate section but it is important to ensure that the version of OAK (Oracle Application Kit) you are deploying is not very far behind.

#/opt/oracle/oak/bin/oakcli show version

If you feel that the software needs patching, please do so prior to the deployment as there are some factors like disk configuration that will change based on the version of the software . These include, but are not limited to, disk redundancy (double vs. triple mirroring) and the introduction of virtualization to the Oracle Database Appliance. The discussion about redeploying an ODA is a 3 part blog post series that can be found at http://bit.ly/VpcWQH . Once you have determined and patched your ODA to the level you want to start out with, the deployment process can start. In the earlier versions of the OAK software, the only option seemed to be using the GUI interface to deploy the software. In later versions (2.3 and above), the copy command has been introduced which allows for a scripted approach to deployment. For a safer GUI deployment it is recommended to use the vnc utility in order to avoid any issues.

As root you can execute

# vncserver :1 will allow for vncserver to run on port 5901.

You will be prompted to setup a password. After which you will gain access to the server via vnc. Below is what a vnc screen will look like on a vnc connect . You can use publically available free vnc viewers like Tight VNC available at

10 108

Engineered Systems

If you have created a deployment configuration file you can copy it to the first node of the host. You can deploy an ODA by entering the configuration information manually, but it is a cumbersome process. The screens below will show a typical deployment using a configuration file prebuilt via the offline configurator

For complete control we will select custom and load the file.

11 108

Engineered Systems

You will provide generic network info in the next page or the information will be prefilled form a config file that is previously created.

12 108

Engineered Systems

The Public network information including which interface to use is the next set of information. You can either use the 10G network (xbond0) or the 1G network (bond0) depending on your network configuration. It is very important that you consult your network team regarding your configuration and ensure cables are plugged into the right ports.

If you have any other networks to configure like a Backup network/NAS network etc . you will get the oppurtunity to configure it in the next screen.

The next screen will walk you through the database configuration. Typically you should select a very small database to get a dummy database in place and create the actual databases that are more customized at a later time.

13 108

Engineered Systems

The Next screen walks you thru the ASR setup process. It is essential to get an inderstanding ASR (Automatic Service Request) functionality and the limitations of setting up the ASR utility via the deployment process. If you have direct access to the internet from the ODA, the ASR deployment should be fairly simple. If you are accessing the internet via a proxy it is important to note that the ODA ASR registration only supports port 80 for the proxy setup and does not support using another port using the deployment GUI. Your ASR deployment might fail if you your proxy uses a different port, due to this at which point you would have to manually setup the ASR.. Later versions of the ODA software (2.5) fixes the proxy port issue.

The final decision is to the setup the ACFS file system - or as Oracle calls it: CloudFS. ACFS is a shared file system that can be used for storing a variety of files. Currently ACFS is a restricted license option. Please consult your licensing docs to get more information.

In conclusion, you will receive a summary of the choices made and you can being the install. The install can take anywhere from 45 mins to an hour after which you will have a fully functional RAC cluster. This concludes the deployment of the Oracle Database Appliance

14 108

Engineered Systems

PATCHING THE ORACLE DATABASE APPLIANCE The Oracle Database appliance (ODA) is unique in the sense that Oracle provides an all-inclusive patching strategy. An ODA patch includes the following components

Infrastructure

1. Integrated Lights Out Management (ILOM)/ BIOS

2. Oracle Appliance Kit (OAK)

3. Hardware Management Pack HMP

4. Automatic Service Request (ASR)

5. Intelligent Platform Management Interface IPMI

6. Operating SSystem

7. Storage Firmware

Grid Infrastructure

The GI patch allows for patching and/or upgrading the Clusterware and ASM software which are now collectively known as the grid infrastructure.

Database

The integrated patching allows for patching and/or upgrading the database software. Oracle currently supports 11.2.0.2.x as well as 11.2.0.3.x on the Database appliance.

Oracle =can also introduce new features in each Oracle Appliance Kit patch. Below is the history of some of the features that were introduced during each patchset.

HISTORY OF OAK/ODA PATCHES ODA 2.1.0.1/2.1.0.2

1. Base Release

2. 11.2.0.2.4

ODA 2.1.0.3/2.1.0.3.1

1. 11.2.0.2.5

2. Added ASR ILOM configuration

3. ILOM/BIOS and Firmware fixes

4. 2.1.0.3.1 patches a critical disk flaw

ODA 2.2.0

15 108

Engineered Systems

1. Introduced 11.2.0.3.2 and provided the means to upgrade from 11.2.0.2

2. Upgraded the kernel to UEK 5.8

3. Odachk introduced perform ODA health checks

ODA 2.3.0

1. Prerequisite of ODA 2.2

2. Multiple Database Homes now supported

3. 11.2.0.2.7 and 11.2.0.3.3 supported in a multi home environment

4. Oakcli command set enhanced to create databases and database homes

ODA 2.4.0

1. Normal redundancy option for new deployment of DATA and RECO(existing deployments require redeployment)

2. Trace File analyzer

3. Grid Infrastructure and Database patching is rolling upgradable.

ODA 2.5.0

1. Virtualization as an option.

2. External ASR option

3. ASR Proxy port change

PATCHING Patching the ODA is performed using the Oracle Appliance Kit (OAK) interface. Each version comes with a comprehensive readme that describes patching behavior. For the purposes of this document we will talk about applying a standard patch. As just described, an ODA patch will patch the OS as well as database and clusterware components. Depending on the criticality of the application, as of ODA 2.4 OS patching still requires downtime. However the GI and Database patching are both rolling upgradable. It depends on your patching strategy and you might want to take a smaller downtime to patch your OS and follow it up with the rolling patching for GI and Database components.

Since the ODA patch includes a lot of components it is important to understand each component and the impact it will have on your system. The patch readme as well as MOS Note 888888.1 does a good job of explaining all the changes in the patch. It is important that the person applying the patch assess the impact and apply accordingly.

Before starting a patching session it is always important to check the version of the patch deployed.

The version will help determine which patchsets are necessary. If you are on a pre ODA 2.2 patch set, it is important to first patch up to 2.2 before moving further. This is due to significant OS changes that were made in ODA patch 2.2.

16 108

#/opt/oracle/oak/bin/oakcli show version

Engineered Systems

Once the patch determination has been made it is important to understand your network components to understand how the patching works. As an example, ILOM and BIOS patching will take significantly longer if your ILOM is on a different network than the database node public network. Typically this is due to network zoning/segmentation.

If you have such a case it is always better to patch the ILOM outside of the ODA patch for speed. To do so you can follow the procedures outlined below. Please note below we are assuming a system with ODA 2.3 installed patching to 2.4. The first 3 steps can be done ahead of time and require no downtime is required

The infrastructure patch will recognize that the ILOM/BIOS has already been patched and will ignore them in the patching cycle.

The downtime related tasks end here if the purpose is patching only the server and not upgrade the grid and database software. The next step is patching the Grid Infrastructure and the patch is rolling upgradable which means a full downtime is not required.

This will patch your grid infrastructure components to 11.2.0.3.4.

The final piece in the patching puzzle is the database and that is rolling upgradable as well. As part of the command you will be asked which Oracle homes to patch.

This concludes the patching exercise on the ODA. As documented patching and maintaining and ODA is much simpler than managing and patching a similar set of components separately

.

17 108

# ./oakcli unpack -package <absolute_path_to_patch_file p14752755_24000_Linux-x86-64.zip> (On Both Nodes)# cd /opt/oracle/oak/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/541-4116-03/3.0.16.22.a-es-r75629/Base (On Both Nodes)

# ipmiflash -v write ILOM-3_0_16_22_a_r75629-Sun_Fire_X4370_M2-rom.pkg force script config delaybios warning=0 (On Both Nodes)# ipmitool chassis power cycle (Will bring down the node and does require downtime. Needs to be run on each node but with a delay)Once the nodes come up. You can patch the infra and it will be significantly faster # /opt/oracle/oak/bin/oakcli update -patch 2.4.0.0.0 –infra

# cd /opt/oracle/oak/bin# ./oakcli update -patch 2.4.0.0.0 –gi

# cd /opt/oracle/oak/bin# ./oakcli update -patch 2.4.0.0.0 –database

Engineered Systems

ONE BUTTON ORACLE DATABASE APPLIANCE Oracle has used the concept of one button for a lot of activities surrounding the Oracle Database Appliance. The Oracle Appliance Kit (OAK) is the management brains of the Oracle Database Appliance. The Oracle Appliance Kit Daemon (oakd) maintains the link between hardware and software in the ODA.

ONE BUTTON DEPLOY

We’ve already discussed deployment but this is one of the key features of the Oracle Database Appliance (ODA). One button deployment allows for rapidly deploying or re-deploying the ODA with one command.

ONE BUTTON VALIDATE

Oracle Database appliance comes with one button validation which should be performed to ensure that the system is performing optimally prior to going live. It is very important to understand the various options that the validation tool offers from the documentation.

The table of options below tells you about the various validation commands you can run. It is essential to execute basic validation on a regular basis to ensure the system is stable.

Oracle Database Appliance Validation Tool Options

Option Purpose

-aRun all system checks, including Disk Calibration. Oracle recommends you use this command to validate system readiness before deployment. Do not run oakcli validate with this option on a loaded system, because the Disk Calibration system check can cause performance issues.

-c Run a comma-delimited list of checks. Use this command to run a specific check or list of checks.

-dRun the default system checks. The default system checks are SystemComponents, OSDiskStorage, SharedStorage, and NetworkComponents

-fCreate an output file. Provide the name of the file after the -f flag. The output for the validation commands is printed to this file. If you do not specify that output is sent to a file, then it is sent to the Page display (stdout).

-h Print help information.

18 108

# cd /opt/oracle/oak/bin# ./oakcli deploy

# cd /opt/oracle/oak/bin# ./oakcli validate -h

Engineered Systems

-v Provide verbose output.

-V Print the validation tool version.

An example of a system problem can be shown here.

A system without a power supply issue would display something like this:

As you can see, even though the system is functioning due to the redundant nature of the system components. If there is an issue, then running an oakcli validate can identify the source of the system or hardware problem. These types of issues can be proactively determined by using ASR (Automatic Service Request).

Validate also provides DiskCalibration data to ensure that the disk is functioning as expected and no disk failures.

An issue with disk access or calibration can also be seen below. In this example, a disk was manually dropped from the diskgroup to simulate this test.

19 108

RESULT: Powersupply PS0 V_IN=Disabled I_IN=Disabled V_OUT=0.88 Volts I_OUT=2 Amps IN_POWER=Disabled OUT_POWER=Disabled – as you can see I_IN and V_IN is disabled which mean there is an issue

RESULT: Powersupply PS1 V_IN=206 Volts I_IN=3.75 Amps V_OUT=12.08 Volts I_OUT=4.40 Amps IN_POWER=720 Watts OUT_POWER=670 Watts

RESULT: Powersupply PS0 V_IN=204 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 Amps IN_POWER=380 Watts OUT_POWER=330 Watts --

RESULT: Powersupply PS1 V_IN=202 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 Amps IN_POWER=350 Watts OUT_POWER=340 Watts

SUCCESS: Both the powersupply are ok and functioning

RESULT: Random read throughput across all 20 SASDisk = 4037 IOPS

RESULT: Random read throughput across all 4 SSDDisk = 16368 IOPS

RESULT: Random read throughput across all 20 SASDisk = 1844 MBPS

RESULT: Random read throughput across all 4 SSDDisk = 941 MBPS

ERROR: Error when running orion for HDD_E0_S01_705927132_MBPS : ORION: ORacle IO Numbers -- Version 11.2.0.3.0

Calibration will take approximately 1 minutes.

Using a large value for -cache_size may take longer.

Error completing IO (storax_aiowait)

ORA-27061: waiting for async I/Os failed

Linux-x86_64 Error: 5: Input/output error

Additional information: -1

Engineered Systems

A Network Check will look at the network components to validate the network speed and Ethernet ports that are currently active. Oracle Database Appliance has a one 10G interface per node as well several 1G interfaces. In the example only 1G interfaces are active.

The oakcli validate one button deployment provides a plethora of diagnostic information that provide insight into the operation of the Oracle Database Appliance. It should be used regularly, but with caution, to ensure the machine is functioning as intended.

ONE BUTTON STORAGE DIAGNOSTICS

Disks can fail, so it is important to ensure that storage diagnostics are key as part of the one button deployment of the ODA. Storage diagnostics are a key component in the oakcli command set. From the ODA documentation, below are some of the commands that are available for storage diagnostic and disk location.

Option Purpose

oakcli locate[-h] [disk diskname on|off]

Locates the disk associated with the ASM disk that you indicate by its ASM disk name, where:

-h outputs the oakcli locate options

disk diskname locates the ASM disk whose name you designate

on turns on the LED of the specified disk

off turns off the LED of the specified disk

oakcli show controller

oakcli show core_config_keyDisplays the configuration for active cores as defined by the registered core key

oakcli show disk Displays disks

20 108

INFO: Doing oak network checks

RESULT: Detected active link for interface eth0 with link speed 1000Mb/s

RESULT: Detected active link for interface eth1 with link speed 1000Mb/s

RESULT: Detected active link for interface eth2 with link speed 1000Mb/s

RESULT: Detected active link for interface eth3 with link speed 1000Mb/s

RESULT: Detected active link for interface eth4 with link speed 1000Mb/s

WARNING: No Link detected for interface eth5

WARNING: No Link detected for interface eth6

WARNING: No Link detected for interface eth7

WARNING: No Link detected for interface eth8

WARNING: No Link detected for interface eth

Engineered Systems

oakcli show diskgroup Displays disk groups

oakcli show expander Displays expanders

oakcli show –h Show help command

oakcli show storage Displays storage device status

Provides a detailed output patch versions for software and firmware on the appliance

It is very important to understand and execute some of the above commands to ensure the stability and understand the state of the disk.

Based on the output above, if there is a bad disk, you can work with Oracle Field Services or your internal hardware resources to replace the disk. The oakcli locate disk helps in ensuring the right bad disk is located and replaced. Also, the show storage option provides a complete picture of the storage which includes the controllers, expanders as well as firmware version of the disks.

ONE BUTTON ODA HEALTHCHECKS

This piece is new since ODA 2.3.x and is not part of the OAK framework. However it is important to note and use as needed. Oracle Database Appliance Checks, or ODAchk as the tool is known, is the Oracle Database Appliance version of the popular RACchk and EXAchk tools.

Most of the OAK built in tools look at hardware and software, but the ODAchk provides more in-depth looks the system from the database perspective and not the appliance perspective. The tool ensures things like RAC, Grid, ASM as well as common database configuration parameters are set up right. ODAchk should be run on a regular basis and is recommended before and after any system change as well as any system patching. It is surprising how sometimes on a running system how many best practices are not being followed.

There is an html file generated with detail information as well as best practices to follow.

21 108

#oakcli show disk

NAME PATH TYPE STATE STATE_DETAILS

pd_00 /dev/sdam HDD ONLINE Good

pd_01 /dev/sdaw HDD ONLINE Good

pd_02 /dev/sdaa HDD ONLINE Good

pd_03 /dev/sdak HDD ONLINE Good

$/opt/oracle/oak/odachk/odachk – Please run as oracle software owner.

FAIL => Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on tst12 instance

FAIL => Database parameter GLOBAL_NAMES is NOT set to recommended value on tst12 instance

FAIL => One or more errors or warnings for OS Disk Storage checks

WARNING => One or more warnings for network and bonding interface checks

Engineered Systems

ONE BUTTON MANAGEMENT DIAGNOSTIC COLLECTION

The OAK framework comes with its own management and diagnostic collection framework including a cleanup script to re-initialize the Oracle Database Appliance and establish “pay as you go” Oracle licensing core counts.

Oakcli manage diagcollect – Collect and package diagnostic files for Oracle Support

Oakcli manage cleanrepo – Remove old versions of ODA patch and version files

CleanupDeploy.pl

If there is ever a need to reimage the box or uninstall all software you can use the CleanupDeploy.pl to do so

Configuring the Oracle licensing core counts

If you are using the pay as go Oracle Licensing scheme you can use oakcli to apply the core cap and install the key you have generated on MOS (My Oracle Support).

22 108

# cd /opt/oracle/oak/bin# ./oakcli manage diagcollect – this will collect diagnostic data from CRS and various DB and OAK components to submit to oracle

#cd/opt/oracle/oak/bin# oakcli manage cleanrepo --ver 2.4.0.0.0

Can not delete the current or latest version repository.

#oakcli manage cleanrepo --ver 2.3.0.0.0 – to cleanup previous version files # oakcli update –patch 2.4.0.0.0 –clean – in 2.4 has been enhanced to cleanup temporary files after patching

1. Log in as root.2. Change directory to /opt/oracle/oak/onecmd.3. Run the command cleanupDeploy.pl.

For example:# ./cleanupDeploy.pl

# oakcli apply –hUsage:oakcli apply core_config_key <key_file>

# oakcli show core_config_keyOptional core_config_key is not applied on this machine yet !

# oakcli show core_config_key Host's serialnumber = ABC123Configured Cores = 20

Engineered Systems

Trace File Analysis

In Oracle Database Appliance 2.4 a new option for TFA was installed. The Trace File Analyzer is a new framework for the collection and analysis of trace files that help the proactive collection of errors in real-time mode and automatically collects the right log files as the problem is happening. More information on TFA can be found in MOS note 1513912.1.

ONE BUTTON PATCHING This paper already discusses patching but in the one button section it is important to mention the capabilities of the one button infrastructure to be able to patch all software or individual components using the one button patching capabilities built into the Oracle Appliance Kit.

While the patching section of this paper discusses more detail, the command above is the but in essence of single button patching, The command above can patch an appliance.

ONE BUTTON DATABASE DEPLOYMENT

Starting with ODA version 2.3 and above, newer releases of the ODA 2.3 and above , the option is now provided to create multiple database homes as well as create new databases. There are options to upgrade database as well delete databases and homes as well.

To create a new database home

To create a new database

To delete a database or database home

To upgrade a database from one version to another (11.2.0.2 to 11.2.0.3)

23 108

# cd /opt/oracle/oak/bin# ./oakcli update -patch version

# oakcli create dbhome -version 11.2.0.3.3 – will create a new dbhome with 11.2.0.3.3

# oakcli create database  -db testdb -version 11.2.0.3.3 – will create a database and a new database home

# oakcli delete database –db olddb # oakcli delete dbhome –oh dbhome_name

.

Engineered Systems

SUMMARY In summary, the Oracle Database Appliance packs a lot of bang for the buck. As the basic model in Oracle engineered Appliance product line, it is the stepping stone to understand and evaluate if an appliance model is a viable option for a company. An Oracle Database Appliance provides ease of setup, patching as well as diagnostics. As an administrator that has previously only had a specific focus, there are a lot of things that an administrator has to be aware of, including the network, hardware and operating system. Oracle has made it easy for people to manage the appliance as one unit by making various aspects easier to manage and maintain using the Oracle Appliance Kit (OAK) framework. There are various monitoring aspects built into the box as well as one button activities to ensure that management of the appliance is simple.

REFERENCES

Oracle Database Appliance - 2.X Supported Versions & Known Issues [ID 888888.1]

Information Center: Oracle Database Appliance [ID 1417713.2]

ODA (Oracle Database Appliance): New Deploy Step-by-Step Short Guide (version 2.3) [ID 1486534.1]

ODA (Oracle Database Appliance): oakcli command reference [ID 1417879.1]

ODA (Oracle Database Appliance): How To Configure xbond,bond1,bond2 post deploy [ID 1422563.1]

ODA (Oracle Database Appliance): The Steps to replace failing disks [ID 1496114.1]

Gathering data for the Oracle Database Appliance server [ID 1358795.1]

ODA (Oracle Database Appliance): Test Plan Outline [ID 1474273.1]

Oracle Database Appliance Setup Poster

TFA Collector- The Preferred Tool for Automatic or ADHOC Diagnostic Gathering Across All Cluster Nodes [ID 1513912.1]

24 108

#oakcli upgrade database {-db <db_names> | -from <source_home>} -to <destination_home>.