with Exadata / GI / ASM / opatch Must-have free scripts ...
Transcript of with Exadata / GI / ASM / opatch Must-have free scripts ...
Franky Weber Faust -- May 4th 2019
Must-have free scripts when working with Exadata / GI / ASM / opatch
Franky Weber Faust -- May 4th 2019
Must-have free scripts when working with Exadata / GI / ASM / opatch
4© The Pythian Group Inc., 2018 4
FRANKY WEBER FAUST
● Senior Oracle Database Consultant at Pythian and PagSeguro
● 28 years old● Writer at OTNLA and Lore Data Blog● Speaker at conferences around the
world● High Availability specialist● Performance researcher● Exadata, RAC, DataGuard,
GoldenGate● AcroYoga practicioner● Guitar player
loredata.com.br
© 2017 Pythian. Confidential 5
Years in Business
20Pythian Experts in 35 Countries
400+Current Clients
Globally
350+
Why this talk ?
● We sometimes expect things to be easy -- but they actually are not or they just don’t exist
● I am lazy and do not like doing repetitive and/or useless non added value tasks
● Fred developed all these scripts for our team to use and then shared with the community
● They have been improved a lot and also for OS different than Linux, a GI version we do not have handy, a new feature idea, etc ...
● All the scripts are free, simply bash made
asmdu.sh
ASM diskgroups sizes and usage
ASM disk usage : the old fashioned way
Non mirrored sizes Mirrored sizes
ASM disk usage : a new & better way with asmdu.sh
asmdu.sh : option -v for a verbose mode
Total non mirrored size
Non mirrored free size
Amount of free space required to be available in order to restore redundancy after one or more disk failures without affecting data availability
asmdu.sh : external redundancy
asmdu.sh : using different units
asmdu.sh : handling subdirectories
for D in `asmcmd ls DATA_DG`do asmcmd du +DATA_DG/$Ddone
asmdu.sh : a subdirectories example
asmdu.sh : another subdirectories example
asmdu.sh : option -h for the syntax
rac-status.shRAC / GI 11g,12c,18c,19c, Oracle Restart
resources status in a glimpse
© 2017 Pythian. Confidential 18
rac-status.sh : the old fashioned way
What is up ? What is supposed to be up ? Where is it up ? etc...
/home/oracle/pythian > ps -ef | grep pmon | wc -l37/home/oracle/pythian > olsnodesanexadatadb01anexadatadb02anexadatadb03anexadatadb04anexadatadb05anexadatadb06anexadatadb07anexadatadb08/home/oracle/pythian > crsctl stat res -p -w "TYPE = ora.database.type" | wc -l18096/home/oracle/pythian > crsctl stat res -v -w "TYPE = ora.database.type" | wc -l3410/home/oracle/pythian >
© 2017 Pythian. Confidential 19
rac-status.sh : a new & better way
© 2017 Pythian. Confidential 20
A La
rge
Impl
emen
tatio
n
© 2017 Pythian. Confidential 21
rac-status.sh : identifying issues
© 2017 Pythian. Confidential 22
rac-status.sh : different DB Types
© 2017 Pythian. Confidential 23
rac-status.sh : also shows the listeners
© 2017 Pythian. Confidential 24
rac-status.sh : also shows the listeners
© 2017 Pythian. Confidential 25
rac-status.sh : also shows the services
© 2017 Pythian. Confidential 26
rac-status.sh : see everything
© 2017 Pythian. Confidential 27
rac-status.sh : Oracle Restart (single instance)
© 2017 Pythian. Confidential 28
rac-status.sh : output customisation (with variables)
● Showing the databases, listeners and services may not be wanted at each execution
● Default output can be customised by modifying variables within the script
SHOW_DB="YES" # Databases #SHOW_DB="NO" SHOW_LSNR="YES" # Listeners#SHOW_LSNR="NO" SHOW_SVC="YES" # Services SHOW_SVC="NO"
● The last uncommented value wins
● The default shipped with the script shows the Databases and the Listeners
© 2017 Pythian. Confidential 29
rac-status.sh : output customisation (from command line)
● Using the command line
-d Revert the behavior defined by SHOW_DB-l Revert the behavior defined by SHOW_LSNR-s Revert the behavior defined by SHOW_SVC
● Show / hide everything
-a Show everything-n Show nothing
© 2017 Pythian. Confidential 30
rac-status.sh : make your maintenances idempotent
● Very useful to ensure being idempotent during a maintenance
● Save the status of the resources before the maintenance
$ ./rac-status.sh -a > status_before_maintenance
● Check the status of the resources after the maintenance
$ ./rac-status.sh -a > status_after_maintenance
● Any difference means something is not back to normal
$ diff status_before_maintenance status_after_maintenance
© 2017 Pythian. Confidential 31
rac-status.sh : output customisation (grep and ungrep)
● You can grep and ungrep from the output
-g Act as “grep” to grep a pattern from the output (key sensitive)-v Act as "grep -v" to ungrep from the output (key sensitive)
● Examples
$ ./rac-status.sh -g 12 # Only grep 12c databases
$ ./rac-status.sh -a -g Shut # Only the lines containing "Shut"
$ ./rac-status.sh -n -l -v Online # Only the listeners that are not Online
© 2017 Pythian. Confidential 32
rac-status.sh : output customisation (colors)
● Option -u for an Uncolored output
© 2017 Pythian. Confidential 33
rac-status.sh : output customisation (colors)
● Option -r to Revert the colors
© 2017 Pythian. Confidential 34
rac-status.sh : environment
● rac-status.sh uses oraenv to set the ASM environment and the uses crsctl
● Option -e if you do not want to use oraenv but the current environment
● You can set USE_ORAENV="NO" in the script to always use the current environment and never use oraenv
© 2017 Pythian. Confidential 35
rac-status.sh : option -h for help
© 2017 Pythian. Confidential 36
rac-status.sh : patch progression
rac-mon.sha GI 11g,12c,18c,19c and Oracle Restart
monitoring tool based on rac-status.sh
rac-mon.sh : how it works
● A monitoring tool based on rac-status.sh
● First, it saves a reference status of your cluster (when everything works as expected)
$ ./rac-status.sh -a > rac-status_reference
● At every execution, it takes the current status of your cluster
$ ./rac-status.sh -a > current_status
● Compare it with the reference
$ diff rac-status_reference current_status
rac-mon.sh : pre requisites
● The rac-status.sh script
RACSTATUS=~/rac-status.sh
● A file to save the reference status of your cluster
REFERENCE=~/rac-status_reference
rac-mon.sh : a first execution
rac-mon.sh : when errros are detected
rac-mon.sh : sending emails on error
rac-mon.sh : sending emails on success
EMAIL_ON_FAILURE="No" # Yes to always send emailsEMAIL_ON_SUCCESS="No" # Yes to always send emails
● Always send emails with no option
rac-mon.sh : email configuration
EMAILTO="[email protected]" # The email to send the alert to
● Email address(es)
● Succes / Failure Emails Subjects
FAILURE_SUBJECT="Error : Cluster status at "`date` # Subject of the email sentSUCCESS_SUBJECT="OK : Cluster status at "`date` # Subject of the email sent
rac-mon.sh : recreate the reference file
● You need to recreate the reference file if you change something on your cluster on purpose
$ ./rac-status.sh -a > ./rac-status_reference
● A quick test
$ ./rac-mon.sh No change has been identified across the cluster, all good !$
rac-mon.sh : cron it
● You may want to cron it to monitor your cluster resources on a regular basis
*/5 * * * * /home/oracle/rac-mon.sh -e >> /var/log/rac-mon.log 2>&1
● It needs to be scheduled on only 1 node of the cluster
rac-mon.sh : option -h for help
cell-status.shExadata Storage Server (Cell)
Bare Metal, OVM, ExaCCStorage overview made simple
cell-status.sh : the old fashioned way
cell-status.sh : a new & better way
cell-status.sh : a new & better way
cell-status.sh : a new & better way
lspatches.shAn Oracle patch reporting tool
© 2017 Pythian. Confidential 54
lspatches.sh : an Oracle patch reporting tool
● Opatch is a nice and easy tool to install and rollback patches
● But it lacks of reporting capabilities
● opatch -lspatches does not provide remote patch information
© 2017 Pythian. Confidential 55
● lsinventory -all_nodes report many lines
lspatches.sh : questions a DBA has to answer
● 472 lines for 1 Home on 2 nodes then almost 2000 lines for 8 nodes
● And you may have dozen Homes then thousands of lines
Is patch 123456 installed on all the nodes ?
Is there a missing patch on a Home ?
© 2017 Pythian. Confidential 56
lspatches.sh : a sample output
© 2017 Pythian. Confidential 57
lspatches.sh : how that works ?
© 2017 Pythian. Confidential 58
lspatches.sh : analyze the Home(s) you want
© 2017 Pythian. Confidential 59
lspatches.sh : show before proceeding (1)
© 2017 Pythian. Confidential 60
lspatches.sh : show before proceeding (2)
© 2017 Pythian. Confidential 61
lspatches.sh : input and output files are supported (1)
© 2017 Pythian. Confidential 62
lspatches.sh : input and output files are supported (2)
© 2017 Pythian. Confidential 63
lspatches.sh : input and output files are supported (3)
© 2017 Pythian. Confidential 64
lspatches.sh : local and remote
● On RAC system, default is remote patch information (-all_nodes or opatchauto -remote)
● Option -l can be used to gather local patch information only
● On non-RAC system, it goes local automatically
© 2017 Pythian. Confidential 65
lspatches.sh : option -h for help
exa-versions.shExadata components versions in a glimpse
exa-versions.sh : the old fashioned way
[root@myclusterdb01 ~]# dcli -g ~/dbs_group -l root imageinfo -ver
myclusterdb01: 12.1.2.2.1.160119
myclusterdb02: 12.1.2.2.1.160119
myclusterdb03: 12.1.2.2.1.160119
myclusterdb04: 12.1.2.2.1.160119
[root@myclusterdb01 ~]#
● dcli is a very good tool to launch many commands in parallel across all the nodes of a cluster
● How to deal with 8 DB nodes, 14 storage cells and 2 IB Switches ?
● exa-versions goal is to clearly show all the versions of each Exadata component
● Syntax for IB Switches is different than DB Servers and Cells
● Are your dbs_group , cell_group and ib_group files up to date ?
exa-versions.sh : a new & better way
exa-versions.sh : detect a status = failure
[root@ahalfrack_db02 ~]# imageinfo
Kernel version: 2.6.39-400.284.1.el6uek.x86_64 #1 SMP Mon Sep 12 19:26:25 PDT 2016 x86_64Image kernel version: 4.1.12-61.57.2.el6uek
Image version: 12.2.1.1.3.171017
Image activated: 2017-12-28 11:42:32 -0600
Image status: failure
System partition on device: /dev/mapper/VGExaDb-LVDbSys1
[root@ahalfrack_db02 ~]#
exa-versions.sh : options
exa-versions.sh : option -h for the syntax
exa-racklayout.sha whole Exadata Rack Layout overview in a glimpse
exa-racklayout.sh : the old fashioned way
● What is the IP of my database server ILOM ?
● What is the Cisco switch IP ?
● What are the cells IP ?
● How many IB Switches do I have in my Rack ?
$ cat /etc/hosts
$ vi or grep /opt/oracle.SupportTools/onecommand/databasemachine.xml
exa-racklayout.sh : a new & better way
exa-racklayout.sh : a new & better way
● -s option to hide the empty U
many-exa-racklayout.shexa-racklayout.sh on many hosts
● executes exa-racklayout.sh on many hosts
LIST="oracle@AN_IP
oracle@ANOTHER_IP"
● SSH keys needed
rac-on_all_db.shExecute a SQL on all databases running on a cluster
rac-on_all_db.sh : a real life example
rac-on_all_db.sh : the old fashioned way
/home/oracle/pythian> ssh anexadata_dbXX
/home/oracle/pythian> . oraenv <<< dbYY
/home/oracle/pythian> sqlplus / as sysdba
/home/oracle/pythian> select name, value . . .
/home/oracle/pythian>
for every database from rac-status.sh do
done
rac-on_all_db.sh : a new & better way
rac-on_all_db.sh : how to customize the queries
exa-howsmart.shHow much your databases use the Exadata features ?
exa-howsmart.sh
exa-howsmart.sh
Where to find the scripts?Fred Denis on Git hub
Thank You Questions ?
© 2018 Pythian. Confidential 87
© 2017 Pythian. Confidential 88
STAY IN TOUCH!
● E-mail: [email protected]● Blog: http://loredata.com.br/blog● Facebook: https://facebook.com/08Franky.Weber● Instagram: https://www.instagram.com/frankyweber/● Twitter: https://twitter.com/frankyweber● LinkedIn: https://linkedin.com/in/frankyweber/en● Oracle ACE profile: https://apex.oracle.com/pls/apex/f?p=19297:4:::NO:4:P4_ID:17002