There is 1 major concern (and security risk) of running exachk on the Oracle Exadata boxes – it required root password for the Database servers, Cell nodes, and the Infiniband Switches!
Another concern is that the exachk script need to be run as the owner of the Oracle software, and in many cases (commonly), the user “oracle” (or any user who run the oracle processes).
And, if your root password contain special characters, the exachk will fails with message:
Enter root password for STORAGE SERVER :- can’t read “password without special char”: no such variable
while executing”send — “correct password\n””
This is documented in MOS ID 1360278.1
After thinking and trying for awhile, I came up with some steps which will enable the user “oracle” to run “exachk” script on the Oracle Exadata Database Machines without entering all the root password for Database Servers, Cell Nodes and InfiniBand Switch (and to avoid the bug of special charactes).
As root/admin user, generate public/private rsa key pair, which is protected with passphrase in 1 of the Database Servers in the Oracle Exadata Database Machines:
#ssh-keygen -f sample-exachk -C “PubKey to run exachk”
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 2048sample-exachk
Enter same passphrase again:
Your identification has been saved in sample-exachk.
Your public key has been saved in sample-exachk.pub.
The key fingerprint is:
52:42:59:48:18:0a:64:9f:6a:e9:12:ae:77:9b:4f:21 PubKey to run exachk
As root/admin user, use “visudo” to add the following line:
oracle ALL=(root) NOPASSWD:/tmp/root_exachk.sh
into all the Database servers. This will enable “oracle” user can run “root_exachk.sh” as root via sudo command without password.
As root/admin user, and assuming the Database server can ssh into the Cell Nodes and InfiniBand Switch, or you have the root password of those devices, transfer the public key generated (sample-exachk.pub) to “/root/.ssh/authorized_keys2“.
The reason I am using “/root/.ssh/authorized_keys2” so that it will not be overwritten by “dcli” command.
- Note: When you run “dcli -k -l root” to the Cell nodes, it will write into “/root/.ssh/authorized_keys”.
- Note: On my half-rack Oracle Exadata Database Machines, “root” ssh pubkeys had been distributed to Cell nodes and InfiniBand switches, so I can login to those machines as “root” using the “root” account in the Database servers.
As root/admin user, copy the passphrase-protected private keys (sample-exachk) into ~oracle/.ssh/, and make sure the ownership of the file belongs to “oracle”.
Now, login as the Oracle software owner “oracle” (or your own Oracle Software user). Here is the part that you will need to load the private keys so that you don’t need to key in the passphrase everytime it go into Cell nodes or InfiniBand Switches.
Here is how I do – start-up ssh-agent and load the private key using “ssh-add”:
$ ssh-agent bash
$ ssh-add ~/.ssh/sample-exachk
Enter passphrase for /home/orcldb/.ssh/sample-exachk:
Identity added: /home/oracle/.ssh/sample-exachk (/home/oracle/.ssh/sample-exachk)
As the Oracle software owner “oracle”, run the “exachk” script. Usually, the exachk script is located in /opt/oracle.SupportTools/exachk, but it could be an older version, so is best to always check the MOS ID ID 1070954.1 and download the latest version from it (You will need to login).
Go grab a coffee, the exachk script will take a while to complete (took me 1 hour+ for half-rack).
In summary, the trick behind this is to generate a passphrase-protected SSH Private/Public keys and distribute it to root user on the Cell nodes and InfiniBand Switches. This will ensure that “oracle” will not be able to login as root in Oracle Exadata Database Machine, but will still be allowed when need to run “exachk”.
LastNote: The public key distributed to the root user for Cell nodes and InfiniBand Switches should be removed after using, to ensure security.
Security Note: Due to this method is using SSH Pubkey authentication for doing “magic”, so it is advise to enable “Loglevel verbose” in the sshd_config. Unfortunately, Oracle don’t allowed any changes to the Oracle Exadata except the DB servers, so please use this at your own risks!
2 June 2013 update: after checking and playing around with Exadata, I discovered that by default Cell nodes already have the “LogLevel verbose” enable, so will just need to enable it on DB nodes.