To provide steps to power down or reboot an Exadata Storage cell
without affecting ASM.
Steps to power down or
reboot a cell without affecting ASM:
When performing maintenance on Exadata Cells, it may be
necessary to power down or reboot the cell. If a storage server is to be shut
down when one or more databases are running, then verify that taking the
storage server offline will not impact Oracle ASM disk group and database
availability. The ability to take Oracle Exadata Storage Server offline without
affecting database availability depends on the level of Oracle ASM redundancy
used on the affected disk groups, and the current status of disks in other
Oracle Exadata Storage Servers that have mirror copies of data as Oracle
Exadata Storage Server to be taken offline.
============================================
1) By default, ASM drops a disk shortly after it is taken
offline; however, you can set the DISK_REPAIR_TIME attribute to prevent this
operation by specifying a time interval to repair the disk and bring it back
online. The default DISK_REPAIR_TIME attribute value of 3.6h should be adequate
for most environments.
(a) To check repair times for all mounted disk groups – log into
the ASM instance and perform the following query:
SQL> select dg.name,a.value from
v$asm_diskgroup
dg, v$asm_attribute a where
dg.group_number=a.group_number and
a.name='disk_repair_time';
(b) If you need to offline the ASM disks for more than the
default time of 3.6 hours then adjust the parameter by issuing the command
below as an example:
SQL> ALTER DISKGROUP DATA SET ATTRIBUTE
'DISK_REPAIR_TIME'='8.5H';
2) Next you will need to check if ASM will be OK if the
grid disks go OFFLINE. The following command should return ‘Yes’ for the grid
disks being listed:
cellcli -e list griddisk attributes
name,asmmodestatus,asmdeactivationoutcome
3) If one or more disks return asmdeactivationoutcome=’No’, you
should wait for some time and repeat step #2. Once all disks return return
asmdeactivationoutcome=’Yes’, you can proceed with taking the griddisk offline
in step #4.
Note: Taking the storage server offline when
one or more grid disks return asmdeactivationoutcome=’No’ will cause Oracle ASM
to dismount the affected disk group, causing the databases to shut down
abruptly.
4) Run cellcli command to Inactivate all grid disks on the cell
you wish to power down/reboot:
cellcli -e alter griddisk all inactive
* Please note – This action could take 10 minutes or longer depending on
activity. It is very important to make sure you were able to offline all the
disks successfully before shutting down the cell services. Inactivating the
grid disks will automatically OFFLINE the disks in the ASM instance.
5) Confirm that the griddisks are now offline by performing the
following actions:
(a) Execute the command below and the output should
show either asmmodestatus=OFFLINE or asmmodestatus=UNUSED and
asmdeactivationoutcome=Yes for all griddisks once the disks are offline in ASM.
Only then is it safe to proceed with shutting down or restarting the cell:
cellcli -e list griddisk attributes
name,asmmodestatus,asmdeactivationoutcome
( there has also been a reported case of
asmmodestatus= OFFLINE: Means Oracle ASM has taken this grid disk offline. This
status is also fine and can proceed with remaining instructions)
(b) List the griddisks to confirm all now show inactive:
cellcli -e list griddisk
6) You can now reboot the cell. Oracle Exadata Storage Servers
are powered off and rebooted using the Linux shutdown command.
(a) The following command will shut down Oracle Exadata Storage
Server immediately: (as root):
#shutdown -h now
(When powering off Oracle Exadata Storage
Servers, all storage services are automatically stopped.)
(b) The following command will reboot Oracle Exadata Storage
Server immediately and force fsck on reboot:
#shutdown -F -r now
7) Once the cell comes back online – you will need to reactive
the griddisks:
cellcli -e alter griddisk all active
8) Issue the command below and all disks should show ‘active':
cellcli -e list griddisk
9) Verify grid disk status:
(a) Verify all grid disks have been successfully put online
using the following command:
cellcli -e list griddisk attributes name,
asmmodestatus
(b) Wait until asmmodestatus is ONLINE for all grid disks. Each
disk will go to a ‘SYNCING’ state first then ‘ONLINE’. The following is an
example of the output:
DATA_CD_00_dm01cel01 ONLINE
DATA_CD_01_dm01cel01 SYNCING
DATA_CD_02_dm01cel01 OFFLINE
DATA_CD_03_dm01cel01 OFFLINE
DATA_CD_04_dm01cel01 OFFLINE
DATA_CD_05_dm01cel01 OFFLINE
DATA_CD_06_dm01cel01 OFFLINE
DATA_CD_07_dm01cel01 OFFLINE
DATA_CD_08_dm01cel01 OFFLINE
DATA_CD_09_dm01cel01 OFFLINE
DATA_CD_10_dm01cel01 OFFLINE
DATA_CD_11_dm01cel01 OFFLINE
(c) Oracle ASM synchronization is only complete when all grid
disks show asmmodestatus=ONLINE.
( Please note: this operation uses Fast
Mirror Resync operation – which does not trigger an ASM rebalance. The Resync
operation restores only the extents that would have been written while the disk
was offline.)
10) Before taking another storage server offline, Oracle ASM
synchronization must complete on the restarted Oracle Exadata Storage Server.
If synchronization is not complete, then the check performed on another storage
server will fail. The following is an example of the output:
CellCLI> list griddisk attributes name where
asmdeactivationoutcome != 'Yes'
DATA_CD_00_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_01_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_02_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_03_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_04_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_05_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_06_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_07_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_08_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_09_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_10_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgroup"
DATA_CD_11_dm01cel02 "Cannot de-activate due
to other offline disks in the diskgr
