CRS Management Commands

June Lee — Mon, 30 Jun 2008 07:21:00 +0000

To check CRS resource status

$ORA_CRS_HOME/bin/crs_stat –t

For example:

[oracle@rac2 css]$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
————————————————————
ora….CL1.srv application ONLINE ONLINE rac1
ora….CL2.srv application ONLINE ONLINE rac2
ora….TEST.cs application ONLINE ONLINE rac1
ora….L1.inst application ONLINE ONLINE rac1
ora….L2.inst application ONLINE ONLINE rac2
ora.ORCL.db application ONLINE ONLINE rac1
ora….SM1.asm application ONLINE ONLINE rac1
ora….C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora….SM2.asm application ONLINE ONLINE rac2
ora….C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

此命令用来启动所有的集群服务，但是启动的前提是资源状态State不能是UNKNOWN

$ORA_CRS_HOME/bin/crs_start –all
$ORA_CRS_HOME/bin/crs_start 集群服务名
（集群服务名可以通过$ORA_CRS_HOME/bin/crs_stat得到）

下面的一组crs命令是用来启动关闭crs服务的

$ORA_CRS_HOME/bin/crs_stop -all
$ORA_CRS_HOME/bin/crs_stop 集群服务名
（集群服务名可以通过$ORA_CRS_HOME/bin/crs_stat得到）

下面通过一个真实的环境，具体讲解这些命令的使用。举例如下：

Oracle RAC在安装完毕后，重启RAC，一些服务不能启动，手动启动也没用。

下面是系统自动启动以后的情况：

[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
————————————————————
ora….CL1.srv application ONLINE UNKNOWN rac1
ora….CL2.srv application ONLINE UNKNOWN rac2
ora….TEST.cs application ONLINE UNKNOWN rac2
ora….L1.inst application ONLINE OFFLINE
ora….L2.inst application ONLINE OFFLINE
ora.ORCL.db application ONLINE UNKNOWN rac1
ora….SM1.asm application ONLINE UNKNOWN rac1
ora….C1.lsnr application ONLINE UNKNOWN rac1
ora.rac1.gsd application ONLINE UNKNOWN rac1
ora.rac1.ons application ONLINE UNKNOWN rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora….SM2.asm application ONLINE UNKNOWN rac2
ora….C2.lsnr application ONLINE UNKNOWN rac2
ora.rac2.gsd application ONLINE UNKNOWN rac2
ora.rac2.ons application ONLINE UNKNOWN rac2
ora.rac2.vip application ONLINE ONLINE rac2

可以看到，只有vip是能自动启动的！其它资源都处于UNKNOWN状态下。

手工用srvctl启动试试！

$ srvctl start nodeapps -n rac1
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource ‘ora.rac1.gsd’ has placement error.
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource ‘ora.rac1.vip’ has placement error.
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource ‘ora.rac1.ons’ has placement error.
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource ‘ora.rac1.LISTENER_RAC1.lsnr’ has placement error.

测试crs：

$crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

[oracle@rac1 bin]$ crsctl start resources
Starting resources.
Successfully started CRS resources

[oracle@rac1 bin]$ pwd
/oracle/app/oracle/oracle/product/10.2.0/crs/bin

[oracle@rac1 bin]$ gsdctl stop
[oracle@rac1 bin]$ gsdctl status
[oracle@rac1 bin]$ ./gsdctl status
[oracle@rac1 bin]$ gsdctl start
[oracle@rac1 bin]$ srvctl start nodeapps -n rac1
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource ‘ora.rac1.gsd’ has placement error.
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource ‘ora.rac1.vip’ has placement error.
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource ‘ora.rac1.ons’ has placement error.
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource ‘ora.rac1.LISTENER_RAC1.lsnr’ has placement error.

[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
————————————————————
ora….CL1.srv application ONLINE UNKNOWN rac1
ora….CL2.srv application ONLINE UNKNOWN rac2
ora….TEST.cs application ONLINE UNKNOWN rac2
ora….L1.inst application ONLINE OFFLINE
ora….L2.inst application ONLINE OFFLINE
ora.ORCL.db application ONLINE UNKNOWN rac1
ora….SM1.asm application ONLINE UNKNOWN rac1
ora….C1.lsnr application ONLINE UNKNOWN rac1
ora.rac1.gsd application ONLINE UNKNOWN rac1
ora.rac1.ons application ONLINE UNKNOWN rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora….SM2.asm application ONLINE UNKNOWN rac2
ora….C2.lsnr application ONLINE UNKNOWN rac2
ora.rac2.gsd application ONLINE UNKNOWN rac2
ora.rac2.ons application ONLINE UNKNOWN rac2
ora.rac2.vip application ONLINE ONLINE rac2

可以看到，仍然是启动后的原始状态。从上面的显示可以知道，每个资源的State显示为UNKNOWN，应该是由于两个节点时间不同步，进而造成crs在启动的时候无法认到系统已经注册的资源造成的。

解决方法：首先，保证两个节点时间保持同步，ORACLE 10g RAC两个节点之间有10几秒的时间误差，不会对应用造成影响。但是过分大的时间误差可能导致某个节点down机，或者节点不断重启。然后执行如下操作：

[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stop –all
Attempting to stop `ora.rac1.vip` on member `rac1`
Attempting to stop `ora.rac2.vip` on member `rac2`
Stop of `ora.rac2.vip` on member `rac2` succeeded.
Stop of `ora.rac1.vip` on member `rac1` succeeded.

[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
————————————————————
ora….CL1.srv application ONLINE UNKNOWN rac1
ora….CL2.srv application ONLINE UNKNOWN rac2
ora….TEST.cs application ONLINE UNKNOWN rac2
ora….L1.inst application OFFLINE OFFLINE
ora….L2.inst application OFFLINE OFFLINE
ora.ORCL.db application OFFLINE UNKNOWN rac1
ora….SM1.asm application OFFLINE UNKNOWN rac1
ora….C1.lsnr application OFFLINE UNKNOWN rac1
ora.rac1.gsd application ONLINE UNKNOWN rac1
ora.rac1.ons application ONLINE UNKNOWN rac1
ora.rac1.vip application OFFLINE OFFLINE
ora….SM2.asm application OFFLINE UNKNOWN rac2
ora….C2.lsnr application OFFLINE UNKNOWN rac2
ora.rac2.gsd application ONLINE UNKNOWN rac2
ora.rac2.ons application ONLINE UNKNOWN rac2
ora.rac2.vip application OFFLINE OFFLINE

[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stat （此命令可以查看各个资源的完整服务名称）
NAME=ora.ORCL.ORATEST.ORCL1.srv
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.ORCL.ORATEST.ORCL2.srv
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.ORCL.ORATEST.cs
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on rac2
NAME=ora.ORCL.ORCL1.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.ORCL.ORCL2.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.ORCL.db
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.rac1.ASM1.asm
TYPE=application
TARGET=OFFLINE
STATE=UNKNOWN on rac1
NAME=ora.rac1.LISTENER_RAC1.lsnr
TYPE=application
TARGET=OFFLINE
STATE=UNKNOWN on rac1
NAME=ora.rac1.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on rac1
NAME=ora.rac1.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on rac1
NAME=ora.rac1.vip
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on rac1
NAME=ora.rac2.ASM2.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.rac2.LISTENER_RAC2.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.rac2.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on rac2
NAME=ora.rac2.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on rac2
NAME=ora.rac2.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

$ORA_CRS_HOME/bin/crs_stop –all只能停掉State为ONLINE的服务，状态为UNKNOWN的资源不能通过这个命令关闭，因此可以通过$ORA_CRS_HOME/bin/crs_stop 服务名的方式单独停止那些状态为UNKNOWN的资源。下面是通过$ORA_CRS_HOME/bin/crs_stop命令依次停止资源状态State为UNKNOWN的服务。

[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stop ora.ORCL.ORATEST.cs
Attempting to stop `ora.ORCL.ORATEST.cs` on member `rac2`
Stop of `ora.ORCL.ORATEST.cs` on member `rac2` succeeded.
[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stop ora.rac1.ASM1.asm
Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`
Stop of `ora.rac1.ASM1.asm` on member `rac1` succeeded.
[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stop ora.rac1.LISTENER_RAC1.lsnr
Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Stop of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stop ora.rac1.ons
Attempting to stop `ora.rac1.ons` on member `rac1`
Stop of `ora.rac1.ons` on member `rac1` succeeded.
[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stop ora.rac1.vip
Attempting to stop `ora.rac1.vip` on member `rac1`
Stop of `ora.rac1.vip` on member `rac1` succeeded.
[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stop ora.rac2.gsd
Attempting to stop `ora.rac2.gsd` on member `rac2`
Stop of `ora.rac2.gsd` on member `rac2` succeeded.
[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stop ora.rac2.ons
Attempting to stop `ora.rac2.ons` on member `rac2`
Stop of `ora.rac2.ons` on member `rac2` succeeded.
[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stop ora.rac1.gsd
Attempting to stop `ora.rac1.gsd` on member `rac1`
Stop of `ora.rac1.gsd` on member `rac1` succeeded.

把所有状态为UNKNOWN的资源全部停止以后，再次查看资源状态：

[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
————————————————————
ora….CL1.srv application OFFLINE OFFLINE
ora….CL2.srv application OFFLINE OFFLINE
ora….TEST.cs application OFFLINE OFFLINE
ora….L1.inst application OFFLINE OFFLINE
ora….L2.inst application OFFLINE OFFLINE
ora.ORCL.db application OFFLINE OFFLINE
ora….SM1.asm application OFFLINE OFFLINE
ora….C1.lsnr application OFFLINE OFFLINE
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application OFFLINE OFFLINE
ora.rac1.vip application OFFLINE OFFLINE
ora….SM2.asm application OFFLINE OFFLINE
ora….C2.lsnr application OFFLINE OFFLINE
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application OFFLINE OFFLINE
ora.rac2.vip application OFFLINE OFFLINE

此时可以重启所有crs资源了

[oracle@rac1 ~]$ $ORA_CRS_HOME/bin/crs_start –all（注意，这里是在oracle用户下操作）
Attempting to start `ora.rac1.vip` on member `rac1`
Attempting to start `ora.rac2.vip` on member `rac2`
Start of `ora.rac2.vip` on member `rac2` succeeded.
Attempting to start `ora.rac2.ASM2.asm` on member `rac2`
Start of `ora.rac1.vip` on member `rac1` succeeded.
Attempting to start `ora.rac1.ASM1.asm` on member `rac1`
Start of `ora.rac2.ASM2.asm` on member `rac2` succeeded.
Attempting to start `ora.ORCL.ORCL2.inst` on member `rac2`
Start of `ora.rac1.ASM1.asm` on member `rac1` succeeded.
Attempting to start `ora.ORCL.ORCL1.inst` on member `rac1`
Start of `ora.ORCL.ORCL2.inst` on member `rac2` succeeded.
Attempting to start `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
Start of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded.
Start of `ora.ORCL.ORCL1.inst` on member `rac1` succeeded.
Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
Attempting to start `ora.ORCL.ORATEST.ORCL1.srv` on member `rac1`
CRS-1002: Resource ‘ora.rac1.ons’ is already running on member ‘rac1′
CRS-1002: Resource ‘ora.ORCL.db’ is already running on member ‘rac1′
CRS-1002: Resource ‘ora.rac2.ons’ is already running on member ‘rac2′
Attempting to start `ora.ORCL.ORATEST.ORCL2.srv` on member `rac2`
Attempting to start `ora.ORCL.ORATEST.cs` on member `rac2`
Attempting to start `ora.rac1.gsd` on member `rac1`
Attempting to start `ora.rac2.gsd` on member `rac2`
Start of `ora.ORCL.ORATEST.cs` on member `rac2` succeeded.
Start of `ora.rac2.gsd` on member `rac2` succeeded.
Start of `ora.ORCL.ORATEST.ORCL2.srv` on member `rac2` succeeded.
Start of `ora.rac1.gsd` on member `rac1` succeeded.
Start of `ora.ORCL.ORATEST.ORCL1.srv` on member `rac1` succeeded.
CRS-0223: Resource ‘ora.ORCL.db’ has placement error.
CRS-0223: Resource ‘ora.rac1.ons’ has placement error.
CRS-0223: Resource ‘ora.rac2.ons’ has placement error.

[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
————————————————————
ora….CL1.srv application ONLINE ONLINE rac1
ora….CL2.srv application ONLINE ONLINE rac2
ora….TEST.cs application ONLINE ONLINE rac2
ora….L1.inst application ONLINE ONLINE rac1
ora….L2.inst application ONLINE ONLINE rac2
ora.ORCL.db application ONLINE ONLINE rac1
ora….SM1.asm application ONLINE ONLINE rac1
ora….C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora….SM2.asm application ONLINE ONLINE rac2
ora….C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

此时，看到crs所有资源全部启动，问题解决。

其它一些常用命令：

$ORA_CRS_HOME/bin/crsctl check crs 用于检查后台进程状态
$ORA_CRS_HOME/bin/crsctl start resources 启动crs资源
$ORA_CRS_HOME/bin/crsctl start crs 启动crs，需要超级用户操作。
$ORA_CRS_HOME/bin/crsctl stop crs 关闭crs，需要超级用户操作。
$ORA_CRS_HOME/bin/crs_unregister 取消注册crs
$ORA_CRS_HOME/bin/ocrcheck 检查ocr设置信息用ocrcheck
$ORA_CRS_HOME/bin/crsctl query css votedisk 检查表决磁盘信息

For Example:

[oracle@rac2 css]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 104344
Used space (kbytes) : 4568
Available space (kbytes) : 99776
ID : 328160432
Device/File Name : /dev/raw/raw4
Device/File integrity check succeeded
Device/File Name : /dev/raw/raw5
Device/File integrity check succeeded
Cluster registry integrity check succeeded

[oracle@rac2 css]$ $ORA_CRS_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 104344
Used space (kbytes) : 4568
Available space (kbytes) : 99776
ID : 328160432
Device/File Name : /dev/raw/raw4
Device/File integrity check succeeded
Device/File Name : /dev/raw/raw5
Device/File integrity check succeeded
Cluster registry integrity check succeeded

[oracle@rac2 css]$ crsctl query css votedisk
0. 0 /dev/raw/raw6
1. 0 /dev/raw/raw7
2. 0 /dev/raw/raw8
located 3 votedisk(s).

注意：
执行crsctl命令的可以是root用户，也可以是oracle用户，如果是root用户要指定绝对路径，但是对于crsctl start crs和crsctl stop crs必须是root用户来执行。

crs主要进程

crsd
负责管理ha操作
管理crs资源，如linstener，vip，ons，gsn等
由root用户管理、启动
ocssd
管理各节点的关系，用于节点间通信
由oracle用户运行管理
oprocd
集群进程管理 —Process monitor for the cluster.
仅在没有使用vendor的集群软件状态下运行
evmd
事件检测进程，由oracle用户运行管理
主要log位置
$ORA_CRS_HOME/log/节点主机名/racg
$ORA_CRS_HOME/log/节点主机名/crsd
$ORA_CRS_HOME/crs/init
$ORA_CRS_HOME/css/log
$ORA_CRS_HOME/css/init
$ORA_CRS_HOME/evm/log
$ORA_CRS_HOME/evm/init
$ORA_CRS_HOME/srvm/log

Reference:

1. [Chinese] http://blog.chinaunix.net/u1/52223/showart_486081.html

Oracle DBA from Beginning » CRS

CRS Management Commands