drbd+heartbeat+nfs实现高可用

drbd nfs heartbeat 阅读量：473

上篇文章介绍了drbd+nfs的架构配置方法。本文通过heartbeat软件实现drbd+nfs的高可用性能。

drbd简单来说就是一个网络raid-1,一般有2到多个node节点，各个节点创建的磁盘块会映射到本地drbd块，而后通过网络对各个节点drbd磁盘块进行互相同步更新。

heartbeat的作用就可以增加drbd的可用性，它能在某节点故障后，自动切换drbd块到备份节点，并自动进行虚IP从新绑定，DRBD块提权，磁盘挂载以及启动NFS等脚本操作，这一系列操作因为只在他后端节点间完成，前端用户访问的是heartbeat的虚IP，所以对用户来说无任何感知。

node1:server.example.com  192.168.213.3

node2:client.example.com 192.168.213.5

vip:192.168.213.20/24

一、安装heartbeat

1.安装epel扩展源：
#rpm -ivh  'http://www.lishiming.net/data/attachment/forum/epel-release-6-8_32.noarch.rpm'

2.两个机器都安装heartbeat / libnet
#yum  install -y heartbeat*   libnet 

3.配置heartbeat
#cd /usr/share/doc/heartbeat-3.0.4/ #cp authkeys ha.cf haresources /etc/ha.d/ #cd /etc/ha.d #vi authkeys //加入 auth 3 3 md5 Hello!

chmod 600 authkeys

vi haresources //加入
server.example.com IPaddr::192.168.213.20/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext4 killnfsd

#vi ha.cf //改为如下内容：
debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 30 warntime 10 initdead 60 udpport 694 ucast eth0 192.168.213.5 auto_failback on node server.example.com node client.example.com ping 192.168.213.2                        //本地网关，尽量选择比较稳定的IP，可以ping通的 respawn hacluster /usr/lib/heartbeat/ipfail

把主上的三个配置拷贝到从上：
#cd /etc/ha.d/
#scp authkeys ha.cf haresources client.example.com:/etc/ha.d/

到从上(client) 编辑ha.cf
#vi /etc/ha.d/ha.cf                     //只需要更改一个地方
ucast eth0 192.168.213.5 改为 ucast eth0 192.168.213.3                //互相配置对方的IP

4.编写脚本

注：上述文件内IPaddr,Filesystem等脚本存放路径在/etc/ha.d/resource.d/下,也可在该目录下存放服务启动脚本（例如：mysql,www）,将相同脚本名称添加到/etc/ha.d/haresources内容中，从而跟随heartbeat启动而启动该脚本。

IPaddr::192.168.7.90/24/eth0：用IPaddr脚本配置浮动VIP

drbddisk::r0：用drbddisk脚本实现DRBD主从节点资源组的挂载和卸载

Filesystem::/dev/drbd0::/data::ext4：用Filesystem脚本实现磁盘挂载和卸载

编辑脚本文件killnfsd,用来重启NFS服务：

注：因为NFS服务切换后，必须重新mount NFS共享出来的目录，否则会报错（待验证）

# vi /etc/ha.d/resource.d/killnfsd

-----------------

killall -9 nfsd; /etc/init.d/nfs restart;exit 0

-----------------

赋予执行权限：

# chmod 755 /etc/ha.d/resource.d/killnfsd

创建DRBD脚本文件drbddisk:(node1,node2)

# vi /etc/ha.d/resource.d/drbddisk

-----------------------

#!/bin/bash

#

# This script is inteded to be used as resource script by heartbeat

#

# Copright 2003-2008 LINBIT Information Technologies

# Philipp Reisner, Lars Ellenberg

#

###

DEFAULTFILE="/etc/default/drbd"

DRBDADM="/sbin/drbdadm"

if [ -f $DEFAULTFILE ]; then

. $DEFAULTFILE

fi

if [ "$#" -eq 2 ]; then

RES="$1"

CMD="$2"

else

RES="all"

CMD="$1"

fi

## EXIT CODES

# since this is a "legacy heartbeat R1 resource agent" script,

# exit codes actually do not matter that much as long as we conform to

#  http://wiki.linux-ha.org/HeartbeatResourceAgent

# but it does not hurt to conform to lsb init-script exit codes,

# where we can.

#  http://refspecs.linux-foundation.org/LSB_3.1.0/

#LSB-Core-generic/LSB-Core-generic/iniscrptact.html

####

drbd_set_role_from_proc_drbd()

{

local out

if ! test -e /proc/drbd; then

ROLE="Unconfigured"

return

fi

dev=$( $DRBDADM sh-dev $RES )

minor=${dev#/dev/drbd}

if [[ $minor = *[!0-9]* ]] ; then

# sh-minor is only supported since drbd 8.3.1

minor=$( $DRBDADM sh-minor $RES )

fi

if [[ -z $minor ]] || [[ $minor = *[!0-9]* ]] ; then

ROLE=Unknown

return

fi

if out=$(sed -ne "/^ *$minor: cs:/ { s/:/ /g; p; q; }" /proc/drbd); then

set -- $out

ROLE=${5%/**}

: ${ROLE:=Unconfigured} # if it does not show up

else

ROLE=Unknown

fi

}

case "$CMD" in

start)

# try several times, in case heartbeat deadtime

# was smaller than drbd ping time

try=6

while true; do

$DRBDADM primary $RES && break

let "--try" || exit 1 # LSB generic error

sleep 1

done

;;

stop)

# heartbeat (haresources mode) will retry failed stop

# for a number of times in addition to this internal retry.

try=3

while true; do

$DRBDADM secondary $RES && break

# We used to lie here, and pretend success for anything != 11,

# to avoid the reboot on failed stop recovery for "simple

# config errors" and such. But that is incorrect.

# Don't lie to your cluster manager.

# And don't do config errors...

let --try || exit 1 # LSB generic error

sleep 1

done

;;

status)

if [ "$RES" = "all" ]; then

echo "A resource name is required for status inquiries."

exit 10

fi

ST=$( $DRBDADM role $RES )

ROLE=${ST%/**}

case $ROLE in

Primary|Secondary|Unconfigured)

# expected

;;

*)

# unexpected. whatever...

# If we are unsure about the state of a resource, we need to

# report it as possibly running, so heartbeat can, after failed

# stop, do a recovery by reboot.

# drbdsetup may fail for obscure reasons, e.g. if /var/lock/ is

# suddenly readonly.  So we retry by parsing /proc/drbd.

drbd_set_role_from_proc_drbd

esac

case $ROLE in

Primary)

echo "running (Primary)"

exit 0 # LSB status "service is OK"

;;

Secondary|Unconfigured)

echo "stopped ($ROLE)"

exit 3 # LSB status "service is not running"

;;

*)

# NOTE the "running" in below message.

# this is a "heartbeat" resource script,

# the exit code is _ignored_.

echo "cannot determine status, may be running ($ROLE)"

exit 4 #  LSB status "service status is unknown"

;;

esac

;;

*)

echo "Usage: drbddisk [resource] {start|stop|status}"

exit 1

;;

esac

exit 0

-----------------------

赋予执行权限：

# chmod 755 /etc/ha.d/resource.d/drbddisk

在两个节点上启动HeartBeat服务，先启动node1：(node1,node2)

# service heartbeat start

# chkconfig heartbeat on

这里能够PING通虚IP 192.168.213.20，表示配置成功.

二、配置NFS
(node1,node2)
# vi /etc/exports
-----------------
/data *(rw,no_root_squash)
-----------------
重启NFS服务：
# service rpcbind restart # service nfs restart # chkconfig rpcbind on # chkconfig nfs off
这里设置NFS开机不要自动运行，因为/etc/ha.d/resource.d/killnfsd 该脚本内容控制NFS的启动。

三.最终测试

在另外一台LINUX的客户端挂载虚IP：192.168.213.20，挂载成功表明NFS+DRBD+HeartBeat大功告成.

# mount -t nfs 192.168.213.20:/data /tmp

# df -h

---------------

......

192.168.213.20:/data   1020M   34M  934M   4% /tmp

---------------

测试DRBD+HeartBeat+NFS可用性：

5.以上都未利用heartbeat实现故障自动转移，当线上DRBD主节点宕机，备份节点是否立即无缝接管，heartbeat+drbd高可用性是否能够实现？

首先先在客户端挂载NFS共享目录

# mount -t nfs 192.168.213.20:/data /tmp

a.模拟将主节点node1 的heartbeat服务停止，则备节点node2是否接管服务？

(node1)

# service drbd status

----------------------------

drbd driver loaded OK; device status:

version: 8.4.3 (api:1/proto:86-101)

GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@drbd1.example.com, 2013-05-27 20:45:19

m:res  cs         ro                 ds                 p  mounted  fstype

0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C  /data    ext4

----------------------------

# service heartbeat stop

(node2)

# service drbd status

----------------------------------------

drbd driver loaded OK; device status:

version: 8.4.3 (api:1/proto:86-101)

GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@drbd2.example.com, 2013-05-27 20:49:06

m:res  cs         ro                 ds                 p  mounted  fstype

0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C  /data    ext4

-----------------------------------------

从机无缝接管，测试客户端是否能够使用NFS共享目录

# cd /tmp

# touch test01

# ls test01

------------------

test01

------------------

测试通过。。。

b.模拟将主节点宕机（直接强行关机），则备节点node2是否接管服务？

(node1)

强制关机,直接关闭node1虚拟机电源

(node2)

# service drbd status

-------------------------------

drbd driver loaded OK; device status:

version: 8.4.3 (api:1/proto:86-101)

GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@drbd2.example.com, 2013-05-27 20:49:06

m:res  cs            ro               ds                 p  mounted  fstype

0:r0   WFConnection  Primary/Unknown  UpToDate/DUnknown  C  /data    ext4

-------------------------------

从机无缝接管，测试客户端是否能够使用NFS共享目录

# cd /tmp

# touch test02

# ls test02

------------------

test02

------------------

待node1恢复启动，查看drbd状态信息：

# service drbd status

------------------------------

drbd driver loaded OK; device status:

version: 8.4.3 (api:1/proto:86-101)

GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@drbd2.example.com, 2013-05-27 20:49:06

m:res  cs         ro                 ds                 p  mounted  fstype

0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C  /data    ext4

-------------------------------

node1已连接上线，处于UpToDate状态，测试通过。。。

注：这里node1的heartbeat有几率在关闭服务时，node2无法接管,所以有一定维护成本，因为本人线上未跑该服务，建议实际使用在上线前多做模拟故障演练，再实际上线。

本文由 Mr Gu 创作，采用 知识共享署名4.0 国际许可协议进行许可
本站文章除注明转载/出处外，均为本站原创或翻译，转载前请务必署名
最后编辑时间为: Aug 26, 2016 at 09:24 pm