Active-active Drbd With Ocfs2 92vv

Active-active DRBD with OCFS2 From Gentoo Linux Wiki Jump to: navigation, search

Contents  

1 Introduction 2 System Configuration



3 Kernel Configuration



4 Installing space tools





o

4.1 DRBD

o

4.2 OCFS2

5 Configuration o

5.1 DRBD

o

5.2 OCFS2

6 Preparing the devices o

6.1 Activating the DRBD resource

o

6.2 Syncing the nodes

o

6.3 Building the OCFS2 filesystem

o

6.4 Switch the DRBD in active-active mode



7 References



8 Resources

[edit] Introduction This is a rewrite of the orginal article by Mathijs and edited by Mars105, most of the credits goes to them. The main change is the re-appearance of the ocfs2 ebuild in the main portage tree and the new built in drbd kernel module (>=2.6.33).

[edit] System Configuration This setup is about running DRBD on 2 machines in active-active mode with OCFS2 as the cluster filesystem to make this possible. The following howto should be followed identical on both nodes, config files should also be identical. Except for the parts where it's explicitly stated. The author used two nodes (Tweedledum and Tweedledee) to build a common storage for Xen. The drbd device is built on top of a lvm volume. This is a choice made by the author but you may install on any block device you want. At the time of writing the author was using =sys-kernel/xen-sources-2.6.34 as kernel and the xen USE flag was on. Others kernels may provide different output for theses commands.

[edit] Kernel Configuration Activate the following options: Linux Kernel Configuration: OCFS Kernel (2.6.34) File systems ---> [*] OCFS2 file system [*] O2CB Kernelspace Clustering [*] OCFS2 statistics [*] OCFS2 logging [ ] OCFS2 expensive checks <*> Distributed Lock Manager (DLM) Device Drivers ---> [*] Block devices ---> [*] DRBD Distributed Replicated Block Device [ ] DRBD fault injection

You may want to leave "OCFS2 expensive checks" unchecked. For the other items, read the help. Alternatively, you can compile these options as modules and then add the following modules to /etc/conf.d/modules: ocfs2 ocfs2_dlmfs configfs ocfs2_dlm

[edit] Installing space tools [edit] DRBD Since kernel 2.6.34, we have builtin DRBD , so we need >=sys-cluster/drbd--8.3.8. Older ebuilds will try to pull the kernel module (sys-cluster/drbd-kernel). After building your kernel with the drbd module, determine the module version and install the corresponding drbd package. modinfo drbd | grep ^version

You may need to unmask the ebuild. (Replace amd64 with your platform) echo "=sys-cluster/drbd- -* ~amd64" >> /etc/portage/package.keywords

Install the drbd tools. emerge -av sys-cluster/drbd

[edit] OCFS2 Unmasking the needed ebuilds. (Replace amd64 with your platform) echo "sys-fs/ocfs2-tools ~amd64" >> /etc/portage/package.keywords

Emerging ocfs2-tools

emerge -av sys-fs/ocfs2-tools

[edit] Configuration [edit] DRBD Now we tell drbd how it must operate. Here we set two nodes (Tweedledum and Tweedledee) on which we created before a disk located at /dev/vg1/xen. Our drbd device will be /dev/drbd1 on both nodes, the resource name is drbd_xen. File: /etc/drbd.d/global_common.conf # Common settings, applies to all resources common { startup { wfc-timeout 60; # Wait for connection timeout degr-wfc-timeout 60;# Wait for connection timeout, if this node was a degraded cluster } disk { on-io-error detach; # Drop the disk on io error } syncer { rate 10M; # Limit sync speed to 10 MByte/s for FastEthernet } protocol C; }

File: /etc/drbd.conf # drbd_xen Our drbd space resource drbd_xen { on Tweedledum { device /dev/drbd1; disk /dev/vg1/xen; address 192.168.0.31:7789; meta-disk internal; } on Tweedledee { device /dev/drbd1; disk /dev/vg1/xen; address 192.168.0.32:7789; meta-disk internal; } }

Warning: rate 10M means 10 MByte/sec, it will transfer with a rate of 80MBit/sec on a FastEthernet port. Note: For more informations see the man page of drbd configuration file man drbd.conf

[edit] OCFS2 Some filesystems must be added to our fstab. The first two are needed by ocfs2 to work. The last is our drbd, this one is mandatory only for automatic boot time mount. File: /etc/fstab

# Needed by ocfs2 none /sys/kernel/config none /sys/kernel/dlm

configfs ocfs2_dlmfs

defaults defaults

# Our DRBD device /dev/drbd1

ocfs2

noatime

/xen

0 0 0 0 0 0

It will be a good idea to mount them now. mount /sys/kernel/config mount /sys/kernel/dlm

Configuration of ocfs2, same nodes as the drbd configuration. File: /etc/ocfs2/cluster.conf node: ip_port = 7777 ip_address = 192.168.0.31 number = 0 name = Tweedledum cluster = ocfs2_xen node:

ip_port = 7777 ip_address = 192.168.0.32 number = 1 name = Tweedledee cluster = ocfs2_xen

cluster: node_count = 2 name = ocfs2_xen

Note: you need to create this file yourself as the current ebuild at the time of this writing does not include it.

Now we must tell the init script which cluster must be started. File: /etc/conf.d/ocfs2 OCFS2_CLUSTER="ocfs2_xen"

Note: Be sure to check your firewall for ports 7777 and 7789, or any ports you set. Otherwise things will not work as expected.

[edit] Preparing the devices [edit] Activating the DRBD resource Now we are going to enable the DRBD resource for the first time. Theses commands must be issued on both nodes. Create the device metadata. This step must be completed only on initial device creation. It initializes DRBD's metadata: drbd create-md drbd_xen

Enable the resource. This step associates the resource with its backing device (or devices, in case of a multivolume resource), sets replication parameters, and connects the resource to its peer: drbd up drbd-xen

Note: Executing theses commands you may receview messages like this one: DRBD module version: 8.3.7 land version: 8.3.8 preferably kernel and land versions should match.

You may discard theses warnings. But make your checks for production use.

Result may be checked by reading /proc/drbd cat /proc/drbd version: 8.3.7 (api:88/proto:86-92) built-in 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r---ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:125825244

Note: The Inconsistent/Inconsistent disk state is expected at this point.

[edit] Syncing the nodes By now, DRBD has successfully allocated both disk and network resources and is ready for operation. What it does not know yet is which of your nodes should be used as the source of the initial device synchronization. Start the initial full synchronization. Warning: This step must be performed on only one node, only on initial resource configuration, and only on the node you selected as the synchronization source drbd -- --overwrite-data-of-peer primary drbd_xen

After issuing this command, the initial full synchronization will commence. You will be able to monitor its progress via /proc/drbd. It may take some time depending on the size of the device. watch cat /proc/drbd version: 8.3.7 (api:88/proto:86-92) built-in 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---ns:20502072 nr:0 dw:0 dr:20510092 al:0 bm:1250 lo:146 pe:2782 ua:2048 ap:0 ep:1 wo:b oos:105334296 [==>.................] sync'ed: 16.3% (102864/122876)M finish: 0:35:27 speed: 49,468 (60,444) K/sec

After this is finished and you see consistent state at both sides, let's create the ocfs2 filesystem on the drbd device.

[edit] Building the OCFS2 filesystem Warning: This is not a raid, you must wait for the sync to be finished before making the filesystem

We can start drbd and ocfs2. /etc/init.d/drbd start /etc/init.d/ocfs2 start

Now, use OCFS2's mkfs implementation to create the file system: mkfs.ocfs2 -N 2 -L ocfs2_xen /dev/drbd1 mkfs.ocfs2 1.4.4 Cluster stack: classic o2cb

Label: ocfs2_xen Features: sparse backup-super unwritten inline-data strict-journal-super Block size: 4096 (12 bits) Cluster size: 4096 (12 bits) Volume size: 128845049856 (31456311 clusters) (31456311 blocks) Cluster groups: 976 (tail covers 6711 clusters, rest cover 32256 clusters) Extent allocator size: 67108864 (16 groups) Journal size: 268435456 Node slots: 2 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Writing backup superblock: 4 block(s) Formatting Journals: done Growing extent allocator: done Formatting slot map: done Writing lost+found: done mkfs.ocfs2 successful

This will create an OCFS2 file system with two node slots on /dev/drbd1, and set the filesystem label to ocfs2_xen. Warning: This step must be done on one node only. The filesystem will be created on the drdb device which is active and will replicate to the nodes.

[edit] Switch the DRBD in active-active mode The final thing we have to do is configure DRBD for active-active mode. Update your /etc/drbd.conf by adding the startup and net sections to your new resource. File: /etc/drbd.conf # Common settings, applies to all resources common { # No changes } # drbd_xen Our drbd space resource drbd_xen { startup { become-primary-on both; } net { allow-two-primaries; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; } # No changes to the nodes specification }

Restart DRBD /etc/init.d/drbd restart

Check the state of the cluster

cat /proc/drbd version: 8.3.7 (api:88/proto:86-92) built-in 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r---ns:0 nr:0 dw:0 dr:408 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

If it looks fine and active-active (Primary/Primary), you can mount your new OCFS2 filesystem: mount /xen

Cluster can be tested by creating a file on one node and reading on the other. We can safely add drbd and ocfs2 on default run level. rc-update add drbd default rc-update add ocfs2 default

Congratulations! You should have a working cluster now.

[edit] References Some references the authors used and the reader may found usefull: http://www.drbd.org/s-guide/ch-ocfs2.html http://www.drbd.org/s-guide-emb/ch-configure.html http://linux.dell.com/wiki/index.php/Set_up_an_OCFS2_cluster_filesystem  Edited by: Xerxes - 2011-10-16 

Edited by: Mars105 - 2010-06-30



Originally written by: Mathijs - 2009-08-16

[edit] Resources This init script come from the package submited by Mathijs in is original article. The only modification made here is to remove the checks for modules, as it will fail when DRBD is built in the kernel. And the mount check will presumably fail if the modules are not loaded. Fix me: Somebody with better init script understanding can maybe improve this script.

File: /etc/init.d/ocfs2 #!/sbin/runscript # Copyright 1999-2006 Gentoo Foundation # Distributed under the of the GNU General Public License v2 depend() { need net localmount before netmount } check_pseudofs() { local retval=0 local HASMOUNT="mount -l -t" if [ -z "`${HASMOUNT} configfs`" ] ; then retval=1

fi if [ -z "`${HASMOUNT} ocfs2_dlmfs`" ] ; then retval=1 fi

0 0"

if [ ${retval} -eq 1 ]; then ewarn "One or more pseudo-filesystes are not mounted." ewarn "Make sure you have following lines in your /etc/fstab:" ewarn "none /config configfs defaults ewarn "none

0 0"

/dlm

ocfs2_dlmfs

defaults

ewarn "For a (in)complete documentation, read /usr/share/doc/ocfs /INSTALL.GENTOO.gz" fi return ${retval} }

start() { check_pseudofs || return $? # A small hack to avoid: # mount.ocfs2: Unable to access cluster service while trying to the group OCFS2_CLUSTER=$( sed -re 's/[[:space:]]+//g' /etc/ocfs2/cluster.conf | grep -A4 "cluster:" | grep 'name=' | cut -d= -f2 )

}

einfo "Starting OCFS2 cluster" for cluster in ${OCFS2_CLUSTER}; do ebegin " - ${cluster}" /sbin/o2cb_ctl -H -n ${cluster} -t cluster -a eend $? done

>/dev/null 2>&1

stop() { # Shamelesly stolen from netmount local ret ebegin "Unmounting OCFS2 filesystems" [ -z "$(umount -art ocfs2 2>&1)" ] ret=$? eend ${ret} "Failed to simply unmount filesystems" [ ${ret} -eq 0 ] && return 0 declare -a siglist=( "TERM" "KILL" "KILL" ) local retry=0 local remaining="go" while [ -n "${remaining}" -a ${retry} -lt 3 ] do remaining="$(awk '$3 ~ /'ocfs2'/ { if ($2 != "/") print $2 }' /proc/mounts | sort -r)" IFS=$'\n' set -- ${remaining//\\040/ } unset IFS [ -z "${remaining}" ] && break ebegin $'\t'"Unmounting ocfs2 filesystems (retry #$((retry+1)))" /bin/f -k -${siglist[$((retry++))]} -m "$@" &>/dev/null sleep 5 umount "$@" &>/dev/null eend $? $'\t'"Failed to unmount filesystems" done einfo "Stopping OCFS2 cluster" for cluster in ${OCFS_CLUSTERS}; do

done

ebegin " - ${cluster}" /sbin/o2cb_ctl -H -n ${cluster} -t cluster -a eend $?

>/dev/null 2>&1

}

Don't forget to make it executable. chmod a+x /etc/init.d/ocfs2

Retrieved from "http://en.gentoo-wiki.com/wiki/Active-active_DRBD_with_OCFS2"