Global heartbeat region can't be found (OCFS2 1.8.6 on Ubuntu 20.04)

[DoobleD]

I'm not sure if this is the right place to ask this, if there's another one, please let me know.

I can't seem to have global heartbeat work, using OCFS2 1.8.6 on Ubuntu (20.04). The local heartbeat mode works fine though.

Here's my setup:

$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0         7:0    0 33.3M  1 loop /snap/amazon-ssm-agent/3552
loop1         7:1    0   25M  1 loop /snap/amazon-ssm-agent/4046
loop2         7:2    0 55.5M  1 loop /snap/core18/1997
loop3         7:3    0 55.4M  1 loop /snap/core18/2128
loop4         7:4    0 70.3M  1 loop /snap/lxd/21029
loop5         7:5    0 32.3M  1 loop /snap/snapd/12704
loop6         7:6    0 32.3M  1 loop /snap/snapd/11588
loop7         7:7    0 70.4M  1 loop /snap/lxd/19647
nvme1n1     259:0    0  128G  0 disk 
nvme0n1     259:1    0   16G  0 disk 
└─nvme0n1p1 259:2    0   16G  0 part /
$ sudo mkfs.ocfs2 --cluster-stack=o2cb --cluster-name=ocfs2 --global-heartbeat /dev/nvme1n1
mkfs.ocfs2 1.8.6
Cluster stack: o2cb
Cluster name: ocfs2
Stack Flags: 0x1
NOTE: Feature extended slot map may be enabled
Label: 
Features: sparse extended-slotmap backup-super unwritten inline-data strict-journal-super xattr indexed-dirs refcount discontig-bg append-dio
Block size: 4096 (12 bits)
Cluster size: 4096 (12 bits)
Volume size: 68719476736 (16777216 clusters) (16777216 blocks)
Cluster groups: 521 (tail covers 4096 clusters, rest cover 32256 clusters)
Extent allocator size: 12582912 (3 groups)
Journal size: 268435456
Node slots: 8
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 3 block(s)
Formatting Journals: done
Growing extent allocator: done
Formatting slot map: done
Formatting quota files: done
Writing lost+found: done
mkfs.ocfs2 successful
$ sudo o2cb add-cluster ocfs2
$ sudo o2cb add-node ocfs2 ip-172-31-32-10 --ip 172.31.32.10
$ sudo o2cb add-node ocfs2 ip-172-31-32-20 --ip 172.31.32.20
$ sudo o2cb add-heartbeat ocfs2 /dev/nvme1n1
$ sudo o2cb heartbeat-mode ocfs2 global
$ cat /etc/ocfs2/cluster.conf
cluster:
	name = ocfs2
	heartbeat_mode = global
	node_count = 2

heartbeat:
	cluster = ocfs2
	region = 1FFE62175A94430EAF3BDD5E28964126

node:
	cluster = ocfs2
	number = 0
	ip_port = 7777
	ip_address = 172.31.32.10
	name = ip-172-31-32-10

node:
	cluster = ocfs2
	number = 1
	ip_port = 7777
	ip_address = 172.31.32.20
	name = ip-172-31-32-20
$ sudo tunefs.ocfs2 -Q "Label = %V\nUUID = %U\nNumSlots =%N\n" /dev/nvme1n1
Label = 
UUID = 1FFE62175A94430EAF3BDD5E28964126
NumSlots =8
$ ll /sys/kernel/config/cluster/ocfs2/heartbeat/
total 0
drwxr-xr-x 2 root root    0 Aug 14 11:14 ./
drwxr-xr-x 4 root root    0 Aug 14 11:14 ../
-rw-r--r-- 1 root root 4096 Aug 14 11:14 dead_threshold
-rw-r--r-- 1 root root 4096 Aug 14 11:20 mode
$ cat /sys/kernel/config/cluster/ocfs2/heartbeat/mode
global

Then if I try to run o2cb I get (online instead of enable gives the same output):

$ sudo service o2cb enable
checking debugfs...
Setting cluster stack "o2cb": OK
Registering O2CB cluster "ocfs2": OK
Setting O2CB cluster timeouts : OK
Starting global heartbeat for cluster "ocfs2": Failed
o2cb: Heartbeat region could not be found 1FFE62175A94430EAF3BDD5E28964126
Stopping global heartbeat on cluster "ocfs2": OK

As you can see the heartbeat region can't be found, despite my heartbeat device being initialized with --global-heartbeat and having the correct UUID. The device has no partition on it.

The nodes configuration seems correct since in local heartbeat mode the same settings work fine.

I've repeated the setup multiple times from fresh installations and I've read multiple guides which all seem to align with this. dmesg doesn't show any error, it just says o2hb: Heartbeat mode set to global as expected.

Any idea on what could be wrong?

[DoobleD]

I've found this thread from 2016 on the exact same symptoms, though at the end no solution is found.