Untitled :: Private Cloud

Piraeus Operator

Wipe talos disk

talosctl -n 192.168.0.31 wipe disk sda --drop-partition

 user: warning: [2025-10-21T10:26:30.149249282Z]: [talos] locking block device "sda"
 user: warning: [2025-10-21T10:26:30.149266282Z]: [talos] wiping block device "sda" with fast method
 user: warning: [2025-10-21T10:26:34.787941282Z]: [talos] block device "sda" wiped by ranges: 536-544

Check linstor

tamaymueller@terra:~$ kubectl -n piraeus-datastore exec deploy/linstor-controller -- linstor node list
+------------------------------------------------------------------------------------+
| Node        | NodeType  | Addresses                                       | State  |
|====================================================================================|
| k8s-home-01 | SATELLITE | 2A02:8070:D88:3680:24E:1FF:FEA2:B06A:3367 (SSL) | Online |
| k8s-home-02 | SATELLITE | 192.168.0.32:3367 (SSL)                         | Online |
| k8s-home-03 | SATELLITE | 192.168.0.33:3367 (SSL)                         | Online |
+------------------------------------------------------------------------------------+

tamaymueller@terra:~$ kubectl -n piraeus-datastore exec deploy/linstor-controller -- linstor storage-pool list
+----------------------------------------------------------------------------------------------------------------------------------------------------+
| StoragePool          | Node        | Driver   | PoolName  | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName                       |
|====================================================================================================================================================|
| DfltDisklessStorPool | k8s-home-01 | DISKLESS |           |              |               | False        | Ok    | k8s-home-01;DfltDisklessStorPool |
| DfltDisklessStorPool | k8s-home-02 | DISKLESS |           |              |               | False        | Ok    | k8s-home-02;DfltDisklessStorPool |
| DfltDisklessStorPool | k8s-home-03 | DISKLESS |           |              |               | False        | Ok    | k8s-home-03;DfltDisklessStorPool |
| zfs-thinpool-1       | k8s-home-01 | ZFS_THIN | zfs-thin1 |   230.62 GiB |       238 GiB | True         | Ok    | k8s-home-01;zfs-thinpool-1       |
| zfs-thinpool-1       | k8s-home-02 | ZFS_THIN | zfs-thin1 |   922.50 GiB |       952 GiB | True         | Ok    | k8s-home-02;zfs-thinpool-1       |
| zfs-thinpool-1       | k8s-home-03 | ZFS_THIN | zfs-thin1 |    14.53 GiB |        15 GiB | True         | Ok    | k8s-home-03;zfs-thinpool-1       |
+----------------------------------------------------------------------------------------------------------------------------------------------------+

kubectl -n piraeus-datastore exec deploy/linstor-controller -- linstor resource list
+-------------------------------------------------------------------------------------------------------------------------+
| ResourceName                             | Node        | Layers       | Usage  | Conns |    State | CreatedOn           |
|=========================================================================================================================|
| pvc-9d3ce6ec-17ac-4d27-87b2-b1a4da05f139 | k8s-home-01 | DRBD,STORAGE | Unused | Ok    | UpToDate | 2025-10-23 22:05:45 |
| pvc-9d3ce6ec-17ac-4d27-87b2-b1a4da05f139 | k8s-home-02 | DRBD,STORAGE | Unused | Ok    | UpToDate | 2025-10-23 22:05:48 |
| pvc-56aabe2e-cc61-4c22-9220-efef3a7ca54a | k8s-home-01 | DRBD,STORAGE | Unused | Ok    | UpToDate | 2025-10-23 22:32:56 |
| pvc-56aabe2e-cc61-4c22-9220-efef3a7ca54a | k8s-home-02 | DRBD,STORAGE | Unused | Ok    | UpToDate | 2025-10-23 22:32:55 |
| pvc-66931031-dc0c-4f8c-a6df-925c10c2fea2 | k8s-home-02 | DRBD,STORAGE | Unused | Ok    | UpToDate | 2025-10-23 22:33:28 |
| pvc-66931031-dc0c-4f8c-a6df-925c10c2fea2 | k8s-home-03 | DRBD,STORAGE | InUse  | Ok    | UpToDate | 2025-10-23 22:32:56 |
+-------------------------------------------------------------------------------------------------------------------------+

DRBD split brain

Recognize the problems:

[home-tamay-cloud|default] tamaymueller@terra:~$ kubectl -n piraeus-datastore exec deploy/linstor-controller -- linstor resource list
+---------------------------------------------------------------------------------------------------------------------------------------------+
| ResourceName                             | Node        | Layers       | Usage  | Conns                   |      State | CreatedOn           |
|=============================================================================================================================================|
| pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c | k8s-home-01 | DRBD,STORAGE | InUse  | Connecting(k8s-home-02) |   UpToDate | 2025-10-28 12:23:53 |
| pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c | k8s-home-02 | DRBD,STORAGE | Unused | Connecting(k8s-home-01) |   Outdated | 2025-10-23 23:24:32 |
| pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c | k8s-home-03 | DRBD,STORAGE | Unused | Ok                      |   Diskless | 2025-10-23 23:24:33 |

Logs in talos dashboard on node k8s-home-01:

 kern:    info: [2025-10-28T14:39:40.529630215Z]: drbd pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c/0 drbd1014 k8s-home-02: drbd_sync_handshake:
 kern:    info: [2025-10-28T14:39:40.529648215Z]: drbd pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c/0 drbd1014 k8s-home-02: self D5DEDF2186AAE01B:0000000000000000:F74843125DD62BEC:2F5A587D703D6F16 bits:0 flags:120
 kern:    info: [2025-10-28T14:39:40.529675215Z]: drbd pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c/0 drbd1014 k8s-home-02: peer D5DEDF2186AAE01A:5E601A97586F46C6:E5BE96C09FAD8156:2F5A587D703D6F16 bits:6144 flags: 1020
 kern:    info: [2025-10-28T14:39:40.529699215Z]: drbd pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c/0 drbd1014 k8s-home-02: uuid_compare()=target-use-bitmap by rule=sync-target-peer-missed-finish
 kern:     err: [2025-10-28T14:39:40.529719215Z]: drbd pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c/0 drbd1014 k8s-home-02: I shall become SyncTarget, but I am primary!
 kern:    info: [2025-10-28T14:39:40.535597215Z]: drbd pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c k8s-home-02: Aborting remote state change 3397858640
 kern:    info: [2025-10-28T14:39:40.591432215Z]: drbd pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c: Preparing remote state change 1189924968: 0->1 role( Secondary ) conn( Connected )
 kern:    info: [2025-10-28T14:39:40.605472215Z]: drbd pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c/0 drbd1014 k8s-home-02: Peer missed end of resync

Connect to the linstor-satellite on the node where the State is Outdated. Then reconnect via drbdadm while discarding the changes.

[home-tamay-cloud|default] tamaymueller@terra:~$ kubectl -n piraeus-datastore exec -it ds/linstor-satellite.k8s-home-02 -- bash
root@k8s-home-02:/# drbdadm disconnect pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c
root@k8s-home-02:/# drbdadm secondary pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c
root@k8s-home-02:/# drbdadm connect --discard-my-data pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c
root@k8s-home-02:/# drbdadm status pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c
pvc-7fdb163c-9d02-4028-8082-8b0f2863f28c role:Secondary
  disk:UpToDate open:no
  k8s-home-01 role:Primary tls:yes
    peer-disk:UpToDate
  k8s-home-03 role:Secondary tls:yes
    peer-disk:Diskless

DRBD Replication via seperate NIC

First add a node interface to each node manually linstor node interface create <nodename> <interface-name> <ip>.

The interface-name can be arbitrary

kubectl -n piraeus-datastore exec -it deployments/linstor-controller -- linstor node interface create k8s-home-01 replication 10.0.0.21
kubectl -n piraeus-datastore exec -it deployments/linstor-controller -- linstor node interface create k8s-home-02 replication 10.0.0.22
kubectl -n piraeus-datastore exec -it deployments/linstor-controller -- linstor node interface create k8s-home-03 replication 10.0.0.23

Next you need to add the object LinstorNodeConnection to your piraeus cluster.

The interface should be the same name as the interface-name from above.

apiVersion: piraeus.io/v1
kind: LinstorNodeConnection
metadata:
  name: dedicated
spec:
  paths:
    - name: dedicated
      interface: replication

DRBD Sync speed

    - name: DrbdOptions/Net/max-buffers
      value: "36864"
    - name: DrbdOptions/Net/rcvbuf-size
      value: "10485760"
    - name: DrbdOptions/Net/sndbuf-size
      value: "10485760"
    - name: DrbdOptions/PeerDevice/c-fill-target
      value: "2048"
    - name: DrbdOptions/PeerDevice/c-max-rate
      value: "440320"
    - name: DrbdOptions/PeerDevice/c-min-rate
      value: "146773"
    - name: DrbdOptions/PeerDevice/resync-rate
      value: "146773"
    - name: DrbdOptions/PeerDevice/c-plan-ahead
      value: "10"

Resource stuck in `Connecting(k8s-home-0X)`

root@linstor-controller-7fcfbf499b-vt9x7:/# linstor r l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node        ┊ Layers       ┊ Usage  ┊ Conns                   ┊      State ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-db66ed74-6b47-45a8-927a-65de3716ce19 ┊ k8s-home-01 ┊ DRBD,STORAGE ┊ InUse  ┊ Connecting(k8s-home-03) ┊   Diskless ┊ 2025-10-29 00:12:10 ┊
┊ pvc-db66ed74-6b47-45a8-927a-65de3716ce19 ┊ k8s-home-02 ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(k8s-home-03) ┊   UpToDate ┊ 2025-10-29 00:12:08 ┊
┊ pvc-db66ed74-6b47-45a8-927a-65de3716ce19 ┊ k8s-home-03 ┊ DRBD,STORAGE ┊        ┊                         ┊    Unknown ┊ 2025-10-29 00:12:10 ┊
┊ pvc-e9d637f2-551a-4280-82f6-62306911781a ┊ k8s-home-01 ┊ DRBD,STORAGE ┊ InUse  ┊ Connecting(k8s-home-03) ┊   UpToDate ┊ 2026-01-04 12:47:36 ┊
┊ pvc-e9d637f2-551a-4280-82f6-62306911781a ┊ k8s-home-02 ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(k8s-home-03) ┊   UpToDate ┊ 2026-01-04 12:47:32 ┊
┊ pvc-e9d637f2-551a-4280-82f6-62306911781a ┊ k8s-home-03 ┊ DRBD,STORAGE ┊        ┊                         ┊    Unknown ┊ 2026-01-04 12:47:35 ┊
┊ pvc-eaeca38c-658d-4ff1-9e75-8687d954339d ┊ k8s-home-01 ┊ DRBD,STORAGE ┊ Unused ┊ Ok                      ┊   Diskless ┊ 2025-10-28 23:25:43 ┊
┊ pvc-eaeca38c-658d-4ff1-9e75-8687d954339d ┊ k8s-home-02 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok                      ┊   UpToDate ┊ 2025-10-28 23:25:42 ┊
┊ pvc-eaeca38c-658d-4ff1-9e75-8687d954339d ┊ k8s-home-03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok                      ┊   UpToDate ┊ 2025-10-28 23:25:43 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Basically just deactivate and activate the resource on the node again

In this example the resource pvc-e9d637f2-551a-4280-82f6-62306911781a is not connected for k8s-home-03. So we just deactive the resource and then activate it again.

linstor resource deactivate k8s-home-03 pvc-e9d637f2-551a-4280-82f6-62306911781a
linstor resource activate k8s-home-03 pvc-e9d637f2-551a-4280-82f6-62306911781a

root@linstor-controller-7fcfbf499b-vt9x7:/# linstor resource deactivate k8s-home-03 pvc-e9d637f2-551a-4280-82f6-62306911781a
SUCCESS:
    Resource deactivated on 'k8s-home-03'
SUCCESS:
    (k8s-home-01) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Resource marked inactivate on 'k8s-home-01'
SUCCESS:
    (k8s-home-02) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Resource marked inactivate on 'k8s-home-02'
SUCCESS:
    Finished deactivation of resource on 'k8s-home-03'
SUCCESS:
    (k8s-home-02) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Finished deactivation of resource on 'k8s-home-02'
SUCCESS:
    (k8s-home-01) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Finished deactivation of resource on 'k8s-home-01'

root@linstor-controller-7fcfbf499b-vt9x7:/# linstor resource list
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node        ┊ Layers       ┊ Usage  ┊ Conns                   ┊      State ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-e9d637f2-551a-4280-82f6-62306911781a ┊ k8s-home-01 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok                      ┊   UpToDate ┊ 2026-01-04 12:47:36 ┊
┊ pvc-e9d637f2-551a-4280-82f6-62306911781a ┊ k8s-home-02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok                      ┊   UpToDate ┊ 2026-01-04 12:47:32 ┊
┊ pvc-e9d637f2-551a-4280-82f6-62306911781a ┊ k8s-home-03 ┊ DRBD,STORAGE ┊        ┊ Ok                      ┊   INACTIVE ┊ 2026-01-04 12:47:35 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

root@linstor-controller-7fcfbf499b-vt9x7:/# linstor resource activate k8s-home-03 pvc-e9d637f2-551a-4280-82f6-62306911781a
SUCCESS:
    (k8s-home-03) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Reactivating resource on 'k8s-home-03'
SUCCESS:
    (k8s-home-02) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Resource updated on 'k8s-home-02'
SUCCESS:
    (k8s-home-01) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Resource updated on 'k8s-home-01'
SUCCESS:
    (k8s-home-03) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Resource activated on 'k8s-home-03'
SUCCESS:
    (k8s-home-01) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Resource activated on 'k8s-home-01'
SUCCESS:
    (k8s-home-02) Resource 'pvc-e9d637f2-551a-4280-82f6-62306911781a' [DRBD] adjusted.
SUCCESS:
    Resource activated on 'k8s-home-02'

root@linstor-controller-7fcfbf499b-vt9x7:/# linstor resource list
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node        ┊ Layers       ┊ Usage  ┊ Conns                   ┊      State ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-e9d637f2-551a-4280-82f6-62306911781a ┊ k8s-home-01 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok                      ┊   UpToDate ┊ 2026-01-04 12:47:36 ┊
┊ pvc-e9d637f2-551a-4280-82f6-62306911781a ┊ k8s-home-02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok                      ┊   UpToDate ┊ 2026-01-04 12:47:32 ┊
┊ pvc-e9d637f2-551a-4280-82f6-62306911781a ┊ k8s-home-03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok                      ┊   Diskless ┊ 2026-01-04 12:47:35 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

If it happens with multiple resources on one node (here k8s-home-03), use this loop to deactivate and activate all:

for i in $(linstor resource list | grep Connecting | cut -f2 -d'|' | uniq); do linstor resource deactivate k8s-home-03 $i; done
for i in $(linstor resource list | grep Connecting | cut -f2 -d'|' | uniq); do linstor resource activate k8s-home-03 $i; done

delpath

 kern:     err: [2026-04-22T11:37:52.151821628Z]: drbd pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88 k8s-home-02: del_path() failed with -16
 kern:     err: [2026-04-22T11:42:52.055195653Z]: drbd pvc-c0a31630-18c3-4c23-af4d-0f8208e8e16b k8s-home-02: del_path() failed with -16 ```



```shell
root@linstor-controller-9b64fd848-7sl5q:/# linstor r l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node        ┊ Layers       ┊ Usage  ┊ Conns                   ┊      State ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88 ┊ k8s-home-01 ┊ DRBD,STORAGE ┊        ┊ Ok                      ┊   DELETING ┊ 2026-04-21 11:42:47 ┊
┊ pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88 ┊ k8s-home-02 ┊ DRBD,STORAGE ┊ Unused ┊ StandAlone(k8s-home-03) ┊   UpToDate ┊ 2026-04-21 11:42:48 ┊
┊ pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88 ┊ k8s-home-03 ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(k8s-home-02) ┊   UpToDate ┊ 2026-04-21 11:42:47 ┊

It’s in deleting state.. Connect to the linstor satellite pod and debug there .

root@k8s-home-02:/# drbdadm del-path pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88
pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88: Failure: (162) Invalid configuration request
additional info from kernel:
unknown connection
Command 'drbdsetup del-path pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88 2 ipv4:10.0.0.22:7003 ipv4:10.0.0.21:7003' terminated with exit code 10
pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88: Failure: (162) Invalid configuration request
additional info from kernel:
unknown connection
Command 'drbdsetup del-path pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88 2 ipv6:[FDE4:79D:3B73::22]:7003 ipv6:[FDE4:79D:3B73::21]:7003' terminated with exit code 10

root@k8s-home-02:/# drbdsetup show pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88
resource "pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88" {
    options {
        on-no-data-accessible   suspend-io;
        on-suspended-primary-outdated   force-secondary;
        _unknown drbd8-api-compatibility;   # not supported by kernel
    }
    _this_host {
        node-id         1;
        volume 0 {
            device          minor 1003;
            disk            "/dev/zvol/zfs-thin-1/pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88_00000";
            meta-disk           internal;
            disk {
                rs-discard-granularity  16384; # bytes
                _unknown bitmap        ;    # not supported by kernel
                block-size          4096;
                _unknown discard-granularity;   # not supported by kernel
            }
        }
    }
    connection {
        _peer_node_id 0;
        path {
            _this_host ipv4 10.0.0.22:7003;
            _remote_host ipv4 10.0.0.23:7003;
        }
        path {
            _this_host ipv6 [fde4:79d:3b73::22]:7003;
            _remote_host ipv6 [fde4:79d:3b73::23]:7003;
        }
        _cstate Connected;
        net {
            sndbuf-size         10485760; # bytes
            rcvbuf-size         10485760; # bytes
            cram-hmac-alg       "sha1";
            shared-secret       "p22ssgVujitmdptnlD+m";
            rr-conflict         retry-connect;
            verify-alg          "crc32c";
            max-buffers         36864;
            tls                 yes;
            _name               "k8s-home-03";
        }
        volume 0 {
            disk {
                resync-rate         146773k; # bytes/second
                c-plan-ahead        10; # 1/10 seconds
                c-fill-target       2048s; # bytes
                c-max-rate          440320k; # bytes/second
                c-min-rate          146773k; # bytes/second
                _unknown resync-without-replication;    # not supported by kernel
                _unknown peer-tiebreaker;   # not supported by kernel
            }
        }
    }
}

What worked:

disconnect the pvc (use the _peer_node_id “0” from the output above)

root@k8s-home-02:/# drbdsetup disconnect pvc-2cb71462-d5f8-4e99-ab0d-1184746a4f88 0