PV Migration

Mit dem Tool pv-migrate kann man den Inhalt von PersistenVolumeClaims zu anderen PVCs (auch in anderen Clustern) migrieren.

Installation wird mittels brew durchgeführt.

brew tap utkuozdemir/pv-migrate
brew install pv-migrate

Migration auf andere StorageClass

  • PVCs haben die StorageClass longhorn

  • PVCs sollen zur StorageClass piraeus-storage migriert werden

  • Namespace ist mittels flux gemanaged

Ablauf

  • Bestenfalls ein Velero backup machen!

  • Flux kustomization stoppen

  • Deployment auf 0 skalieren (wenn RWO PVC, da pv-migrate die PVC mountet)

  • neue temporäre PVC erstellen (pvc-original -> pvc-original2)

  • pv-migrate von pvc-original auf pvc-original2

  • pvc-original löschen

  • pvc-original auf neuer StorageClass anlegen

  • pv-migrate von temporärer PVC ausführen

  • Deployment wieder auf >0 skalieren

  • Flux kustomization wieder starten

  • überprüfen, dass alles geklappt hat

  • temporäre PVC löschen

Beispiel

Die beiden PVCs im Namespace baikal sollen von longhorn auf piraeus-storage migriert werden.

tamaymueller@terra:~$ kubectl -n baikal get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
baikal-config     Bound    pvc-a80e8779-d1d4-4f6b-94b8-1c97c2351655   1Gi        RWO            longhorn       <unset>                 26d
baikal-specific   Bound    pvc-8660e6a9-c92e-4108-aa45-bad4066ba096   200Mi      RWO            longhorn       <unset>                 26d

Zuerst stoppt man den Sync durch flux

tamaymueller@terra:~$ flux suspend kustomization baikal
► suspending kustomization baikal in flux-system namespace
✔ kustomization suspended

Dann skaliert man das Deployment auf 0

tamaymueller@terra:~/migrate$ kubectl -n baikal scale deployment baikal --replicas 0
deployment.apps/baikal scaled

tamaymueller@terra:~/migrate$ kubectl -n baikal get deployments.apps
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
baikal   0/0     0            0           26d

Dann die beiden temporären PVCs anlegen

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: baikal-config-tmp
  namespace: baikal
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: piraeus-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: baikal-specific-tmp
  namespace: baikal
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 200Mi
  storageClassName: piraeus-storage
tamaymueller@terra:~/migrate$ kubectl -n baikal get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
baikal-config         Bound    pvc-a80e8779-d1d4-4f6b-94b8-1c97c2351655   1Gi        RWO            longhorn          <unset>                 26d
baikal-config-tmp     Bound    pvc-c08cca00-ada3-4313-8ee9-3e1cb5f7204d   1Gi        RWO            piraeus-storage   <unset>                 11s
baikal-specific       Bound    pvc-8660e6a9-c92e-4108-aa45-bad4066ba096   200Mi      RWO            longhorn          <unset>                 26d
baikal-specific-tmp   Bound    pvc-af8c4088-b2d7-4cd7-917c-57357d7c8ad8   200Mi      RWO            piraeus-storage   <unset>                 11s

Jetzt migriert man die beiden PVCs

tamaymueller@terra:~/migrate$
Mar 20 22:30:27.055 INF 🚀 Starting migration
Mar 20 22:30:27.170 INF 💭 Attempting migration source=baikal/baikal-specific dest=baikal/baikal-specific-tmp strategies=mnt2,svc,lbsvc
Mar 20 22:30:27.170 INF 🚁 Attempt using strategy source=baikal/baikal-specific dest=baikal/baikal-specific-tmp attempt_id=babcb strategy=mnt2
W0320 22:30:27.623913   71424 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "rsync" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "rsync" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "rsync" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "rsync" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
📂 Copying data... 100% |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (154 MB/s)
Mar 20 22:30:41.554 INF 🧹 Cleaning up source=baikal/baikal-specific dest=baikal/baikal-specific-tmp attempt_id=babcb strategy=mnt2
Mar 20 22:30:41.884 INF ✨ Cleanup done source=baikal/baikal-specific dest=baikal/baikal-specific-tmp attempt_id=babcb strategy=mnt2
Mar 20 22:30:41.884 INF ✅ Migration succeeded source=baikal/baikal-specific dest=baikal/baikal-specific-tmp attempt_id=babcb strategy=mnt2
tamaymueller@terra:~/migrate$ pv-migrate \
  --source-namespace=baikal \
  --source=baikal-config \
  --dest-namespace=baikal \
  --dest=baikal-config-tmp
Mar 20 22:33:26.312 INF 🚀 Starting migration
Mar 20 22:33:26.487 INF 💭 Attempting migration source=baikal/baikal-config dest=baikal/baikal-config-tmp strategies=mnt2,svc,lbsvc
Mar 20 22:33:26.487 INF 🚁 Attempt using strategy source=baikal/baikal-config dest=baikal/baikal-config-tmp attempt_id=abeba strategy=mnt2
W0320 22:33:27.010428   71857 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "rsync" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "rsync" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "rsync" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "rsync" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
📂 Copying data... 100% |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (217 kB/s)
Mar 20 22:33:40.695 INF 🧹 Cleaning up source=baikal/baikal-config dest=baikal/baikal-config-tmp attempt_id=abeba strategy=mnt2
Mar 20 22:33:41.029 INF ✨ Cleanup done source=baikal/baikal-config dest=baikal/baikal-config-tmp attempt_id=abeba strategy=mnt2
Mar 20 22:33:41.029 INF ✅ Migration succeeded source=baikal/baikal-config dest=baikal/baikal-config-tmp attempt_id=abeba strategy=mnt2

Jetzt löscht man die originale PVCs

tamaymueller@terra:~/migrate$ kubectl -n baikal delete pvc baikal-config baikal-specific
persistentvolumeclaim "baikal-config" deleted
persistentvolumeclaim "baikal-specific" deleted

Dann legt man die neuen PVCs an

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: baikal-config
  namespace: baikal
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: piraeus-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: baikal-specific
  namespace: baikal
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 200Mi
  storageClassName: piraeus-storage

Dann migriert man von den temporären PVCs zu den neuen PVCs

tamaymueller@terra:~/migrate$ pv-migrate \
  --source-namespace=baikal \
  --source=baikal-specific-tmp \
  --dest-namespace=baikal \
  --dest=baikal-specific
Mar 20 22:37:19.702 INF 🚀 Starting migration
Mar 20 22:37:19.818 INF 💭 Attempting migration source=baikal/baikal-specific-tmp dest=baikal/baikal-specific strategies=mnt2,svc,lbsvc
Mar 20 22:37:19.818 INF 🚁 Attempt using strategy source=baikal/baikal-specific-tmp dest=baikal/baikal-specific attempt_id=adaed strategy=mnt2
W0320 22:37:20.311958   72742 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "rsync" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "rsync" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "rsync" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "rsync" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
📂 Copying data... 100% |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (64 MB/s)
Mar 20 22:37:26.853 INF 🧹 Cleaning up source=baikal/baikal-specific-tmp dest=baikal/baikal-specific attempt_id=adaed strategy=mnt2
Mar 20 22:37:27.259 INF ✨ Cleanup done source=baikal/baikal-specific-tmp dest=baikal/baikal-specific attempt_id=adaed strategy=mnt2
Mar 20 22:37:27.259 INF ✅ Migration succeeded source=baikal/baikal-specific-tmp dest=baikal/baikal-specific attempt_id=adaed strategy=mnt2
tamaymueller@terra:~/migrate$ pv-migrate \
  --source-namespace=baikal \
  --source=baikal-config-tmp \
  --dest-namespace=baikal \
  --dest=baikal-config
Mar 20 22:37:30.426 INF 🚀 Starting migration
Mar 20 22:37:30.533 INF 💭 Attempting migration source=baikal/baikal-config-tmp dest=baikal/baikal-config strategies=mnt2,svc,lbsvc
Mar 20 22:37:30.533 INF 🚁 Attempt using strategy source=baikal/baikal-config-tmp dest=baikal/baikal-config attempt_id=dbbdb strategy=mnt2
W0320 22:37:31.012920   72768 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "rsync" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "rsync" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "rsync" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "rsync" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
📂 Copying data... 100% |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (74 kB/s)
Mar 20 22:37:37.902 INF 🧹 Cleaning up source=baikal/baikal-config-tmp dest=baikal/baikal-config attempt_id=dbbdb strategy=mnt2
Mar 20 22:37:38.226 INF ✨ Cleanup done source=baikal/baikal-config-tmp dest=baikal/baikal-config attempt_id=dbbdb strategy=mnt2
Mar 20 22:37:38.226 INF ✅ Migration succeeded source=baikal/baikal-config-tmp dest=baikal/baikal-config attempt_id=dbbdb strategy=mnt2

Jetzt skaliert man das Deployment wieder auf 1

tamaymueller@terra:~/migrate$ kubectl -n baikal scale deployment baikal --replicas 1
deployment.apps/baikal scaled
tamaymueller@terra:~/migrate$ kubectl -n baikal get deployments.apps
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
baikal   1/1     1            1           26d

Dann überprüft man, ob die Nutzdaten vorhanden sind, und falls ja, kann man die temporären PVCs löschen.

tamaymueller@terra:~/migrate$ kubectl -n baikal delete pvc baikal-config-tmp baikal-specific-tmp
persistentvolumeclaim "baikal-config-tmp" deleted
persistentvolumeclaim "baikal-specific-tmp" deleted

tamaymueller@terra:~/migrate$ kubectl -n baikal get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
baikal-config     Bound    pvc-634ff52d-ca80-435b-905d-19c88429ef30   1Gi        RWO            piraeus-storage   <unset>                 3m43s
baikal-specific   Bound    pvc-45f57e7f-ab0d-4165-832e-65926d6cf4cb   200Mi      RWO            piraeus-storage   <unset>                 3m43s