k8s的pvc扩容（ceph-rbd模式）

发表评论
4259 views

k8s的pvc扩容（ceph-rbd模式）

k8s中的pvc扩容(ceph-rbd)

前言

在k8s的1.11版本中，k8s默认集成了pv的扩容接口。通过编辑pvc来达到扩容的目的。需要开启feature gate和ExpandPersistentVolumes， PresistentVolumeClaimResize。在1.11版本之后这个特性默认开启。

目前支持的volume插件: AWS-EBS,GCE-PD,AZURE DISK, Azure File, Clusterfs, Cinder, Portworx, Ceph RBD。管理员通过allowVolumeExpansion字段值为true来开启volume扩展功能

# cat rbd-class.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ceph-rbd
provisioner: ceph.com/rbd
#provisioner: kubernetes.io/rbd
parameters:
  monitors: 10.0.7.52:6789
  pool: kube
  adminId: admin
  adminSecretNamespace: kube-system
  adminSecretName: ceph-admin-secret
  userId: kube
  userSecretNamespace: kube-system
  userSecretName: ceph-secret
  fsType: ext4
  imageFormat: "2"
  imageFeatures: layering
allowVolumeExpansion: true
reclaimPolicy: Retain

目前扩容的文件系统为XFS, EXT3, EXT4

开启功能

采用kubeadm安装的话，默认feture-gate都是开启的。如果是二进制安装的话，需要手动开启。

apiserver

KUBE_ADMISSION_CONTROL="--admission-control=PersistentVolumeClaimResize,Initializers,
 NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,
  NodeRestriction,ResourceQuota"
  --feature-gates=ExpandPersistentVolumes=true

controller

--feature-gates=ExpandPersistentVolumes=true

官方文档测试(未成功)

创建pvc

# cat pvc-test.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ceph-rbd
  resources:
    requests:
      storage: 2Gi

创建pod

pod创建

# cat pod-test.yaml  | grep -v "^$"
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ceph-rbd-pvc-test
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ceph-rbd-pvc-test
    spec:
      containers:
      - name: ceph-rbd-pvc-test
        image: busybox:latest
        command:
        - sleep
        - "360000"
        volumeMounts:
        - name: ceph-rbd-test
          mountPath: /data/ceph-rbd
          readOnly: false 
      volumes:
      - name: ceph-rbd-test 
        persistentVolumeClaim:
          claimName: test-pvc

在pod中实验

# kubectl get pod 
NAME                                 READY   STATUS    RESTARTS   AGE
ceph-rbd-pvc-test-59d4869cfb-g9kb8   1/1     Running   0          70s
rbd-provisioner-77cc657d45-l6hfw     1/1     Running   95         55d

[root@k8s-ceshi-01.novalocal 17:48 ~/k8s/ceph/rbd/deploy/rbac/test]
# kubectl exec -it ceph-rbd-pvc-test-59d4869cfb-g9kb8 /bin/sh
/ # df -hT
Filesystem           Type            Size      Used Available Use% Mounted on
overlay              overlay        40.0G     30.8G      9.2G  77% /
tmpfs                tmpfs          64.0M         0     64.0M   0% /dev
tmpfs                tmpfs           5.8G         0      5.8G   0% /sys/fs/cgroup
/dev/vda1            xfs            40.0G     30.8G      9.2G  77% /dev/termination-log
/dev/rbd0            ext4            2.9G      9.0M      2.9G   0% /data/ceph-rbd
/dev/vda1            xfs            40.0G     30.8G      9.2G  77% /etc/resolv.conf
/dev/vda1            xfs            40.0G     30.8G      9.2G  77% /etc/hostname
/dev/vda1            xfs            40.0G     30.8G      9.2G  77% /etc/hosts
shm                  tmpfs          64.0M         0     64.0M   0% /dev/shm
tmpfs                tmpfs           5.8G     12.0K      5.8G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                tmpfs           5.8G         0      5.8G   0% /proc/acpi
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/kcore
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/keys
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/timer_list
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/timer_stats
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                tmpfs           5.8G         0      5.8G   0% /proc/scsi
tmpfs                tmpfs           5.8G         0      5.8G   0% /sys/firmware
/ # cd /data/ceph-rbd/
/data/ceph-rbd # ls
lost+found
/data/ceph-rbd # vi abc
/data/ceph-rbd # echo "1234" >> 123
/data/ceph-rbd # ls -lt
total 24
-rw-r--r--    1 root     root             5 Sep 18 09:49 123
-rw-r--r--    1 root     root            13 Sep 18 09:49 abc
drwx------    2 root     root         16384 Sep 18 09:47 lost+found

pvc扩容

修改pvc的容量

# cat pvc-test.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ceph-rbd
  resources:
    requests:
      storage: 5Gi

查看pvc状态,可以看到状态在resizing状态

# kubectl get pvc test-pvc  -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"test-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"ceph-rbd"}}
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: ceph.com/rbd
  creationTimestamp: "2019-09-18T09:47:22Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: test-pvc
  namespace: default
  resourceVersion: "43700029"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/test-pvc
  uid: 502a6637-d9f9-11e9-a3ea-fa163e8116b5
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: ceph-rbd
  volumeMode: Filesystem
  volumeName: pvc-502a6637-d9f9-11e9-a3ea-fa163e8116b5
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 3Gi
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-09-18T09:51:14Z"
    status: "True"
    type: Resizing
  phase: Bound

过一会儿等待文件扩容完成

# kubectl get pvc test-pvc -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"test-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Gi"}},"storageClassName":"ceph-rbd"}}
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: ceph.com/rbd
  creationTimestamp: "2019-09-23T03:57:09Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: test-pvc
  namespace: default
  resourceVersion: "44746226"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/test-pvc
  uid: 37512c5c-ddb6-11e9-a3ea-fa163e8116b5
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: ceph-rbd
  volumeMode: Filesystem
  volumeName: pvc-37512c5c-ddb6-11e9-a3ea-fa163e8116b5
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 3Gi
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-09-23T08:38:44Z"
    message: Waiting for user to (re-)start a pod to finish file system resize of
      volume on node.
    status: "True"
    type: FileSystemResizePending
  phase: Bound

然后重启pod既可以达到扩容pod的目的。

在线扩容

需要开启ExpandInUsePersistentVolumes=true。同样支持GCE-PD, AWS-EBS, Cinder, Ceph RBD。目前还处于测试阶段。该方案扩容后不需要重启pod。

通过rbd命令扩容

该方式直接通过rbd命令直接操作pod的rbd镜像，来达到扩容的目的。

  • 获取pv的镜像
    # kubectl get pv pvc-d4aeea1a-ddde-11e9-a3ea-fa163e8116b5 -o yaml | grep image
  image: kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203
  • 查看pv的rbd信息 ```

[root@k8s-ceshi-01.novalocal 16:46 ~/k8s/ceph/rbd/deploy/rbac]

rbd list -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube kubernetes-dynamic-pvc-50444df4-d9f9-11e9-94f6-12f1cfd47f5e

kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203

[root@k8s-ceshi-01.novalocal 16:54 ~/k8s/ceph/rbd/deploy/rbac]

rbd info kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube

rbd image 'kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203': size 3 GiB in 768 objects order 22 (4 MiB objects) id: acb36b8b4567 block_name_prefix: rbd_data.acb36b8b4567 format: 2 features: layering op_features: flags: create_timestamp: Mon Sep 23 16:47:53 2019

- 扩容rbd

[root@k8s-ceshi-01.novalocal 16:25 ~/k8s/ceph/rbd/deploy/rbac]

rbd resize --image kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 --size 51200 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube

Resizing image: 100% complete...done.

- 在挂载节点上面扩容

[root@k8s-ceshi-01.novalocal 16:54 ~/k8s/ceph/rbd/deploy/rbac/test]

kubectl get pod -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ceph-rbd-pvc-test-59d4869cfb-kpl25 1/1 Running 0 7m39s 172.88.18.87 k8s-ceshi-05.novalocal

[root@k8s-ceshi-05.novalocal 16:30 /var/lib/kubelet]

df -hT | grep kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203

/dev/rbd0 ext4 2.9G 9.0M 2.9G 1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-image-kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203

[root@k8s-ceshi-05.novalocal 16:30 /var/lib/kubelet]

blockdev --getsize64 /dev/rbd0

53687091200

[root@k8s-ceshi-05.novalocal 16:30 /var/lib/kubelet]

resize2fs /dev/rbd0

resize2fs 1.42.9 (28-Dec-2013) Filesystem at /dev/rbd0 is mounted on /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-image-kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203; on-line resizing required old_desc_blocks = 5, new_desc_blocks = 7 The filesystem on /dev/rbd0 is now 13107200 blocks long.

如果是xfs的话，采用xfs_growfs -d /mnt来进行扩容

- 查看pod状态

[root@k8s-ceshi-01.novalocal 17:01 ~/k8s/ceph/rbd/deploy/rbac/test]

kubectl exec -it ceph-rbd-pvc-test-59d4869cfb-kpl25 /bin/sh

/ # df -hT Filesystem Type Size Used Available Use% Mounted on overlay overlay 40.0G 31.0G 9.0G 78% / tmpfs tmpfs 64.0M 0 64.0M 0% /dev tmpfs tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup /dev/rbd0 ext4 4.9G 12.0M 4.8G 0% /data/ceph-rbd

- 修改pv
修改pv的状态为5G

[root@k8s-ceshi-01.novalocal 17:06 ~/k8s/ceph/rbd/deploy/rbac/test]

kubectl get pv pvc-d4aeea1a-ddde-11e9-a3ea-fa163e8116b5 -o yaml

apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/provisioned-by: ceph.com/rbd rbdProvisionerIdentity: ceph.com/rbd creationTimestamp: "2019-09-23T08:47:53Z" finalizers:

  • kubernetes.io/pv-protection name: pvc-d4aeea1a-ddde-11e9-a3ea-fa163e8116b5 resourceVersion: "44749887" selfLink: /api/v1/persistentvolumes/pvc-d4aeea1a-ddde-11e9-a3ea-fa163e8116b5 uid: d4d41995-ddde-11e9-a3ea-fa163e8116b5 spec: accessModes:
  • ReadWriteOnce capacity: storage: 5Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: test-pvc namespace: default resourceVersion: "44747590" uid: d4aeea1a-ddde-11e9-a3ea-fa163e8116b5 persistentVolumeReclaimPolicy: Retain rbd: fsType: ext4 image: kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203 keyring: /etc/ceph/keyring monitors:
    • 10.0.7.52:6789 pool: kube secretRef: name: ceph-secret namespace: kube-system user: kube storageClassName: ceph-rbd volumeMode: Filesystem status: phase: Bound ```
    • 查看pvc状态 ``` # kubectl get pvc test-pvc -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"test-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Gi"}},"storageClassName":"ceph-rbd"}} pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: ceph.com/rbd creationTimestamp: "2019-09-23T03:57:09Z" finalizers:
  • kubernetes.io/pvc-protection name: test-pvc namespace: default resourceVersion: "44746226" selfLink: /api/v1/namespaces/default/persistentvolumeclaims/test-pvc uid: 37512c5c-ddb6-11e9-a3ea-fa163e8116b5 spec: accessModes:
  • ReadWriteOnce resources: requests: storage: 5Gi storageClassName: ceph-rbd volumeMode: Filesystem volumeName: pvc-37512c5c-ddb6-11e9-a3ea-fa163e8116b5 status: accessModes:
  • ReadWriteOnce capacity: storage: 3Gi conditions:
  • lastProbeTime: null lastTransitionTime: "2019-09-23T08:38:44Z" message: Waiting for user to (re-)start a pod to finish file system resize of volume on node. status: "True" type: FileSystemResizePending phase: Bound
    - 重启pod
    # kubectl delete pod ceph-rbd-pvc-test-59d4869cfb-cfgzw pod "ceph-rbd-pvc-test-59d4869cfb-cfgzw" deleted
    - 查看状态
    # kubectl get pvc,pv NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-pvc Bound pvc-37512c5c-ddb6-11e9-a3ea-fa163e8116b5 50Gi RWO ceph-rbd 4h45m

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-37512c5c-ddb6-11e9-a3ea-fa163e8116b5 50Gi RWO Retain Bound default/test-pvc ceph-rbd 4h45m

## 扩展

cbd命令说明

- 查看info信息

rbd info kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube --format=json

{"name":"kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203","id":"ac726b8b4567","size":3221225472,"objects":768,"order":22,"object_size":4194304,"block_name_prefix":"rbd_data.ac726b8b4567","format":2,"features":["layering"],"op_features":[],"flags":[],"create_timestamp":"Mon Sep 23 11:57:12 2019"}

- 删除image

[root@k8s-ceshi-01.novalocal 16:46 ~/k8s/ceph/rbd/deploy/rbac]

rbd remove kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube

Removing image: 100% complete...done.

- 列出images

rbd list -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube kubernetes-dynamic-pvc-50444df4-d9f9-11e9-94f6-12f1cfd47f5e

- 扩容images

rbd resize --image kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 --size 51200 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube

Resizing image: 100% complete...done.

## 排错

注意：该错误排查，仅仅应用于官方文档的扩容方式

### rbd文件找不到

报错

Warning VolumeResizeFailed 3m (x75 over 4h) volume_expand Error expanding volume "default/resize" of plugin kubernetes.io/rbd : rbd info failed, error: executable file not found in $PATH

分析：  
在controller中找不到rbd这个命令。


解决办法：  
我们要在controller中安装rbd-common这个包。如果fs的话，安装ceph-fs-common


### 执行输出报错

E0923 07:36:36.319172 1 nestedpendingoperations.go:267] Operation for "\"37512c5c-ddb6-11e9-a3ea-fa163e8116b5\"" failed. No retries permitted until 2019-09-23 07:38:38.319000883 +0000 UTC m=+156611.810101860 (durationBeforeRetry 2m2s). Error: "error expanding volume \"default/test-pvc\" of plugin \"kubernetes.io/rbd\": rbd info failed, error: parse rbd info output failed: 2019-09-23 07:36:36.239778 7f913ad24100 -1 did not load config file, using default settings.\n{\"name\":\"kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203\",\"size\":3221225472,\"objects\":768,\"order\":22,\"object_size\":4194304,\"block_name_prefix\":\"rbd_data.ac726b8b4567\",\"format\":2,\"features\":[\"layering\"],\"flags\":[]}, invalid character '-' after top-level value"

分析：
rbd中解析输出的时候报错。
正常情况

[root@k8s-ceshi-01.novalocal 15:10 ~/k8s/ceph/rbd/deploy/rbac]

rbd info kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube --format=json

{"name":"kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203","id":"ac726b8b4567","size":3221225472,"objects":768,"order":22,"object_size":4194304,"block_name_prefix":"rbd_data.ac726b8b4567","format":2,"features":["layering"],"op_features":[],"flags":[],"create_timestamp":"Mon Sep 23 11:57:12 2019"}

而异常的情况

[root@k8s-ceshi-01.novalocal 15:10 ~/k8s/ceph/rbd/deploy/rbac]

rbd info kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube --format=json

did not load config file, using default settings. 2019-09-23 15:43:47.500 7f6d1861e840 -1 Errors while parsing config file! 2019-09-23 15:43:47.500 7f6d1861e840 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.500 7f6d1861e840 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.500 7f6d1861e840 -1 parse_file: cannot open ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.501 7f6d1861e840 -1 Errors while parsing config file! 2019-09-23 15:43:47.501 7f6d1861e840 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.501 7f6d1861e840 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.501 7f6d1861e840 -1 parse_file: cannot open ceph.conf: (2) No such file or directory {"name":"kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203","id":"ac726b8b4567","size":3221225472,"objects":768,"order":22,"object_size":4194304,"block_name_prefix":"rbd_data.ac726b8b4567","format":2,"features":["layering"],"op_features":[],"flags":[],"create_timestamp":"Mon Sep 23 11:57:12 2019"} ``` 然后在解析的时候报错了。导致rbd的info命令执行失败。