简介
Velero 是一款可以安全的备份、恢复和迁移 Kubernetes 集群资源和持久卷等资源的备份恢复软件。
Velero 实现的 kubernetes 资源备份能力，可以轻松实现 Kubernetes 集群的数据备份和恢复、复制 kubernetes 集群资源到其他 kubernetes 集群或者快速复制生产环境到测试环境等功能，这种备份就类似于把资源的 yaml 文件进行整体备份，从而保障资源的完整性。
Velero 对存储的支持较好，可以支持很多种存储资源，比如 AWS S3、Azure Blob、Google Cloud Storage、Alibaba Cloud OSS、Swift、MinIO 等等.
流程图
注意：本文档采用 MinIO 作为后端存储。
Docker 部署 MinIO
因为我们要保证存储和集群隔离，这样能保证在集群出问题的时候，不影响备份，所以就将 MinIO 部署在其他服务器上，与 Kubernetes 环境隔离。避免因意外情况宕机的时候影响更多。
mkdir -pv /data/{minio,config}
docker pull minio/minio
docker pull minio/mc:latest
docker run -d -p 9000:9000 -p 9090:9090 --name=minio --restart=always -e "MINIO_ROOT_USER=admin" -e "MINIO_ROOT_PASSWORD=admin123456" -v /data/minio:/data -v /data/config:/root/.minio minio/minio server /data --console-address ":9090" --address ":9000"
# 对minio 授权
docker run -it --entrypoint=/bin/sh -v /data/config:/root/.minio minio/mc
mc config host add velero http://127.0.0.1:9000 admin admin123456 --api S3v4
mc mb -p velero/velero
注意，mc 使用的是 api 端口，并不是 console 界面端口，所以 mc 在授权的时候需要使用 API 端口。
然后浏览器通过配置的账号和 password 即可登录：
安装 velero 在 k8s master 上
下载 velero：
https://github.com/vmware-tanzu/velero/releases/download/v1.9.0/velero-v1.9.0-linux-amd64.tar.gz。
1、创建密钥并安装velero
cat > credentials-velero <<EOF
[default]
aws_access_key_id = admin
aws_secret_access_key = admin123456
EOF
# 安装velero，配置自动补全
cp velero /usr/local/bin/
echo 'source <(velero completion bash)' >> /etc/profile && source /etc/profile
2、K8s集群安装velero
# 需要提前下载镜像 velero/velero-plugin-for-aws:v1.2.1 和velero/velero:v1.9.0
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.2.1 \
--bucket velero \
--secret-file ./credentials-velero \
--use-volume-snapshots=false \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.0.2.196:9000
安装时产生的日志。
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource client
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
验证是否安装完毕。
[root@master ~]# kubectl get pod -n velero
NAME READY STATUS RESTARTS AGE
velero-cf79784ff-zdfp9 1/1 Running 0 47s
[root@master ~]# kubectl api-versions | grep velero
velero.io/v1
3、velero 命令
velero create backup NAME [flags]
# 剔除 namespace
--exclude-namespaces stringArray namespaces to exclude from the backup
# 剔除资源类型
--exclude-resources stringArray resources to exclude from the backup, formatted as resource.group, such as storageclasses.storage.k8s.io
# 包含集群资源类型
--include-cluster-resources optionalBool[=true] include cluster-scoped resources in the backup
# 包含 namespace
--include-namespaces stringArray namespaces to include in the backup (use '*' for all namespaces) (default *)
# 包含 namespace 资源类型
--include-resources stringArray resources to include in the backup, formatted as resource.group, such as storageclasses.storage.k8s.io (use '*' for all resources)
# 给这个备份加上标签
--labels mapStringString labels to apply to the backup
-o, --output string Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. 'table' is not valid for the install command.
# 对指定标签的资源进行备份
-l, --selector labelSelector only back up resources matching this label selector (default <none>)
# 对 PV 创建快照
--snapshot-volumes optionalBool[=true] take snapshots of PersistentVolumes as part of the backup
# 指定备份的位置
--storage-location string location in which to store the backup
# 备份数据多久删掉
--ttl duration how long before the backup can be garbage collected (default 720h0m0s)
# 指定快照的位置，也就是哪一个公有云驱动
--volume-snapshot-locations strings list of locations (at most one per provider) where volume snapshots should be stored
4、测试
velero 在安装包中有一个测试demo，我们可以利用测试demo来进行测试验证。
[root@master ~]# cd velero-v1.9.0-linux-amd64/examples/nginx-app/
[root@master nginx-app]# ls
base.yaml README.md with-pv.yaml
# 注意修改 with-pv 中的 storageclass
storageClassName: longhorn
# 修改 svc 类型为 ClusterIP
type: ClusterIP
[root@master nginx-app]# kubectl apply -f with-pv.yaml
namespace/nginx-example created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-deployment created
service/my-nginx created
[root@master nginx-app]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx ClusterIP 10.108.189.178 <none> 80/TCP 79s
[root@master nginx-app]# kubectl get all -n nginx-example
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-6779884c68-4nv2s 2/2 Running 0 2m16s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-nginx ClusterIP 10.108.189.178 <none> 80/TCP 2m16s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 1/1 1 1 2m16s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-6779884c68 1 1 1 2m16s
5、备份
[root@master nginx-app]# velero backup create nginx-backup --include-namespaces nginx-example
Backup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.
[root@master nginx-app]# velero backup describe nginx-backup
Name: nginx-backup
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.23.5
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=23
Phase: Failed (run `velero backup logs nginx-backup` for more information)
Errors: 0
Warnings: 0
Namespaces:
Included: nginx-example
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2022-06-30 17:44:49 +0800 CST
Completed: 2022-06-30 17:44:49 +0800 CST
Expiration: 2022-07-30 17:44:49 +0800 CST
Velero-Native Snapshots: <none included>
[root@master nginx-app]# velero backup logs nginx-backup
An error occurred: gzip: invalid header
登录 minio 查看，备份记录。
6、恢复测试
删除 nginx-app：
[root@master velero-v1.9.0-linux-amd64]# kubectl delete -f examples/nginx-app/base.yaml恢复 nginx：
[root@master velero-v1.9.0-linux-amd64]# velero restore create --from-backup nginx-backup --wait
Restore request "nginx-backup-20220630192521" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe nginx-backup-20220630192521` and `velero restore logs nginx-backup-20220630192521`.
[root@master velero-v1.9.0-linux-amd64]# kubectl get pods -n nginx-example
NAME READY STATUS RESTARTS AGE
nginx-deployment-6779884c68-4nv2s 2/2 Running 0 27s
恢复会新建一个 restores。
目录注意：velero restore 恢复不会覆盖已有的资源，只恢复当前集群中不存在的资源。已有的资源不会回滚到之前的版本，如需要回滚，需在restore之前提前删除现有的资源。
备份所有命名空间：
velero backup logs k8s-all
7、定时备份
比较推荐做定时全量备份，这个能保证我们是一直全量最新备份的状态。
# 每天12点备份一次
velero create schedule k8s-all --schedule="0 0 * * *"
查看备份：
velero get backup #备份查看
velero get schedule #查看定时备份
velero get restore #查看已有的恢复
velero get plugins #查看插件
