ceph简介及部署安装

Ceph 分布式存储简介Ceph 简介Ceph 是一款开源、分布式、软件定义存储 。Ceph 具备极高的可用性、 扩展性和易用性 用于存储海量数据。Ceph 存储可部署在通用服务器上 这些服务器的CPU可以是x86架构 也可以是ARM架构。Ceph 支持在同一集群内既有x86主机又有ARM 主机。软件定义 software-defined storage SDSCeph访问方式-简介Ceph提供以下访问Ceph集群的方法Ceph原生API(librados)Ceph块设备(RBD、librbd)也称为 RADOS块设备(RBD)镜像Ceph对象网关(RADOSGW、librgw)Ceph文件系统(CephFS、libcephfs)Ceph存储后端组件-简介监控器(MON)维护集群状态映射。它们可帮助其他守护进程互相协调。对象存储设备(OSD)存储数据并处理数据复制、恢复和重新平衡。管理器(MGR)通过基于浏览器的仪表板和RESTAPI跟踪运行时指标并公开集群信息。元数据服务器(MDS)存储CephFS使用的元数据而非对象存储或块存储以便客户端能够高效运行 POSIX命令。Ceph 分布式存储 部署过程Ceph 集群环境说明部署方法cephadm操作系统Centos Stream 8最小化安装硬件配置2 cpu、4G memory、1个系统盘3个20G scsi数据盘准备虚拟机模板基于 CentOS-Stream-8-template 模板克隆出 ceph-template# 1 配置主机名解析[rootlocalhost ~]# cat /etc/hosts EOF###### ceph ######192.168.108.10 client.zhu.cloud client192.168.108.11 ceph1.zhu.cloud ceph1192.168.108.12 ceph2.zhu.cloud ceph2192.168.108.13 ceph3.zhu.cloud ceph3 EOF# 2 关闭 SELinux[rootlocalhost ~]# sed -ri s/^SELINUX.*/SELINUXdisabled/g/etc/selinux/config# 3 关闭防火墙[rootlocalhost ~]# systemctl disable firewalld --now# 4 配置yum仓库[rootlocalhost ~]# cat EOF /etc/yum.repos.d/ceph.repo[Ceph]nameCephbaseurlhttps://mirrors.aliyun.com/centos-vault/8-stream/storage/x86_64/cephpacificenabled1gpgcheck0EOF# 5 安装基础软件包[rootlocalhost ~]# dnf install -y bash-completion vim lrzsz unzip rsync sshpass tar# 6 配置时间同步[rootlocalhost ~]# dnf install -y chrony[rootlocalhost ~]# systemctl enable chronyd --now# 7 安装 cephadm[rootlocalhost ~]# dnf install -y cephadm[rootlocalhost ~]# cephadm --helpusage: cephadm[-h][--image IMAGE][--docker][--data-dir DATA_DIR][--log-dir LOG_DIR][--logrotate-dir LOGROTATE_DIR][--sysctl-dir SYSCTL_DIR][--unit-dir UNIT_DIR][--verbose][--timeout TIMEOUT][--retry RETRY][--env ENV][--no-container-init]{version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rmcluster,run,shell,enter,ceph-volume,zap-osds,unit,logs,bootstrap,deploy,checkhost,prepare-host,add-repo,rm-repo,install,registry-login,gatherfacts,exporter,host-maintenance,disk-rescan}... Bootstrap Ceph daemons with systemd and containers. positional arguments:{version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rmcluster,run,shell,enter,ceph-volume,zap-osds,unit,logs,bootstrap,deploy,checkhost,prepare-host,add-repo,rm-repo,install,registry-login,gatherfacts,exporter,host-maintenance,disk-rescan}sub-command version get ceph version from container pull pull the default container image inspect-image inspectlocalcontainer imagelslist daemon instances on thishost......# 安装 cephadm 的时候会自动安装官方推荐的容器引擎 podman[rootlocalhost ~]# rpm -q podmanpodman-4.9.4-0.1.module_el89713d3df00d.x86_64# 8 提前下载镜像[rootlocalhost ~]#podmanpull quay.io/ceph/ceph:v16podmanpull quay.io/ceph/ceph-grafana:8.3.5podmanpull quay.io/prometheus/node-exporter:v1.3.1podmanpull quay.io/prometheus/alertmanager:v0.23.0podmanpull quay.io/prometheus/prometheus:v2.33.4# 准备配置主机脚本[rootlocalhost ~]# cat /usr/local/bin/sethost EOF#/bin/bashhostnamectl set-hostname ceph$1.zhu.cloud nmcli connection modify ens160 ipv4.method manual ipv4.addresses192.168.108.1$1/24 ipv4.gateway192.168.108.2 ipv4.dns192.168.108.2 init0EOF##添加可执行权限[rootlocalhost ~]# chmod x /usr/local/bin/sethost关机虚拟机并打快照准备集群节点克隆出其他虚拟机并配置主机名和IP地址。##之前模板配置好脚本直接运行脚本配置#ceph1到ceph3按以下修改[rootlocalhost ~]# sethost 1 #ceph1用1Ceph 集群初始化[rootceph1 ~]# cephadm bootstrap --mon-ip 192.168.108.11 --allow-fqdn-hostname --initial-dashboard-user admin --initial-dashboard-password xxxxx --dashboard-password-noupdate##初始化时会输出内容网站用户名密码会显示在输出内容里面–mon-ip 192.168.108.11指定 monitor ip。–allow-fqdn-hostname指定允许使用长名称。当主机名是长名称时初始化时必须使用该参数。–initial-dashboard-user admin指定 Web UI 登录的管理员账户。–initial-dashboard-password 指定 Web UI 登录的管理员账户对应密码。–dashboard-password-noupdate指定不要更新 Web UI 登录密码。随后登陆网站添加节点添加节点过程Ceph采用共享秘钥进行身份验证 使用命令“ceph cephadm get-pub-key” 获取到主机接入集群时所需的ssh 公钥。获取到公钥后 使用该公钥实现对节点的免密ssh管理。使用命令“ceph orch host add” 添加主机。# 为了配置方便我们在ceph1上安装ceph客户端工具 ceph-common[rootceph1 ~]# dnf install -y ceph-common# 获取集群公钥[rootceph1 ~]# ceph cephadm get-pub-key ~/ceph.pub# 推送公钥到其他节点[rootceph1 ~]# ssh-copy-id -f -i ~/ceph.pub rootceph2.zhu.cloud[rootceph1 ~]# ssh-copy-id -f -i ~/ceph.pub rootceph3.zhu.cloud# 添加节点[rootceph1 ~]# ceph orch host add ceph2.zhu.cloudAddedhostceph2.zhu.cloudwith addr192.168.108.12[rootceph1 ~]# ceph orch host add ceph3.zhu.cloudAddedhostceph3.zhu.cloudwith addr192.168.108.13[rootceph1 ~]# ceph orch host lsHOST ADDR LABELS STATUS ceph1.zhu.cloud192.168.108.11 _admin ceph2.zhu.cloud192.168.108.12 ceph3.zhu.cloud192.168.108.133hostsincluster# 等待自动部署服务到其他节点部署完成后效果如下[rootceph1 ~]# ceph orch lsNAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,90941/1 8m ago 9m count:1 crash3/3 8m ago 9m * grafana ?:30001/1 8m ago 9m count:1 mgr2/2 8m ago 9m count:2 mon3/5 8m ago 9m count:5 node-exporter ?:91003/3 8m ago 9m * prometheus ?:90951/1 8m ago 9m count:1# crash 3/3个# mgr 2/2个# mon 3/5个# node-exporter 3/3个部署 mon 和 mgr# 禁用 mon 和 mgr 服务的自动扩展功能[rootceph1 ~]# ceph orch apply mon --unmanagedtrue[rootceph1 ~]# ceph orch apply mgr --unmanagedtrue[rootceph1 ~]# ceph orch lsNAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,90941/1 56s ago 12m count:1 crash3/3 57s ago 12m * grafana ?:30001/1 56s ago 12m count:1 mgr2/2 57s ago 3sunmanagedmon3/5 57s ago 8sunmanagednode-exporter ?:91003/3 57s ago 12m * prometheus ?:90951/1 56s ago 12m count:1# mon 和 mgr 的 PLACEMENT 状态为 unmanaged# 配置主机标签ceph2 和 ceph3 添加标签“ _admin”[rootceph1 ~]# ceph orch host label add ceph2.zhu.cloud _adminAdded label _admin tohostceph2.zhu.cloud[rootceph1 ~]# ceph orch host label add ceph3.zhu.cloud _adminAdded label _admin tohostceph3.zhu.cloud[rootceph1 ~]# ceph orch host lsHOST ADDR LABELS STATUS ceph1.zhu.cloud192.168.108.11 _admin ceph2.zhu.cloud192.168.108.12 _admin ceph3.zhu.cloud192.168.108.13 _admin3hostsincluster# 将 mon 和 mgr 组件部署到具有_admin标签的节点上[rootceph1 ~]# ceph orch apply mon --placementlabel:_adminScheduled mon update...[rootceph1 ~]# ceph orch apply mgr --placementlabel:_adminScheduled mgr update...#观察现象[rootceph1 ~]# ceph orch ls | egrep mon|mgrmgr3/3 2m ago 14s label:_admin mon3/3 2m ago 28s label:_admin部署 OSD# 将所有主机上闲置的硬盘添加为 OSD[rootceph1 ~]# ceph orch apply osd --all-available-devicesScheduled osd.all-available-devices update...验证查看集群中部署的服务[rootceph1 ~]# ceph orch lsNAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,90941/1 42s ago 2h count:1 crash3/3 42s ago 2h * grafana ?:30001/1 42s ago 2h count:1 mgr3/3 42s ago 2h label:_admin mon3/3 42s ago 2h label:_admin node-exporter ?:91003/3 42s ago 2h * osd.all-available-devices942s ago 2h * prometheus ?:90951/1 42s ago 2h count:1查看集群状态[rootceph1 ~17:08:27]# ceph -scluster: id: bdee4f9e-5996-11f1-8dce-000c294cc7b5 health: HEALTH_OK##okservices: mon:3daemons, quorum ceph1.zhu.cloud,ceph2,ceph3(age 52m)mgr: ceph1.zhu.cloud.fvwqgp(active, since 52m), standbys: ceph3.nytnqr, ceph2.noelbi osd:9osds:9up(since 52m),9in(since 2h)data: pools:1pools,1pgs objects:0objects,0B usage:2.6GiB used,177GiB /180GiB avail pgs:1activeclean#HEALTH_OK表示健康状态良好#HEALTH_WARN表示集群存在告警需进行排查处理后可转为HEALTH_OK#HEALTH_ERR表示集群存在比较严重的错误需要立即处理查看集群 osd 结构[rootceph1 ~17:10:48]# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-10.17537root default-30.05846hostceph10hdd0.01949osd.0 up1.000001.000003hdd0.01949osd.3 up1.000001.000006hdd0.01949osd.6 up1.000001.00000-70.05846hostceph22hdd0.01949osd.2 up1.000001.000005hdd0.01949osd.5 up1.000001.000008hdd0.01949osd.8 up1.000001.00000-50.05846hostceph31hdd0.01949osd.1 up1.000001.000004hdd0.01949osd.4 up1.000001.000007hdd0.01949osd.7 up1.000001.00000查看集群组件集群中运行的主要组件mgrceph 管理程序monitorceph 监视器osdceph 对象存储进程rgwceph 对象存储网关其他组件crash崩溃数据收集模块prometheus监控组件grafana监控数据展示dashboardalertmanagerprometheus告警组件node_exporterprometheus节点数据收集组件