【Prometheus监控Linux系统】

提示本文原创作品良心制作干货为主简洁清晰一看就会文章目录前言一、环境介绍二、安装node_exporter2.1 安装docker2.2 安装docker-compose2.3 安装node_exporter三、修改prometheus配置3.1 修改prometheus.yml3.2 设置触发器三、查看监控数据前言在复杂的运维体系中Linux服务的稳定性与性能直接关系到业务可用性。Prometheus作为云原生时代的监控利器以其多维数据模型和强大的PromQL查询语言正成为守护Linux服务的最佳选择。本文将带你快速落地Prometheus监控方案构建可观测性基线一、环境介绍主机名ip地址服务备注prometheus192.168.13.141docker、docker-compose、prometheus、alertmanager、node-exporter、grafana监控端已安装ubuntu192.168.13.142docker、docker-compose、node-exporter被监控端未安装关于监控端的服务我已经安装好了prometheus有两种安装方式二进制安装和docker安装本次实验使用的容器安装的后续被监控端我也统一使用容器部署大家可以自行选择关于监控端的服务如何安装这里不在赘述有不懂的同学可以查看此篇文章Prometheus二进制安装https://blog.csdn.net/m0_63756214/article/details/161196428?spm1001.2014.3001.5501Prometheus容器安装https://blog.csdn.net/m0_63756214/article/details/161225636?spm1001.2014.3001.5501二、安装node_exporter2.1 安装docker我的操作系统是ubuntu的centos安装docker可以参考下面这篇文章的1.1小节https://blog.csdn.net/m0_63756214/article/details/158650241?spm1001.2014.3001.5501被监控端ubuntu主机安装docker## 1. 安装必要的一些系统工具rootubuntu:~# apt-get updaterootubuntu:~# apt-get install ca-certificates curl gnupg## 2. 信任docker的gpg公钥rootubuntu:~# install -m 0755 -d /etc/apt/keyringsrootubuntu:~# curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpgrootubuntu:~# chmod ar /etc/apt/keyrings/docker.gpg## 3. 写入软件源信息rootubuntu:~# echo \deb[arch$(dpkg--print-architecture) signed-by/etc/apt/keyrings/docker.gpg]https://mirrors.aliyun.com/docker-ce/linux/ubuntu \ $(. /etc/os-releaseecho $VERSION_CODENAME) stable|\ sudo tee /etc/apt/sources.list.d/docker.list/dev/null## 4. 安装dockerrootubuntu:~# apt-get updaterootubuntu:~# apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-pluginrootubuntu:~# docker --versionDocker version 24.0.2,build cb74dfc rootubuntu:~# systemctl start docker systemctl enable docker## 5. 配置docker镜像加速器rootubuntu:~# vim /etc/docker/daemon.json{registry-mirrors:[https://a88uijg4.mirror.aliyuncs.com,https://docker.lmirror.top,https://docker.m.daocloud.io,https://hub.uuuadc.top,https://docker.anyhub.us.kg,https://dockerhub.jobcher.com,https://dockerhub.icu,https://docker.ckyl.me,https://docker.awsl9527.cn,https://docker.laoex.link]}rootubuntu:~# systemctl daemon-reloadrootubuntu:~# systemctl restart docker2.2 安装docker-composeGitHub安装地址https://github.com/docker/compose/releases下拉选择适合自己系统的版本我选的是v2.38.0被监控端ubuntu主机安装docker-compose# 安装docker-composerootubuntu:~# lsdocker-compose-linux-x86_64 rootubuntu:~# mv docker-compose-linux-x86_64 /usr/bin/ #将包移到/usr/binrootubuntu:~# mv /usr/bin/docker-compose-linux-x86_64 /usr/bin/docker-compose #改名成docker-composerootubuntu:~# chmod x /usr/bin/docker-composerootubuntu:~# docker-compose version #查看docker-compose版本Docker Compose version v2.38.02.3 安装node_exporter被监控端ubuntu主机安装node_exporterrootubuntu:~# mkdir -p /opt/prometheusrootubuntu:~# cd /opt/prometheusrootubuntu:/opt/prometheus# vim docker-compose.yamlversion:3.8services:node_exporter:image:prom/node-exporter:v1.11.1container_name:node-exporterrestart:alwaysnetwork_mode:hostvolumes:-/proc:/host/proc:ro-/sys:/host/sys:ro-/:/rootfs:rocommand:---web.listen-address:9100---path.procfs/host/proc---path.sysfs/host/sys---path.rootfs/rootfs---collector.filesystem.ignored-mount-points^/(sys|proc|dev|host|etc|rootfs/var/lib/docker)($$|/)rootubuntu:/opt/prometheus# docker-compose up -d三、修改prometheus配置3.1 修改prometheus.yml监控端prometheus主机修改prometheus.yml文件rootprometheus:~# vim /opt/prometheus/prometheus/prometheus.ymlscrape_configs:# 增加一个名为node-exporter-linux的job-job_name:node-exporter-linuxscrape_interval:15sstatic_configs:# 填写linux的地址-targets:[192.168.13.142:9100]labels:instance:Linux服务器 rootprometheus:~# curl -X POST http://localhost:9090/-/reload浏览器访问prometheus ip:90090可以看到prometheus的实例多了一个node-exporter-linux3.2 设置触发器监控端prometheus主机修改alert.yml文件rootprometheus:~# vim /opt/prometheus/prometheus/alert.ymlgroups:-name:Prometheus alertrules:# 对任何实例超过30秒无法联系的情况发出警报-alert:服务告警expr:up 0for:30slabels:severity:criticalannotations:summary:服务异常,实例:{{ $labels.instance }}description:{{ $labels.job }} 服务已关闭-name:node-exporterrules:-alert:HostOutOfMemoryexpr:node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 10for:2mlabels:severity:warningannotations:summary:主机内存不足,实例:{{ $labels.instance }}description:内存可用率10%当前值{{ $value }}-alert:HostMemoryUnderMemoryPressureexpr:rate(node_vmstat_pgmajfault[1m])1000for:2mlabels:severity:warningannotations:summary:内存压力不足,实例:{{ $labels.instance }}description:节点内存压力大。 重大页面错误率高当前值为{{ $value }}-alert:HostUnusualNetworkThroughputInexpr:sum by (instance) (rate(node_network_receive_bytes_total[2m])) / 1024 / 1024100for:5mlabels:severity:warningannotations:summary:异常流入网络吞吐量,实例:{{ $labels.instance }}description:网络流入流量 100 MB/s当前值{{ $value }}-alert:HostUnusualNetworkThroughputOutexpr:sum by (instance) (rate(node_network_transmit_bytes_total[2m])) / 1024 / 1024100for:5mlabels:severity:warningannotations:summary:异常流出网络吞吐量实例:{{ $labels.instance }}description:网络流出流量 100 MB/s当前值为{{ $value }}-alert:HostUnusualDiskReadRateexpr:sum by (instance) (rate(node_disk_read_bytes_total[2m])) / 1024 / 102450for:5mlabels:severity:warningannotations:summary:异常磁盘读取,实例:{{ $labels.instance }}description:磁盘读取 50 MB/s当前值{{ $value }}-alert:HostUnusualDiskWriteRateexpr:sum by (instance) (rate(node_disk_written_bytes_total[2m])) / 1024 / 102450for:2mlabels:severity:warningannotations:summary:异常磁盘写入,实例:{{ $labels.instance }}description:磁盘写入 50 MB/s当前值{{ $value }}-alert:HostOutOfDiskSpaceexpr:(node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes 10 and ON (instance,device,mountpoint) node_filesystem_readonly 0for:2mlabels:severity:warningannotations:summary:磁盘空间不足告警,实例:{{ $labels.instance }}description:剩余磁盘空间 10% 当前值{{ $value }}-alert:HostDiskWillFillIn24Hoursexpr:(node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes 10 and ON (instance,device,mountpoint) predict_linear(node_filesystem_avail_bytes{fstype!~tmpfs}[1h],24 * 3600) 0 and ON (instance,device,mountpoint) node_filesystem_readonly 0for:2mlabels:severity:warningannotations:summary:磁盘空间将在24小时内耗尽,实例:{{ $labels.instance }}description:以当前写入速率预计磁盘空间将在 24 小时内耗尽当前值{{ $value }}-alert:HostOutOfInodesexpr:node_filesystem_files_free{mountpoint /}/ node_filesystem_files{mountpoint/}* 100 10 and ON (instance,device,mountpoint) node_filesystem_readonly{mountpoint/} 0for:2mlabels:severity:warningannotations:summary:磁盘Inodes不足,实例:{{ $labels.instance }}description:剩余磁盘 inodes 10%当前值 {{ $value }}-alert:HostUnusualDiskReadLatencyexpr:rate(node_disk_read_time_seconds_total[1m]) / rate(node_disk_reads_completed_total[1m])0.1 and rate(node_disk_reads_completed_total[1m])0for:2mlabels:severity:warningannotations:summary:异常磁盘读取延迟,实例:{{ $labels.instance }}description:磁盘读取延迟 100ms当前值{{ $value }}-alert:HostUnusualDiskWriteLatencyexpr:rate(node_disk_write_time_seconds_total[1m]) / rate(node_disk_writes_completed_total[1m])0.1 and rate(node_disk_writes_completed_total[1m])0for:2mlabels:severity:warningannotations:summary:异常磁盘写入延迟,实例:{{ $labels.instance }}description:磁盘写入延迟 100ms当前值{{ $value }}-alert:high_loadexpr:node_load14for:2mlabels:severity:pageannotations:summary:CPU1分钟负载过高,实例:{{ $labels.instance }}description:CPU1分钟负载4已经持续2分钟。当前值为{{ $value }}-alert:HostCpuIsUnderUtilizedexpr:100-(avg by(instance) (rate(node_cpu_seconds_total{modeidle}[2m])) * 100)80for:1mlabels:severity:warningannotations:summary:cpu负载高,实例:{{ $labels.instance }}description:cpu负载 80%当前值{{ $value }}-alert:HostCpuStealNoisyNeighborexpr:avg by(instance) (rate(node_cpu_seconds_total{modesteal}[5m])) * 10010for:0mlabels:severity:warningannotations:summary:CPU窃取率异常,实例:{{ $labels.instance }}description:CPU 窃取率 10%。 嘈杂的邻居正在扼杀 VM 性能或者 Spot 实例可能失去信用当前值{{ $value }}-alert:HostSwapIsFillingUpexpr:(1-(node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes)) * 10080for:2mlabels:severity:warningannotations:summary:磁盘swap空间使用率异常,实例:{{ $labels.instance }}description:磁盘swap空间使用率80%-alert:HostNetworkReceiveErrorsexpr:rate(node_network_receive_errs_total[2m]) / rate(node_network_receive_packets_total[2m])0.01for:2mlabels:severity:warningannotations:summary:异常网络接收错误,实例:{{ $labels.instance }}description:网卡{{ $labels.device }}在过去2分钟接收错误率大于0.01当前值:{{ $value }}-alert:HostNetworkTransmitErrorsexpr:rate(node_network_transmit_errs_total[2m]) / rate(node_network_transmit_packets_total[2m])0.01for:2mlabels:severity:warningannotations:summary:异常网络传输错误,实例:{{ $labels.instance }}description:网卡{{ $labels.device }}在过去2分钟传输错误率大于0.01当前值:{{ $value }}-alert:HostNetworkInterfaceSaturatedexpr:(rate(node_network_receive_bytes_total{device!~^tap.*}[1m]) rate(node_network_transmit_bytes_total{device!~^tap.*}[1m])) / node_network_speed_bytes{device!~^tap.*}0.8 10000for:1mlabels:severity:warningannotations:summary:异常网络接口饱和,实例:{{ $labels.instance }}description:网卡{{ $labels.device }}正在超载当前值{{ $value }}-alert:HostConntrackLimitexpr:node_nf_conntrack_entries / node_nf_conntrack_entries_limit0.8for:5mlabels:severity:warningannotations:summary:异常连接数,实例:{{ $labels.instance }}description:连接数过大当前连接数{{ $value }}-alert:HostClockSkewexpr:(node_timex_offset_seconds0.05 and deriv(node_timex_offset_seconds[5m]) 0) or (node_timex_offset_seconds -0.05 and deriv(node_timex_offset_seconds[5m]) 0)for:2mlabels:severity:warningannotations:summary:异常时钟偏差,实例:{{ $labels.instance }}description:检测到时钟偏差时钟不同步。值为{{ $value }}-alert:HostClockNotSynchronisingexpr:min_over_time(node_timex_sync_status[1m]) 0 and node_timex_maxerror_seconds 16for:2mlabels:severity:warningannotations:summary:时钟不同步,实例:{{ $labels.instance }}description:时钟不同步-alert:NodeFileDescriptorLimitexpr:node_filefd_allocated / node_filefd_maximum * 10080for:1mlabels:severity:warningannotations:summary:预计内核将很快耗尽文件描述符限制description:{{ $labels.instance }}已分配的文件描述符数超过了限制的80%当前值为{{ $value }}## 重新加载配置rootprometheus:~# curl -X POST http://localhost:9090/-/reload浏览器访问prometheus ip:90090可以查看prometheus的报警设置三、查看监控数据登录grafana没有导入过prometheus数据源需要先导入数据源然后再创建dashboard不会操作的同学可以查看本文的第四节https://blog.csdn.net/m0_63756214/article/details/161225636?spm1001.2014.3001.5501导入过的只需要直接选择相应的job查看即可注文中若有疏漏欢迎大家指正赐教。本文为100%原创转载请务必标注原创作者尊重劳动成果。求赞、求关注、求评论你的支持是我更新的最大动力评论区等你