关键词：
写在前面
- 学习
K8s监控涉及到 - 网上的教程大都不全或者有些旧，所以整理分享给小伙伴。
- 博文内容为
K8s集群通过helm方式创建kube-prometheus-stack监控平台教程 - 折腾了一晚上，搞定了，一开始一直用
prometheus-operator这个chart来装，报错各种找问题，后来才发现我的集群版本太高了，1.22的版本，而且prometheus-operator之后的版本改变了名字kube-prometheus-stack，旧的版本可能不兼容。
人生不尽美好，追求自我注定孤独，而这就是生命的意义 ——黑塞《彼得卡门青》
环境版本
我的K8s集群版本
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$kubectl get nodes
NAME STATUS ROLES AGE VERSION
vms81.liruilongs.github.io Ready control-plane,master 34d v1.22.2
vms82.liruilongs.github.io Ready <none> 34d v1.22.2
vms83.liruilongs.github.io Ready <none> 34d v1.22.2
hrlm版本
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$helm version
version.BuildInfoVersion:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"
prometheus-operator(旧名字)安装出现的问题
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$helm search repo prometheus-operator
NAME CHART VERSION APP VERSION DESCRIPTION
ali/prometheus-operator 8.7.0 0.35.0 Provides easy monitoring definitions for Kubern...
azure/prometheus-operator 9.3.2 0.38.1 DEPRECATED Provides easy monitoring definitions...
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$helm install liruilong ali/prometheus-operator
Error: failed to install CRD crds/crd-alertmanager.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$helm pull ali/prometheus-operator
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$
解决办法:新版本安装
直接下载kube-prometheus-stack(新)的chart包，通过命令安装：
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$ls
index.yaml kube-prometheus-stack-30.0.1.tgz liruilonghelm liruilonghelm-0.1.0.tgz mysql mysql-1.6.4.tgz
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
解压chart包kube-prometheus-stack-30.0.1.tgz
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$tar -zxf kube-prometheus-stack-30.0.1.tgz
创建新的命名空间
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$cd kube-prometheus-stack/
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$kubectl create ns monitoring
namespace/monitoring created
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$kubectl config set-context $(kubectl config current-context) --namespace=monitoring
Context "kubernetes-admin@kubernetes" modified.
进入文件夹，直接通过helm install liruilong .安装
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$ls
Chart.lock charts Chart.yaml CONTRIBUTING.md crds README.md templates values.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$helm install liruilong .
kube-prometheus-admission-create对应Pod的相关镜像下载不下来问题
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$kubectl get pods
NAME READY STATUS RESTARTS AGE
liruilong-kube-prometheus-admission-create--1-bn7x2 0/1 ImagePullBackOff 0 33s
查看pod详细信息，发现是谷歌的一个镜像国内无法下载
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$kubectl describe pod liruilong-kube-prometheus-admission-create--1-bn7x2
Name: liruilong-kube-prometheus-admission-create--1-bn7x2
Namespace: monitoring
Priority: 0
Node: vms83.liruilongs.github.io/192.168.26.83
Start Time: Sun, 16 Jan 2022 02:43:07 +0800
Labels: app=kube-prometheus-stack-admission-create
app.kubernetes.io/instance=liruilong
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/part-of=kube-prometheus-stack
app.kubernetes.io/version=30.0.1
chart=kube-prometheus-stack-30.0.1
controller-uid=2ce48cd2-a118-4e23-a27f-0228ef6c45e7
heritage=Helm
job-name=liruilong-kube-prometheus-admission-create
release=liruilong
Annotations: cni.projectcalico.org/podIP: 10.244.70.8/32
cni.projectcalico.org/podIPs: 10.244.70.8/32
Status: Pending
IP: 10.244.70.8
IPs:
IP: 10.244.70.8
Controlled By: Job/liruilong-kube-prometheus-admission-create
Containers:
create:
Container ID:
Image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068
Image ID:
Port: <none>
Host Port:
。。。。。。。。。。。。。。。。。。。。。。。。。。。
在dokcer仓库里找了一个类似的，通过 kubectl edit修改
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0 替换为 ： docker.io/liangjw/kube-webhook-certgen:v1.1.1
或者也可以修改配置文件从新install(记得要把sha注释掉)
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$ls
index.yaml kube-prometheus-stack kube-prometheus-stack-30.0.1.tgz liruilonghelm liruilonghelm-0.1.0.tgz mysql mysql-1.6.4.tgz
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$cd kube-prometheus-stack/
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$ls
Chart.lock charts Chart.yaml CONTRIBUTING.md crds README.md templates values.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$cat values.yaml | grep -A 3 -B 2 kube-webhook-certgen
enabled: true
image:
repository: docker.io/liangjw/kube-webhook-certgen
tag: v1.1.1
#sha: "f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068"
pullPolicy: IfNotPresent
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$helm del liruilong;helm install liruilong .
之后其他的相关pod正常创建中
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$kubectl get pods
NAME READY STATUS RESTARTS AGE
liruilong-grafana-5955564c75-zpbjq 0/3 ContainerCreating 0 27s
liruilong-kube-prometheus-operator-5cb699b469-fbkw5 0/1 ContainerCreating 0 27s
liruilong-kube-state-metrics-5dcf758c47-bbwt4 0/1 ContainerCreating 0 27s
liruilong-prometheus-node-exporter-rfsc5 0/1 ContainerCreating 0 28s
liruilong-prometheus-node-exporter-vm7s9 0/1 ContainerCreating 0 28s
liruilong-prometheus-node-exporter-z9j8b 0/1 ContainerCreating 0 28s
kube-state-metrics这个pod的镜像也没有拉取下来。应该也是相同的原因
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$kubectl get pods
NAME READY STATUS RESTARTS AGE
alertmanager-liruilong-kube-prometheus-alertmanager-0 2/2 Running 0 3m35s
liruilong-grafana-5955564c75-zpbjq 3/3 Running 0 4m46s
liruilong-kube-prometheus-operator-5cb699b469-fbkw5 1/1 Running 0 4m46s
liruilong-kube-state-metrics-5dcf758c47-bbwt4 0/1 ImagePullBackOff 0 4m46s
liruilong-prometheus-node-exporter-rfsc5 1/1 Running 0 4m47s
liruilong-prometheus-node-exporter-vm7s9 1/1 Running 0 4m47s
liruilong-prometheus-node-exporter-z9j8b 1/1 Running 0 4m47s
prometheus-liruilong-kube-prometheus-prometheus-0 2/2 Running 0 3m34s
同样 k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0 这个镜像没办法拉取
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$kubectl describe pod liruilong-kube-state-metrics-5dcf758c47-bbwt4
Name: liruilong-kube-state-metrics-5dcf758c47-bbwt4
Namespace: monitoring
Priority: 0
Node: vms82.liruilongs.github.io/192.168.26.82
Start Time: Sun, 16 Jan 2022 02:59:53 +0800
Labels: app.kubernetes.io/component=metrics
app.kubernetes.io/instance=liruilong
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kube-state-metrics
app.kubernetes.io/part-of=kube-state-metrics
app.kubernetes.io/version=2.3.0
helm.sh/chart=kube-state-metrics-4.3.0
pod-template-hash=5dcf758c47
release=liruilong
Annotations: cni.projectcalico.org/podIP: 10.244.171.153/32
cni.projectcalico.org/podIPs: 10.244.171.153/32
Status: Pending
IP: 10.244.171.153
IPs:
IP: 10.244.171.153
Controlled By: ReplicaSet/liruilong-kube-state-metrics-5dcf758c47
Containers:
kube-state-metrics:
Container ID:
Image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0
Image ID:
Port: 8080/TCP
。。。。。。。。。。。。。。。。。。。。。。
同样的，我们通过docker仓库找一下相同的,然后通过kubectl edit pod修改一下
k8s.gcr.io/kube-state-metrics/kube-state-metrics 替换为： docker.io/dyrnq/kube-state-metrics:v2.3.0
可以先在节点机上拉取一下
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ ansible node -m shell -a "docker pull dyrnq/kube-state-metrics:v2.3.0"
192.168.26.82 | CHANGED | rc=0 >>
v2.3.0: Pulling from dyrnq/kube-state-metrics
e8614d09b7be: Pulling fs layer
53ccb90bafd7: Pulling fs layer
e8614d09b7be: Verifying Checksum
e8614d09b7be: Download complete
e8614d09b7be: Pull complete
53ccb90bafd7: Verifying Checksum
53ccb90bafd7: Download complete
53ccb90bafd7: Pull complete
Digest: sha256:c9137505edaef138cc23479c73e46e9a3ef7ec6225b64789a03609c973b99030
Status: Downloaded newer image for dyrnq/kube-state-metrics:v2.3.0
docker.io/dyrnq/kube-state-metrics:v2.3.0
192.168.26.83 | CHANGED | rc=0 >>
v2.3.0: Pulling from dyrnq/kube-state-metrics
e8614d09b7be: Pulling fs layer
53ccb90bafd7: Pulling fs layer
e8614d09b7be: Verifying Checksum
e8614d09b7be: Download complete
e8614d09b7be: Pull complete
53ccb90bafd7: Verifying Checksum
53ccb90bafd7: Download complete
53ccb90bafd7: Pull complete
Digest: sha256:c9137505edaef138cc23479c73e46e9a3ef7ec6225b64789a03609c973b99030
Status: Downloaded newer image for dyrnq/kube-state-metrics:v2.3.0
docker.io/dyrnq/kube-state-metrics:v2.3.0
修改完之后，会发现所有的pod都创建成功
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$kubectl get pods
NAME READY STATUS RESTARTS AGE
alertmanager-liruilong-kube-prometheus-alertmanager-0 2/2 Running 0 61m
liruilong-grafana-5955564c75-zpbjq 3/3 Running 0 62m
liruilong-kube-prometheus-operator-5cb699b469-fbkw5 1/1 Running 0 62m
liruilong-kube-state-metrics-5dcf758c47-bbwt4 1/1 Running 7 (32m ago) 62m
liruilong-prometheus-node-exporter-rfsc5 1/1 Running 0 62m
liruilong-prometheus-node-exporter-vm7s9 1/1 Running 0 62m
liruilong-prometheus-node-exporter-z9j8b 1/1 Running 0 62m
prometheus-liruilong-kube-prometheus-prometheus-0 2/2 Running 0 61m
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$
然后我们需要修改liruilong-grafana SVC的类型为NodePort，这样，物理机就可以访问了
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]
└─$kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 33m
liruilong-grafana ClusterIP 10.99.220.121 <none> 80/TCP 34m
liruilong-kube-prometheus-alertmanager ClusterIP 10.97.193.228 <none> 9093/TCP 34m
liruilong-kube-prometheus-operator ClusterIP 10.101.106.93 <none> 443/TCP 34m
liruilong-kube-prometheus-prometheus ClusterIP 10.105.176.19 <none> 9090/TCP 34m
liruilong-kube-state-metrics ClusterIP 10.98.94.55 <none> 8080/TCP 34m
liruilong-prometheus-node-exporter ClusterIP 10.110.216.215 <none> 9100/TCP 34m
prometheus-operated ClusterIP None <none> 9090/TCP 33m
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]
└─$kubectl edit svc liruilong-grafana
service/liruilong-grafana edited
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]
└─$kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 35m
liruilong-grafana NodePort 10.99.220.121 <none> 80:30443/TCP 36m
liruilong-kube-prometheus-alertmanager ClusterIP 10.97.193.228 <none> 9093/TCP 36m
liruilong-kube-prometheus-operator ClusterIP 10.101.106.93 <none> 443/TCP 36m
liruilong-kube-prometheus-prometheus ClusterIP 10.105.176.19 <none> 9090/TCP 36m
liruilong-kube-state-metrics ClusterIP 10.98.94.55 <none> 8080/TCP 36m
liruilong-prometheus-node-exporter ClusterIP 10.110.216.215 <none> 9100/TCP 36m
prometheus-operated ClusterIP None <none> 9090/TCP 35m
| 物理机访问 |
|---|
 |
通过secrets解密获取用户名密码
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]
└─$kubectl get secrets | grep grafana
liruilong-grafana Opaque 3 38m
liruilong-grafana-test-token-q8z8j kubernetes.io/service-account-token 3 38m
liruilong-grafana-token-j94p8 kubernetes.io/service-account-token 3 38m
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]
└─$kubectl get secrets liruilong-grafana -o yaml
apiVersion: v1
data:
admin-password: cHJvbS1vcGVyYXRvcg==
admin-user: YWRtaW4=
ldap-toml: ""
kind: Secret
metadata:
annotations:
meta.helm.sh/release-name: liruilong
meta.helm.sh/release-namespace: monitoring
creationTimestamp: "2022-01-15T18:59:40Z"
labels:
app.kubernetes.io/instance: liruilong
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: grafana
app.kubernetes.io/version: 8.3.3
helm.sh/chart: grafana-6.20.5
name: liruilong-grafana
namespace: monitoring
resourceVersion: "1105663"
uid: c03ff5f3-deb5-458c-8583-787f41034469
type: Opaque
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]
└─$kubectl get secrets liruilong-grafana -o jsonpath='.data.admin-user'| base64 -d
adminbase64: 输入无效
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]
└─$kubectl get secrets liruilong-grafana -o jsonpath='.data.admin-password'| base64 -d
prom-operatorbase64: 输入无效
得到用户名密码：admin/prom-operator
| 正常登录，查看监控信息 |
|---|
 |
 |
生活加油 ^ _ ^
通过kubernetes监控探索应用架构，发现预期外的流量
...系统，是Kubernetes集群软件栈端到端可观测性的一体化解决方案，在Kubernetes监控中可以同时看到关联的所有层的观测数据。我们希望通过Kubernetes监控的一系列最佳实践，让大家能够使用Kubernetes监控解决Kubernetes环境 查看详情
通过kubernetes监控探索应用架构，发现预期外的流量
...系统，是Kubernetes集群软件栈端到端可观测性的一体化解决方案，在Kubernetes监控中可以同时看到关联的所有层的观测数据。我们希望通过Kubernetes监控的一系列最佳实践，让大家能够使用Kubernetes监控解决Kubernetes环境下 查看详情
使用 Prometheus 监控 Kubernetes 集群中的 Spring Boot 应用程序
】使用Prometheus监控Kubernetes集群中的SpringBoot应用程序【英文标题】：UsingPrometheustomonitorSpringBootApplicationsinKubernetesCluster【发布时间】：2020-05-0605:28:39【问题描述】：我在本地kubernetes集群中部署了SpringBoot驱动的微服务。微服务使... 查看详情
kubernetes集群监控解决方案kube-prometheus-stack(prometheus-operator)helm安装(代码片段)
写在前面学习K8s监控涉及到网上的教程大都不全或者有些旧，所以整理分享给小伙伴。博文内容为K8s集群通过helm方式创建kube-prometheus-stack监控平台教程折腾了一晚上，搞定了，一开始一直用prometheus-operator这个chart来... 查看详情
kubernetes集群监控解决方案kube-prometheus-stack(prometheus-operator)helm安装(代码片段)
写在前面学习K8s监控涉及到网上的教程大都不全或者有些旧，所以整理分享给小伙伴。博文内容为K8s集群通过helm方式创建kube-prometheus-stack监控平台教程折腾了一晚上，搞定了，一开始一直用prometheus-operator这个chart来... 查看详情
用weavescope监控集群-每天5分钟玩转docker容器技术（175）
创建Kubernetes集群并部署容器化应用只是第一步。一旦集群运行起来，我们需要确保一起正常，所有必要组件就位并各司其职，有足够的资源满足应用的需求。Kubernetes是一个复杂系统，运维团队需要有一套工具帮助他们获知集群... 查看详情
kubernetes集群搭建zabbix监控平台(代码片段)
kubernetes集群搭建Zabbix监控平台一、zabbix介绍1.zabbix简介2.zabbix特点3.zabbix的主要功能4.zabbix架构图二、检查本地k8s环境1.检查系统pod运行状态2.检查node节点状态三、配置nfs共享存储1.安装nfs2.创建共享目录3.配置共享目录4..启动相关... 查看详情
如何专业化监控一个kubernetes集群？
简介：本文会介绍Kubernetes可观测性系统的构建，以及基于阿里云云产品实现Kubernetes可观测系统构建的最佳实践。作者：佳旭阿里云容器服务技术专家 引言 Kubernetes在生产环境应用的普及度越来越广、复杂度越来越高，随... 查看详情
系统架构面临的三大挑战，看kubernetes监控如何解决？
...，我们就借助三个具体场景聊聊这一问题以及相应的解决方案。作者｜炎寻大家好ÿ 查看详情
系统架构面临的三大挑战，看kubernetes监控如何解决？
...，我们就借助三个具体场景聊聊这一问题以及相应的解决方案。作者｜炎寻审核&校对 查看详情
GCP Kubernetes 对象监控选项
】GCPKubernetes对象监控选项【英文标题】：GCPkubernetesobjectsmonitoringoptions【发布时间】：2020-07-2304:13:12【问题描述】：我们正在尝试找出哪些监控选项适合我们的环境。我们在GCP中有两个集群，我们在两个集群中都安装了Istio（带... 查看详情
kubernetes集群稳定性
...。次要的原因是服务器作用不明朗和发版流程的不稳定。解决方案发版流程不稳定重构发版流程。业务全面Kubernetes化，构建以Kubernetes为核心的CI/CD流程。发版流程有关发版流程如下：浅析：研发人员提交代码到develope... 查看详情
使用 VisualVM 通过 Kubernetes 集群后面的 JMX 监控 dockerised Spring Boot 应用程序内部
】使用VisualVM通过Kubernetes集群后面的JMX监控dockerisedSpringBoot应用程序内部【英文标题】：MonitordockerisedSpringBootapplicationinternalsviaJMXbehindaKubernetesclusterwithVisualVM【发布时间】：2020-10-0315:18:16【问题描述】：我想在Docker中为我的SpringBo... 查看详情
kubernetes集群中要监控哪些组件？采集哪些指标？
导读：让我们看一看在Kubernetes集群中应该监控哪些组件。作者：BrendanBurns,EddieVillalba,DaveStrebel,LachlanEvenson译者：陈亮王磊张扬来源：大数据DT（ID：hzdashuju）Kubernetes集群由控制平面和工作节点两类组件... 查看详情
有道kubernetes容器api监控系统设计和实践(代码片段)
...道技术团队微信公众号作者/郭超容王伟静编辑/hjy1.背景Kubernetes已经成为事实上的编排平台的领导者、下一代分 查看详情
kubernetes监控手册06-监控apiserver(代码片段)
写在前面如果是用的公有云托管的Kubernetes集群，控制面的组件都交由云厂商托管的，那作为客户的我们就省事了，基本不用操心APIServer的运维。个人也推荐使用云厂商这个服务，毕竟Kubernetes还是有点复杂的，升级也不好搞，我... 查看详情
prometheus在kubernetes集群的搭建教程
Prometheus在kubernetes集群的搭建一、Prometheus介绍1.Prometheus简介2.Prometheus介绍3.Prometheus与市面的监控系统区别4.Prometheus特点5.Prometheus的网站6.Prometheus原理图7.Prometheus监控示意图二、集群环境检查1.检查kubernetes的集群状态2.Prometheus使用... 查看详情
k8s集群稳定性评估与大规模基础服务部署调优(代码片段)
...细数kubernetes常见20道问题彻底搞懂K8SPodPending故障原因及解决方案Kubernetes架构核心点详细总结！顶级的Kubernetes日志监控工具Kubernetes集群需要重点关注的6个指标如何优雅的限制Kubernetes节点上运行的Pod数量容器与Pod到底有什么... 查看详情