Questions tagged [prometheus]
The Prometheus monitoring system.
153
questions
0
votes
1
answer
54
views
Use let's encrypt certificates with the prometheus node exporter
I'm trying to configure a prometheus node exporter with HTTPS using the website certificates generated by let's encrypt.
There is certificates with permissions default let's encrypt permission
But I ...
- 103
0
votes
0
answers
24
views
How to track PersistentVolume usage with prometheus for non gp2 storage class?
I am running a kubenetes cluster in an ubuntu node. I have created a persistent volume (storage class: openebs-hostpath) and corresponding claim. I want to track how much of the claim is being used by ...
- 1
1
vote
1
answer
226
views
Sending Prometheus data to local and remote storage
Having a PI running to monitor internet connectivity locally is good, but it also comes with the drawback of having limited storage, especially when other stuff is running on it, too.
I am looking for ...
- 164
0
votes
0
answers
117
views
SNMP scraping returned HTTP status 500 Internal Server Error on Prometheus
I installed Prometheus and SNMP exporter on Ubuntu Server, then configuring for scraping with
/etc/prometheus/prometheus.yml
- job_name: 'Mikrotik'
static_configs:
- targets:
- 10.10.106.210 ...
0
votes
2
answers
22
views
How do I get the boot time for a node given the instance information from dockerswarm_sd_config
I am trying to get the up time from node exporter but I want to have it driven by the nodes provided by dockerswarm_sd
- job_name: dockernodes
scrape_interval: 5s
dockerswarm_sd_configs:
- host: ...
0
votes
0
answers
15
views
Can Prometheus scale horizontally with a single data source?
I'm trying to run Prometheus on an unrealistic system (a Raspberry Pi) and I find that it keeps on losing connectivity which I presume is load related. I was wondering if it can be scaled ...
0
votes
0
answers
48
views
Puppet syntax issue when configuring a Prometheus scrape on an SNMP exporter
Some context
I have an SNMP exporter running as a service on a server which allows me to access the data of some PDU using the following URL: http://my.host.name:9116/snmp?target=10.0.1.200&auth=...
- 101
0
votes
0
answers
169
views
Auto-Instrumentation of application using OpenTelemetry
I have an AKS cluster where I am running a test Python-Django based web application.
I also have Grafana and Prometheus configured. I need to use OpenTelemetry to get the metrics data from the test ...
- 63
0
votes
0
answers
145
views
OpenTelemetry Collector Data not being fetched by Prometheus in Grafana
I have a requirement where I have some container workload in Azure AKS cluster and I need to use OpenTelemetry to gather data like metrics, logs and traces. I also have Grafana as the visualisation ...
- 63
0
votes
0
answers
64
views
OpenTelemetry K8s Operator Collector - Exporter Configuration for Prometheus
I have some container workloads in Azure AKS cluster. I need to use OpenTelemetry to get the metrics, logs and trace data from the container workload and get it collected by the OTEL collector.
I have ...
- 63
0
votes
2
answers
247
views
How can i monitor Openstack with prometheus and grafana
I installed openstack with devstack on Ubuntu 20 and installed prometheus and grafana with node exporter and it showed on grafana no issues, but how to monitor openstack components Nova.... There is a ...
- 3
0
votes
0
answers
68
views
nginx - can't reach status endpoint on localhost
I try to setup the nginx exporter for prometheus but have difficulties with the proper configuration of my nginx.
I have a default server block that redirects all requests from port 80 to 443. So all ...
- 211
0
votes
0
answers
77
views
I'm trying to expose a folder size in Node exporter but messes up. any suggestions?
This wasn't the case for a few months but suddenly my Node exporter in the server service messed up.
Oct 02 10:13:01 nodefs01 node_exporter[117252]: time="2023-10-02T10:13:01Z" level=error ...
0
votes
0
answers
38
views
Write_Prometheus plugin with basic auth
I am testing the below Write_Prometheus plugin to send the cassandra metrics to prometheus server which is external machine but i see that all metrics are sending without any basic auth happening. so ...
0
votes
0
answers
42
views
Spot instance sometime slow down and loss connection
I have a system deployed in AWS EKS, sometimes spot instances metrics is down, and API call to these nodes are very slow. Here is my system:
1 EKS cluster
1 on-demand node group
1 Karpenter v0.29.2 ...
- 21
0
votes
0
answers
133
views
Why Kube state metrics only shows metrics related to the namespace where it is running?
I have AWS EKS cluster with kube-state-metrics installed in a namespace called "monitoring". This installation is using service monitor and other components (see yaml files below).
In this ...
1
vote
0
answers
173
views
Prometheus does not start with WAL segment loaded
There is a Prometheus installed in Kubernetes Azure AKS with default storage StandardSSD.
Prometheus version v2.47.0
The volume storage size is 1 Gb. Data retention is configured for 1 hour.
Maximum ...
- 135
0
votes
0
answers
69
views
Correct way to setup a multinode LGTM stack
I have a 4 nodes clusters:
MonitoringCenter, hosting
a Grafana, connected to all Prometheus and Loki instances + local AlertManager
a Prometheus scraping local NodeExporter/AlertManager/Loki/...
- 199
0
votes
0
answers
61
views
How to add configuration for fluent-plugin-prometheus in Fluentd deployed via Fleet in Rancher?
I'm using Rancher to manage my Kubernetes cluster and have added a logging system (cattle-logging-system) via Fleet. I now need to add monitoring for Fluentd using the fluent-plugin-prometheus.
Here's ...
0
votes
0
answers
90
views
Thanos Receiver not deleting old data in Persistent Volume (PV) after retention is exceeded
I have set up Kube Prometheus Stack with Thanos on my Kubernetes cluster, and I'm using the Thanos Receiver instead of the sidecar approach. I have also configured the Thanos Compactor and Minio for ...
0
votes
0
answers
77
views
Spike in Cadvisor container_network_receive_bytes_total Metric in a Kubernetes Cluster
Summary:
I'm using Cadvisor with Prometheus in multiple Kubernetes (k8s) clusters to monitor network traffic usage. I utilize the container_network_receive_bytes_total metric in a query to calculate ...
0
votes
0
answers
88
views
Prometheus Server Pod Suddenly Crashed (unexpected fault address 0x7f911b1795d4)
With traffic Prometheus server pod getting restarting with below error stack. This is happend when the live traffic of the system. But could not be able to reproduced with the load testing.
Grafana ...
1
vote
1
answer
44
views
How to safely update an existing Kubernetes server without original configurations
I was handed a Kubernetes cluster with no config files, and was not setup with helm. The author said they just created everything from the cmd line. It is a small/new cluster for a single API server ...
- 13
0
votes
0
answers
39
views
Restrict access to a Prometheus server in AKS can only be achieved with nginx-ingress?
Prometheus server with its respective Loadbalancer in AKS.
I wanted to secure the access to /metrics through network rules...but it doesn't work. I can still acess to the endpoint with any device.
...
- 1
0
votes
0
answers
131
views
How can I know request waiting time in Nginx Ingress Controller?
We use Kubernetes with Nginx Ingress Controller to run our platform with various backend services. We also use New Relic (& Prometheus, Grafana) for our Observability dashboards & alerts. ...
- 101
0
votes
1
answer
127
views
prometheus-operator when configuring alertmanager config for PagerDuty
Have next issue:
When try to set up alertmanager configuration over CRD, get wrong configuration on pod.
Problem look like:
- routing_key: |
*****
routing_key parameter after all CD use ...
0
votes
1
answer
148
views
promtool tsdb create-blocks-from openmetrics doesnt show
I have a file like
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total{NODE_APP_INSTANCE="...
- 101
0
votes
0
answers
70
views
how to configure a prometheus installation of openshift to include the labels of node role in the exporter?
How to configure a prometheus installation of openshift to include the labels of node role in the exporter?
We want to alert based on cpu requests only on worker nodes
namespace_cpu:...
0
votes
0
answers
123
views
kube-prometheus-stack redundancy across multiple clusters
I currently use kube-prometheus-stack to monitor several kubernetes clusters. Each cluster has its own deployment of the kube-prometheus-stack, however, there is currently only one cluster (a) that ...
- 31
1
vote
0
answers
102
views
my prometheus dashboard shows 0 on node_export instance
I have installed prometheus on a server and node_exporter on another server
I configured my prometheus.yml to interact with the node exporter
The connection works fine but the up query shows 0 which ...
- 113
0
votes
0
answers
50
views
How to export Prometheus metrics from GCE into stack driver
We currently have some software that contains Prometheus data on a GCE. The said GCE already has the legacy logging agent using fluentD running, this is currently incompatible with the google ops ...
- 1
0
votes
0
answers
210
views
Persist Prometheus data from a Docker Container
I'm running Prometheus and Grafana from docker-compose and basically it works. But I don't get my prometheus data to persist on my host machine.
As soon as I switch my volume settings from
---
version:...
0
votes
0
answers
473
views
How to scrape Prometheus secured with OAuth2-proxy and Keycloak
I have 2 prometheuses, both are with forward-auth via the oauth2-proxy, which have the same client credentials in a single keycloak.
I would like one prometheus to federate the other one. This is my ...
- 373
1
vote
2
answers
77
views
Monitor Azure Kubernetes Service (AKS) with Prometheus servers running independently
Can I deploy Prometheus to each Azure Kubernetes Service (AKS), and monitor each Prometheus from a Prometheus created by a virtual machine outside the AKS? In other words, I want to connect the ...
- 33
3
votes
2
answers
1k
views
Debugging Prometheus OOMkilled despite 6Gi limits
I'm at the end of my patience with a prometheus setup leveraging kube-prometheus-stack 44.3.0 (latest being 45).
I have two environments, staging and prod. In staging, my prometheus runs smoothly. In ...
- 141
0
votes
0
answers
45
views
How to avoid duplicate entries in DNS Bind cache db file?
While running command "rndc dumpdb -cache" it has huge amount of data some duplicate entries like one domain has multiple entries with same target but different TTL, how to reduce like ...
- 1
0
votes
1
answer
259
views
How to remove labels in alert template?
I found a template for Prometheus Alert Manager for alerts, but I receive many unnecessary labels in notification. How to remove labels in template? This code is responsible for labels (look below). I ...
- 131
0
votes
0
answers
573
views
How to remove information about old firing alerts in notification?
When 1 alert is firing, AlertManager sends a notification, then 2nd alert is firing or resolved, AM sends a notification with information about 2 alerts. How to remove information about 1st alert in ...
- 131
0
votes
1
answer
906
views
Monitoring of MySQL replication and alerting when replication is not working
I have a question about monitoring of MySQL replication and in particular about alerting using Grafana/Prometheus.
We have 2 MySQL(MariaDB to be exact) instances for which replication is configured: ...
- 11
0
votes
0
answers
93
views
Nginx `ngx_http_stub_status_module` with multiple sites on the same server (VM)
Server is a VM running Debian 11 with nginx-extras package.
I have the ngx_http_stub_status_module setup in /etc/nginx/conf.d/stub_status.conf as shown below.
server {
listen 127.0.0.1:8080;
...
- 101
1
vote
1
answer
466
views
Grafana graph value as a percentage of the initial value
I am new to grafana & prometheus. I want to graph the change in a value (here disk space used), but as a percentage of the initial value, which one commonly sees for inflation charts.
For example, ...
- 31.9k
0
votes
1
answer
549
views
Prometheus expose any metrics on itself? (e.g. disk space it uses)
Does Prometheus expose any metrics on itself? i.e. how much disk space it's using etc.
I want to start fine tuning our prometheus server, so need to monitor what's currently there. I want to be able ...
- 31.9k
0
votes
0
answers
107
views
Agent prometheus does not send metrics
I am trying to configure a Prometheus agent with the 'remote_write' function. The Prometheus agent configuration file has these parameters.
global: scrape_interval: 30s external_labels:
...
- 1
0
votes
1
answer
1k
views
kube-api server high cpu
I want to know how I can check why one of my ctrl node and kubernetes consumes more cpu than the others.
I have a cluster with 3 ctrl nodes and 4 worker nodes.
I have an nginx load balancer with the ...
PeterWegner90
0
votes
0
answers
66
views
Accessing local host Dashboards (GCP VM) from my Laptop
I have 2 VM instances on Google Cloud Platform.
1st VM is used as a Kubernetes Master Node and 2nd as Worker Node.
I have Prometheus and Grafana Deployed on the cluster. I want to access the ...
- 13
0
votes
1
answer
1k
views
Kubernetes upgrade from 1.21 to 1.22 caused Prometheus to fail
We recently upgraded Kubernetes 1.21 to 1.22 version on aws eks. The upgrade was successful. However, the associated prometheus deployments fails with error
$ kubectl -n monitoring logs prometheus-...
- 103
0
votes
0
answers
25
views
Prometheus Metrics - Multiple boxes in one file
My existing custom exporter for Prometheus generates:
this_metric_one 10.3
this_metric_two 11.4
Currently this is running on two boxes, Box_A and Box_B. INSTEAD, I want to have Box_C query both (...
- 101
0
votes
0
answers
480
views
Alertmanager does not send resolved email
I have configured an alertmanager and it sends alerts properly But it does not send resolved email. I have tried different amount of time for resovled timeout but it did not work
here is my ...
0
votes
0
answers
55
views
My mysql Exportir not appears on my Prometheus localhost
So I have 2 VMs in my virtualbox, there are Monitor and Target. So, in the Monitor VM, I install Grafana as monitoring and in Target VM, I install Prometheus as target monitoring.
For now I've ...
1
vote
1
answer
657
views
Ceph alertmanager config
I've installed ceph using cephadm
Including monitoring stack prometheus, alertmanager, node-exporter
Currently I'm trying to add a telegram receiver (Telegram is supported from v0.24.0, so I've ...
- 63