# Prometheus Integration

# Prerequisites

Install and set up Prometheus (opens new window).

# How to configure Prometheus to integrate with yCrash?

# 1. Configure yCrash webhook endpoint

Add yCrash webhook endpoint in alertmanager.yaml file. yCrash will listen to webhook notifications whenever an alert is triggered.

"receivers":
- "name": "ycrash"
  "webhook_configs":
  - "url": "http://yc-web:8080/alert-receiver?source=prometheus"
1
2
3
4

Note: Change http://yc-web:8080 with your yCrash server URL.

# 2. Route alerts

You can route some alerts or all alerts to the yCrash webhook endpoint. In order to do that you need to add label ycrash: capture in alertmanager.yaml file.

"route":
      "receiver": "ycrash"           # all alerts
    "routes":                   # match some alerts
      - "match":
          "ycrash": "capture"   # alerts with this criteria
           "receiver": "ycrash"
1
2
3
4
5
6

For more information visit https://prometheus.io/docs/alerting/latest/configuration/ (opens new window)

# How to add alerts?

Let's say you are running an application in Kubernetes and you want to trigger alerts whenever the CPU utilization of the pod goes beyond 80%. In that case, your alert rule will look like this:

groups:
  - name: ycrash.rules
    rules:
    - alert: ContainerCpuUsageHigh
      annotations:
        description: 'CPU usage is high in Pod {{ $labels.pod }} container {{ $labels.container }}.'
        summary: High container cpu usage.
      expr: |
        rate(container_cpu_usage_seconds_total{name!~".*prometheus.*", image!="", image !~ ".*pause.*", container_name!="POD"}[1m])
          + on (namespace, pod) group_left(pod_ip) 0 * kube_pod_info
          > 80
      for: 2m
      labels:
          severity: critical
          ycrash: capture
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Here is another example to trigger an alert when the memory usage of pod is over 80%.

groups:
  - alert: ContainerMemoryUsageHigh
      annotations:
        description: 'Memory usage is high in Pod {{ $labels.pod }} container {{ $labels.container }}.'
        summary: High container memory usage.
      expr: |
(100 * avg by (namespace, pod, container) (container_memory_working_set_bytes{name!~".*prometheus.*", image!="", image !~ ".*pause.*", container_name!="POD"})
          / avg by (namespace, pod, container) ( container_spec_memory_limit_bytes{name!~".*prometheus.*", image!="", image !~ ".*pause.*", container_name!="POD"} > 0)
        )
        + on (namespace, pod) group_left(pod_ip) 0 * kube_pod_info > 80
      for: 2m
      labels:
          severity: critical
           ycrash: capture
1
2
3
4
5
6
7
8
9
10
11
12
13
14

Important note:

In kubernetes, yCrash requires pod_ip in the alert's field. To include that, use the below line in your alert expr.

"+ on (namespace, pod) group_left(pod_ip) 0 * kube_pod_info"

The label ycrash: capture is used as a match criteria to route some routes to ycrash. If you route all alerts (use ycrash webhook endpoint as default), this label is not needed.

# Configure monitoring-tools.json file

The monitoring-tools.json file is used to configure the custom properties for the integration with monitoring tools. Inorder to integrate yCrash with Prometheus you may want to configure the below 2 properties in monitoring-tools.json file:

  • ycAgentPort - Configure the port number where your yc-360 script is listening to.
  • ycAgentProtocol - Protocol (http/https) that is configured for the yc-360 script.

By default the value of ycAgentPort will be 8085 and ycAgentProtocol will be https. In case if you don’t configure these properties in the monitoring-tools.json file then the yCrash will use the default values. Here are the steps to create monitoring-tools.json:

  1. Create a monitoring-tools.json file and add the above 2 properties into it.

  2. Place this file in the 'upload' directory.

  3. Here is a sample of the monitoring-tools.json file.

{
    "ycAgentPort": "8080",
    "ycAgentProtocol": "http"
}
1
2
3
4

← AppDynamics →