# Prestop Hook

# Capture Artifacts Before Pod Restarts

In Kubernetes' world, pods are considered to be relatively ephemeral (rather than durable) entities. Means, we cannot expect a pod to be a long running resource. When there are any availability issues or SLA breaches, pods will get recycled. Before they get recycled, we should capture troubleshooting artifacts (such as GC logs, thread dump, heap dump, netstat, iostat, top, …) to troubleshoot the problem. In this post we will show you how to capture and analyze these troubleshooting artifacts.

Kubernetes provides Containers with lifecycle hooks. The hooks enable Containers to be aware of events in their management lifecycle. Kubernetes offers postStart and preStop hooks. Containers can access these hooks by implementing and registering a handler for that hook. There are two types of hook handlers that can be implemented:

  • Exec - Executes a specific command or script pre-stop.sh inside the container.

  • HTTP (httpGet) - Executes an HTTP request against a specific endpoint on the container.

In this example, we used the yCrash-k8s (opens new window) project from our official GitHub repository. In the kubernetes/buggyapp-deployment.yaml file we included preStop hook and implemented Exec handler and to execute yc-360 script to capture artifacts from the target application before pod restarts.

Here is an example of kubernetes/buggyapp-deployment.yaml file:

















 



 
 
 
 



apiVersion: apps/v1
kind: Deployment
metadata:
  name: buggyapp
  labels:
    app: buggyapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: buggyapp
  template:
    metadata:
      labels:
        app: buggyapp
    spec:
      terminationGracePeriodSeconds: 180
      containers:
      - name: buggyapp
        image: ycrash/buggyapp:latest
        lifecycle:
          preStop:
            exec:
              command: ["/bin/bash","-c","/opt/workspace/yc-360-script/linux/yc -c /opt/workspace/yc-360-script/linux/yc-config.yaml -p buggyapp.war"]
			  
			...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

As described above, We are executing yc-360-script in the preStop hook. If you notice in the above YAML file definition, we have configured terminationGracePeriodSeconds to 180 seconds. It means kubernetes will wait for 180 seconds before it is killed forcefully. Generally the yc-360-script takes less than 60 seconds to run. However if you are capturing heap dump using yc-360-script then based on your heap size (i.e. -Xmx value of your application) it may take more than 60 seconds to complete. Thus we have given a grace period of 120 seconds. You can increase it, in case your heap size is very large.

# Debugging Hook Handlers

If you are facing problem while running yc-360-script in preStop hook. You may want to check following steps:

  1. The logs for a Hook handler are not exposed in Pod events. If a handler fails for some reason, it broadcasts an event. For preStop, this is the FailedPreStopHook event. For more information please visit here (opens new window)

  2. If the handler is executing successfully but you are not seeing any incident in the yCrash calendar dashboard page then you can try to execute yc-360-script manually from inside a pod and refresh the calendar page. In case if the issue still persists, you can reach out to support@tier1app.com.

← Run yc-360 Script in Sidecar Container →