How to Integrate yCrash with Cisco AppDynamics (Step-by-Step)

Before proceeding, We would suggest to read the related documentation:

• Alert and Respond (opens new window)
• Policies (opens new window)
• Health Rules (opens new window)
• Actions (opens new window)
• Remediation Actions (opens new window)
• Prerequisites for Local Script Actions (opens new window)

1. Install the machine agent in the node (opens new window):

Please follow the below steps to install the machine agent if you haven’t installed it yet. In this documentation we have used an Ubuntu machine. Here are instructions to install the machine agent -

• OS: Ubuntu 22.04 LTS

• Download Machine-agent:

Go to: https://YOURACCOUNT.saas.appdynamics.com/controller/#/location=GETTING_STARTED_SERVER&timeRange=last_1_hour.BEFORE_NOW.-1.-1.60 (opens new window)

Note: Replace YOURACCOUNT with your account subdomain in the above URL.

1. Click the button Click here to download to download the agent.

2. Copy the downloaded file to the machine. (you can use rsync / scp / ftp)

• Installation steps:

1. Execute the following commands to install the machine agent.

apt update && apt install default-jre default-jdk unzip

mkdir /opt/appdynamics/machine-agent -p; cd /opt/appdynamics/machine-agent

# Copy the downloaded agent file to this directory (/opt/appdynamics/machine-agent)
cp <DOWNLOADED_AGENT_FILE> /opt/appdynamics/machine-agent

# Unzip the file you just copied using the below command
unzip machineagent.zip

2. The downloaded machine-agent bundle is shipped with JRE. However, using the bundled JRE gives us the following errors:

Using java executable at /opt/appdynamics/machine-agent-2/jre/bin/java                                                                                                                   

./bin/machine-agent: 90: exec: /opt/appdynamics/machine-agent-2/jre/bin/java: Exec format error

So, let’s use the system default-jdk and delete the bundled jre directory using the below command:

rm -rf jre

• Run as systemd service:

useradd -m appdynamics-machine-agent
chown appdynamics-machine-agent:appdynamics-machine-agent /opt/appdynamics/ -R
cp etc/systemd/system/appdynamics-machine-agent.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable appdynamics-machine-agent.service
systemctl start appdynamics-machine-agent.service
systemctl status appdynamics-machine-agent.service

2. Install the app agent in the node (opens new window)

Please follow the below steps to install the app agent if you haven’t installed yet. Here are the installation instructions:

1. Goto Applications from the main menu on the top.

2. Click Create Application.

3. Choose Create an Application using the Getting Started Wizard.

4. On Getting Started Wizard, choose: Applications: Java.

5. Then on Section 4. Download the Agent Installer, click Click Here to Download button.

6. Upload the agent file to the machine using scp / rsync / ftp.

7. Create a new directory and put the agent file in /opt/appdynamics/appagent directory.

mkdir /opt/appdynamics/appagent -p; cd /opt/appdynamics/appagent

8. Unzip the file using the below command.

unzip appagent.zip

9. Run hostnamecommand to get your host name. Note it for the next step.

hostname

10. Edit /opt/appdynamics/appagent/conf/controller-info.xml to update the <node-name> value.

<node-name>YOUR_HOST_NAME</node-name>

11. Follow the specific instructions for your Application server on section 5. Install the App Agent. In this document, we have selected Standalone JVM as we are using BuggyApp as a target java application.

3. Start your Java application with the app-agent enabled, follow step 11 above

Here we are using BuggyApp (opens new window) as a java application. You can replace BuggyApp with your java application.

Run the below commands to download and run the BuggyApp.

1. Create a new directory for Buggyapp.

mkdir /opt/ycrash/buggyapp -p; cd /opt/ycrash/buggyapp 

2. Download Buggyapp and unzip the file once it's downloaded.

wget https://tier1app.com/dist/buggyapp/buggyapp-latest.zip
unzip buggyapp-latest.zip 

3. Modify launch.sh script.

• Because BuggyApp is a standalone JVM, by following step 11 of previous step, add the "-javaagent" argument in launch.sh file as shown below:

java -javaagent:"/opt/appdynamics/appagent/javaagent.jar" -Xmx2g -DlogDir=. -DuploadDir=. -jar webapp-runner-8.0.33.4.jar --port 9010 buggyapp.war

4. To run the BuggyApp application execute the launch.sh script.

launch.sh

Wait a few minutes until the application shows up in the Applications menu.

4. Create / Review Health Rules

1. Goto Alert & Respond from the main menu on the top.

2. Click Health Rules from the left sidebar.

3. Select your application from the Applications dropdown.

4. There are about 7 default health rules. For example - Business Transaction Health, CLR Garbage Collection Time is High, CPU utilization is high, etc…

You can define your own health rules, but in this document we will just use the default JVM Heap Utilization is too high.

5. The default rules have a quite long attribute Wait Time after Violation: 30 minutes. You can change this value based on your requirement. Please follow the below steps to change the value:

a) Right click and select Edit option.

b) Wait Time after Violation: 30 minutes.

c) Critical Criteria -> Specific Value: update from 90% to 70%.

5. Create an Action

1. Go to Alert & Respond from the main menu on the top.

2. Click Actions menu from the left sidebar.

3. Select your application from the Applications dropdown. In the below screenshot you will see the “BuggyApp” application is selected.

4. Now click the +Create button.

5. Choose Remediation: Run a script or executable on problematic Nodes.

6. Click OK.

7. Then fill out the next form which is Create Remediation Script Action form.

a) Enter invoke-yc-360-script in the name field.

b) Enter invoke-yc-360-script.sh in the Relative Path to Script field.

8. Then click Save.

6. Install the invoke-yc-360-script.sh

1. Add the below script in invoke-yc-360-script.sh file and save this file to /opt/appdynamics/machine-agent/local-scripts/ directory.

#!/bin/bash
if [ -z "${ProcessID}" ]
then
      printf "Could not get PID from ProcessID\n"
      exit 1
else
      printf "Got PID ${ProcessID} from ProcessID\n"
      printf "Going to execute: yc-360 script -c yc-config.yaml -p ${ProcessID}\n"

      # TODO: Update yc-360 script path to your installation
    # See https://docs.ycrash.io/
      # /opt/yc-360-script/yc -c yc-config.yaml -p ${ProcessID}
fi

2. Set it as an executable:

chmod +x /opt/appdynamics/machine-agent/local-scripts/invoke-yc-360-script.sh

3. Check the file owner, it should match with the parent directory.

7. Create a Policy

1. Goto Alert & Respond from the main menu on the top.

2. Click Policies from the left sidebar.

3. Select your application from the Applications dropdown.

4. Click the Create Policy Manually button.

5. Fill the form:

a) On tab Trigger

1. Enter ycrash-integration in the name field.
2. We would suggest to uncheck Execute actions in batch.
3. Health Rule Violation Events:
   1. Enable Health Rule Violation Started - Critical.
   2. Enable Health Rule Violation Continues - Critical.
   3. Enable Health Rule Violation Upgraded - Warning to Critical.

b) On tab Health Rule Scope

1. Choose These Health Rules.
2. Click + button.
3. Select the health rules. In this document, we are going to use JVM Heap Utilization is too high.

c) On tab Object Scope

1. Choose These Specified Objects.
2. Click the + Add Objects button.
3. Select Tiers and Nodes.
4. Select your relevant Tiers or Nodes or select all tiers if all your servers are running this setup.

d) On tab Actions

1. On Actions to Execute, click + button.
2. Select the one you created previously: invoke-yc-360-script.
3. Click the Select button.
4. Execute Action on Affected Nodes:
   • Execute Action on: 100% of the Nodes
5. Click OK button.

6. Click Save button.

8. Review Health Rules Events and Invocations

1. Go to Alert & Respond from the main menu on the top.

2. Click Health Rules from the left sidebar.

3. Select your application from the Applications dropdown.

4. Select the Health Rule you just attached to the policy.

5. Check the Evaluation Events on the right side.

6. In this document, we are using BuggyApp so we are triggering a memory leak to violate JVM Heap Utilization is too high.

7. Once the rule is violated, the event shows up there. Click the Details and you will see the invoked script and the output.

8. At this point, if the script works, it should trigger the yc-360 script on demand command.