A simple way to load test your To-Do List Manager application could be to run a curl query that targets the application load balancer URL, from within your Cloud9 IDE’s terminal.

For hash generation, we can leverage the RANDOM variable and derive a md5 checksum value from it, using the following command:

echo $RANDOM | md5sum | head -c 20

We can utilize the hash generation inside a curl command that triggers a POST request on the application endpoint:

curl –location –request POST ‘http://chapt-farga-65sza285ppo1-1511829492.eu-central-1.elb.amazonaws.com/’ \

–form “todo=echo $RANDOM | md5sum | head -c 20” \ –form ‘tasktype=”Un-important”‘

To prepare this for infinite execution (until it’s manually interrupted), we can wrap up the command with a while loop:

while true; do curl –location –request POST ‘http://chapt-farga-65sza285ppo1-1511829492.eu-central-1.elb.amazonaws.com/’ –form “todo=echo $RANDOM | md5sum | head -c 20” –form ‘tasktype=”Un-important”‘; done

Leave this code snippet running for some time so that enough data is generated in our observability tools.

Observing data to understand application behavior

Debugging operational incidents often requires correlating information across several metrics and deriving insights from the underlying data. Our test application stack publishes different types of data in different tools. We have infrastructure metrics and logs available in CloudWatch, and additionally, we are publishing metrics in Prometheus. Let’s take a look in CloudWatch to see if we got some logs as part of the logging that we enabled in the Flask decorator functions.

Monitoring logs with Amazon CloudWatch Logs

To check CloudWatch Logs, you can go to the CloudWatch service console and click on the relevant log group. This seamless log ingestion is provided by the integration between ECS and CloudWatch. As a best practice, you should always emit the log stream on STDOUT so that the data can be easily ingested into other platforms.

As you may recall, we had a logging configuration in each CDK construct that represented one of the ECS containers. For the application container, the configuration looked something like this:

logging: ecs.LogDrivers.awsLogs({streamPrefix: “aws-devops-simplified-app-container”})

This single line of config ingests data from the app container and forwards it to the CloudWatch service, on the predefined stream prefix. Checking the logs for this log stream on CloudWatch shows some interesting insights, as highlighted in Figure 8.8:

Figure 8.8 – Application logs captured in CloudWatch Logs

Let’s take a closer look:

1. GET /metrics: HTTP invocations from the OpenTelemetry collector scrape application metrics at a predefined interval and publish them on the Prometheus server. As we can see, the source of the request is 127.0.0.1, which is expected because the collector runs inside the same task definition as the application.
2. GET /: These are invocations on the application root that seem to originate from two IP addresses, 10.0.3.223 and 10.0.2.30. These are the application load balancer interfaces and trigger regular health check requests against the application.
3. POST: This is a request on the application domain that is a result of thecurl command that we triggered from our local Terminal to generate some load. You will see many of these, depending on how long you keep the command running.

CloudWatch also allows you to create metrics from these logs based on some patterns. So, a user could track the number of POST calls coming in a time window of 1 minute and raise an alarm if it crosses the expected threshold. Furthermore, you could also track the source IP addresses from such requests and figure out the highest contributors to application load in a certain time frame. As you can see, the possibilities are endless, and it all depends on how deep you want to go with such insights.

Next, let’s have a look at our Prometheus metrics.