Combining AWS Monitoring Services: A Basic Example

Overview workflow

In previous article we have discussed about the difference and usage of different Monitor services in AWS. Next we will combine CloudTrail, EventBridge, CloudWatch, X‑Ray, and SNS together around a simple Lambda–S3 workflow. In this example:

• CloudTrail records management (control‐plane) events such as S3 “PutObject.”

• EventBridge watches for a CloudTrail event (when someone uploads an object to S3) and triggers an SNS notification.

• Lambda is configured to run when a new S3 object is created. It is instrumented with X‑Ray (so here we can see an end‑to‑end trace of the request, including a simulated S3 call) and automatically sends logs and metrics to CloudWatch.

• SNS sends a notification when the EventBridge rule fires.

We have 3 workflows here:

1. S3 Upload → CloudTrail → EventBridge → SNS → Email

2. S3 Upload → Lambda → X-Ray

3. S3 Upload → Lambda → CloudWatch (Logs/Metrics)

1st route is for triggering notifications based on S3 API events

2nd for distributed tracing of a processing function

and 3rd for operational monitoring via logs and metrics.

Now we will have a step by step instruction on AWS console panel.

1. Create a CloudTrail Trail

here we will record events for Management events and Data events, which will both record some management actions like the IAM create, update functions, and also record some data related actions like put/get data/file from s3 bucket.

1. Open CloudTrail.

2. In the sidebar, choose Trails and then Create trail.

3. Name: monitor-demo-trail.

4. Event Type choose Data events and Management events

5. Edit Gerneral Details:

   • Data Event Type: S3.

   • Create a new Buckets for storage.

6. Addtional Settings:

   • Remember to ENABLE SNS notification delivery

7. Event Types:

   • Management events and Data events: to record All object-level API events (or we can restrict to just Put events if available).

8. Under Management events, ensure that Read/Write events is set to All.

9. Under S3 Data Event: choose resource type as s3, and log all events

10. Ok, Create.

2. Create an SNS Topic

1. Go to the SNS console. Choose Topics on left sidebar, then Create topic.

2. Select Topic type: Standard.

3. Name the topic: demo-event-sns.

4. Create topic.

5. Then click Create subscription, choose Email as the protocol. Remember go to the email and confirm the subscription.

3. Create an EventBridge Rule to Trigger SNS on S3 “PutObject” Events

1. Navigate to the EventBridge console.

2. Click Rules on left sidebar, then Create rule.

3. Name: demo-s3-put-obj-eb.

4. Event Bus: Use the default event bus.

5. Rule State: Ensure it is Enabled.

6. Creation Method: Choose Custom pattern (or “Use pattern form” and then switch to the JSON editor).

7. Define Event Pattern:

   •     {
           "source": ["aws.s3"],
           "detail-type": ["AWS API Call via CloudTrail"],
           "detail": {
             "eventName": ["PutObject"]
           }
         }
     

This pattern ensures that whenever an S3 object is uploaded (a PutObject event), the rule matches.

8. Select Targets:

   • Target type: SNS topic.

   • Choose above we created SNS topic demo-event-sns.

Now when a new object is added to S3 (and CloudTrail records the event), EventBridge will send an SNS notification.

4. Create an S3 Bucket

1. Enter a Bucket Name: Use a unique name monitor-demo-bucket.

2. Leave other as default, Create bucket.

5. Create a Lambda Function Instrumented with X‑Ray

5.1 Create the Function

1. Open the Lambda console.

2. Click Create function and select Author from scratch.

3. Name: MyXRayFunction.

4. Choose Runtime: Node.js

5. Permissions: Allow Lambda to create a new role(we talk details below 5.2).

6. Create function.

5.2 Grant Get Permission to role of Lambda

In function’s Configuration tab, find the Permissions section. Then click on the role name (e.g., MyXRayFunction-role-abcdefg) to open it in the IAM console.

In the IAM Console: here we just “Add inline policy”, and choose the JSON tab in the policy editor.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::monitor-demo-bucket/*"
    }
  ]
}

here

• "Action": "s3:GetObject" allows Lambda function to retrieve objects.

• "Resource": "arn:aws:s3:::monitor-demo-bucket/*" specifies all objects in the bucket named monitor-demo-bucket.

• give a name as: AllowS3GetObject

5.3 Enable X‑Ray Tracing

1. In Lambda function page, choose Configuration tab.

2. In the sidebar, choose Monitoring and operations tools.

3. Under Lambda service traces, toggle the switch to Enabled.

5.4 Add S3 as a Trigger

1. Still on Configuration tab sidebar, click Triggers.

2. Select Trigger Type: S3.

3. Configure Trigger:

   • Bucket: Choose the bucket we created.

   • Event type: Select All object create events (or simply “PUT” events).

5.5 Make the Lambda Code file

As we need AWS-XRay-SDK and AWS-SDK Package as the Dependency in the Lambda Code, we need to install these packages on our local computer and then upload the zip file to lambda.

Create a Local Project Folder:

mkdir my-lambda-function
cd my-lambda-function

Create a file named index.js with our code. (this simple example uses the X‑Ray SDK to capture a subsegment around a simulated S3 call):

// Import the AWS X-Ray SDK and wrap the AWS SDK
const AWSXRay = require('aws-xray-sdk');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));

exports.handler = async (event) => {
  // Log the received event to CloudWatch Logs
  console.log("Received event:", JSON.stringify(event, null, 2));

  // Get the current X-Ray segment
  const segment = AWSXRay.getSegment();

  // If the event is from S3, process it
  if (event.Records && event.Records[0].s3) {
    const bucket = event.Records[0].s3.bucket.name;
    const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));

    // Create an S3 client and simulate a getObject call
    const s3 = new AWS.S3();
    let subsegment;
    try {
      // Create a subsegment to trace the S3 call
      subsegment = segment.addNewSubsegment('s3-getObject');
      const data = await s3.getObject({ Bucket: bucket, Key: key }).promise();
      subsegment.close();
      console.log("S3 object retrieved, size:", data.ContentLength);
    } catch (err) {
      // Record the error in X-Ray
      if (subsegment) {
        subsegment.addError(err);
        subsegment.close(err);
      }
      console.error("Error retrieving S3 object:", err);
    }
  }

  return {
    statusCode: 200,
    body: JSON.stringify({ message: "Lambda executed successfully" })
  };
};

Initialize the Project and Install Dependencies:

Run the following commands inside the project folder:

npm init -y
npm install aws-xray-sdk aws-sdk

Package the Function:

Zip the project folder contents (ensure that index.js and the node_modules/ directory are at the root of the zip). So from within project folder:

zip -r ../my-lambda-function.zip .

Under the Code tab, upload my-lambda-function.zip file.

6. Test

1. Upload a File to S3:

   • CloudTrail logs the S3 PutObject event.

   • EventBridge picks up the CloudTrail event and sends an SNS notification.

   • The S3 event also triggers the Lambda function.

2. Verify CloudTrail:

   • In the CloudTrail console, find Event history on the left side panel.

   • Look for a recent event with "eventName": "PutObject". This confirms CloudTrail is recording management events.

3. Check SNS and Email

4. View CloudWatch Logs and Metrics:

   • In the CloudWatch console, check the Logs group for our Lambda function to see the console output.

   • In the Metrics section, review Lambda invocation metrics.

5. Inspect X‑Ray Traces:

   • Open the X‑Ray console.

   • In the Service map or Trace list, we should see traces from our Lambda function.

   • Drill into a trace to view details (such as the custom “s3-getObject” subsegment), which shows the latency and any errors.

Summary

• Workflow A: S3 Upload → CloudTrail → EventBridge → SNS → Email

  • S3 Upload: When we upload an object to an S3 bucket, if CloudTrail is configured to capture data events (object-level events), it logs the API call (e.g., a PutObject event).

  • CloudTrail: This event is recorded by CloudTrail.

  • EventBridge: An EventBridge rule matching that CloudTrail event (based on criteria like eventSource and eventName) is triggered.

  • SNS: The rule sends the event to an SNS topic.

  • Email: SNS, in turn, delivers a notification to the email.

  • Note:
    To make this work, we have to enabled S3 data events in CloudTrail (object-level logging). Without that, only management events are logged by default.

• Workflow B: S3 Upload → Lambda → X-Ray

  • S3 Upload Trigger: An S3 bucket can be configured to send event notifications (e.g., on object creation) to trigger a Lambda function.

  • Lambda Execution: The Lambda function processes the event. When it’s instrumented with the X-Ray SDK (or has Lambda service traces enabled which is previous “active tracing”), every request gets traced.

  • X-Ray: X-Ray collects trace segments (and subsegments) that show the end-to-end flow of the request (for example, how long it takes for the Lambda to perform a GetObject call on S3 or make other downstream calls).

• Workflow C: S3 Upload → Lambda → CloudWatch (Logs/Metrics)

  • S3 Trigger: The same S3 event can trigger a Lambda function.

  • Lambda Logging: The Lambda function automatically writes logs (e.g., console.log outputs, errors) to CloudWatch Logs and its performance metrics (invocation count, duration, error rate) are automatically sent to CloudWatch Metrics.

  • Analysis: We can then use CloudWatch Logs Insights or CloudWatch dashboards/alarms to monitor the behavior of our Lambda and the overall system health.