Introduction

In the realm of cloud architecture and API management, monitoring and analyzing access to APIs are crucial tasks for ensuring security, performance, and compliance. Amazon Web Services (AWS) offers a powerful solution to this challenge through its Amazon API Gateway Access Logs feature. In this blog post, we'll delve into what Amazon API Gateway Access Logs are, explore their usage scenarios, provide a step-by-step demonstration of implementation, discuss alternatives, and highlight important caveats to consider.

Why API Gateway Access Logs are Important

Amazon API Gateway Access Logs offer significant advantages in various scenarios:

1. Security Monitoring: Access logs allow you to track and analyze every request made to your API, enabling you to detect and respond to suspicious or unauthorized activity effectively.
2. Compliance and Auditing: For organizations subject to regulatory requirements, access logs provide a valuable audit trail, documenting API usage for compliance purposes.
3. Performance Optimization: By analyzing access logs, you can identify performance bottlenecks, optimize API usage patterns, and improve overall system efficiency.

API Gateway Access Logs VS. Execution Logs

Access logs and execution logs are two types of API logging options in CloudWatch for Amazon API Gateway.

Access logs focus on who accessed your API and how they did it by recording detailed information about every request made to your API Gateway such as source IP address, request path, HTTP status code, and response size. The logs typically contain information related to the incoming HTTP request and corresponding responses sent back. This is useful in understanding who is accessing your API, how often they are accessing it, and whether there are any errors or issues during those interactions (such as 4XX errors).

Execution logs provide detailed information on the execution of the API Gateway resource itself such as how it processes requests, integrates with other AWS services in the backend, and executes any associated business logic.

Execution logs are managed by API Gateway by creating log groups and log streams and reporting requests and responses to the log streams. The logs capture information about the internal workings of API Gateway such as request-response transformations, authentication and authorization processes, and errors encountered during request processing.

Access logs are like a visitor logbook at a resort, recording who came and when while execution logs are like security cameras recording everything that happens in the public spaces for a specific visitor.

Step-by-Step Demonstration

Let's walk through the process of setting up Amazon API Gateway Access Logs for an API deployed on AWS using Terraform.

Prerequisites

• An AWS account with permissions to create and configure API Gateway resources.
• An API deployed on Amazon API Gateway.

Configuration Steps

Step 1: Set Up Log Destination

Choose an existing CloudWatch log group or create a new one to store the access logs. Ensure that the appropriate permissions are set to allow API Gateway to write logs to this log group.

resource "aws_cloudwatch_log_group" "claim" {
  name              = "API-Gateway-Execution-Logs_${aws_api_gateway_rest_api.this.id}/${var.stage_name}"
  retention_in_days = 7

}

Step 2: Permissions for CloudWatch Logging

You must grant API Gateway permission to read and write logs to the CloudWatch log group. We will use the AmazonAPIGatewayPushToCloudWatchLogs AWS Managed Policy which has all the required permissions.

First, we create the role and a trust policy for the 'apigateway.amazonaws.com' service:

resource "aws_iam_role" "apigw_exec_role" {
  name = "apigw-exec-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Sid    = ""
      Principal = {
        Service = "apigateway.amazonaws.com"
      }
      }
    ]
  })
}

‍Then we attached the AWS Managed policy, to the role:

resource "aws_iam_role_policy_attachment" "apigw_cloudwatch" {
  role = aws_iam_role.apigw_exec_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs"
  
}

Using AWS-managed policies is a convenient way to assign appropriate permissions to IAM principals without writing the permissions line-by-line yourself.

But in case you must define a custom policy, here are the permissions required in the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents",
                "logs:GetLogEvents",
                "logs:FilterLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Step 3: Enable CloudWatch Logging for API Gateway Account

We need to associate an IAM role with our API Gateway account, which is necessary to enable CloudWatch logging for API Gateway.

resource "aws_api_gateway_account" "this" {
  cloudwatch_role_arn = aws_iam_role.apigw_exec_role.arn
}

The 'aws_api_gateway_account'  resource block is an important part of the Terraform configuration for enabling CloudWatch logging for your Amazon API Gateway.

Without this resource, the API Gateway would not be able to push logs to the specified CloudWatch log group, even if you have configured the access_log_settings in your 'aws_api_gateway_stage' resource.

Step 3: Setup Access Logging Settings

Here we set up access logging in the 'access_log_settings' for the 'dev' stage and direct the logs to our CloudWatch log group:

resource "aws_api_gateway_stage" "dev" {
  deployment_id = aws_api_gateway_deployment.this.id
  rest_api_id   = aws_api_gateway_rest_api.this.id
  stage_name    = "${var.stage_name}"

    access_log_settings {
    destination_arn = aws_cloudwatch_log_group.claim.arn
    format          = "$context.error.message $context.httpMethod $context.identity.sourceIp $context.integration.error $context.integration.integrationStatus $context.integration.latency $context.integration.requestId $context.integration.status $context.path $context.requestId $context.responseLatency $context.responseLength $context.stage $context.status" 

  }

  depends_on = [aws_api_gateway_account.this]
  
}

In the format field, we define the format of the access logs. In the code snippet above, we are using the Common Log Format (CLF).

Each field in the format corresponds to a specific attribute or context variable related to the incoming HTTP request. Here's a breakdown of each field:

1. '$context.identity.sourceIp': The source IP address of the client that made the request.
2. '$context.identity.caller': The identity of the caller, typically associated with authentication information (e.g., IAM user or role ARN).
3. '$context.identity.user': The user associated with the request, if applicable.
4. '[$context.requestTime]': The timestamp indicating when the request was received by API Gateway.
5. '"$context.httpMethod $context.resourcePath $context.protocol"': The HTTP method, resource path, and protocol (e.g., "GET /example/resource HTTP/1.1") of the request.
6. '$context.status': The HTTP status code returned by the API Gateway for the request.
7. '$context.responseLength': The length of the response body returned by the API Gateway.
8. '$context.requestId': The unique identifier assigned to the request by API Gateway.
9. '$context.extendedRequestId': The extended request ID, which includes additional information about the request and its processing.

For the log format output, JSON, CSV, and XML formats are also supported. You can refer to the API Gateway documentation for more information on the different access log formats. Here is also a full list of the $context variables available for API Gateway access logging.

Step 4: Verifications

After the API receives a couple of requests, we can head to the CloudWatch Logs console to verify that access logs are being generated and stored correctly.

When we select our CloudWatch Log Group, we can see all the various Log Streams for the Log Group.

With many Log Streams, it is difficult to know exactly which one you might have interesting information but you can click “Search all log streams” and enter a filter pattern in the search bar.

Better still, you can use CloudWatch Logs Insights to query for something like all logs with HTTP 4XX codes as you can see in the image below:

Caveats with Amazon API Gateway Access Logs

• Cost Considerations: Storing and analyzing access logs in CloudWatch Logs may incur additional costs, especially for high-volume APIs.
• Performance Impact: Enabling access logging can introduce overhead, potentially affecting API latency and throughput. Careful configuration and monitoring are necessary to mitigate performance impacts.

Conclusion

Amazon API Gateway Access Logs are a valuable tool for monitoring and securing APIs deployed on AWS. By enabling access logging, organizations can gain insights into API usage patterns, enhance security posture, and ensure compliance with regulatory requirements. While there are alternative approaches to API logging, Amazon API Gateway Access Logs offer a convenient and integrated solution for AWS users. By understanding their capabilities, limitations, and best practices, organizations can harness the full potential of API Gateway Access Logs to optimize their API management workflows.

References

https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html

https://www.alexdebrie.com/posts/api-gateway-access-logs/

https://repost.aws/knowledge-center/api-gateway-errors-cloudwatch-logs

https://docs.aws.amazon.com/apigateway/latest/developerguide/monitoring-cloudwatch.html

https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html