Getting started if you’re running applications in AWS
AWS, or Amazon Web Services, is a cloud computing platform provided by Amazon. It offers a wide range of services, including computing power, storage, databases, machine learning, and more, all on a pay-as-you-go basis. AWS is widely used by businesses, governments, and individuals to build and scale applications without the need to manage physical hardware. AWS started out as a way to sell access to the platform Amazon had built to support its ecommerce operations and now accounts a significant fraction of amazons profits.
Amazon provides built-in tools for accessing and viewing logs and metrics. For advanced needs, data is typically sent using one of three methods. However, if you're using Elastic Kubernetes Service (EKS) or Elastic Container Service (ECS), it may be better to collect data directly by following Kubernetes, Linux, or Windows tutorials. Alternatively, data can be periodically polled from the CloudWatch API, but this method should be used sparingly, as it generates excessive and often unnecessary data.
Data sent directly
If you are running an EKS cluster or ECS, generating and sending that data directly is likely the best option.
To get started with data sent directly from Windows click here.
To get started with data sent directly from Linux click here.
To get started with data sent directly from Kubernetes click here.
Via an S3 bucket
S3 (simple storage service) was one of AWS’s earliest products and has become a common option for exporting data from other Amazon Services. Log files get saved into an S3 bucket and then consumed from there into a Logit stack.CloudFront uses a network of globally distributed edge locations to cache content closer to end users, improving performance and reducing load on origin servers. To learn more about using an S3 bucket to send logs and metrics to Logit.io click here.
Here are some examples of Amazon Services that can send data to Logit.io using an S3 bucket:
CloudFront
Amazon CloudFront is a content delivery network (CDN) service provided by AWS. It securely delivers data, videos, applications, and APIs to users with low latency and high transfer speeds.
To find out more about how to log from CloudFront using an S3 bucket click here.
CloudTrail
Amazon CloudTrail is a service that enables governance, compliance, and operational and risk auditing for your AWS account. It records and logs detailed information about API calls and actions taken in your AWS environment, including who performed the action, when it occurred, and the resources involved.
To find out more on how to log from CloudTrail using an S3 bucket click here.
ELB Application (ELB)
Amazon Elastic Load Balancer (ELB) Application Load Balancer (ALB) is a service that distributes incoming application traffic across multiple targets, such as Elastic Compute Cloud (EC2) instances, containers, or IP addresses. It operates at the application layer (Layer 7 of the OSI model), enabling advanced routing based on request attributes like URLs, headers, and query strings.
To find out more on how to log from ELB using an S3 bucket click here.
Elastic Load Balancer (ELB) Classic
Amazon Elastic Load Balancer (ELB) Classic Load Balancer is a legacy load balancing service that distributes incoming traffic across multiple Amazon EC2 instances. Operating at both the transport layer (Layer 4) and application layer (Layer 7) of the OSI model, it supports basic routing and load-balancing features, such as SSL termination and sticky sessions.
To find out more on how to log from ELB Classic using an S3 bucket click here.
Relational Database Service (RDS)
Amazon Relational Database Service (RDS) is a managed database service provided by AWS that simplifies the setup, operation, and scaling of relational databases in the cloud. It supports multiple database engines, including MySQL, PostgreSQL, MariaDB, Oracle, SQL Server, and Amazon Aurora. RDS automates routine database tasks such as provisioning, patching, backups, and scaling, freeing users to focus on application development.
To find out more on how to log from RDS using an S3 bucket click here.
Virtual Private Cloud (VPC)
Amazon Virtual Private Cloud (VPC) is a service that allows users to create isolated, customizable networks within the AWS cloud. With VPC, you can define your network's IP address range, create subnets, and configure route tables, gateways, and security settings to control traffic. It enables secure communication between AWS resources and on-premises networks via VPNs or Direct Connect.
To find out more on how to log from VPC using an S3 bucket click here.
Via a Lambda function
AWS CloudWatch Lambda integration enables monitoring and observability for AWS Lambda functions. It collects performance metrics such as invocation count, duration, and error rates, as well as custom logs generated by Lambda functions. Lambda functions are part of AWS’s serverless platform, where all the hosting is handled by aws and the customer simply provides the code they want to run.
These could generate and send logs/metrics directly to Logit, or be called by a third service acting as a forwarder for messages, potentially preprocessing messages or adding additional information to messages.
To learn more about sending data to Logit.io via a Lambda function click here.
Via an SQS queue
SQS (Services Simple Queue Service) is a message queue, messages are generated within AWS and consumed by Logit to transfer data to your stack.
It is better suited to those situations where there are lower volumes of data with low latency requirements. Data sent via an S3 bucket often is batched and compressed, which increases the latency between message generation and its availability in a Logit stack, but this also can result in reduced costs.
To learn more about sending data to Logit.io using SQS click here.
From cloudwatch
Amazon CloudWatch is a monitoring and observability service from AWS that provides insights into your cloud resources and applications. It collects and tracks metrics, logs, and events from AWS services and custom sources, enabling you to visualize performance, identify issues, and set up automated responses.
The cloudwatch API exposes lots of metrics from within AWS, it should be used sparingly to monitor specific functions and metrics. Attempting to consume everything rarely provides meaningful insight, whereas monitoring and alerting on specific metrics can help identify issues before they interfere with normal performance.
To learn more about sending data to Logit.io using Cloudwatch click here.