Managing AWS Distro for OpenTelemetry Collector with AWS Systems Manager Distributor

AWS Systems Manager Distributor simplifies the distribution, installation, and update process for software packages on managed instances at scale. AWS Systems Manager also provides a secured and centralized repository with version control for these software packages. With AWS Systems Manager Distributor, users can install and update their software packages for a large set of managed instances on-demand with a few simple steps.

AWS Distro for OpenTelemetry (ADOT) Collector v0.9.0 is now available with Systems Manager Distributor support. We now have a Systems Manager package for the ADOT Collector that any user can use on Amazon Elastic Compute Cloud (Amazon EC2) as well as managed on-premises instances by running simple Systems Manager commands. Systems Manager helps reduce the operational cost on ADOT Collector deployments for users and ensures a better onboarding user experience for service observability.

In this blog post, we’ll discuss the design for packaging and publishing the ADOT Collector for regional Systems Manager repositories from AWS Observability GitHub repository via GitHub Actions. Furthermore, we’ll show how to utilize Systems Manager Distributor to manage the deployment lifecycle of ADOT Collector for a large fleet of managed Amazon EC2 hosts.

Enabling ADOT Collector in AWS Systems Manager

The ADOT project is open source and hosted in GitHub. ADOT implements its continuous integration (CI) and continuous delivery (CD) workflows using GitHub Actions. In our approach of enabling ADOT Collector in Systems Manager Distributor, the end-to-end packaging and publishing workflow was implemented in GitHub Actions. The diagram below describes a high-level view of the workflow implementation.

As shown in the diagram, the continuous integration workflow in GitHub includes the following steps:

1. Build and upload ADOT Collector artifacts to Amazon Simple Storage Service (Amazon S3) bucket.
2. Call AWS Systems Manager to create an AWS Systems Manager Distributor package named AWSDistroOTel-Collector package. AWSDistroOTel-Collector package is Systems Manager format of ADOT Collector installer.
3. AWS Systems Manager downloads the released artifacts of ADOT Collector from an Amazon S3 bucket and creates a Systems Manager Distributor package named AWSDistroOTel-Collector. This package is published with a public Systems Manager command, later used by a Systems Manager public document. Once the AWSDistroOTel-Collector package is published publicly, it is available to everyone in the Systems Manager Distributor console. This package helps install the ADOT Collector for a large group of hosts. Behind the scenes, AWS Systems Manager utilizes the predefined command document named AWS-ConfigureAWSPackage to install the AWSDistroOTel-Collector we uploaded to AWS Systems Manager. At the same time, AWS Systems Manager can also help configure the ADOT Collector during the installation.
4. The provided customized configuration will be stored in the Systems Manager Parameter store.
5. With the uploaded ADOT Collector package and customized configuration, the Systems Manager service will notify the Systems Manager Agent on the hosts to download ADOT Collector installer and the configuration from the Systems Manager service.
6. The Systems Manager Agent will invoke the ADOT Collector installer to install the Collector on the target EC2 and on-premises hosts.
7. Additionally, ADOT Collector can be deployed on the EC2 instances managed by Auto Scaling groups; it utilizes the AWS Systems Manager State Manager to run auto-installation on the newly added hosts in the Auto Scaling group.
8. The Systems Manager State Manager will periodically scan EC2 instance tags of Auto Scaling groups and run AWS-ConfigureAWSPackage, which will install the ADOT Collector on the new EC2 instances managed by Auto Scaling groups.

Getting started

In this section, we walk through how to deploy and update ADOT Collector on Amazon EC2 and managed on-premises instances via AWS Systems Manager Distributor. For this setup, the target instances need to meet the following requirements:

• SSM Agent version 2.3.1550.0 or later installed.
• Permissions policy AmazonSSMManagedInstanceCore attached in IAM Role of Amazon EC2 instance.

Install ADOT Collector via AWS Systems Manager

To start, you’ll need to set up your own ADOT Collector YAML configuration. Refer to the ADOT Collector Configuration documentation for more details.

Note that Systems Manager parameters don’t support multi-line string text including YAML, so you’ll need to encode the ADOT Collector YAML configuration to Base64 and copy the Base64 string to the AWS Systems Manager Parameter Store. Run the following command to encode YAML to a Base64 string:

For Linux: base64 -w0 config.yaml > aoc-config-base64.txt

For Windows: certutil.exe -f -encodehex config.yaml aoc-config-base64.txt 0x40000001

For macOS: base64 -i config.yaml -o aoc-config-base64.txt

Next, navigate to the SSM Distributor console and open the Owned by Amazon tab. Select AWSDistroOTel-Collector and choose Install.

Place the Systems Manager parameter you created in the first step in the Additional Arguments field. For example, if the Systems Manager parameter name is aoc-config-base64, the following value should be used:

{"SSM_CONFIG": "{{ssm:aoc-config-base64}}"}

Install ADOT Collector to the target Amazon EC2 instances by selecting EC2 instance tags, EC2 instance IDs, or Resource Group in the Targets section and select Run to start the installation. Refer to Install or update packages for more installation options.

Conclusion

In this blog post, we walked through how to enable the ADOT Collector in AWS Systems Manager using a GitHub Actions workflow to provide a simplified installation experience for users. We also walked through steps for how to deploy the ADOT Collector to a large group of managed EC2 instances with AWS Systems Manager. To learn more, check out the AWS Distro for OpenTelemetry repos and developer portal for detailed technical documentation.