How to Use Webhooks to Automate Red Hat OpenShift App Rebuilds from AWS CodeCommit

By Mehdi Salehi, Sr. Partner Solutions Architect – AWS
By Boris Jelic, APAC OpenShift and Middleware Leader – IBM

Red Hat

For more than a decade, Red Hat has been an Amazon Web Services (AWS) strategic partner. In 2019, IBM acquired Red Hat, which has furthered IBM’s expertise in enabling the next-generation of enterprise applications on AWS.

IBM maintains an extensive set of AWS competencies, validations, and certifications, and builds on Red Hat’s open hybrid cloud vision with vast experience in modernizing and transforming traditional applications.

One of the offerings to help joint AWS and Red Hat customers build for an open hybrid cloud future is Red Hat OpenShift Service on AWS (ROSA), a fully managed OpenShift service, jointly supported by both Red Hat and AWS.

Installation, management, maintenance, and upgrades are performed by Red Hat site reliability engineers (SRE) with joint Red Hat and Amazon support.

In this post, IBM and AWS focus on a single but common use case often observed in the field where a joint enterprise customer has made investments in the AWS Cloud and is now considering expanding and adopting ROSA service as part of the AWS portfolio.

We’ll also demonstrate how customers can automate their OpenShift builds when a change is pushed into their code repositories on AWS CodeCommit.

Overview

Source-to-Image (S2I) helps developers simplify and speed up container-based application development. For this to happen, S2I needs to be able to download the application source code from a source code repository. In this post, we’ll explore how a code update on AWS CodeCommit can initiate an application rebuild on ROSA automatically.

S2I needs a source clone secret to clone the code from CodeCommit. The following diagram shows the direction of the request and how the new code version is “pulled” from CodeCommit. To learn more about integrating ROSA with CodeCommit, see this AWS blog post.

Figure 1 – ROSA clones the code from AWS CodeCommit using secrets.

This works when we initiate a rebuild from the OpenShift side manually. In the following sections, we’ll explore how this process can be automated.

Mechanics of OpenShift Builds

We have an S2I application running in cluster1 within the rosaproject1 namespace:

$ rosa list clusters
ID				        NAME      STATE
1lk70d1hicri6r4rnuke49r926vuadgv  cluster1   ready
$ oc config set-context --current --namespace=rosaproject1
$ oc get deployment 
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
demo-app1   1/1     1            1           1d

We have also exposed the application at a public URL, so a Route object has been created:

$ oc get route
NAME        HOST/PORT                                                         PATH   SERVICES    PORT       TERMINATION   WILDCARD
demo-app1   demo-app1-rosaproject1.apps.cluster10.p92r.p1.openshiftapps.com          demo-app1   8080-tcp                 None

Using the public URL of the route, we can reach the application from a web browser, which shows the output:

Welcome to Rosa! This is app v1.0

In OpenShift, a build configuration (or BuildConfig) describes a single build definition and set of triggers for when a new build is created. A BuildConfig is characterized by a build strategy and one or more sources. In our scenario, the source is a CodeCommit repository:

$ oc get buildconfigs
NAME		TYPE	FROM	LATEST
demo-app1   	Source	Git	1

$ oc describe buildconfig demo-app1 | grep vcs-uri
		app.openshift.io/vcs-uri=https://git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/ruby-hello-world

According to OpenShift documentation, a build is the process which is used to transform input parameters or source code into a runnable image. A BuildConfig object is the definition of the entire build process. In other words, for any BuildConfig there will be one or more builds:

$ oc get builds 
NAME          TYPE     FROM          STATUS     STARTED          DURATION
demo-app1-1   Source   Git@f3aff56   Complete   27 minutes ago   37s

If the BuildConfig is modified, a new build is created. We can also instruct the BuildConfig to initiate a new build. In both cases, the application source will be downloaded again from the repository. In the next section, we’ll explore how rebuilds can be configured to trigger automatically.

Application Rebuild Using a Webhook

To start a rebuild after an update is pushed to the code repository, a webhook should be configured. Creating and configuring a webhook for AWS CodeCommit needs a generic webhook trigger to kick off a new S2I build.

As shown in the following diagram, after a code update is pushed to CodeCommit, a notification mechanism is needed to inform ROSA that the code has changed.

Figure 2 – Trigger the build from AWS CodeCommit.

A unique BuildConfig is created for each S2I application. A BuildConfig describes a single build definition and a set of triggers for when a new build should be created. In fact, by describing a BuildConfig, we can extract the webhook URL:

1. Get the name of the BuildConfig:

   $ oc get buildconfigs
   NAME        TYPE     FROM   LATEST
   demo-app1   Source   Git    2
   

2. Extract the webhook URL from the BuildConfig:

   $ oc describe buildconfig demo-app1 | awk '/webhooks/ {print $2}'
   https://api.cluster10.p92r.p1.openshiftapps.com:6443/apis/build.openshift.io/v1/namespaces/rosaproject1/buildconfigs/demo-app1/webhooks/<secret>/generic
   

3. Before the URL can be used as a webhook, the <secret> (which is part of the above URL) needs to be replaced with the right WebHookSecretKey. To find the WebHookSecretKey, the secret name should be determined first:

   $ oc get buildconfig demo-app1 -o yaml | grep ' triggers' -A
     triggers:
     - generic:
         secretReference:
           name: demo-app1-generic-webhook-secret
       type: Generic
   

   Alternatively:

   $ oc get buildconfig demo-app1 -o jsonpath='{.spec.triggers[].generic.secretReference.name}'
   demo-app1-generic-webhook-secret
   

4. Now that the secret name is known, the secret key can be extracted:

   $ oc get secrets demo-app1-generic-webhook-secret -o jsonpath='{.data.WebHookSecretKey}'
    M2MzMWE5MmE0N2RlODdlNQ==
   

   However, this is the Base64-encoded version of the secret, so it needs to be decoded:

   $ oc get secrets demo-app1-generic-webhook-secret -o jsonpath='{.data.WebHookSecretKey}' | base64 -d
    3c31a92a47de87e5
   

5. The decoded secret (3c31a92a47de87e5) needs to be embedded into the webhook URL we extracted in Step 2:

   https://api.cluster10.p92r.p1.openshiftapps.com:6443/apis/build.openshift.io/v1/namespaces/rosaproject1/buildconfigs/demo-app1/webhooks/3c31a92a47de87e5/generic
   

6. Let’s verify. By sending a POST HTTP request to this webhook (even without a payload), we should be able to trigger a build:

   $ curl -kX POST 'https://api.cluster10.p92r.p1.openshiftapps.com:6443/apis/build.openshift.io/v1/namespaces/rosaproject1/buildconfigs/demo-app1/webhooks/3c31a92a47de87e5/generic' 2>/dev/null |grep '"code":'
     "code": 200
   

   The success response code is a good indication, but this can also be verified by listing the builds. The following command shows we have successfully triggered a new build using the webhook, just a few seconds ago:

   $ oc get build
   NAME          TYPE     FROM          STATUS     STARTED         DURATION
   demo-app1-1   Source   Git@f3aff56   Complete   13 hours ago    40s
   demo-app1-2   Source   Git@f3aff56   Complete   13 hours ago    39s
   demo-app1-3   Source   Git           Running    5 seconds ago
   

   We can even describe the build (demo-app1-3) and get the build trigger cause. The following command shows a generic webhook—which is the POST HTTP request in step 6—has initiated this build:

   $ oc describe builds demo-app1-5 | grep -i 'Build trigger cause'
   Build trigger cause:	Generic WebHook
   

   The new app version will be deployed after a minute or so, showing the output:

   Welcome to ROSA! This is app v2.0
   

Automation

What we just saw in the previous section is still manual. The goal is to automate the build process, such that AWS CodeCommit can invoke the webhook URL and notify the OpenShift BuildConfig automatically. There are several ways to address this, but for simplicity we are going to use AWS Lambda to invoke the hardcoded webhook URL.

Here’s the high-level sequence of the events:

1. Developer sends a push request to the code repository on AWS CodeCommit.
2. The update on CodeCommit triggers a Lambda function.
3. Lambda function sends a POST HTTP request to the ROSA BuildConfig using the webhook URL.
4. ROSA pulls a fresh copy of the application code from CodeCommit.
5. A new build is created; therefore, a new version of the application becomes available.

In the following sections, we dive deeper into Steps 2 and 3 and see how they can be accomplished.

For Step 1, we assume the developer knows how to interact with Git repositories. And the last two steps (4 and 5) are automatically performed by S2I.

Define a Lambda Function with AWS CodeCommit as Event Source

An example and the steps of this scenario are in the AWS CodeCommit User Guide. As shown below, we have created a Lambda function and added a CodeCommit trigger to it.

Figure 3 – Adding AWS CodeCommit as a trigger for the Lambda function.

In the simplest form, if the Lambda function sends a POST request to the webhook URL, a new build should be created in ROSA. Figure 4 shows the configuration of a simple Lambda function that does this job using a few lines of code.

Figure 4 – The AWS Lambda code that invokes the webhook URL.

As we can see in the output, it has taken 252.87 milliseconds for the Lambda function to invoke the webhook URL and print the 200 HTTP response:

Figure 5 – The successful HTTP response after invoking the webhook URL.

We should also be able to see a new build that has been initiated by a webhook:

oc get builds | tail -1
demo-app1-4   Source   Git@8070f0e   Running    16 seconds ago   

oc describe builds demo-app1-4 | grep -i 'Build trigger cause'
Build trigger cause:	Generic WebHook

Finally, when the new build is completed, we can see the expected output in the front-page of the application:

This is app v3.0 (Lambda triggered the build)

Using AWS CI/CD Tools

So far, we have manually generated the webhook URL and hardcoded it into a Lambda function. But these manual steps can be performed as part of an AWS CodeBuild stage as well.

For a fully-automated mechanism, we also need to be able to log on to the ROSA cluster using credentials or a token. This challenge can be addressed by securely storing this information in AWS Secrets Manager. The continuous integration and continuous delivery (CI/CD) build stage will then be able to retrieve the credentials (or the token) from Secrets Manager programmatically.

Here is an example using the AWS Command Line Interface (CLI):

$ aws secretsmanager get-secret-value --secret-id ROSA --output=text | head -1 | cut -f4 -d'"'
oc login https://api.cluster1.XXXXX.openshiftapps.com:6443 --username rosa-user1 --password XXXXXXXXX

The following diagram shows the different steps of the solution—updating the code, initiating a build, retrieving the secrets, generating the webhook URL, and invoking it:

Figure 6 – The proposed solution using AWS CodePipeline.

The following sample BuildSpec shows how the AWS CodeBuild part of the solution works:

version: 0.2

phases:
  install:
	commands:
  	- apt-get install wget curl
  	- wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux.tar.gz
  	- tar xzf openshift-client-linux.tar.gz
  	- cp oc /usr/local/bin
  build:
	commands:
  	- oclogin=`aws secretsmanager get-secret-value --secret-id ROSA --output=text | head -1 | cut -f4 -d'"'`
  	- $oclogin
  	- oc config set-context --current --namespace=rosaproject1
  	- buildConfig="demo-app1"
  	- secret_name=`oc get bc $buildConfig -o jsonpath='{.spec.triggers[?(@.generic)].generic.secretReference.name}'`
  	- webhookSecretKey=`oc get secret $secret_name -o jsonpath='{.data.WebHookSecretKey}' | base64 -d`
  	- oc describe bc $buildConfig |grep webhooks | awk '{print $2}' | sed -e s/\<secret\>/$webhookSecretKey/
  	- webhookURL=`oc describe bc $buildConfig |grep webhooks | awk '{print $2}' | sed -e s/\<secret\>/$webhookSecretKey/`
  	- curl -kX POST $webhookURL
artifacts:
  files:
	- '**/*'

As a result, if we change application code, a new OpenShift build will be initiated through AWS CodePipeline and show the following output:

This is app v4.0 (CodePipeline triggered the build)

Conclusion

Full integration of Red Hat OpenShift on AWS (ROSA) with AWS CodeCommit allows customers to keep their code and CI/CD pipelines within AWS in a fully managed environment.

In this post, we demonstrated how ROSA can pull the application codes from a private Git repository on CodeCommit, and how code updates on CodeCommit can trigger a new application build on ROSA.

If you’re interested in getting hands-on experience with ROSA, this workshop is a good start. The Red Hat OpenShift Service on AWS documentation also provides more information about the service, its architecture, and how to operate it.

.

.

Red Hat – AWS Partner Spotlight

Red Hat is an AWS Competency Partner that helps customers get the most of the AWS Cloud by guiding them along their modernization journeys.

Contact Red Hat | Partner Overview | AWS Marketplace

*Already worked with Red Hat? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.