RFC 31: Integration testing for CDK apps #378

nija-at · 2021-09-03T16:51:07Z

This is a request for comments about Integration testing for CDK apps.
See #31 for additional details.

Readable version: https://github.com/aws/aws-cdk-rfcs/blob/nija-at/integ-tests/text/0031-integ-testing.md

APIs are signed off by @{BAR_RAISER}.

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache-2.0 license

rix0rrr

Interesting approach. I'm a little concerned that expressivity might suffer, and I'm also a little concerned about the runtime performance of these tests.

Ideally you'd want to keep test cases separated, but as doing so will require a full CloudFormation cycle for each tests (~multiple minutes per test) people will NOT do that and combine as many assertions as they can into a single test to avoid the performance overhead. Leading to worse tests, imo...

rix0rrr · 2021-09-07T13:06:01Z

text/0031-integ-testing.md

+
+### contest
+
+🌩️ contest tests can verify that your CDK app can be deployed successfully on AWS with the desired resources.


Are we also taking care of cleanup? I think that'd be a selling point.

Agreed. I'll add.

rix0rrr · 2021-09-07T13:07:37Z

text/0031-integ-testing.md

+
+## Working Backwards - README.md
+
+### contest


Haaaaaaaaaaaah!

Not a fan of the name... I'd just call this cdk test/@aws-cdk/testing

rix0rrr · 2021-09-07T13:08:28Z

text/0031-integ-testing.md

+
+* `assertAws`: execute AWS service APIs and assert the response.
+* `invokeLambda`: invoke an AWS lambda function and assert the response.
+* `curlApi`: Execute the 'curl' command on an API endpoint. Used typically to test API Gateway resources.


Too low level? Maybe just httpRequest ?

text/0031-integ-testing.md

BenChaimberg · 2021-09-07T15:38:23Z

text/0031-integ-testing.md

+repoTest.assertAws.
+  s3.putObject({ bucket: stack.artifacts, key: '...', body: '...' });


This feels like test setup, which makes the "assert" part of this statement read a bit awkwardly. I'm wondering if it should be more like test.aws.module.action(...) in general and then test.aws.module.action(...).assertReturns and test.aws.module.action(...).assertThrows for assertions. Of course the test should fail if any call fails

On the other hand, this may lead users to attempt to modify the return values of the SDK calls in the test file, which is not correct since the test file is executed at synthesis time, not at runtime. I think this will be the most confusing part of the paradigm you are describing here.

Maybe there is some way to make this look less like a normal assertions library (using object construction instead of function calls). The increased friction could actually be a good thing here since it would give users pause

This feels like test setup, which makes the "assert" part of this statement read a bit awkwardly.

Hmm yes, you make a good point. Maybe just dropping the "assert" phrasing in the API will make this cleaner. I'll take a stab at this in the next rev.

this may lead users to attempt to modify the return values of the SDK calls in the test file
I think this will be the most confusing part of the paradigm you are describing here.

Good observation. I came to the similar realization after I published this. There's further work needed here.

nija-at · 2021-09-08T08:43:44Z

@rix0rrr

I'm a little concerned that expressivity might suffer

Can you expand?

also a little concerned about the runtime performance of these tests.

My approach was going to involve creating snapshots (app + assertions) for all tests by default and not "running" the test if the snapshots matched. Would this sufficiently address your concern?

skorfmann

I think this topic would be hugely beneficial beyond just AWS CDK. In CDK for Terraform were having similar thoughts and even beyond that, I think that whomever is provisioning AWS resources would benefit from having a strong testing support library which could be plugged into whatever testing framework people might be using.

It would be fantastic to have a polyglot testing support library, which abstracts away common use cases when interacting with AWS resources in test scenarios, like (off the top of my head, there's certainly more):

triggering a Lambda function
publishing / receiving messages (SQS, EventBridge)
invoking Cloudfront Functions and fetching results
Trigger StepFunction workflows / analyzing its state
Dynamically override (stub) Lambda Function (e.g. in StepFunctions or API Gateway)
AppSync resolvers
Easily access logs and traces

I'd leave the actual assertions to the respective testing framework, let them run the tests locally by default. There could still be an glue layer on top, which integrates this neatly into AWS CDK, CDK for Terraform, etc...

Two questions:

What is the motivation to run the tests as Lambda functions deployed via a Cloudformation Stack?
Do you see away to extract the test helpers into an independent library to make these available beyond AWS CDK?

skorfmann · 2021-09-08T09:12:56Z

text/0031-integ-testing.md

+This would imply a simpler and a more familiar experience of writing tests.
+
+However, to achieve this effectively, the test cases will need to deploy the app it defines.
+We do not have a way to deploy CDK applications programmatically.


Deploying programmatically seems to be a low hanging fruit and it's something people are asking for as well #300 (there are similar requests in cdktf)

I actually built a jest test suite which deploys multiple AWS CDK test stack the other day - granted, it's all Typescript but the experience is quite good.

If we did get programmatic deploy-ability, this could work out.

As far as the AWS CDK goes, this will mean taking the 'deploy' logic in the AWS CDK CLI and converting it into an independent jsii module (so as to make this polyglot). I would not consider this low hanging fruit, but an rfc of its own.
However, this does not mean it's not worth pursuing.

@rix0rrr - given you know the most about the AWS CDK CLI, what do you think of this?

Definitely not an easy change to do this in a jsii-compatible way.

Of course, we could hack it and shell out to the CLI. But if we're going to do this, I'd be very interested in clearly defining and controlling the user experience. Having every test suite run wait for CFN deployments is not going to be great.

nija-at · 2021-09-08T14:23:21Z

@skorfmann - thanks for your review.

What is the motivation to run the tests as Lambda functions deployed via a Cloudformation Stack?

A few things -

It allows for users, typically enterprise users, to test "private" resources, such as a lambda function in a private vpc.

The biggest problem with this type of testing is the time of test execution which is going to be really long. The mitigation to this is snapshot testing (this section is due in the rfc and I hope to have it in the next rev). If the snapshot includes the app and the test, then we run the test only if the app and the test have not changed.

It also allows for predictable test execution environments, which allows for stable assertion execution. Let's say, an assertion depends on the sed command. If this is run on the Mac, it will run BSD sed and on unix, this will be GNU sed, which have subtly different behaviours.
With a lambda function, the execution environment and how to bring additional dependencies is well known.

Finally, a nice side effect is that we can use existing CLIs to deploy and run these tests, without any additional runtime logic.

This is a great question. I'll need to feed this back into the RFC.

Do you see away to extract the test helpers into an independent library to make these available beyond AWS CDK?

What do you mean by test helpers?

text/0031-integ-testing.md

eladb · 2021-09-09T09:32:45Z

text/0031-integ-testing.md

+
+## Working Backwards - README.md
+
+### contest


Not a fan of the name... I'd just call this cdk test/@aws-cdk/testing

text/0031-integ-testing.md

nija-at · 2021-09-09T13:44:39Z

I think this topic would be hugely beneficial beyond just AWS CDK. In CDK for Terraform were having similar thoughts and even beyond that, I think that whomever is provisioning AWS resources would benefit from having a strong testing support library which could be plugged into whatever testing framework people might be using.

@skorfmann - agreed that this would be hugely beneficial.

I'm currently looking to find the right solution that works for one CDK, specifically the AWS CDK. Once we arrive at a happy solution, I intend to see what parts of the solution can be expanded to benefit other CDKs.

BenChaimberg · 2021-09-14T16:53:34Z

text/0031-integ-testing.md

-test.invokeLambda(...).throws({ ... });
+new AwsAssertion(test, 'GetObject', {
+  request: AwsAssertionCall.S3.getObject({ ... }),
+  returns: { Data: ... },


to support the use case of allowing chained calls (call 1: StartExecution, call 2: GetExecutionStatus(executionId = ...)), can we use the Capture model?

Let's not focus too deeply on this. We can get into a discussion about the exact API modelling when we get to implement (and implementation design of) this.
We can get as innovative on this as needed over time or even build service specific Assertion classes that do chaining, etc.

The idea here is to capture the overall direction in terms of API, to see there are no major gaps, or innovation blockers.

I would like to focus this RFC on areas around the designs of the test execution framework, assertion model, interaction between CLI and the test frameworks, etc.

Sure – this was I guess a poorly phrased way of saying: I would like to see first class support for call chaining in this design. I don't think it's explicitly mentioned but I hope it will be, since I think it's integral to the use of the framework.

Ah, now I see what you mean.

This feels like the feature of the aws-assertions "module". If we look at the aws assertions as its own piece separate from the core testing system, we need to build some form of chaining (something like Capture perhaps as you previously alluded to) into the aws assertions part.

It feels to me that this is part of the 'aws assertions' part, and not something first class in the 'core assertion' part of the system. But I'm also having a hard time judging that at this level - need to get my hands dirty to see how this will pan out.

Does that make sense?
Or do you see a piece that we need to add to the 'core assertion' part of the system that will enable this for aws assertions?

I do agree chaining is important for the aws assertions.

I guess I don't yet see the dividing line between "core" and "aws" assertions – I'm sure this is in the works as the RFC takes shape! However, it seems to me that regardless of the [infrastructure manager (AWS/Azure/...)/call provider (AWS SDK/HTTP/...)], call chaining would be very useful. Not being able to track "state" in the assertions seems to really restrict what kind of tests the library can support. But, again, this is based just off my assumptions about core vs. aws. Excited to see this progress!

I've addressed this in the latest change.

skorfmann · 2021-09-22T08:44:32Z

Do you see away to extract the test helpers into an independent library to make these available beyond AWS CDK?

What do you mean by test helpers?

Was aiming at what @eladb mentioned here #378 (comment)

skorfmann · 2021-09-22T09:38:39Z

I like where this is going 👍 What are your thoughts on traceability and logs for tests and their assertions?

I've been playing around with jest and xray integration. My goal there was to get visibility into the resources and their potential logs, in the context of the tests they are running in. I still have to streamline this a bit, but then gonna publish a full code example.

The rough idea goes like this:

// the jest test function wrapped into an xray starting segment 
// A setup block deployed a testing stack already, that's where the 'output' is coming from
  xray.test("render index file", async () => {
    const post = await fs.readFile(path.join(__dirname, 'fixtures', 'simple.md'), 'utf8');
    const index = await fs.readFile(path.join(__dirname, 'fixtures', 'index.md'), 'utf8');
    const bucketArn = output.renderOlap

   // the AWS SDK clients are instrumented with xray
    await putObject(bucketArn, `posts/simple.md`, post, 'text/markdown');
    await putObject(bucketArn, `index.md`, index, 'text/markdown');
    const result = await getObject(bucketArn, `index.md`);
    expect(result.toString()).toMatchSnapshot();
  }, 20_000);

Which allows to see the test execution for each single test as a full xray segment, including downstream calls of the tested infrastructure components.

And more importantly, it's possible to collect all (cloudwatch) logs of the individual testing xray segments - here's an excerpt from my logging output (there's a delay of roughly 5 - 10 seconds for the logs to show up after the tests finished)

I know that this is not that easy to pull off across all testing frameworks / languages, but perhaps it might be possible to keep this in mind and enable users to achieve something similar when they want to.

nija-at · 2021-09-23T11:55:52Z

text/0031-integ-testing.md

+This setup will enable defining a separate script that runs the apptest suite - npm script,
+Ant target, Maven phase, etc. - by using `*.apptest.*` as the file filter in your test runner.
+
+The test will be executed when the `run()` method is called on the `apptest.Test` class.


jsii (currently) does not support throwing error messages across languages.

Best to return a result that can be then asserted on. It may also be the best outcome to get good test reports.

Not sure what you mean by that? You can throw an exception in JavaScript and it will propagate to the host language...

polothy · 2021-09-27T15:44:23Z

text/0031-integ-testing.md

+This feature is beneficial to users who own a CDK app and want to run assertions on the deployed
+app in their pre-production stages as part of a CI/CD pipeline.


So these tests are not deployed as part of the main application? If not, then they cannot trigger a rollback if any of the tests fail, which seems like a nice feature.

pre-production stages

Why not allow tests against production? Just curious :)

We are currently using step function and/or lambdas to run integration tests. Since they are part of the stack, they can roll it back if anything fails and we can run it against any deployment stage. Maybe this RFC isn't meant to address this use case?

This RFC is very interesting! It'll be helpful no matter what. Just trying to grasp how it is run and when (EG: how would this run in CodePipeline?).

Why not allow tests against production? Just curious :)

You are right. Take a look at the latest revision. I've included this.

Think I caught your updates, looks like it is possible to include these in your main stack in order to trigger rollbacks, thanks! 👍

For when it isn't part of your main stack, what's the expected workflow in a pipeline like CodePipeline?

CloudFormation Action to deploy the main application that we want to test

CodeBuild that runs CDK deploy/destroy or Another CloudFormation Action to deploy... and maybe a CloudFormation Action to destroy? (or maybe just keep it? No real cost if all serverless tech)

Think my concern about CodeBuild is if the deployment isn't in the same account, we have to do some role assumes.

what's the expected workflow in a pipeline like CodePipeline?

You would run the tests as you would any other tests in your pipeline, which is probably CodeBuild.

A future addendum to this RFC will address this when using CDK Pipelines and make this trivial. I've taken this out of scope to keep the RFC contained.

If you use your own CI/CD mechanism, you will need to set up cross-account assume role, yes.

nija-at · 2021-09-29T12:36:27Z

@skorfmann -

What are your thoughts on traceability and logs for tests and their assertions?

This should be possible with the apptest framework being proposed here. To get into the details on this, this is an entire topic we can go very deep on, so I've left it out.

If we build the right abstraction for an 'assertion', we can build assertions based on logs and, perhaps even, metrics.

eladb · 2021-09-30T07:53:25Z

text/0031-integ-testing.md

+import { ArtifactRepo } from './records';
+import { AwsAssertionCall, AwsAssertion, Test } from '@aws-cdk/apptest';
+
+const repo = new ArtifactRepo(app, 'RequestRecord');


Where is app coming from?

eladb · 2021-09-30T07:53:54Z

text/0031-integ-testing.md

+  returns: { Messages: [...] }
+});
+
+const testResult = repoTest.run();


This is weird... What does run() do in this context?

It executes the test, i.e., deploys the stack with the construct and assertions, checks for its status, collects results and reports back.

eladb · 2021-09-30T07:54:04Z

text/0031-integ-testing.md

+
+const testResult = repoTest.run();
+if (testResult.failures > 0) {
+  // notify the test runner to fail


What do you mean by that?

call jest.fail() for jest test or junit.fail() in junit test, etc.

eladb · 2021-09-30T07:55:06Z

text/0031-integ-testing.md

+
+## Working Backwards - README.md
+
+### `@aws-cdk/apptest`


Not sure I understand the semantics "app test". This is not just for testing full applications... It can also test parts of apps (constructs). Something is still not accurate about the terminology.

Let's brainstorm on the name synchronously. It needs to be good.

eladb · 2021-09-30T08:01:42Z

text/0031-integ-testing.md

+* `AwsAssertion`: execute AWS service APIs and assert the response.
+* `InvokeLambda`: invoke an AWS Lambda Function and assert its response.
+* `HttpRequest`: Execute an HTTP request against an API endpoint and assert the response.


The naming patterns are not consistent here (some use verbs, some use nouns). Also seems like all of them are "assertions", so either they all should have an Assertion suffix or none.

Here are some options:

AwsApiAssertion, LambdaAssertion, HttpAssertion

AwsApiRequest, LambdaInvoke, HttpRequest

AssertAwsApi, AssertLambda, AssertHttp

etc...

eladb · 2021-09-30T08:15:53Z

text/0031-integ-testing.md

+import { App } from 'cdk-deploy';
+
+const app = new App('/path/to/cdk/app');
+app.deploy();


How do you plan to implement this synchronously?

eladb · 2021-09-30T08:17:18Z

text/0031-integ-testing.md

+```ts
+import { App } from 'cdk-deploy';
+
+const app = new App('/path/to/cdk/app');


Is this a path to a cloud assembly or to a directory with cdk.json? Kind of feels like it should accept cdk.App instance instead, no?

eladb · 2021-09-30T08:17:49Z

text/0031-integ-testing.md

+
+### Credentials
+
+This module uses the AWS SDK for Javascript under the hood to communicate with CloudFormation.


Suggested change

This module uses the AWS SDK for Javascript under the hood to communicate with CloudFormation.

This module uses the AWS SDK for Javascript under the hood to communicate with AWS services such as S3 and CloudFormation.

eladb · 2021-09-30T08:21:45Z

text/0031-integ-testing.md

+The AWS CDK CLI (aka toolkit) is responsible for deploying and destroying AWS CDK apps.
+We will extract these two parts from the CLI into a separate jsii module.
+
+The `run()` method on the class `apptest.Test` will depend on this module to deploy and


Feels like this is a bit of a rabbit hole not directly needed to support integration tests. Why can't integration tests be simple CDK apps (like cdk-integ works) and then we can just use the CDK CLI (or a tool that wraps the CDK CLI) to deploy them?

eladb · 2021-09-30T08:26:02Z

text/0031-integ-testing.md

+Users will now need to learn how to organize apptest tests, learn how to execute it and read
+results. We are not leveraging existing "well known" test frameworks.
+
+Although this looks basic today, as the scope of apptest expands and more features are added,


I understand the desire to reuse execution&reporting from existing frameworks but I am not sure this makes total sense because users won't be able to use any assertion functions and the programming experience will likely be confusion.

As an exercise, can you provide a more complete example/prototype of a jest-based test suite that uses this? I am curious how this will look?

I understand the desire to reuse execution&reporting from existing frameworks but I am not sure this makes total sense because users won't be able to use any assertion functions and the programming experience will likely be confusion.

Why wouldn't it make sense to use native testing assertions, outside of the scope of the AWS CDK assertions?

misterjoshua · 2021-10-03T19:21:22Z

text/0031-integ-testing.md

+});
+
+new AwsAssertion(repoTest, 'MessageReceived', {
+  request: AwsAssertionCall.Sqs.receiveMessage({ queue: stack.publishNotifs }),


When does this assertion issue the first request? How does it know when to stop long polling? (Does it send exactly one request?)

How might I handle a situation where the message takes 5 minutes to arrive?

misterjoshua · 2021-10-03T19:49:13Z

text/0031-integ-testing.md

+
+new AwsAssertion(repoTest, 'MessageReceived', {
+  request: AwsAssertionCall.Sqs.receiveMessage({ queue: stack.publishNotifs }),
+  returns: { Messages: [...] }


Are we supporting other than exact matches in returns? If the messages contain id numbers, I may not know those in advance, so I might want a language to match patterns.

Here's something like what we get in expect for jest:

new AwsAssertion(repoTest, 'MessageReceived', { request: AwsAssertionCall.Sqs.receiveMessage({ queue: stack.publishNotifs }), returns: Equals.objectContaining({ Messages: Equals.arrayContaining([ Equals.objectContaining({ foo: 'bar' }), ]), }), });

I'd find this useful for the HttpInvoke as well - on some of my websites, there's smoke if the front page doesn't contain a certain keyword. The test for that might look like:

new HttpInvoke(repoTest, 'MissingKeyword', { endpoint: 'https://example.com', responseBody: Equals.stringContaining('mykeyword'), });

We can definitely support different configuration for these assertions. However, we're most likely going to start with some basic assertions and simple configurations and then build on as needed.

The model is also going to allow for users to bring their own assertions, so if something is not available in the assertion shipped by the CDK, you will be able to bring your own, as well as, publish your own construct libraries with assertions.

misterjoshua · 2021-10-03T20:16:57Z

text/0031-integ-testing.md

+* `InvokeLambda`: invoke an AWS Lambda Function and assert its response.
+* `HttpRequest`: Execute an HTTP request against an API endpoint and assert the response.
+  Used typically to test API Gateway resources.
+


What about these assertions:

AwsAlarmStatusAssertion: a CloudWatch alarm does (or does not) transition to 'ALARM' in a given timeframe.
AwsStepFunctionsAssertion: execute a Step Functions state machine - when it's done, check for a successful execution status.

(The latter could enable the former - here's an example from a similar library)

see #378 (comment)

Niranjan Jayakar added 6 commits August 1, 2021 12:42

drafting...

f43e340

next update

42612c3

remove requirements

77e6669

more

ecd30fc

test execution

126606d

assertions

2a49d5e

nija-at self-assigned this Sep 3, 2021

rix0rrr reviewed Sep 7, 2021

View reviewed changes

nija-at commented Sep 7, 2021

View reviewed changes

text/0031-integ-testing.md Outdated Show resolved Hide resolved

BenChaimberg reviewed Sep 7, 2021

View reviewed changes

skorfmann reviewed Sep 8, 2021

View reviewed changes

eladb reviewed Sep 9, 2021

View reviewed changes

Niranjan Jayakar added 3 commits September 9, 2021 15:23

explained alternative to assertions

c0d6312

snapshots

b0b09f1

more updates

3d09e72

BenChaimberg reviewed Sep 14, 2021

View reviewed changes

Niranjan Jayakar added 2 commits September 16, 2021 15:14

rename to app-test

4806267

bunch of rewrites

3d5ed23

nija-at force-pushed the nija-at/integ-tests branch from 75805e9 to 87a5353 Compare September 16, 2021 17:21

Niranjan Jayakar added 2 commits September 17, 2021 10:22

more changes

0f5d43a

cdk-deploy

b63bdfb

nija-at force-pushed the nija-at/integ-tests branch from 87a5353 to b63bdfb Compare September 17, 2021 10:14

nija-at commented Sep 23, 2021

View reviewed changes

polothy reviewed Sep 27, 2021

View reviewed changes

app owners vs library owners

561b95b

smoke tests

cdaf349

nija-at force-pushed the nija-at/integ-tests branch from 9f07121 to cdaf349 Compare September 28, 2021 15:00

danieldreier mentioned this pull request Sep 28, 2021

Design CDK for Terraform integration testing hashicorp/terraform-cdk#1093

Open

nija-at marked this pull request as ready for review September 29, 2021 12:28

adjustments

44f6b5f

nija-at force-pushed the nija-at/integ-tests branch from fd367e2 to 44f6b5f Compare September 29, 2021 12:32

eladb suggested changes Sep 30, 2021

View reviewed changes

misterjoshua reviewed Oct 3, 2021

View reviewed changes

nija-at mentioned this pull request Oct 14, 2021

(assertions): support snapshot testing of the cloud assembly aws/aws-cdk#16343

Closed

2 tasks

nija-at mentioned this pull request Nov 16, 2021

feat(apigatewayv2): http api - IAM authorizer support aws/aws-cdk#17519

Merged

nija-at removed their assignment Nov 25, 2021

Chriscbr mentioned this pull request Sep 28, 2022

Integration tests that can be run on all targets winglang/wing#205

Closed

nija-at closed this Apr 20, 2023

mrgrain deleted the nija-at/integ-tests branch August 24, 2023 18:38


		### contest

		🌩️ contest tests can verify that your CDK app can be deployed successfully on AWS with the desired resources.

		repoTest.assertAws.
		s3.putObject({ bucket: stack.artifacts, key: '...', body: '...' });

		This feature is beneficial to users who own a CDK app and want to run assertions on the deployed
		app in their pre-production stages as part of a CI/CD pipeline.


		### Credentials

		This module uses the AWS SDK for Javascript under the hood to communicate with CloudFormation.

RFC 31: Integration testing for CDK apps #378

RFC 31: Integration testing for CDK apps #378

Conversation

nija-at commented Sep 3, 2021 • edited

rix0rrr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nija-at commented Sep 8, 2021

skorfmann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nija-at commented Sep 8, 2021

Choose a reason for hiding this comment

nija-at commented Sep 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nija-at Sep 16, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skorfmann commented Sep 22, 2021

skorfmann commented Sep 22, 2021

nija-at Sep 23, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nija-at commented Sep 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterjoshua Oct 3, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterjoshua Oct 3, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nija-at commented Sep 3, 2021 •

edited

nija-at Sep 16, 2021 •

edited

nija-at Sep 23, 2021 •

edited

misterjoshua Oct 3, 2021 •

edited

misterjoshua Oct 3, 2021 •

edited