This Proof of Concept (POC) bidding service application will demonstrate the use of AWS Nitro Enclaves to perform computation on multiple sensitive datasets. We will utilize Nitro Enclaves with AWS Key Management Service (KMS) to create an isolated compute environment, allow the environment to process encrypted datasets from multiple parties, and return an output. The POC application will be centered around the scenario of real estate bidding where a bidding service will take in encrypted bids from two buyers and determine the highest bid on each property without disclosing the bid amounts to each buyer. Instructions are given for three different roles: Buyer1, Buyer2, and the Bidding Service. You can create three separate accounts for each role or use a single account for all the roles.
The POC bidding service application is a single python script that contains the implementation for both the parent instance and the enclave instance. The parent instance is responsible for retrieving the encrypted bids from S3, sending a Decrypt message to the enclave instance with the encrypted bids, and writing the output of the Decrypt message to its own S3 bucket. The enclave instance is responsible for calling KMS to decrypt the bids, determining the highest bidder, and returning the output to the parent. Communication between the parent and the enclave is done through a VSockHandler class which handles low level functions of a VSock connection. A detailed look at the code is shown below:
Retrieving encrypted bids from S3.
data = s3client.get_object(Bucket=bucketBuyer1, Key="encrypted.csv")
for row in csv.DictReader(codecs.getreader("utf-8")(data["Body"])):
buyer1data.append(row['[].bid'])
data2 = s3client.get_object(Bucket=bucketBuyer2, Key="encrypted.csv")
for row in csv.DictReader(codecs.getreader("utf-8")(data2["Body"])):
buyer2data.append(row['[].bid'])
The parent will take the encrypted bids and combine them in a Decrypt message that is sent to the enclave through an instance of the VsockHandler class.
result = VsockConnection.request(0,"Decrypt,"+buyer1data[i] + "," + buyer2data[i],True)
When all the results have been received, the results will be aggregrated and written to an output file in the bidding service's S3 bucket.
response = s3client.put_object(
Bucket=bucketBiddingService, Key="output.csv", Body=csv_buffer.getvalue()
)
Decrypting the bids and determining the winner.
plaintext1 = self.decryptText(dataStr[2]);
plaintext2 = self.decryptText(dataStr[3]);
returnMsg = "Buyer1 Wins"
if int(plaintext2)>int(plaintext1):
returnMsg = "Buyer2 Wins"
The decryptText function will call KMS to decrypt the bids through the KMS proxy running on the parent instance.
def decryptText(self, data):
proc = subprocess.Popen(
[
"/usr/src/app/kmstool_enclave_cli",
"--region", "us-west-2",
"--proxy-port", "8000",
"--aws-access-key-id", self.accessKey,
"--aws-secret-access-key", self.secretKey,
"--aws-session-token", self.sessionKey,
"--ciphertext", data,
],
stdout=subprocess.PIPE
)
plaintext = proc.communicate()[0].decode()
base64_bytes = plaintext.encode('ascii')
message_bytes = base64.b64decode(base64_bytes)
message = message_bytes.decode('ascii')
return message
The result of the bidding is returned to the parent through an instance of the VsockHandler class.
self.vsockhandle.request(dataStr[0],returnMsg,False)
The VsockHandler class handles the low level functions of a VSock connction. This includes having a separate thread for sending and receiving messages, message queuing, and optional waiting on responses.
Separate threads for sending and receiving messages. The threads will utilize message queues in the VsockHandler class to track sent and received messages.
def listener_vsock_thread(self, port):
self.listenerObject = VsockListener(self)
self.listenerObject.bind(port)
self.listenerObject.recv_data(self.enclave,port)
def sender_vsock_thread(self, cid, port):
while self.run:
for i in list(self.requestQueue):
self.requestQueue.remove(i);
msg = str(i.msgID) + "," + i.msg
print("SEND: Sending msg: "+msg+" to "+str(cid)+":"+str(port))
client = VsockStream(self)
endpoint = (cid, port)
client.connect(endpoint)
client.send_data(msg.encode())
time.sleep(0.5)
Message queuing to ensure message delivery ordering.
self.requestQueue.append(socketMessage(msgID,msg))
...
self.responseQueue.append(socketMessage(msgID,msg))
When sending a message, waiting on the response is controlled by the waitResp parameter. This is useful as the parent needs to wait on the response from the enclave for a Decypt message while the enclave would not need to wait on the response from the parent when sending the result of the bidding.
def request(self, msgID, msg, waitResp):
if waitResp:
msgID = self.requestID
self.requestID += 1
self.requestQueue.append(socketMessage(msgID,msg))
msgNotReady = waitResp
returnMsg = ""
while msgNotReady:
for i in list(self.responseQueue):
if int(i.msgID) == int(msgID):
returnMsg = i.msg
self.responseQueue.remove(i)
msgNotReady = False
time.sleep(0.5)
return returnMsg
- AWS CLI
- One or more AWS Account(s)
We will be creating AWS Resources for each of the roles and then creating our encrypted bid files which will be used during the bidding process.
- Create a S3 bucket. See these instructions Note the name and ARN.
- Create a KMS Customer managed key (CMK). See these instructions for more details. Note the KeyID and ARN of this key.
- In this POC we will be making three bids on three different properties. Determine your bids for each property and then encrypt the bids using aws-cli. See AWS documentation for more details about aws-cli. For the example I will be bidding $100,000 on the first property, $200,000 on the second property, and $150,000 on the third property.
aws kms encrypt --key-id <KMS CMK KeyID> --cli-binary-format raw-in-base64-out --plaintext "100000"
Returns: <Encrypted Bid 1>
aws kms encrypt --key-id <KMS CMK KeyID> --cli-binary-format raw-in-base64-out --plaintext "200000"
Returns: <Encrypted Bid 2>
aws kms encrypt --key-id <KMS CMK KeyID> --cli-binary-format raw-in-base64-out --plaintext "150000"
Returns: <Encrypted Bid 3>
If you need to verify the bid is encrypted correctly, you can use the following command to decrypt it:
aws kms decrypt --key-id <KMS CMK KeyID> --ciphertext-blob "<Encrypted Bid>" | jq -r .Plaintext | base64 --decode
- Now create a file called encrypted.csv:
[].contract,[].bid
1,<Encrypted Bid 1>
2,<Encrypted Bid 2>
3,<Encrypted Bid 3>
Alternatively, you can run
bash scripts/generate_bidder_1_bids.shfor step 3 and 4.
- Place this file in the S3 bucket you created earlier.
Buyer2 should repeat the steps above to create their AWS resources and generate their own file. Ensure the bid amounts are different.
- Create a S3 bucket. See these instructions
- Note the name and ARN.
- Follow these instructions to create an IAM EC2 instance role.
- Note the ARN for the IAM role.
At this point you should have the following ARNs from the previous steps:
- Buyer1 BUCKET ARN: S3 bucket arn for Buyer1 account.
- Buyer2 BUCKET ARN: S3 bucket arn for Buyer2 account.
- Bidding Service BUCKET ARN: S3 bucket arn for Bidding Service account.
- INSTANCE ROLE ARN: IAM role arn assigned to EC2 instance.
- Buyer1 KMS CMK ARN: KMS CMK arn for Buyer1.
- Buyer2 KMS CMK ARN: KMS CMK arn for Buyer2.
- Follow these instructions to add the following bucket policy to the Buyer's S3 bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "<INSTANCE ROLE ARN>"
},
"Action": "s3:*",
"Resource": "<Buyer's BUCKET ARN>/*"
}
]
}
- Follow these instructions to add the following IAM policy to the IAM EC2 instance role:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"sts:AssumeRole",
"kms:Decrypt"
],
"Resource": [
"<Buyer1 KMS CMK ARN>",
"<Buyer2 KMS CMK ARN>",
"<Buyer1 BUCKET ARN>/*",
"<Buyer2 BUCKET ARN>/*",
"<Bidding Service BUCKET ARN>/*",
"<INSTANCE ROLE ARN>"
]
}
]
}
- Follow these instructions to add the following trust policy to the IAM EC2 instance role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com",
"AWS": "<INSTANCE ROLE ARN>"
},
"Action": "sts:AssumeRole"
}
]
}
- Launch an EC2 instance with the following settings:
- AMI: Amazon Linux 2 AMI (HVM), SSD Volume Type, x86
- Instance type: c5.2xlarge
- IAM role: Choose the instance role created earlier
- Enclave: Enable
- Login to the EC2 instance.
- Setup Docker and nitro-cli
sudo amazon-linux-extras install -y docker
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -a -G docker ec2-user
sudo amazon-linux-extras enable aws-nitro-enclaves-cli
sudo yum install -y aws-nitro-enclaves-cli aws-nitro-enclaves-cli-devel
- Allocate more memory to Nitro Enclaves by modifying /etc/nitro_enclaves/allocator.yaml:
memory_mib: 4096
- Run this command to allocate the memory
sudo systemctl start nitro-enclaves-allocator.service && sudo systemctl enable nitro-enclaves-allocator.service
- Setup Python3 and Pip
sudo yum install -y python3 python3-pip
- Install Python dependencies
sudo pip3 install boto3 pandas
- Start the vsock-proxy to allow KMS communication from the Enclave
sudo systemctl start nitro-enclaves-vsock-proxy.service
sudo systemctl enable nitro-enclaves-vsock-proxy.service
- Install git and clone this repository.
sudo yum install -y git
git clone https://github.com/aws-samples/aws-nitro-enclaves-bidding-service.git
- Build the kmstool-enclave-cli by following the instructions here: https://github.com/aws/aws-nitro-enclaves-sdk-c/tree/main/bin/kmstool-enclave-cli
- After building the kmstool-enclave-cli, copy kmstool_enclave_cli and libnsm.so to your nitro-enclave-bidding-service directory.
- Modify vsock-poc.py with the S3 bucket names and the IAM EC2 instance role. Note that the bucket names are defined instead of the ARNs.
...
bucketBuyer1 = "<Buyer1 BUCKET NAME>"
bucketBuyer2 = "<Buyer2 BUCKET NAME>"
bucketBiddingService = "<Bidding Service BUCKET NAME>"
instanceRoleARN = "<INSTANCE ROLE ARN>"
...
Also remember to modify the AWS region in the file if you are not using the default region.
For step 5 to 7, you can run
script/update_enclave.sh.
- Build the container
docker build -t vsock-poc .
- Build the Enclave image
sudo nitro-cli build-enclave --docker-uri vsock-poc --output-file ~/vsock_poc.eif
If successful you should see output similar to below:
Enclave Image successfully created.
{ "Measurements":
{ "HashAlgorithm": "Sha384 { ... }",
"PCR0": "287b24930a9f0fe14b01a71ecdc00d8be8fad90f9834d547158854b8279c74095c43f8d7f047714e98deb7903f20e3dd",
"PCR1": "aca6e62ffbf5f7deccac452d7f8cee1b94048faf62afc16c8ab68c9fed8c38010c73a669f9a36e596032f0b973d21895",
"PCR2": "0315f483ae1220b5e023d8c80ff1e135edcca277e70860c31f3003b36e3b2aaec5d043c9ce3a679e3bbd5b3b93b61d6f"
}
}
- Save the PCR values for setting up the KMS key policies later in this document.
- Follow these instructions to modify the KMS CMK key policy for each Buyer. You will want to add the following statement to the key policy, or replace the existing section with
"Sid": "Allow use of the key",. Note that the value of kms:RecipientAttestation:ImageSha384 is a series of zeroes as we will be running in debug mode for troubleshooting purposes:
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "<INSTANCE ROLE ARN>"
},
"Action": "kms:Decrypt",
"Resource": "*",
"Condition": {
"StringEqualsIgnoreCase": {
"kms:RecipientAttestation:ImageSha384": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
}
}
}
- Start the Enclave in debug mode. In debug mode, you will beable to connect to the enclave's console to view its output for troubleshooting. Also the PCR0 value sent to KMS will be a series of zeroes used instead of the actual PCR0 value.
sudo nitro-cli run-enclave --eif-path ~/vsock_poc.eif --cpu-count 2 --memory 4096 --debug-mode
If successful, you will see similar output below:
Start allocating memory...
Started enclave with enclave-cid: 19, memory: 4096 MiB, cpu-ids: [1, 5]
{
"EnclaveName": "vsock_poc",
"EnclaveID": "i-0c3d696ac3c1f00dc-enc1802f51427d0db9",
"ProcessID": 18647,
"EnclaveCID": 19,
"NumberOfCPUs": 2,
"CPUIDs": [
1,
5
],
"MemoryMiB": 4096
}
Note the EnclaveID and EnclaveCID.
- Connect to the enclave console using the EnclaveID:
sudo nitro-cli console --enclave-id <EnclaveID>
This console will continuously print out the logs from inside the enclave. If you see similar output below, the application is ready to receive requests:
...
[ 0.807515] nsm: loading out-of-tree module taints kernel.
[ 0.807870] nsm: module verification failed: signature and/or required key missing - tainting kernel
[ 0.812592] random: python3: uninitialized urandom read (24 bytes read)
Starting VsockConnection
- Open another SSH session (or use a terminal multiplexer like tmux to create two separate terminals). Run the parent instance application
python3 vsock-poc.py parent <EnclaveCID> 5005
Output:
Starting VsockConnection
SEND: Sending msg: 0,SetCredential,<CredentialData> to 19:5005
RECEIVE: 0,1
Property 1
SEND: Sending msg: 1,Decrypt,<Buyer1 Property 1 encrypted bid>,<Buyer2 Property 1 encrypted bid> to 19:5005
RECEIVE: 1,Buyer1 Wins
Result: Buyer1 Wins
Property 2
SEND: Sending msg: 2,Decrypt,<Buyer1 Property 2 encrypted bid>,<Buyer2 Property 2 encrypted bid> to 19:5005
RECEIVE: 2,Buyer2 Wins
Result: Buyer2 Wins
Property 3
SEND: Sending msg: 3,Decrypt,<Buyer1 Property 3 encrypted bid>,<Buyer2 Property 3 encrypted bid> to 19:5005
RECEIVE: 3,Buyer1 Wins
Result: Buyer1 Wins
Successful S3 put_object response. Status - 200
- An output file: output.csv should have been generated in the Bidding Service S3 bucket.
- Change the key policies to use the actual PCR measurement values generated from the enclave image.
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "<INSTANCE ROLE ARN>"
},
"Action": "kms:Decrypt",
"Resource": "*",
"Condition": {
"StringEqualsIgnoreCase": {
"kms:RecipientAttestation:ImageSha384": "<PCR0 VALUE>",
"kms:RecipientAttestation:PCR1":"<PCR1 VALUE>",
"kms:RecipientAttestation:PCR2":"<PCR2 VALUE>"
}
}
}
You can run
script/generate_key_policy.shto generate the above policy with PCRs pre-filled. Remember to replace the<INSTANCE ROLE ARN>manually.
- Repeat the steps for running the bidding service application but remove the debug-mode flag. You will also not be able to connect to the enclave console.
sudo nitro-cli run-enclave --eif-path ~/vsock_poc.eif --cpu-count 2 --memory 4096
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.