Introduction
This page details an example architecture around a serverless event driven framework oriented around AWS services.
Analysis
This example serverless application will involve a collaboration portal - passing documents between two or more users - using the AWS subsystem to facilitate upload, retrieval, movement. The simplest version of this collaboration portal would be a stateless messaging system where a source user posts a message/event and a target user consumes it - the event is removed from the system.
There are 3 actors in the system - source user, target user and target system.
There are several goal of the POC - reduced FinOps, Infrastructure as Code and event based serverless applications.
Requirements
Document upload, validation, transfer between 2 users
R | Details |
---|---|
R1 | Portal: API - Provide Northbound API endpoints for the following functions |
R1.1 | Document upload |
R1.2 | Document view |
R1.3 | Document transfer |
R2 | Provide Front end GUI on top of NBI for portal use cases |
R3 | Artifact upload |
R3.1 | Historical artifact upload (on demand/batch) - not required |
R3.2 | Live artifact upload (scheduled/event driven) |
R3.3 | Provide document validation/text parsing capability (Textract) |
R4 | User Management (Add/Modify/Delete) |
Constraints
C | Constraint Type | Details | IaC | Systems |
---|---|---|---|---|
Accessibility | ||||
Access | ||||
Auditing | ||||
Authorization | 2FA supported by KeyCloak MFA/2FA in KeyCloak - https://www.keycloak.org/docs/latest/server_admin/index.html#_webauthn | |||
Authentication | ||||
Encryption in Transit | ||||
Encryption at Rest | ||||
FinOps | Continuous Cost Optimization (added to CI/CD) | |||
Governance | SLA | |||
Identity | Need account creation/governance | AWS Organizations | ||
Logging | ||||
Observability Monitoring | Metrics, healthcheck, error handling auto-scaling triggers | |||
Policies | ||||
Privacy | ||||
Provisioning | ||||
Reporting | ||||
Scaling | ||||
Security - Static analysis | CVE | |||
Security - Dynamic | DDoS, Exploits | |||
Validation - S3 bucket upload | MD5 validation is on by default from AWS SFTP to S3 - but not from the client to the SFTP server. However there is an API to upload the MD5 hash first to SFTP that ensures validation all the way from the client to the S3 bucket |
Throughput
S3 Upload Verification
Provide an initial upload speed test to target various upload speeds and notify the client.
S3 Upload Quadrant
S3 Upload Options
Note: S3 Multipart upload up to 5 MB to 5 TB in x GB parts
Turn on encrypt/decrypt, singed lambda,
Option | Client | AWS end | Max Mbps | Max files | Max 1 file | Max Session size | sync/async | session time | FinOps | Error handling | Test cases | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Reference AWS S3 console (account holder) | browser | S3 | 5 GB | 160 GB 2.3Kb parts | Network error on wired network for 5GB file 25% into 20GB upload | Not recommended for external clients | ||||||
browser - custom aws portal using pre-signed HTTPS PUT | browser | S3 | 5 GB | 160 GB? Multipart | reuse URL for session | Non CLI client limited to 160GB | ||||||
Above using S3 Transfer Acceleration Check elasticache integration | CloudFront | https://aws.amazon.com/premiumsupport/knowledge-center/cloudfront-serve-static-website/ Transfer acceleration is cost effective only when there is a distance between the client and the site. For example in CA we get 0-1% increase in speed to ca-central-a or us-east-2 | ||||||||||
browser - portal | browser | S3 | ||||||||||
Transfer sftp managed | SCP | AWS Transfer to S3 | $0.30 per hour $0.04 /GB (normally 0) | Handles account info | ||||||||
scp custom | SCP | container/vm to S3 | custom work - use SFTP | |||||||||
PrivateLink for S3 | AWS VPN, Site to Site or DirectConnect required | |||||||||||
snowball | Courier to S3 | 3rd party not responsible | ||||||||||
custom s3 CLI sync client | any java/python | S3 | 5 | AmazonS3Client | not recommended install | |||||||
commercial s3 client cloudberry... | commercial | S3 | not recommended install | |||||||||
BitTorrent | Not recommended because of secure distribution |
Discussion as part of
will parallel buckets help on top of multipart upload
https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html
Interfaces
A subset of 1:1 interfaces in the system as we POC out the use cases.
Interface | source | target | Collaborators | diagram | Notes |
---|---|---|---|---|---|
create user | portal | SSO (keycloak) | |||
Use Cases
UC 1: Upload Documents
source user logs into portal documents are uploaded system validates docs, stores them in object storage user page refreshes when documents are validated/stored message sent to target user with storage access key |
UC 2: Transfer Documents
target user logs into portal (via email) document metadata viewed documents pushed to 3rd party system (including option for local) documents removed from system |
Architecture
Interfaces
interface | source | target | collaborators | diagram |
---|---|---|---|---|
load portal page | browser | portal |
| |
create user | portal | SSO keycloak | ||
upload file | portal | storage | ||
view file |
Activity Diagrams
Sequence Diagrams
Design
AWS VPC Architecture
- AWS-10Getting issue details... STATUS
AWS 2 subnet Inventory
Artifact | Dependencies | ID | Parameters | automation | Placement | |
---|---|---|---|---|---|---|
vpc | nacl rt-main dhcp-os | vpc-0110dc64629c21ce3 | 10.0.0.0/16 | VPC wiz | ||
dhcp options set | dopt-737d6711 | VPC wiz | ||||
rt (main) pri | none | rtb-039ae7e0d5c872abf pri | vpc, pri 10.0.0.0/16 = local target 0.0.0.0/0 = nat-06e5b7505848cbcf4 target | VPC wiz | ||
rt pub | pub subnet | rtb-045de6b6456e40599 pub | 10.0.0.0/16 = local target 0.0.0.0/0 = igw-0fc957d91c8b8ab18 route target | VPC wiz | ||
nacl | pub subnet pri subnet | acl-008431e23c1dd6555 | in 0.0.0.0/0 allow out 0.0.0.0/0 allow | VPC wiz | ||
pub subnet | rt acl igw | subnet-07d9a87b786e915f6 | 10.0.0.0/24 us-west-2c | VPC wiz | ||
igw | igw-0fc957d91c8b8ab18 | VPC wiz | ||||
igw-ni | igw-ni | eni-034236b5593034c80 | VPC wiz | |||
Pri subnet | 10.0.1.0/24 us-west-2c | VPC wiz | ||||
NAT-gw | eip pub subnet | nat-06e5b7505848cbcf4 | VPC wiz | |||
nat EIP | eipalloc-0999a0e7468b2e980 | 35.80.198.217 | console | |||
nat ENI | pub subnet | eni-0cedd448f03a3348b | VPC wiz | |||
bastion EIP | prereq | console | ||||
route 53 A record | prereq | console | ||||
EC2 pri test | for testing out the bastion, the nat and the igw | console | ||||
EC2 pri sg | prereq | allopen | console | |||
ssh key | console | |||||
bastion sg | prereq | bastionSg | console | |||
ec2 bastion | prereq | bastion | console | |||
AWS VPC via Console
Prerequisites | Cost | Details | Cloud Formation | SDK | |
---|---|---|---|---|---|
public/private VPC wizard | |||||
AWS account with admin access | |||||
EIP NAT + Bastion | create EIPs for NAT and Bastion nat-uswest2.packet.global and bastion-uswest2.packet.global | ||||
VPC artifacts created | |||||
VPC | 0 | ||||
Public Subnet | 0 | ||||
Private Subnet | 0 | ||||
NAT Gateway | 35/month | ||||
ACL | |||||
Launch Bastion EC2 | |||||
Associate EIP with bastion | |||||
Launch private EC2 - test bastion | test bastion to private instance connectivity and NAT GW egress $ ssh ubuntu@bastion-uswest2.packet.global ubuntu@ip-10-0-0-192:~$ ls _bastion ubuntu@ip-10-0-0-192:~$ exit $ ssh -J ubuntu@bastion-uswest2.packet.global ubuntu@private-uswest2.packet.global ubuntu@ip-10-0-1-96:~$ ls _private ubuntu@ip-10-0-1-96:~$ ping www.google.com PING www.google.com (172.217.3.196) 56(84) bytes of data. 64 bytes from sea15s12-in-f4.1e100.net (172.217.3.196): icmp_seq=1 ttl=97 time=7.20 ms ubuntu@ip-10-0-1-96:~$ wget www.google.com (www.google.com)|172.217.3.196|:80... connected. HTTP request sent, awaiting response... 200 OK index.html [ <=> ] 12.69K --.-KB/s in 0s | ||||
VPC Endpoint | 7.50/month |
AWS VPC via Cloud Formation
see
Design Issues
DI 1: Content validation during S3/SFTP upload
Cloud to Ground and Ground to Cloud#AWSS3SecureFileUploadOptions
AWS S3 Use Cases#S3TransferOptions
DI 2: Investigate Dynamic website via S3 and Lambda
following https://aws.amazon.com/blogs/architecture/create-dynamic-contact-forms-for-s3-static-websites-using-aws-lambda-amazon-api-gateway-and-amazon-ses/
https://docs.aws.amazon.com/lambda/latest/dg/deploying-lambda-apps.html
Lambda | Including generated lambda IAM role REPORT RequestId: b60e1ed6-aeb0-41ac-9449-22751b8d4bdd Duration: 17.45 ms Billed Duration: 18 ms Memory Size: 128 MB Max Memory Used: 64 MB Init Duration: 148.93 ms logs |
Amplify | Design Issues#WebApplicationsonAWSAmplify |
DI 3: Connect SSO to KeyCloak Pod
Reference Architecture#Keycloak
DI 4: Lambda placement VPC public or private
Q) by default where is a lambda function placed? A test would be to verify internet access through the NAT Gateway.
DevOps
Developer Setup
AWS account | Get an account either via the Landing zone - or yourself via https://signin.aws.amazon.com Use us-west-2 (oregon) for manual POC experimentation Use use-west-1 (N. California) for automated Infrastructure as Code - CloudFormation Ohio (us-east-2) is for the other project IaC Virginia (us-east-1) is for manual pocs |
AWS funding | Use the Always free tier limits and request founders funding https://aws.amazon.com/activate/ $1-100K |
AWS CLI | Install the V2 AWS CLI - https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html biometric:~ michaelobrien$ python --version Python 2.7.16 biometric:~ michaelobrien$ python3 --version Python 3.9.4 $ pip --version pip 20.2.3 from c:\users\micha\appdata\local\programs\python\python39\lib\site-packages\pip (python 3.9) https://awscli.amazonaws.com/AWSCLIV2.msi biometric:refarch michaelobrien$ aws --version aws-cli/2.1.29 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off |
CloudFormation | Run CF templates either from the CLI or in the console via S3 upload see https://github.com/obrienlabs/refarch/tree/master/reference-aws-lambda/src/cloudformation biometric:cloudformation michaelobrien$ aws cloudformation deploy --template-file vpc-cfn.yaml --stack-name vpc --region=us-west-1 Waiting for changeset to be created.. Waiting for stack create/update to complete Successfully created/updated stack - vpc update the yaml - rerun deploy to UPDATE the stack delete the stack aws cloudformation delete-stack --stack-name vpc --region=us-west-1 For stacks over 50Kb - upload to S3 first aws s3 mb s3://cf-uswest1-packet-global --region=us-west-1 https://s3.us-west-1.amazonaws.com/cf-us-west-1-packet-global/main.template.yaml |
Terraform | |
Docker |
Deployment
Work Items
WI | ||
---|---|---|
Processing files in Lambda (avoid textract) | ||
S3 upload options via SFTP (AWS Transfer) | AWS S3 Use Cases#S3TransferOptions | |
S3 upload Options via preSigned URLs | https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-UsingHTTPPOST.html Use the following python lambda function (boto will target AWS services) import uuid import boto3 def lambda_handler(event, context): # Get the service client. s3 = boto3.client('s3') # Generate a random S3 key name upload_key = uuid.uuid4().hex # Generate the presigned URL for put requests presigned_url = s3.generate_presigned_url( ClientMethod='put_object', Params={ 'Bucket': 'lambda.input.uswest2.packet.global', 'Key': upload_key }) return {"upload_url": presigned_url} Add default S3 permissions to the IAM role { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": "*" }]}
Run the function { "upload_url": "https://s3.us-west-2.amazonaws.com/lambda.input.uswest2.packet.global/4166f6cae477482ab6b7264fedcea60f?AWSAccessKeyId=ASIA.....D&Signature=UvR..........3D&Expires=1620017193" } Function Logs REPORT RequestId: 5c90fef3-5be0-4da4-8ed0-00ab3d6658fa Duration: 1261.11 ms Billed Duration: 1262 ms Memory Size: 128 MB Max Memory Used: 76 MB Init Duration: 261.36 ms Upload a file biometric:uipath michaelobrien$ curl --request PUT --upload-file ~/Orion_Head_to_Toe.jpg 'https://s3.us-west-2.amazonaws.com/lambda.input.uswest2.packet.global/4166f6...' Check file by downloading and renaming Check email from function "sendMailNodeJs"
| |
S3 upload using API Gateway | Create a bucket for a domain and enable website hosting (http = public, https = not public) http://portals3.cloudlift.systems.s3-website-us-west-2.amazonaws.com/ Set the A record (alias=on) in route53 - select region then the s3 website dropdown https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/RoutingToS3Bucket.html upload S3 objects - make the bucket and the objects public Enable CORS Create OPTIONS method Add 200 Method Response with Empty Response Model to OPTIONS method Add Mock Integration to OPTIONS method Add 200 Integration Response to OPTIONS method Add Access-Control-Allow-Headers, Access-Control-Allow-Methods, Access-Control-Allow-Origin Method Response Headers to OPTIONS method Add Access-Control-Allow-Headers, Access-Control-Allow-Methods, Access-Control-Allow-Origin Integration Response Header Mappings to OPTIONS method Add Access-Control-Allow-Origin Method Response Header to POST method Add Access-Control-Allow-Origin Integration Response Header Mapping to POST method still getting Access to XMLHttpRequest at 'https://o2d2a1yr22.execute-api.us-west-2.amazonaws.com/test' from origin 'http://portals3.cloudlift.systems' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. reran cors config - redeployed api gateway function - that was it If manually editing static html - add the following tag to the very top of the header to enable CORS - reduce scope as required <meta http-equiv="Access-Control-Allow-Origin" content="*"/> http://portals3.cloudlift.systems/lambda.html posts an email using the form | |
S3 upload using AWS Snowball | Detail the entire process after the courier arrives - down to the bucket proxy and lambda triggers |
Links
https://medium.com/@ashan.fernando/upload-files-to-aws-s3-using-signed-urls-fa0a0cf489db
https://sookocheff.com/post/api/uploading-large-payloads-through-api-gateway/
https://www.trek10.com/blog/cloudformation-nested-stacks-primer
https://itnext.io/how-to-build-a-serverless-app-with-s3-and-lambda-in-15-minutes-b14eecd4ea89
1 Comment
Michael O'Brien
s3 transfer rate
Total remaining: 1 file: 502.7 MB(56.11%)
Estimated time remaining: 3 minutes
Transfer rate: 2.8 MB/s
Uploading
53%
Total remaining: 1 file: 420.8 MB(46.97%)
Estimated time remaining: 2 minutes
Transfer rate: 2.9 MB/s
Total remaining: 1 file: 284.1 MB(31.71%)
Estimated time remaining: 2 minutes
Transfer rate: 3.0 MB/s
24