Page tree
Skip to end of metadata
Go to start of metadata

Introduction

This page details an example architecture around a serverless event driven framework oriented around AWS services.

Analysis

This example serverless application will involve a collaboration portal - passing documents between two or more users - using the AWS subsystem to facilitate upload, retrieval, movement.  The simplest version of this collaboration portal would be a stateless messaging system where a source user posts a message/event and a target user consumes it - the event is removed from the system.

There are 3 actors in the system - source user, target user and target system.

There are several goal of the POC - reduced FinOps, Infrastructure as Code and event based serverless applications.

Requirements

Document upload, validation, transfer between 2 users 

RDetails
R1Portal: API - Provide Northbound API endpoints for the following functions
R1.1Document upload
R1.2Document view
R1.3Document transfer
R2Provide Front end GUI on top of NBI for portal use cases
R3Artifact upload
R3.1Historical artifact upload (on demand/batch) - not required
R3.2Live artifact upload (scheduled/event driven)
R3.3Provide document validation/text parsing capability (Textract)
R4User Management (Add/Modify/Delete)



Constraints


CConstraint TypeDetailsIaCSystems

Accessibility



Access



Auditing



Authorization

2FA supported by KeyCloak
Reference Architecture#KeyCloak

https://www.keycloak.org/

MFA/2FA in KeyCloak - https://www.keycloak.org/docs/latest/server_admin/index.html#_webauthn




Authentication



Encryption in Transit



Encryption at Rest



FinOpsContinuous Cost Optimization (added to CI/CD)


GovernanceSLA


IdentityNeed account creation/governanceAWS Organizations

Logging



Observability
Monitoring

Metrics, healthcheck, error handling

auto-scaling triggers




Policies



Privacy



Provisioning



Reporting



Scaling



Security - Static analysisCVE


Security - DynamicDDoS, Exploits


Validation - S3 bucket uploadMD5 validation is on by default from AWS SFTP to S3 - but not from the client to the SFTP server.  However there is an API to upload the MD5 hash first to SFTP that ensures validation all the way from the client to the S3 bucket


Throughput

S3 Upload Verification

Provide an initial upload speed test to target various upload speeds and notify the client.

S3 Upload Quadrant












S3 Upload Options

AWS S3 Use Cases

Note: S3 Multipart upload up to 5 MB to 5 TB in x GB parts
Turn on encrypt/decrypt, singed lambda, 

OptionClientAWS endMax MbpsMax filesMax
1 file
Max Session
size
sync/asyncsession timeFinOpsError handlingTest cases
Reference AWS S3 console
(account holder)
browserS3

5 GB160 GB
2.3Kb parts




Network error on wired network for 5GB file 25% into 20GB uploadNot recommended for external clients
browser - custom aws portal
using pre-signed HTTPS PUT
browserS3

5 GB160 GB?
Multipart

reuse URL for session


Non CLI client limited to 160GB
Above using S3 Transfer Acceleration
Check elasticache integration

CloudFront








https://aws.amazon.com/premiumsupport/knowledge-center/cloudfront-serve-static-website/

Transfer acceleration is cost effective only when there is a distance between the client and the site.  For example in CA we get 0-1% increase in speed to ca-central-a or us-east-2

browser -  portalbrowserS3









Transfer sftp managedSCPAWS Transfer
to S3







$0.30 per hour
$0.04 /GB (normally 0)

Handles account info
scp customSCPcontainer/vm
to S3









custom work - use SFTP
PrivateLink for S3










AWS VPN, Site to Site or DirectConnect required
snowball
Courier to
S3









3rd party not responsible 
custom s3 CLI sync clientany java/pythonS3

5




AmazonS3Clientnot recommended install
commercial s3 client
cloudberry...
commercialS3








not recommended install
BitTorrent










Not recommended because of secure distribution

Discussion as part of 

will parallel buckets help on top of multipart upload

https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html

Interfaces

A subset of 1:1 interfaces in the system as we POC out the use cases.

InterfacesourcetargetCollaboratorsdiagramNotes
create userportalSSO (keycloak)








Use Cases

UC 1: Upload Documents




source user logs into portal

documents are uploaded

system validates docs, stores them in object storage

user page refreshes when documents are validated/stored

message sent to target user with storage access key

UC 2: Transfer Documents




target user logs into portal (via email)

document metadata viewed

documents pushed to 3rd party system (including option for local)

documents removed from system


Architecture

Interfaces

interfacesourcetargetcollaboratorsdiagram
load portal pagebrowserportal

static-site-serving

create userportalSSO keycloak

upload fileportalstorage

view file



Activity Diagrams


Sequence Diagrams


Design


AWS VPC Architecture

AWS-10 - Getting issue details... STATUS

AWS 2 subnet Inventory

ArtifactDependenciesIDParametersautomationPlacement
vpcnacl
rt-main
dhcp-os
vpc-0110dc64629c21ce310.0.0.0/16VPC wiz

dhcp options set
dopt-737d6711
VPC wiz

rt
(main)
pri
nonertb-039ae7e0d5c872abf
pri
vpc, pri
10.0.0.0/16 = local target
0.0.0.0/0 = nat-06e5b7505848cbcf4 target
VPC wiz

rt
pub
pub subnetrtb-045de6b6456e40599
pub
10.0.0.0/16 = local target
0.0.0.0/0 = igw-0fc957d91c8b8ab18 route target
VPC wiz

naclpub subnet
pri subnet
acl-008431e23c1dd6555in 0.0.0.0/0 allow
out 0.0.0.0/0 allow
VPC wiz

pub
subnet
rt
acl
igw
subnet-07d9a87b786e915f610.0.0.0/24
us-west-2c
VPC wiz

igw
igw-0fc957d91c8b8ab18

VPC wiz

igw-niigw-nieni-034236b5593034c80
VPC wiz

Pri
subnet


10.0.1.0/24
us-west-2c
VPC wiz

NAT-gweip
pub subnet
nat-06e5b7505848cbcf4
VPC wiz

nat EIP
eipalloc-0999a0e7468b2e98035.80.198.217console

nat ENIpub subnet

eni-0cedd448f03a3348b


VPC wiz

bastion EIPprereq


console

route 53
A record
prereq


console

EC2 pri
test


for testing out the bastion, the nat and the igw

console

EC2 pri sgprereqallopen
console

ssh key


console

bastion
sg
prereqbastionSg
console

ec2 bastionprereqbastion
console









AWS VPC via Console







Prerequisites CostDetails
Cloud FormationSDK
public/private VPC wizard





AWS account with admin access




EIP NAT + Bastion
create EIPs for NAT and Bastion
nat-uswest2.packet.global and bastion-uswest2.packet.global



VPC artifacts created




VPC0



Public Subnet0



Private Subnet0



NAT Gateway35/month



ACL










Launch Bastion EC2




Associate EIP with bastion




Launch private EC2 - test bastion

test bastion to private instance connectivity and NAT GW egress

$ ssh ubuntu@bastion-uswest2.packet.global
ubuntu@ip-10-0-0-192:~$ ls
_bastion
ubuntu@ip-10-0-0-192:~$ exit
$ ssh -J ubuntu@bastion-uswest2.packet.global ubuntu@private-uswest2.packet.global

ubuntu@ip-10-0-1-96:~$ ls
_private
ubuntu@ip-10-0-1-96:~$ ping www.google.com
PING www.google.com (172.217.3.196) 56(84) bytes of data.
64 bytes from sea15s12-in-f4.1e100.net (172.217.3.196): icmp_seq=1 ttl=97 time=7.20 ms
ubuntu@ip-10-0-1-96:~$ wget www.google.com (www.google.com)|172.217.3.196|:80... connected.
HTTP request sent, awaiting response... 200 OK
index.html [ <=> ] 12.69K --.-KB/s in 0s






VPC Endpoint 7.50/month



AWS VPC via Cloud Formation

see 












Design Issues

DI 1: Content validation during S3/SFTP upload

Cloud to Ground and Ground to Cloud#AWSS3SecureFileUploadOptions

AWS S3 Use Cases#S3TransferOptions

DI 2: Investigate Dynamic website via S3 and Lambda

following https://aws.amazon.com/blogs/architecture/create-dynamic-contact-forms-for-s3-static-websites-using-aws-lambda-amazon-api-gateway-and-amazon-ses/
https://docs.aws.amazon.com/lambda/latest/dg/deploying-lambda-apps.html

https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/




Lambda

Including

generated lambda IAM role

testing

https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/eventStreamPortalProcessNodejs?tab=monitoring

REPORT RequestId: b60e1ed6-aeb0-41ac-9449-22751b8d4bdd	Duration: 17.45 ms	Billed Duration: 18 ms	Memory Size: 128 MB	Max Memory Used: 64 MB	Init Duration: 148.93 ms


logs

Amplify

Design Issues#WebApplicationsonAWSAmplify



DI 3: Connect SSO to KeyCloak Pod

Reference Architecture#Keycloak

DI 4: Lambda placement VPC public or private

Q) by default where is a lambda function placed?  A test would be to verify internet access through the NAT Gateway.


DevOps

Developer Setup




AWS account

 Get an account either via the Landing zone - or yourself via  https://signin.aws.amazon.com 

Use us-west-2 (oregon) for manual POC experimentation

Use use-west-1 (N. California) for automated Infrastructure as Code - CloudFormation

Ohio (us-east-2) is for the other project IaC

Virginia (us-east-1) is for manual pocs

AWS fundingUse the Always free tier limits and request founders funding
https://aws.amazon.com/activate/ $1-100K
AWS CLI

Install the V2 AWS CLI - https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
Python first (optional) 3.9.2 - https://www.python.org/downloads/

biometric:~ michaelobrien$ python --version
Python 2.7.16
biometric:~ michaelobrien$ python3 --version
Python 3.9.4
$ pip --version
pip 20.2.3 from c:\users\micha\appdata\local\programs\python\python39\lib\site-packages\pip (python 3.9)

https://awscli.amazonaws.com/AWSCLIV2.msi

biometric:refarch michaelobrien$ aws --version
aws-cli/2.1.29 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off
CloudFormation

Run CF templates either from the CLI or in the console via S3 upload

see https://github.com/obrienlabs/refarch/tree/master/reference-aws-lambda/src/cloudformation

biometric:cloudformation michaelobrien$ aws cloudformation deploy --template-file vpc-cfn.yaml --stack-name vpc --region=us-west-1
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - vpc
update the yaml - rerun deploy to  UPDATE  the stack
delete the stack
aws cloudformation delete-stack --stack-name vpc --region=us-west-1

For stacks over 50Kb - upload to S3 first
aws s3 mb s3://cf-uswest1-packet-global --region=us-west-1


https://s3.us-west-1.amazonaws.com/cf-us-west-1-packet-global/main.template.yaml
Terraform
Docker

Deployment

Work Items


WI

Processing files in Lambda (avoid textract)

https://docs.aws.amazon.com/pinpoint/latest/developerguide/tutorials-importing-data-lambda-function-process-incoming.html



S3 upload options via SFTP (AWS Transfer)AWS S3 Use Cases#S3TransferOptions
S3 upload Options via preSigned URLs

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-UsingHTTPPOST.html

Use the following python lambda function (boto will target AWS services)

import uuid
import boto3
def lambda_handler(event, context):
    # Get the service client.
    s3 = boto3.client('s3')
    # Generate a random S3 key name
    upload_key = uuid.uuid4().hex
    # Generate the presigned URL for put requests
    presigned_url = s3.generate_presigned_url(
        ClientMethod='put_object',
        Params={
            'Bucket': 'lambda.input.uswest2.packet.global',
            'Key': upload_key
        })
    return {"upload_url": presigned_url}


Add default S3 permissions to the IAM role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }]}


Create an API Gateway REST endpoint - for now via console - later via OpenAPI import via CloudFormation



Run the function
https://kvzryfi6hc-xb71.execute-api.us-west-2.amazonaws.com/test

{
  "upload_url": "https://s3.us-west-2.amazonaws.com/lambda.input.uswest2.packet.global/4166f6cae477482ab6b7264fedcea60f?AWSAccessKeyId=ASIA.....D&Signature=UvR..........3D&Expires=1620017193"
}

Function Logs
REPORT RequestId: 5c90fef3-5be0-4da4-8ed0-00ab3d6658fa	Duration: 1261.11 ms	Billed Duration: 1262 ms	Memory Size: 128 MB	Max Memory Used: 76 MB	Init Duration: 261.36 ms

Upload a file

biometric:uipath michaelobrien$ curl --request PUT --upload-file ~/Orion_Head_to_Toe.jpg 'https://s3.us-west-2.amazonaws.com/lambda.input.uswest2.packet.global/4166f6...'


Check file by downloading and renaming


Check email from function "sendMailNodeJs"


Email


S3 upload using API Gateway

Create a bucket for a domain and enable website hosting (http = public, https = not public)

http://portals3.cloudlift.systems.s3-website-us-west-2.amazonaws.com/

Set the A record (alias=on) in route53 - select region then the s3 website dropdown

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/RoutingToS3Bucket.html

upload S3 objects - make the bucket and the objects public


Enable CORS


Create OPTIONS method
Add 200 Method Response with Empty Response Model to OPTIONS method
Add Mock Integration to OPTIONS method
Add 200 Integration Response to OPTIONS method
Add Access-Control-Allow-Headers, Access-Control-Allow-Methods, Access-Control-Allow-Origin Method Response Headers to OPTIONS method
Add Access-Control-Allow-Headers, Access-Control-Allow-Methods, Access-Control-Allow-Origin Integration Response Header Mappings to OPTIONS method
Add Access-Control-Allow-Origin Method Response Header to POST method
Add Access-Control-Allow-Origin Integration Response Header Mapping to POST method

still getting
Access to XMLHttpRequest at 'https://o2d2a1yr22.execute-api.us-west-2.amazonaws.com/test' from origin 'http://portals3.cloudlift.systems' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
reran cors config - redeployed api gateway function - that was it

If manually editing static html - add the following tag to the very top of the header to enable CORS - reduce scope as required

<meta http-equiv="Access-Control-Allow-Origin" content="*"/>


http://portals3.cloudlift.systems/lambda.html posts an email using the form

https://github.com/obrienlabs/eventstream/commit/04086ff120864bb8bb1a17485db83ba6ed25fbef#diff-d140e2a59c8e94736cd57bb401dfe78a545cbac82ea87d5b42f2d866eff69582R4







S3 upload using AWS Snowball

Detail the entire process after the courier arrives - down to the bucket proxy and lambda triggers



Links

https://medium.com/@ashan.fernando/upload-files-to-aws-s3-using-signed-urls-fa0a0cf489db

https://sookocheff.com/post/api/uploading-large-payloads-through-api-gateway/

https://www.trek10.com/blog/cloudformation-nested-stacks-primer

https://itnext.io/how-to-build-a-serverless-app-with-s3-and-lambda-in-15-minutes-b14eecd4ea89



  • No labels

1 Comment

  1. s3 transfer rate



    Total remaining: 1 file: 502.7 MB(56.11%)

    Estimated time remaining: 3 minutes

    Transfer rate: 2.8 MB/s



    Uploading



    53%

    Total remaining: 1 file: 420.8 MB(46.97%)

    Estimated time remaining: 2 minutes

    Transfer rate: 2.9 MB/s


    Total remaining: 1 file: 284.1 MB(31.71%)

    Estimated time remaining: 2 minutes

    Transfer rate: 3.0 MB/s


    24