Provide Temporary AWS Credentials limited to single S3 Bucket Upload and Download
20210320:1600
The driver for this use case it to prototype AWS Organizations and Landing Zone automation around providing 2 S3 limited accounts on AWS - one account to read/write to specific S3 bucket - the second account with S3 bucket read only permissions.
Options for S3 access
| Access | |
|---|---|
| IAM (quick but insecure) | https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:getBucketVersioning",
"s3:*"
],
"Resource": "arn:aws:s3:::vesna2021"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject" ],
"Resource": "arn:aws:s3:::vesna2021/*"
}
]}
|
| Full console access | |
| Full CLI access | |
| AWS Security Token Service | https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html |
| Temporary Credentials and Signed URLs | https://medium.com/@ashan.fernando/upload-files-to-aws-s3-using-signed-urls-fa0a0cf489db |
Options for S3 Upload
| Option | Details | Example |
|---|---|---|
AWS CLI s3 put | Include muiltipart upload (perf) https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html | |
AWS Console (not recommended) | need temporary access creds | |
| Transfer SFTP | finops: $0.30 per hour + $0.04 /GB (normally 0) https://aws.amazon.com/blogs/aws/new-aws-transfer-for-sftp-fully-managed-sftp-service-for-amazon-s3/
|
Options for S3 Download
| Options | Details | Example |
|---|---|---|
| AWS CLI s3 cp | ||
Create S3 bucket
// get ARNarn:aws:s3:::bucket2021
Create IAM Users
Provide S3 Upload URLs
Provide S3 Download URLs
Upload Large Datasets to S3
S3 Upload Options
Use the same region for S3 and consuming services like EC2
https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html
AWS CLI
https://aws.amazon.com/premiumsupport/knowledge-center/s3-upload-large-files/
https://aws.amazon.com/s3/pricing/
AWS S3 Transfer Accelleration
AWS SFTP
https://aws.amazon.com/aws-transfer-family/pricing/
Costly
AWS Snowball