Requirements
I need to bring up a system that uses the pre-kubernetes ECS as the base for a set of java based microservices. I am constrained from using either a native kubernetes cluster (using RKE) or the managed AWS EKS elastic kubernetes service.
R# | A# | Key | Details |
---|---|---|---|
R1 | Deployment is unaffected by OSX/Windows Dev environments | ||
R7 | ECS | ECS cluster using EC2 | |
R7.1 | in/out | uService containers can communicate privately | |
uService containers are reachable from the NLB | |||
uService containers are reachable from the API-GW | |||
The ECS cluster must be in an auto scaling group | |||
R9 | VPC | The ECS cluster must be in a VPC with private/public subnets | |
R10 | API-GW | The public facing URLs must be through the AWS API Gateway | |
R10.1 | The ECS cluster is fronted by an NLB to work with the API Gateway (not an ALB) | ||
R10.2 | The public facing URLs must be obfuscated from the real ECS NLB urls using a reverse proxy inside the subnet | ||
R11 | |||
Work Items
WI | Description | Jira |
---|---|---|
Spring boot application | ||
Docker build/deploy infrastructure | ||
AWS Cloud infrastructure/Automation | ||
Add systems manager agent manually to ec2 instances amzn2-ami-ecs-hvm-2.0.20191212-x86_64-ebs (ami-00afc256a955c31b5) | ||
Upgrade spring boot from 2.1.3 to 2.2.4 |
Repositories
Github
Analysis
Architecture
ECS AWS Infrastructure Diagram
20200114
Angular Front End
Spring Boot Application Backend
AWS ECS infrastructure
ECS Service Creation
In order of dependency
Step | |||
---|---|---|---|
Pre | Create ECS Cluster | ||
Create ecsTaskExecutionRole 20191222-4:CreatecustomcontainerecsTaskExecutionRole | |||
Design
ECS Network Design
We have 3 options 1= (all containers in a single task (same as a pod in kubernetes) - they use localhost because of the private namespace), - not recommended 2=route53 service discovery (I have this enabled as a secondary option - A records (per ENI) are maintained by the scheduler) and 3=awsvpc (essentially a simulated localhost namespace of a single task by launching the pause container (same thing as in Kubernetes) first - it gets associated with the 2nd trunk ENI and all other containers are launched into it's private namespace)
ECS Task Networking
see https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html and https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#network_mode and
ECS Bridge Networking Mode
ECS Host Networking Mode
ECS Awsvpc Networking Mode
DI 1: Base Case 1: 2 node EC2 ECS cluster in a VPC with a single public subnet - verify uService comms
DI 2: Base Case 2: 2 node EC2 ECS cluster in a VPC with public/private subnets
Implementation
Infrastructure
Quickstart Sandbox
Existing 2 subnet VPC with IG, Nat
Enable awsVpctrunking on ECS settings - to enable awsvpc network mode
Create ECS cluster with at least 2 instances
Testing
Implementation Timeline
20191215-1: Minimal Maven project structure for a Spring Boot Application
The microservice exercising the ECS project will start as a standard maven java 11 spring boot project.
Initial checkin branch for a clean project compile and spring boot run under
-
ECS-4Getting issue details...
STATUS
https://github.com/obrienlabs/ecs-app-source/issues/1
https://github.com/obrienlabs/ecs-app-source/commit/225cdd21e5a57eae156210ff8ceb9d6f29c0687b
Run NbiApplication as spring boot application on OSX
:: Spring Boot :: (v2.1.3.RELEASE) 2019-12-15 11:36:17.673 INFO 8724 --- [ restartedMain] systems.cloudlift.nbi.NbiApplication : Starting NbiApplication on biometric.local with PID 8724 (/Users/michaelobrien/wse_github/ecs-app-source/ecs-app-source-nbi/target/classes started by michaelobrien in /Users/michaelobrien/wse_github/ecs-app-source/ecs-app-source-nbi) 2019-12-15 11:36:17.902 INFO 8724 --- [ restartedMain] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.16] 2019-12-15 11:36:18.077 INFO 8724 --- [ restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '/nbi' 2019-12-15 11:36:18.077 INFO 8724 --- [ restartedMain] systems.cloudlift.nbi.NbiApplication : Started NbiApplication in 0.714 seconds (JVM running for 215.995) 2019-12-15 11:36:33.278 INFO 8724 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 3 ms 1 systems.cloudlift.nbi.ApiController
Check the REST API via curl
biometric:main michaelobrien$ curl http://127.0.0.1:8080/nbi/api {"id":1,"content":"PASS"}
Run NbiApplication on Windows 10
PS C:\WINDOWS\system32> curl http://127.0.0.1:8080/nbi/api StatusCode : 200 Content : {"id":1,"content":"PASS"}
20191215-2: Add Docker Build and Deploy Infrastructure
Preliminary testing
DockerFile
For a Java 11 LTS based spring boot application use the openjdk base docker image - not the alpine one that stops at Java 8
#FROM anapsix/alpine-java:jre8 FROM openjdk:11 ARG USERVICE_HOME=/opt/app/ # Build up the deployment folder structure RUN mkdir -p $USERVICE_HOME ADD ecs-app-source-nbi-*.jar $USERVICE_HOME/lib/ecs-app-source-nbi.jar ADD startService.sh $USERVICE_HOME/bin/ CMD ["/opt/app/bin/startService.sh"]
startService.sh
using a separate script is optional for the entrypoint but decouples the start allowing for multiple sh commands on image start.
#!/bin/bash cd /opt/app #-Dlogging.config=config/logback.xml if [ -z "${java_runtime_arguments}" ]; then java -Xms128m -Xmx1536m -jar /opt/app/lib/ecs-app-source-nbi.jar else java $java_runtime_arguments -jar /opt/app/lib/ecs-app-source-nbi.jar fi
Build script prototype
The docker build folder needs to be outside of the src folder to avoid git picking them up.
#!/bin/bash BUILD_ID=10001 mkdir ../../docker TARGET_DIR=../../docker/$BUILD_ID mkdir $TARGET_DIR CONTAINER_IMAGE=ecs-app-source-nbi cp ../../target/*.jar $TARGET_DIR cp DockerFile $TARGET_DIR cp startService.sh $TARGET_DIR cd $TARGET_DIR docker build --no-cache --build-arg build-id=$BUILD_ID -t $CONTAINER_IMAGE -f DockerFile . docker tag $CONTAINER_IMAGE:latest $CONTAINER_IMAGE:latest docker stop $CONTAINER_IMAGE docker rm $CONTAINER_IMAGE echo "starting: $CONTAINER_IMAGE" docker run --name $CONTAINER_IMAGE -d -p 8888:8080 -e os.environment.configuration.dir=/ -e os.environment.ecosystem=sbx $CONTAINER_IMAGE:latest cd ../../src/docker
Running the docker container locally
biometric:docker michaelobrien$ ./build.sh Sending build context to Docker daemon 24.54MB Step 1/6 : FROM openjdk:11 ---> 243e95d792e3 Step 2/6 : ARG MICROSERVICE_HOME=/opt/app/ ---> Running in 9b5100430ac0 Removing intermediate container 9b5100430ac0 ---> 5866713a904b Step 3/6 : RUN mkdir -p $MICROSERVICE_HOME ---> Running in 94671e9d8341 Removing intermediate container 94671e9d8341 ---> b3482d330179 Step 4/6 : ADD ecs-app-source-nbi-*.jar $MICROSERVICE_HOME/lib/ecs-app-source-nbi.jar ---> f9d078171e04 Step 5/6 : ADD startService.sh $MICROSERVICE_HOME/bin/ ---> 9e7f8845f2d7 Step 6/6 : CMD ["/opt/app/bin/startService.sh"] ---> Running in c445f2c3871b Removing intermediate container c445f2c3871b ---> 626d1c21f152 [Warning] One or more build-args [build-id] were not consumed Successfully built 626d1c21f152 Successfully tagged ecs-app-source-nbi:latest ecs-app-source-nbi 129f22ccb92f0b9189706ba50bbd6d5ec62de2170a3d6474999eb84aa996d8f9 biometric:docker michaelobrien$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 129f22ccb92f ecs-app-source-nbi:latest "/opt/app/bin/startS…" About a minute ago Up About a minute 0.0.0.0:8888->8080/tcp ecs-nbi biometric:docker michaelobrien$ curl http://127.0.0.1:8888/nbi/api {"id":1,"content":"PASS"}
Verify Docker Container runs in Windows under Docker Desktop on Hyper-V as well
Issue: any sh scripts that run inside a docker container need to be in unix LF format - not CR/LF.
This can be fixed by putting in a *.sh override in .gitattributes.
Before
michaelobrien@biometrics MINGW64 /f/wse_github/ecs-app-source/ecs-app-source-nbi/src/docker (master) $ ./build.sh Sending build context to Docker daemon 24.54MB Step 1/6 : FROM openjdk:11 11: Pulling from library/openjdk db8414d1baad: Pull complete Digest: sha256:0e871bfa64e4eab7453fecb9b555da938ca3496c89f19de9eeb4bced684bf66f Status: Downloaded newer image for openjdk:11 ---> 243e95d792e3 Step 2/6 : ARG USERVICE_HOME=/opt/app/ ---> Running in e37270d17866 Removing intermediate container e37270d17866 ---> eee7698fd555 Step 3/6 : RUN mkdir -p $USERVICE_HOME ---> Running in 2ea24b69d2cc Removing intermediate container 2ea24b69d2cc ---> 5598e6b253f8 Step 4/6 : ADD ecs-app-source-nbi-*.jar $USERVICE_HOME/lib/ecs-app-source-nbi.jar ---> 80cfbe1e7834 Step 5/6 : ADD startService.sh $USERVICE_HOME/bin/ ---> 3d45f937a3ff Step 6/6 : CMD ["/opt/app/bin/startService.sh"] ---> Running in 5529387ab214 Removing intermediate container 5529387ab214 ---> 884e985c1c60 [Warning] One or more build-args [build-id] were not consumed Successfully built 884e985c1c60 Successfully tagged ecs-app-source-nbi:latest SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories. Error response from daemon: No such container: ecs-app-source-nbi Error: No such container: ecs-app-source-nbi starting: ecs-app-source-nbi 5c013d61c996c9cc9c52cf971e4cda27498a390e01bf8e055de8e25841c4a68c container failed michaelobrien@biometrics MINGW64 /f/wse_github/ecs-app-source/ecs-app-source-nbi/src/docker (master) $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES check logs michaelobrien@biometrics MINGW64 /f/wse_github/ecs-app-source/ecs-app-source-nbi/src/docker (master) $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5c013d61c996 ecs-app-source-nbi:latest "/opt/app/bin/startS…" About a minute ago Exited (1) About a minute ago ecs-app-source-nbi michaelobrien@biometrics MINGW64 /f/wse_github/ecs-app-source/ecs-app-source-nbi/src/docker (master) $ docker logs -f ecs-app-source-nbi standard_init_linux.go:211: exec user process caused "no such file or directory" This actually means the file is either not 755 chmod or it is in CR/LF format not LF.
After
https://github.com/obrienlabs/ecs-app-source/commit/4de82b6dcc1ef77c5817b2392fa5f9d9755c52f5
- ECS-10Getting issue details... STATUS
We manually create services via the console before using AWS CLI, CloudFormation or Terraform.
20191215-3: AWS ECS infrastructure via Console before CloudFormation
Figure out our minimal ECS infrastructure to get internal calls running between containers
Create 2 subnet VPC
AWS Inventory
Artifact | Dependencies | ID | Parameters | automation | Placement | |
---|---|---|---|---|---|---|
vpc | nacl rt-main | ecs-dev-vpc vpc-081dde8eafa7d26a3 | 10.0.0.0/16 | VPC wiz | ||
rt (main) | rtb-0d91d6ea933d26672 pri | vpc, pri 10.0.0.0/16 = local target 0.0.0.0/0 = nat-0326e83e479ce1586 target | VPC wiz | |||
rt | rtb-0f02d96a393ff81d8 pub | 10.0.0.0/16 = local target 0.0.0.0/0 = igw-0b312c8909f0cde7f target | VPC wiz | |||
nacl | acl-0d99d1a1c43b91e11 | in 0.0.0.0/0 allow out 0.0.0.0/0 allow | VPC wiz | |||
subnet | ecs-dev-vpc-pub | 10.0.0.0/24 us-east-1f | VPC wiz | |||
igw | igw-0b312c8909f0cde7f | VPC wiz | ||||
igw-ni | igw-ni | eni-034236b5593034c80 | VPC wiz | |||
subnet | ecs-dev-vpc-pri | 10.0.1.0/24 us-east-1f | VPC wiz | |||
NAT-gw | nat-0326e83e479ce1586 | 3.215.46.53 | VPC wiz | |||
nat EIP | eipalloc-0999a0e7468b2e980 | 3.215.46.53 | console | |||
bastion EIP | 3.227.98.40 | console | ||||
route 53 A record | on cloudlift.systems | dig ecs-dev-bastion.cloudlift.systems | console | |||
EC2 | ecs-dev-vpc-private-instance | for testing out the bastion, the nat and the igw | console | |||
EC2 | ecs-dev-vpc-instance-pri-sg | console | ||||
ssh key | console | |||||
sg | ecs-dev-vpc-bastion-sg | console | ||||
ec2 | bastion | console | ||||
ECS | ecs-dev-vpc | ECS wiz | ||||
CF stack | EC2ContainerService-ecs-dev-vpc | |||||
ECS | ECS Cluster setting | |||||
sg | ecs-dev- sg-0d77b9ffd6177deee | |||||
spot | spot fleet request id sfr-aaba1ace-6771-49e4-96da-aa8968ac7d51 | |||||
EC2 image | amzn2-ami-ecs-hvm-2.0.20191212-x86_64-ebs (ami-00afc256a955c31b5) | |||||
ECS Role | ecsInstanceRole | |||||
ECS Role | ecsTaskExecutionRole | |||||
ECS Role | ecs-codedeploy-role | |||||
ECS Task Definition | ||||||
ECS Service | ||||||
NLB | internal private subnet | |||||
ASG | ||||||
Auto Scaling security group | ||||||
Target Group |
20191222-0: Create VPC with public and private subnets
Just run the AWS VPC wizard in the console which will generate/run a CloudFormation stack
2019-12-22 21:40:11 UTC-0500 | EC2ContainerService-ecs-dev-vpc2 | CREATE_COMPLETE | - |
2019-12-22 21:40:08 UTC-0500 | EcsInstanceAsg | CREATE_COMPLETE | - |
2019-12-22 21:40:07 UTC-0500 | EcsInstanceAsg | CREATE_IN_PROGRESS | Resource creation Initiated |
2019-12-22 21:40:07 UTC-0500 | EcsInstanceAsg | CREATE_IN_PROGRESS | - |
2019-12-22 21:40:03 UTC-0500 | EcsInstanceLc | CREATE_COMPLETE | - |
2019-12-22 21:40:02 UTC-0500 | EcsInstanceLc | CREATE_IN_PROGRESS | Resource creation Initiated |
2019-12-22 21:40:01 UTC-0500 | EcsInstanceLc | CREATE_IN_PROGRESS | - |
2019-12-22 21:39:56 UTC-0500 | EC2ContainerService-ecs-dev-vpc2 | CREATE_IN_PROGRESS | User Initiated |
20191222-1: Push docker image to dockerhub repository - for ECS consumption
Add a repository to dockerhub
https://hub.docker.com/repository/docker/obrienlabs/ecs-app-source-nbi/general
Run the following in the automated script.
docker build --no-cache --build-arg build-id=$BUILD_ID -t obrienlabs/$CONTAINER_IMAGE -f DockerFile . #docker tag $CONTAINER_IMAGE:latest $CONTAINER_IMAGE:latest docker tag obrienlabs/$CONTAINER_IMAGE obrienlabs/$CONTAINER_IMAGE:0.0.1 # dockerhub docker push obrienlabs/$CONTAINER_IMAGE:0.0.1 Successfully built 899825fb179f Successfully tagged obrienlabs/ecs-app-source-nbi:latest The push refers to repository [docker.io/obrienlabs/ecs-app-source-nbi] 4dc175544461: Pushed 231c72dcbe90: Pushed 74c538929695: Pushed 25efa461ccff: Mounted from library/openjdk .... e4b20fcc48f4: Mounted from library/openjdk 0.0.1: digest: sha256:668c99361d5d6225c1aafde34b7d4f414c0a31946e9e156cdb45dff00133ad8c size: 2421
20191222-2: Create ECS cluster with 2 instances
Make sure to put your ssh key on the launch configuration, select latest AMI and put the EC2 instances in the private subnet.
20191222-3: Verify ssh access to private ECS EC2 container via bastion
biometric:~ michaelobrien$ ssh ubuntu@3.227... ubuntu@ip-10-0-0-48:~$ ssh -i onap_rsa ec2-user@10.0.1.91 __| __| __| _| ( \__ \ Amazon Linux 2 (ECS Optimized) ____|\___|____/ [ec2-user@ip-10-0-1-91 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f7c4231f7ca1 amazon/amazon-ecs-agent:latest "/agent" 27 minutes ago Up 27 minutes (healthy) ecs-agent
20191222-4: Create custom container ecsTaskExecutionRole
20191222-5: Create ECS Task Definition
use the obrienlabs/ecs-dev-source-nbi:0.0.1 image
Create Task Definition Container
select the image, tag and port
20191222-6: Create IAM ECS CodeDeploy role - generic
A CodeDeploy IAM Service role must be created before using a Blue/Green deployment model for an ECS Service.
This codeDeploy role can be used for any task/service
20191222-7: Create private NLB
An internal facing NLB on port 80, instance - not ip type.
20191222-8: Create Auto Scaling Group
Create Auto Scaling Group Security Group in VPC
Create ECS Launch Configuration
Choose the following AMI - amzn2-ami-ecs-hvm-2.0.20191212-x86_64-ebs (ami-00afc256a955c31b5)
20191222-9: Create ECS Capacity Provider for Auto Scaling
There already is an inherent ASG running the other 2 instances launched with the ECS cluster we can use - I should have picked the ECSCon* ASG - not created the ecs-dev-asg below
So instead
20191222-10: Create ECS Service with NLB
An ECS service needs an existing NLB or ALB
Note: health check grace period should be non-zero to avoid restarts. - I use 40sec
Select "add listener"
Next time set the "network" to awsvpc - see later
Check ECS Tasks
ubuntu@ip-10-0-0-48:~$ ssh -i onap_rsa ec2-user@10.0.1.91 Last login: Tue Dec 24 03:31:05 2019 from ip-10-0-0-48.ec2.internal __| __| __| _| ( \__ \ Amazon Linux 2 (ECS Optimized) ____|\___|____/ [ec2-user@ip-10-0-1-91 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2209f328d43f obrienlabs/ecs-app-source-nbi:0.0.1 "/opt/app/bin/startS…" 28 minutes ago Up 28 minutes 0.0.0.0:8080->8080/tcp ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-ccfea89fb5b2b1d6dd01 f7c4231f7ca1 amazon/amazon-ecs-agent:latest "/agent" 25 hours ago Up 25 hours (healthy) ecs-agent [ec2-user@ip-10-0-1-91 ~]$ docker logs -f 2209f328d43f . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.1.3.RELEASE) 2019-12-24 03:04:43.986 INFO 6 --- [ main] systems.cloudlift.nbi.NbiApplication : Starting NbiApplication v0.0.1-SNAPSHOT on 2209f328d43f with PID 6 (/opt/app/lib/ecs-app-source-nbi.jar started by root in /opt/app) 2019-12-24 03:04:44.002 INFO 6 --- [ main] systems.cloudlift.nbi.NbiApplication : No active profile set, falling back to default profiles: default 2019-12-24 03:04:50.084 INFO 6 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) ubuntu@ip-10-0-0-48:~$ ssh -i onap_rsa ec2-user@10.0.1.234 __| __| __| _| ( \__ \ Amazon Linux 2 (ECS Optimized) ____|\___|____/ [ec2-user@ip-10-0-1-234 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e9d385f9eb2 obrienlabs/ecs-app-source-nbi:0.0.1 "/opt/app/bin/startS…" 28 minutes ago Up 28 minutes 0.0.0.0:8080->8080/tcp ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-dcf8d3dae1dc9e876600 87251d55a242 amazon/amazon-ecs-agent:latest "/agent" 25 hours ago Up 25 hours (healthy) ecs-agent [ec2-user@ip-10-0-1-234 ~]$ docker logs -f 2e9d385f9eb2 . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.1.3.RELEASE) 2019-12-24 03:04:44.017 INFO 6 --- [ main] systems.cloudlift.nbi.NbiApplication : Starting NbiApplication v0.0.1-SNAPSHOT on 2e9d385f9eb2 with PID 6 (/opt/app/lib/ecs-app-source-nbi.jar started by root in /opt/app) 2019-12-24 03:04:44.029 INFO 6 --- [ main] systems.cloudlift.nbi.NbiApplication : No active profile set, falling back to default profiles: default 2019-12-24 03:04:50.314 INFO 6 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) 2019-12-24 03:04:50.454 INFO 6 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2019-12-24 03:04:50.507 INFO 6 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.16] 2019-12-24 03:04:50.522 INFO 6 --- [ main] o.a.catalina.core.AprLifecycleListener : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib] 2019-12-24 03:04:50.747 INFO 6 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/nbi] : Initializing Spring embedded WebApplicationContext 2019-12-24 03:04:50.747 INFO 6 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 6489 ms 2019-12-24 03:04:52.558 INFO 6 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor' 2019-12-24 03:04:53.449 INFO 6 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator' 2019-12-24 03:04:53.652 INFO 6 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '/nbi' 2019-12-24 03:04:53.658 INFO 6 --- [ main] systems.cloudlift.nbi.NbiApplication : Started NbiApplication in 11.835 seconds (JVM running for 13.27)
Verify Inter-app REST communication between tasks on different EC2 VMs
check individual REST endpoints ec2-user@ip-10-0-1-91 ~]$ curl http://127.0.0.1:8080/nbi/api {"id":1,"content":"PASS"} [ec2-user@ip-10-0-1-234 ~]$ curl http://127.0.0.1:8080/nbi/api {"id":1,"content":"PASS"} check internal EC2 endpoints - inter-app communication FROM [ec2-user@ip-10-0-1-234 ~]$ curl http://127.0.0.1:8080/nbi/api {"id":1,"content":"PASS"} [ec2-user@ip-10-0-1-234 ~]$ curl http://10.0.1.234:8080/nbi/api {"id":2,"content":"PASS"} [ec2-user@ip-10-0-1-234 ~]$ curl http://10.0.1.91:8080/nbi/api {"id":3,"content":"PASS"} [ec2-user@ip-10-0-1-234 ~]$ curl http://10.0.1.91:8080/nbi/api [ec2-user@ip-10-0-1-234 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:80/nbi/api {"id":11,"content":"PASS"} [ec2-user@ip-10-0-1-234 ~]$ curl http://10.0.1.91:8080/nbi/api {"id":12,"content":"PASS"} TO 2019-12-24 03:36:57.121 INFO 6 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet : Completed initialization in 12 ms ... 11 systems.cloudlift.nbi.ApiController 12 systems.cloudlift.nbi.ApiController ^C [ec2-user@ip-10-0-1-91 ~]$
Verify traffic between a container out of the awsvpc container namespace
# the 111 IP is the pause container private ENI for awsvpc # the 125 ip is the subnet EC2 ENI for this particular EC2 instance # traffic from 111 (target) to 125 (source) # inside the 111 target container (awsvpc mode) biometric:difference-kubernetes michaelobrien$ ssh ubuntu@ecs-dev-bastion.cloudlift.systems [ec2-user@ip-10-0-1-125 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6ffd98a23d73 obrienlabs/ecs-app-source-nbi:0.0.1 "/opt/app/bin/startS…" 5 days ago Up 5 days 0.0.0.0:8080->8080/tcp ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-eac6f2d08eb8c0fa6400 0ea26a38b729 obrienlabs/ecs-app-source-nbi:0.0.1 "/opt/app/bin/startS…" 5 days ago Up 5 days ecs-ecs-app-target-nbi-1-ecs-app-target-nbi-f899a981d7c19b89d701 e2b4334fa7c3 amazon/amazon-ecs-pause:0.1.0 "./pause" 5 days ago Up 5 days ecs-ecs-app-target-nbi-1-internalecspause-909cf3a19aa18ed1d801 dd183992dbaf amazon/amazon-ecs-agent:latest "/agent" 5 days ago Up 5 days (healthy) ecs-agent [ec2-user@ip-10-0-1-125 ~]$ docker exec -it 0ea26a38b729 bash root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api {"id":3,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api {"id":4,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api {"id":5,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api {"id":6,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.125:8080/nbi/api {"id":1,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.125:8080/nbi/api {"id":2,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api {"id":7,"content":"PASS"}root@ip-10-0-1-111:/# exit exit # inside the non-awsvpc source container {"id":8,"content":"PASS"}root@6ffd98a23d73:/# curl http://10.0.1.125:8080/nbi/api # logs on the 111 awsvpc target container [ec2-user@ip-10-0-1-125 ~]$ docker logs -f 0ea26a38b729 2020-01-08 04:44:44.835 INFO 6 --- [nio-8080-exec-5] o.s.web.servlet.DispatcherServlet : Completed initialization in 9 ms 1 systems.cloudlift.nbi.ApiController 2 systems.cloudlift.nbi.ApiController 3 systems.cloudlift.nbi.ApiController 4 systems.cloudlift.nbi.ApiController 5 systems.cloudlift.nbi.ApiController 6 systems.cloudlift.nbi.ApiController 7 systems.cloudlift.nbi.ApiController 8 systems.cloudlift.nbi.ApiController # logs on the 125 non-awsvpc source container [ec2-user@ip-10-0-1-125 ~]$ docker logs -f 6ffd98a23d73 2020-01-13 15:53:09.055 INFO 6 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet : Completed initialization in 10 ms 1 systems.cloudlift.nbi.ApiController 2 systems.cloudlift.nbi.ApiController 3 systems.cloudlift.nbi.ApiController
(remove)
20191222-11: Create CodeDeploy Application
Create a CodeDeploy application of compute platform ECS.
20191222-11: Create CodeDeploy DeploymentGroup
A CodeDeploy DeploymentGroup needs an existing ECS service definition
Create the VPC and 2 subnets
Put a bastion in the public subnet
Put a test instance in the private subnet
Verify private instance initiated web traffic
We need to verify that instances in the private subnet can reach github or any other public repos.
# better to tunnel - but for now scp your key to the bastion $ scp ~/.ssh/rsa ubuntu@ecs-dev-bastion.cloudlift.systems:~/ rsa 675 # ssh into the bastion $ ssh ubuntu@ecs-dev-bastion.cloudlift.systems ubuntu@ip-10-0-0-48:~$ sudo chmod 400 rsa ubuntu@ip-10-0-0-48:~$ sudo cp rsa ~/.ssh/ ubuntu@ip-10-0-0-48:~$ sudo chown ubuntu:ubuntu ~/.ssh/rsa # test connectivity ubuntu@ip-10-0-0-48:~$ curl www.google.com <!doctype html><html itemscope=""... # ssh from the bastion into a private test instance ubuntu@ip-10-0-0-48:~$ ssh -i ~/.ssh/rsa ubuntu@10.0.1.233 # initiate web traffic ubuntu@ip-10-0-1-233:~$ curl www.google.com <!doctype html><html itemscope="".... OK
Add ECS cluster to VPC with Auto Scaling
The ECS wizard can be used to create the cluster, select...
Type | Linux + EC2 |
Instance | spot, lowest price, t3a.large, $0.08, 3 instances, EBS storage=22 default, key-pair, |
networking | vpc=ecs-dev-vpc, security-group=new, subnet=private, cidr-incoming=0.0.0.0/0 0-65535 |
container instance role | role= ecsInstanceRole (default) |
spot fleet role | new role |
20200107-1: Implement ECS awsvpc Task Networking
- ECS-12Getting issue details... STATUS
Note: awsvpc simulates running containers in a single task - where they are all in the same private namespace.
“containers that belong to the same task can communicate over the localhost
interface.” - this is similar to the pod in Kubernetes - even more so because the ENI is attached to the pause container.
Issues:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html and scenario 4 of https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/
enabled ECS awsVpcTrunking - this affects any new EC2 launched from now on in the cluster (essentially doubles the ENI's per instance - one extra for trunking) - see 4th checkbox for root accounts in https://ca-central-1.console.aws.amazon.com/ecs/home?region=ca-central-1#/settings
If there are issues with EC2 size limits we can managed this flag - we should be OK as we switched to default 16g xlarge vms which have a higher allocation.
awsVpcTrunking is an opt-in feature https://aws.amazon.com/blogs/compute/optimizing-amazon-ecs-task-density-using-awsvpc-network-mode/
NLB attachment only occurs during service creation.
Enable awsVpcTrunking in ECS
see https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_PutAccountSetting.html "When awsvpcTrunking
is specified, the elastic network interface (ENI) limit for any new container instances that support the feature is changed. If awsvpcTrunking
is enabled, any new container instances that support the feature are launched have the increased ENI limits available to them. For more information, see Elastic Network Interface Trunking in the Amazon Elastic Container Service Developer Guide."
follow https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-eni.html
# Check for existing role $ aws2 iam list-roles | grep RoleName ... "RoleName": "AWSServiceRoleForElastiCache", # already there $ aws2 iam create-service-linked-role --aws-service-name ecs.amazonaws.com An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation: Service role name AWSServiceRoleForECS has been taken in this account, please try a different suffix. # enable the setting via AWS console | ECS settings | or use CLI $ aws2 ecs put-account-setting-default --name awsvpcTrunking --value enabled --region us-east-1 { "setting": { "name": "awsvpcTrunking", "value": "enabled", "principalArn": "arn:aws:iam::2...:root" } }
Terminate and Recreate ECS instances to enable awsvpc mode
Make sure the ASG has a minimum of 2 not 0
All gone - the ASG will restart them
Create awsvpc network mode task definition
View the task definition
Create the awsvpc ECS service
Verify network mode
check json task definition
{ "ipcMode": null, "executionRoleArn": "arn:aws:iam::249302271888:role/ecsTaskExecutionRole", "containerDefinitions": [ { "dnsSearchDomains": null, "logConfiguration": null, "entryPoint": null, "portMappings": [ { "hostPort": 8080, "protocol": "tcp", "containerPort": 8080 } ], "command": null, "linuxParameters": null, "cpu": 0, "environment": [], "resourceRequirements": null, "ulimits": null, "dnsServers": null, "mountPoints": [], "workingDirectory": null, "secrets": null, "dockerSecurityOptions": null, "memory": null, "memoryReservation": null, "volumesFrom": [], "stopTimeout": null, "image": "obrienlabs/ecs-app-source-nbi:0.0.1", "startTimeout": null, "firelensConfiguration": null, "dependsOn": null, "disableNetworking": null, "interactive": null, "healthCheck": null, "essential": true, "links": null, "hostname": null, "extraHosts": null, "pseudoTerminal": null, "user": null, "readonlyRootFilesystem": null, "dockerLabels": null, "systemControls": null, "privileged": null, "name": "ecs-app-target-nbi" } ], "placementConstraints": [], "memory": "2048", "taskRoleArn": "arn:aws:iam::249302271888:role/ecsTaskExecutionRole-ecs-app-source-nbi", "compatibilities": [ "EC2", "FARGATE" ], "taskDefinitionArn": "arn:aws:ecs:us-east-1:249302271888:task-definition/ecs-app-target-nbi:1", "family": "ecs-app-target-nbi", "requiresAttributes": [ { "targetId": null, "targetType": null, "value": null, "name": "com.amazonaws.ecs.capability.task-iam-role" }, { "targetId": null, "targetType": null, "value": null, "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18" }, { "targetId": null, "targetType": null, "value": null, "name": "ecs.capability.task-eni" } ], "pidMode": null, "requiresCompatibilities": [ "EC2" ], "networkMode": "awsvpc", "cpu": "1024", "revision": 1, "status": "ACTIVE", "inferenceAccelerators": null, "proxyConfiguration": null, "volumes": [] }
Check for 2 ENI's on the ECS EC2 Instances
Private IPs 10.0.1.125, 10.0.1.111 Network interfaces eth0 eth1
Check the EC2 instance that contains the task networking container
ec2-user@ip-10-0-1-125 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0ea26a38b729 obrienlabs/ecs-app-source-nbi:0.0.1 "/opt/app/bin/startS…" 15 minutes ago Up 15 minutes ecs-ecs-app-target-nbi-1-ecs-app-target-nbi-f899a981d7c19b89d701 e2b4334fa7c3 amazon/amazon-ecs-pause:0.1.0 "./pause" 15 minutes ago Up 15 minutes ecs-ecs-app-target-nbi-1-internalecspause-909cf3a19aa18ed1d801 dd183992dbaf amazon/amazon-ecs-agent:latest "/agent" 2 hours ago Up 2 hours (healthy) ecs-agent # the pause container is used by awsvpc network mode as a private namespace to launch the application container into # see the default docker0 network and eth0, notice ecs-bridge on the awsvpc/awsvpcTrunk enabled instance # with a single ENI [ec2-user@ip-10-0-1-116 ~]$ ifconfig docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001 inet 10.0.1.116 netmask 255.255.255.0 broadcast 10.0.1.255 vethc4d90cc: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::90bb:8fff:fe15:c698 prefixlen 64 scopeid 0x20<link> # with two ENI's - the 10.0.1.111 IP does not appear here [ec2-user@ip-10-0-1-125 ~]$ ifconfig docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255 ecs-bridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 169.254.172.1 netmask 255.255.252.0 broadcast 0.0.0.0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001 inet 10.0.1.125 netmask 255.255.255.0 broadcast 10.0.1.255 vetha420835f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::a413:8dff:feb4:e529 prefixlen 64 scopeid 0x20<link>
Switch first service to awsvpc network mode
Note: ssh sessions will get bounced as the 2nd ENI is attached via the pause container
[ec2-user@ip-10-0-1-125 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6ffd98a23d73 obrienlabs/ecs-app-source-nbi:0.0.1 "/opt/app/bin/startS…" 40 seconds ago Up 39 seconds 0.0.0.0:8080->8080/tcp ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-eac6f2d08eb8c0fa6400 0ea26a38b729 obrienlabs/ecs-app-source-nbi:0.0.1 "/opt/app/bin/startS…" About an hour ago Up About an hour ecs-ecs-app-target-nbi-1-ecs-app-target-nbi-f899a981d7c19b89d701 e2b4334fa7c3 amazon/amazon-ecs-pause:0.1.0 "./pause" About an hour ago Up About an hour ecs-ecs-app-target-nbi-1-internalecspause-909cf3a19aa18ed1d801 dd183992dbaf amazon/amazon-ecs-agent:latest "/agent" 2 hours ago Up 2 hours (healthy) ecs-agent
We have 3 options 1= (all containers in a single task (same as a pod in kubernetes) - they use localhost because of the private namespace), - not recommended 2=route53 service discover (I have this enabled as a secondary option - the 2-level SRV/A record is maintained by the scheduler) and 3=awsvpc (essentially a simulated localhost namespace of a single task by launching the pause container (same thing as in Kubernetes) first - it gets associated with the 2nd trunk ENI and all other containers are launched into it's private namespace) - I have created the infrastructure around this locally and am testing connectivity from exec to exec in the containers
EC2-1 [ec2-user@ip-10-0-1-125 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6ffd98a23d73 obrienlabs/ecs-app-source-nbi:0.0.1 "/opt/app/bin/startS…" 8 minutes ago Up 8 minutes 0.0.0.0:8080->8080/tcp ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-eac6f2d08eb8c0fa6400 0ea26a38b729 obrienlabs/ecs-app-source-nbi:0.0.1 "/opt/app/bin/startS…" About an hour ago Up About an hour ecs-ecs-app-target-nbi-1-ecs-app-target-nbi-f899a981d7c19b89d701 e2b4334fa7c3 amazon/amazon-ecs-pause:0.1.0 "./pause" About an hour ago Up About an hour ecs-ecs-app-target-nbi-1-internalecspause-909cf3a19aa18ed1d801 dd183992dbaf amazon/amazon-ecs-agent:latest "/agent" 3 hours ago Up 3 hours (healthy) ecs-agent [ec2-user@ip-10-0-1-125 ~]$ curl http://10.0.1.111:8080/nbi/api {"id":1,"content":"PASS"} EC2-2 [ec2-user@ip-10-0-1-116 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8f1d75f601ab amazon/amazon-ecs-agent:latest "/agent" 3 hours ago Up 3 hours (healthy) ecs-agent [ec2-user@ip-10-0-1-116 ~]$ curl http://10.0.1.111:8080/nbi/api {"id":2,"content":"PASS"} the EC2 gets modified on the first awsvpc service launch with the 2nd ENI/IP - notice that the awsvpc container app-target has no port mappings - they are removed - it has the 2nd ENI ip 10.0.1.111Still need to test in-container and dynamically get the address from the service and also test using the nlb.
20200114-1: Verify Inter app REST calls through private NLB
# console 1: perform REST calls from one of the EC2 cluster nodes [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":6,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":6,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":7,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":8,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":9,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":10,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":11,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":7,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":12,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":13,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":8,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":9,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api {"id":14,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"} [ec2-user@ip-10-0-1-116 ~]$ # console 2: logs from 1 of 2 target containers 6 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 7 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 8 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 9 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com # console 3: logs for 2nd of 2 target containers 6 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 7 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 8 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 9 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 10 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 11 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 12 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 13 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com 14 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
FAQ
Why not route inter container calls through the API gateway
I might not have every detail but the main point is avoiding calls out through the NAT or IG to avoid public routing outside the VPC (the APIGW is outside the VPC RT) - so even though calls route back in through private VPC endpoints the request is public routed (so private traffic would need to be https - it is going to be anyway in the future). The 2nd trunk ENI's on the task networking enabled VMs are private RT routed, also the SG attached to the task using trunking can have granular per/service extra security than just using the host ENI sg.
We have 3 options 1= (all containers in a single task (same as a pod in kubernetes) - they use localhost because of the private namespace), - not recommended 2=route53 service discover (I have this enabled as a secondary option - the 2-level SRV/A record is maintained by the scheduler) and 3=awsvpc (essentially a simulated localhost namespace of a single task by launching the pause container (same thing as in Kubernetes) first - it gets associated with the 2nd trunk ENI and all other containers are launched into it's private namespace) - I have created the infrastructure around this locally and am testing connectivity from exec to exec in the containers
ECS awsvpc network mode compatible EC2 instance types
I have been using t3.xlarge but the list is officially more specific - switching to m5.xlarge - see https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-eni.html