Requirements

I need to bring up a system that uses the pre-kubernetes ECS as the base for a set of java based microservices. I am constrained from using either a native kubernetes cluster (using RKE) or the managed AWS EKS elastic kubernetes service.

R#	Key	Details
R1		Deployment is unaffected by OSX/Windows Dev environments

R7	ECS	ECS cluster using EC2
R7.1	in/out	uService containers can communicate privately
		uService containers are reachable from the NLB
		uService containers are reachable from the API-GW
		The ECS cluster must be in an auto scaling group
R9	VPC	The ECS cluster must be in a VPC with private/public subnets

R10	API-GW	The public facing URLs must be through the AWS API Gateway
R10.1		The ECS cluster is fronted by an NLB to work with the API Gateway (not an ALB)
R10.2		The public facing URLs must be obfuscated from the real ECS NLB urls using a reverse proxy inside the subnet
R11

Work Items

WI	Description	Jira
	Spring boot application	ECS-4 - Getting issue details... STATUS
	Docker build/deploy infrastructure	ECS-2 - Getting issue details... STATUS
	AWS Cloud infrastructure/Automation	ECS-6 - Getting issue details... STATUS
	Add systems manager agent manually to ec2 instances amzn2-ami-ecs-hvm-2.0.20191212-x86_64-ebs (ami-00afc256a955c31b5)
	Upgrade spring boot from 2.1.3 to 2.2.4	ECS-13 - Getting issue details... STATUS

Repositories

Github


https://github.com/obrienlabs/ecs-app-source
https://github.com/obrienlabs/ecs-devops

Analysis

Architecture

ECS AWS Infrastructure Diagram

20200114

Angular Front End

Spring Boot Application Backend

AWS ECS infrastructure

ECS Service Creation

In order of dependency

Step
Pre	Create ECS Cluster


	Create ecsTaskExecutionRole 20191222-4:CreatecustomcontainerecsTaskExecutionRole

Design

ECS Network Design

We have 3 options 1= (all containers in a single task (same as a pod in kubernetes) - they use localhost because of the private namespace), - not recommended 2=route53 service discovery (I have this enabled as a secondary option - A records (per ENI) are maintained by the scheduler) and 3=awsvpc (essentially a simulated localhost namespace of a single task by launching the pause container (same thing as in Kubernetes) first - it gets associated with the 2nd trunk ENI and all other containers are launched into it's private namespace)

ECS Task Networking

see https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html and https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#network_mode and

ECS Bridge Networking Mode

ECS Host Networking Mode

ECS Awsvpc Networking Mode

DI 1: Base Case 1: 2 node EC2 ECS cluster in a VPC with a single public subnet - verify uService comms

DI 2: Base Case 2: 2 node EC2 ECS cluster in a VPC with public/private subnets

Implementation

Infrastructure

Quickstart Sandbox

Existing 2 subnet VPC with IG, Nat

Enable awsVpctrunking on ECS settings - to enable awsvpc network mode

Create ECS cluster with at least 2 instances

Testing

Implementation Timeline

20191215-1: Minimal Maven project structure for a Spring Boot Application

The microservice exercising the ECS project will start as a standard maven java 11 spring boot project.

Initial checkin branch for a clean project compile and spring boot run under ECS-4 - Getting issue details... STATUS
https://github.com/obrienlabs/ecs-app-source/issues/1
https://github.com/obrienlabs/ecs-app-source/commit/225cdd21e5a57eae156210ff8ceb9d6f29c0687b

Run NbiApplication as spring boot application on OSX

:: Spring Boot ::        (v2.1.3.RELEASE)
2019-12-15 11:36:17.673  INFO 8724 --- [  restartedMain] systems.cloudlift.nbi.NbiApplication     : Starting NbiApplication on biometric.local with PID 8724 (/Users/michaelobrien/wse_github/ecs-app-source/ecs-app-source-nbi/target/classes started by michaelobrien in /Users/michaelobrien/wse_github/ecs-app-source/ecs-app-source-nbi)
2019-12-15 11:36:17.902  INFO 8724 --- [  restartedMain] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.16]
2019-12-15 11:36:18.077  INFO 8724 --- [  restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path '/nbi'
2019-12-15 11:36:18.077  INFO 8724 --- [  restartedMain] systems.cloudlift.nbi.NbiApplication     : Started NbiApplication in 0.714 seconds (JVM running for 215.995)
2019-12-15 11:36:33.278  INFO 8724 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 3 ms
1 systems.cloudlift.nbi.ApiController

Check the REST API via curl

biometric:main michaelobrien$ curl http://127.0.0.1:8080/nbi/api
{"id":1,"content":"PASS"}

Run NbiApplication on Windows 10

PS C:\WINDOWS\system32> curl http://127.0.0.1:8080/nbi/api                                                                                                                                                       
StatusCode        : 200
Content           : {"id":1,"content":"PASS"}

20191215-2: Add Docker Build and Deploy Infrastructure

Preliminary testing

DockerFile

For a Java 11 LTS based spring boot application use the openjdk base docker image - not the alpine one that stops at Java 8

#FROM anapsix/alpine-java:jre8
FROM openjdk:11
ARG USERVICE_HOME=/opt/app/
# Build up the deployment folder structure
RUN mkdir -p $USERVICE_HOME
ADD ecs-app-source-nbi-*.jar $USERVICE_HOME/lib/ecs-app-source-nbi.jar
ADD startService.sh $USERVICE_HOME/bin/
CMD ["/opt/app/bin/startService.sh"]

startService.sh

using a separate script is optional for the entrypoint but decouples the start allowing for multiple sh commands on image start.

#!/bin/bash
cd /opt/app
#-Dlogging.config=config/logback.xml
if [ -z "${java_runtime_arguments}" ]; then
  java  -Xms128m -Xmx1536m -jar /opt/app/lib/ecs-app-source-nbi.jar
else
  java  $java_runtime_arguments -jar /opt/app/lib/ecs-app-source-nbi.jar
fi

Build script prototype

The docker build folder needs to be outside of the src folder to avoid git picking them up.

#!/bin/bash

BUILD_ID=10001
mkdir ../../docker
TARGET_DIR=../../docker/$BUILD_ID
mkdir $TARGET_DIR
CONTAINER_IMAGE=ecs-app-source-nbi

cp ../../target/*.jar $TARGET_DIR
cp DockerFile $TARGET_DIR
cp startService.sh $TARGET_DIR
cd $TARGET_DIR
docker build --no-cache --build-arg build-id=$BUILD_ID -t $CONTAINER_IMAGE -f DockerFile .
docker tag $CONTAINER_IMAGE:latest $CONTAINER_IMAGE:latest
docker stop $CONTAINER_IMAGE
docker rm $CONTAINER_IMAGE
echo "starting: $CONTAINER_IMAGE"
docker run --name $CONTAINER_IMAGE -d -p 8888:8080 -e os.environment.configuration.dir=/ -e os.environment.ecosystem=sbx $CONTAINER_IMAGE:latest

cd ../../src/docker

Running the docker container locally

biometric:docker michaelobrien$ ./build.sh 
Sending build context to Docker daemon  24.54MB
Step 1/6 : FROM openjdk:11
 ---> 243e95d792e3
Step 2/6 : ARG MICROSERVICE_HOME=/opt/app/
 ---> Running in 9b5100430ac0
Removing intermediate container 9b5100430ac0
 ---> 5866713a904b
Step 3/6 : RUN mkdir -p $MICROSERVICE_HOME
 ---> Running in 94671e9d8341
Removing intermediate container 94671e9d8341
 ---> b3482d330179
Step 4/6 : ADD ecs-app-source-nbi-*.jar $MICROSERVICE_HOME/lib/ecs-app-source-nbi.jar
 ---> f9d078171e04
Step 5/6 : ADD startService.sh $MICROSERVICE_HOME/bin/
 ---> 9e7f8845f2d7
Step 6/6 : CMD ["/opt/app/bin/startService.sh"]
 ---> Running in c445f2c3871b
Removing intermediate container c445f2c3871b
 ---> 626d1c21f152
[Warning] One or more build-args [build-id] were not consumed
Successfully built 626d1c21f152
Successfully tagged ecs-app-source-nbi:latest
ecs-app-source-nbi
129f22ccb92f0b9189706ba50bbd6d5ec62de2170a3d6474999eb84aa996d8f9

biometric:docker michaelobrien$ docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED              STATUS              PORTS                    NAMES
129f22ccb92f        ecs-app-source-nbi:latest   "/opt/app/bin/startS…"   About a minute ago   Up About a minute   0.0.0.0:8888->8080/tcp   ecs-nbi

biometric:docker michaelobrien$ curl http://127.0.0.1:8888/nbi/api
{"id":1,"content":"PASS"}

Verify Docker Container runs in Windows under Docker Desktop on Hyper-V as well

Issue: any sh scripts that run inside a docker container need to be in unix LF format - not CR/LF.

This can be fixed by putting in a *.sh override in .gitattributes.

Before

michaelobrien@biometrics MINGW64 /f/wse_github/ecs-app-source/ecs-app-source-nbi/src/docker (master)
$ ./build.sh
Sending build context to Docker daemon  24.54MB
Step 1/6 : FROM openjdk:11
11: Pulling from library/openjdk                                                                                                                                                                                                db8414d1baad: Pull complete                                                                                                                                                                                                Digest: sha256:0e871bfa64e4eab7453fecb9b555da938ca3496c89f19de9eeb4bced684bf66f
Status: Downloaded newer image for openjdk:11
 ---> 243e95d792e3
Step 2/6 : ARG USERVICE_HOME=/opt/app/
 ---> Running in e37270d17866
Removing intermediate container e37270d17866
 ---> eee7698fd555
Step 3/6 : RUN mkdir -p $USERVICE_HOME
 ---> Running in 2ea24b69d2cc
Removing intermediate container 2ea24b69d2cc
 ---> 5598e6b253f8
Step 4/6 : ADD ecs-app-source-nbi-*.jar $USERVICE_HOME/lib/ecs-app-source-nbi.jar
 ---> 80cfbe1e7834
Step 5/6 : ADD startService.sh $USERVICE_HOME/bin/
 ---> 3d45f937a3ff
Step 6/6 : CMD ["/opt/app/bin/startService.sh"]
 ---> Running in 5529387ab214
Removing intermediate container 5529387ab214
 ---> 884e985c1c60
[Warning] One or more build-args [build-id] were not consumed
Successfully built 884e985c1c60
Successfully tagged ecs-app-source-nbi:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
Error response from daemon: No such container: ecs-app-source-nbi
Error: No such container: ecs-app-source-nbi
starting: ecs-app-source-nbi
5c013d61c996c9cc9c52cf971e4cda27498a390e01bf8e055de8e25841c4a68c

container failed
michaelobrien@biometrics MINGW64 /f/wse_github/ecs-app-source/ecs-app-source-nbi/src/docker (master)
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

check logs
michaelobrien@biometrics MINGW64 /f/wse_github/ecs-app-source/ecs-app-source-nbi/src/docker (master)
$ docker ps -a
CONTAINER ID        IMAGE                       COMMAND                  CREATED              STATUS                          PORTS               NAMES
5c013d61c996        ecs-app-source-nbi:latest   "/opt/app/bin/startS…"   About a minute ago   Exited (1) About a minute ago                       ecs-app-source-nbi

michaelobrien@biometrics MINGW64 /f/wse_github/ecs-app-source/ecs-app-source-nbi/src/docker (master)
$ docker logs -f ecs-app-source-nbi
standard_init_linux.go:211: exec user process caused "no such file or directory"

This actually means the file is either not 755 chmod or it is in CR/LF format not LF.

After

https://github.com/obrienlabs/ecs-app-source/commit/4de82b6dcc1ef77c5817b2392fa5f9d9755c52f5

ECS-10 - Getting issue details... STATUS

We manually create services via the console before using AWS CLI, CloudFormation or Terraform.

20191215-3: AWS ECS infrastructure via Console before CloudFormation

Figure out our minimal ECS infrastructure to get internal calls running between containers

Create 2 subnet VPC

AWS Inventory

Artifact	Dependencies	ID	Parameters	automation	Placement
vpc	nacl rt-main	ecs-dev-vpc vpc-081dde8eafa7d26a3	10.0.0.0/16	VPC wiz
rt (main)		rtb-0d91d6ea933d26672 pri	vpc, pri 10.0.0.0/16 = local target 0.0.0.0/0 = nat-0326e83e479ce1586 target	VPC wiz
rt		rtb-0f02d96a393ff81d8 pub	10.0.0.0/16 = local target 0.0.0.0/0 = igw-0b312c8909f0cde7f target	VPC wiz
nacl		acl-0d99d1a1c43b91e11	in 0.0.0.0/0 allow out 0.0.0.0/0 allow	VPC wiz
subnet		ecs-dev-vpc-pub	10.0.0.0/24 us-east-1f	VPC wiz
igw		igw-0b312c8909f0cde7f		VPC wiz
igw-ni	igw-ni	eni-034236b5593034c80		VPC wiz
subnet		ecs-dev-vpc-pri	10.0.1.0/24 us-east-1f	VPC wiz
NAT-gw		nat-0326e83e479ce1586	3.215.46.53	VPC wiz
nat EIP		eipalloc-0999a0e7468b2e980	3.215.46.53	console
bastion EIP			3.227.98.40	console
route 53 A record		on cloudlift.systems	dig ecs-dev-bastion.cloudlift.systems	console
EC2		ecs-dev-vpc-private-instance	for testing out the bastion, the nat and the igw	console
EC2		ecs-dev-vpc-instance-pri-sg		console
ssh key				console
sg		ecs-dev-vpc-bastion-sg		console
ec2		bastion		console

ECS		ecs-dev-vpc		ECS wiz
CF stack		EC2ContainerService-ecs-dev-vpc
ECS		ECS Cluster setting
sg		ecs-dev- sg-0d77b9ffd6177deee
spot		spot fleet request id sfr-aaba1ace-6771-49e4-96da-aa8968ac7d51
EC2 image		amzn2-ami-ecs-hvm-2.0.20191212-x86_64-ebs (ami-00afc256a955c31b5)
ECS Role		ecsInstanceRole
ECS Role		ecsTaskExecutionRole
ECS Role		ecs-codedeploy-role
ECS Task Definition
ECS Service
NLB					internal private subnet
ASG
Auto Scaling security group
Target Group

20191222-0: Create VPC with public and private subnets

Just run the AWS VPC wizard in the console which will generate/run a CloudFormation stack

2019-12-22 21:40:11 UTC-0500	EC2ContainerService-ecs-dev-vpc2	CREATE_COMPLETE	-
2019-12-22 21:40:08 UTC-0500	EcsInstanceAsg	CREATE_COMPLETE	-
2019-12-22 21:40:07 UTC-0500	EcsInstanceAsg	CREATE_IN_PROGRESS	Resource creation Initiated
2019-12-22 21:40:07 UTC-0500	EcsInstanceAsg	CREATE_IN_PROGRESS	-
2019-12-22 21:40:03 UTC-0500	EcsInstanceLc	CREATE_COMPLETE	-
2019-12-22 21:40:02 UTC-0500	EcsInstanceLc	CREATE_IN_PROGRESS	Resource creation Initiated
2019-12-22 21:40:01 UTC-0500	EcsInstanceLc	CREATE_IN_PROGRESS	-
2019-12-22 21:39:56 UTC-0500	EC2ContainerService-ecs-dev-vpc2	CREATE_IN_PROGRESS	User Initiated

20191222-1: Push docker image to dockerhub repository - for ECS consumption

Add a repository to dockerhub

https://hub.docker.com/repository/docker/obrienlabs/ecs-app-source-nbi/general

Run the following in the automated script.

docker build --no-cache --build-arg build-id=$BUILD_ID -t obrienlabs/$CONTAINER_IMAGE -f DockerFile .
#docker tag $CONTAINER_IMAGE:latest $CONTAINER_IMAGE:latest
docker tag obrienlabs/$CONTAINER_IMAGE obrienlabs/$CONTAINER_IMAGE:0.0.1
# dockerhub
docker push obrienlabs/$CONTAINER_IMAGE:0.0.1

Successfully built 899825fb179f
Successfully tagged obrienlabs/ecs-app-source-nbi:latest
The push refers to repository [docker.io/obrienlabs/ecs-app-source-nbi]
4dc175544461: Pushed 
231c72dcbe90: Pushed 
74c538929695: Pushed 
25efa461ccff: Mounted from library/openjdk 
....
e4b20fcc48f4: Mounted from library/openjdk 
0.0.1: digest: sha256:668c99361d5d6225c1aafde34b7d4f414c0a31946e9e156cdb45dff00133ad8c size: 2421

20191222-2: Create ECS cluster with 2 instances

Make sure to put your ssh key on the launch configuration, select latest AMI and put the EC2 instances in the private subnet.

20191222-3: Verify ssh access to private ECS EC2 container via bastion

biometric:~ michaelobrien$ ssh ubuntu@3.227...
ubuntu@ip-10-0-0-48:~$ ssh -i onap_rsa ec2-user@10.0.1.91
   __|  __|  __|
   _|  (   \__ \   Amazon Linux 2 (ECS Optimized)
 ____|\___|____/
[ec2-user@ip-10-0-1-91 ~]$ docker ps
CONTAINER ID        IMAGE                            COMMAND             CREATED             STATUS                    PORTS               NAMES
f7c4231f7ca1        amazon/amazon-ecs-agent:latest   "/agent"            27 minutes ago      Up 27 minutes (healthy)                       ecs-agent

20191222-4: Create custom container ecsTaskExecutionRole

20191222-5: Create ECS Task Definition

use the obrienlabs/ecs-dev-source-nbi:0.0.1 image

Create Task Definition Container

select the image, tag and port

20191222-6: Create IAM ECS CodeDeploy role - generic

A CodeDeploy IAM Service role must be created before using a Blue/Green deployment model for an ECS Service.

This codeDeploy role can be used for any task/service

20191222-7: Create private NLB

An internal facing NLB on port 80, instance - not ip type.

20191222-8: Create Auto Scaling Group

Create Auto Scaling Group Security Group in VPC

Create ECS Launch Configuration

Choose the following AMI - amzn2-ami-ecs-hvm-2.0.20191212-x86_64-ebs (ami-00afc256a955c31b5)

20191222-9: Create ECS Capacity Provider for Auto Scaling

There already is an inherent ASG running the other 2 instances launched with the ECS cluster we can use - I should have picked the ECSCon* ASG - not created the ecs-dev-asg below

So instead

20191222-10: Create ECS Service with NLB

An ECS service needs an existing NLB or ALB

Note: health check grace period should be non-zero to avoid restarts. - I use 40sec

Select "add listener"

Next time set the "network" to awsvpc - see later

Check ECS Tasks

ubuntu@ip-10-0-0-48:~$ ssh -i onap_rsa ec2-user@10.0.1.91
Last login: Tue Dec 24 03:31:05 2019 from ip-10-0-0-48.ec2.internal
   __|  __|  __|
   _|  (   \__ \   Amazon Linux 2 (ECS Optimized)
 ____|\___|____/
[ec2-user@ip-10-0-1-91 ~]$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                  PORTS                    NAMES
2209f328d43f        obrienlabs/ecs-app-source-nbi:0.0.1   "/opt/app/bin/startS…"   28 minutes ago      Up 28 minutes           0.0.0.0:8080->8080/tcp   ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-ccfea89fb5b2b1d6dd01
f7c4231f7ca1        amazon/amazon-ecs-agent:latest        "/agent"                 25 hours ago        Up 25 hours (healthy)                            ecs-agent

[ec2-user@ip-10-0-1-91 ~]$ docker logs -f 2209f328d43f

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.3.RELEASE)

2019-12-24 03:04:43.986  INFO 6 --- [           main] systems.cloudlift.nbi.NbiApplication     : Starting NbiApplication v0.0.1-SNAPSHOT on 2209f328d43f with PID 6 (/opt/app/lib/ecs-app-source-nbi.jar started by root in /opt/app)
2019-12-24 03:04:44.002  INFO 6 --- [           main] systems.cloudlift.nbi.NbiApplication     : No active profile set, falling back to default profiles: default
2019-12-24 03:04:50.084  INFO 6 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)

ubuntu@ip-10-0-0-48:~$ ssh -i onap_rsa ec2-user@10.0.1.234
   __|  __|  __|
   _|  (   \__ \   Amazon Linux 2 (ECS Optimized)
 ____|\___|____/
[ec2-user@ip-10-0-1-234 ~]$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                  PORTS                    NAMES
2e9d385f9eb2        obrienlabs/ecs-app-source-nbi:0.0.1   "/opt/app/bin/startS…"   28 minutes ago      Up 28 minutes           0.0.0.0:8080->8080/tcp   ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-dcf8d3dae1dc9e876600
87251d55a242        amazon/amazon-ecs-agent:latest        "/agent"                 25 hours ago        Up 25 hours (healthy)                            ecs-agent

[ec2-user@ip-10-0-1-234 ~]$ docker logs -f 2e9d385f9eb2
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.3.RELEASE)
2019-12-24 03:04:44.017  INFO 6 --- [           main] systems.cloudlift.nbi.NbiApplication     : Starting NbiApplication v0.0.1-SNAPSHOT on 2e9d385f9eb2 with PID 6 (/opt/app/lib/ecs-app-source-nbi.jar started by root in /opt/app)
2019-12-24 03:04:44.029  INFO 6 --- [           main] systems.cloudlift.nbi.NbiApplication     : No active profile set, falling back to default profiles: default
2019-12-24 03:04:50.314  INFO 6 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2019-12-24 03:04:50.454  INFO 6 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2019-12-24 03:04:50.507  INFO 6 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.16]
2019-12-24 03:04:50.522  INFO 6 --- [           main] o.a.catalina.core.AprLifecycleListener   : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
2019-12-24 03:04:50.747  INFO 6 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/nbi]    : Initializing Spring embedded WebApplicationContext
2019-12-24 03:04:50.747  INFO 6 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 6489 ms
2019-12-24 03:04:52.558  INFO 6 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2019-12-24 03:04:53.449  INFO 6 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2019-12-24 03:04:53.652  INFO 6 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path '/nbi'
2019-12-24 03:04:53.658  INFO 6 --- [           main] systems.cloudlift.nbi.NbiApplication     : Started NbiApplication in 11.835 seconds (JVM running for 13.27)

Verify Inter-app REST communication between tasks on different EC2 VMs

check individual REST endpoints
ec2-user@ip-10-0-1-91 ~]$ curl http://127.0.0.1:8080/nbi/api
{"id":1,"content":"PASS"}

[ec2-user@ip-10-0-1-234 ~]$ curl http://127.0.0.1:8080/nbi/api
{"id":1,"content":"PASS"}

check internal EC2 endpoints - inter-app communication
FROM
[ec2-user@ip-10-0-1-234 ~]$ curl http://127.0.0.1:8080/nbi/api
{"id":1,"content":"PASS"}
[ec2-user@ip-10-0-1-234 ~]$ curl http://10.0.1.234:8080/nbi/api
{"id":2,"content":"PASS"}
[ec2-user@ip-10-0-1-234 ~]$ curl http://10.0.1.91:8080/nbi/api
{"id":3,"content":"PASS"}
[ec2-user@ip-10-0-1-234 ~]$ curl http://10.0.1.91:8080/nbi/api

[ec2-user@ip-10-0-1-234 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:80/nbi/api
{"id":11,"content":"PASS"}
[ec2-user@ip-10-0-1-234 ~]$ curl http://10.0.1.91:8080/nbi/api
{"id":12,"content":"PASS"}

TO
2019-12-24 03:36:57.121  INFO 6 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Completed initialization in 12 ms
...
11 systems.cloudlift.nbi.ApiController
12 systems.cloudlift.nbi.ApiController
^C
[ec2-user@ip-10-0-1-91 ~]$

Verify traffic between a container out of the awsvpc container namespace

# the 111 IP is the pause container private ENI for awsvpc
# the 125 ip is the subnet EC2 ENI for this particular EC2 instance
# traffic from 111 (target) to 125 (source)

# inside the 111 target container (awsvpc mode)
biometric:difference-kubernetes michaelobrien$ ssh ubuntu@ecs-dev-bastion.cloudlift.systems
[ec2-user@ip-10-0-1-125 ~]$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                PORTS                    NAMES
6ffd98a23d73        obrienlabs/ecs-app-source-nbi:0.0.1   "/opt/app/bin/startS…"   5 days ago          Up 5 days             0.0.0.0:8080->8080/tcp   ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-eac6f2d08eb8c0fa6400
0ea26a38b729        obrienlabs/ecs-app-source-nbi:0.0.1   "/opt/app/bin/startS…"   5 days ago          Up 5 days                                      ecs-ecs-app-target-nbi-1-ecs-app-target-nbi-f899a981d7c19b89d701
e2b4334fa7c3        amazon/amazon-ecs-pause:0.1.0         "./pause"                5 days ago          Up 5 days                                      ecs-ecs-app-target-nbi-1-internalecspause-909cf3a19aa18ed1d801
dd183992dbaf        amazon/amazon-ecs-agent:latest        "/agent"                 5 days ago          Up 5 days (healthy)                            ecs-agent
[ec2-user@ip-10-0-1-125 ~]$ docker exec -it 0ea26a38b729 bash
root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api
{"id":3,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api
{"id":4,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api
{"id":5,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api
{"id":6,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.125:8080/nbi/api
{"id":1,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.125:8080/nbi/api
{"id":2,"content":"PASS"}root@ip-10-0-1-111:/# curl http://10.0.1.111:8080/nbi/api
{"id":7,"content":"PASS"}root@ip-10-0-1-111:/# exit
exit

# inside the non-awsvpc source container
{"id":8,"content":"PASS"}root@6ffd98a23d73:/# curl http://10.0.1.125:8080/nbi/api

# logs on the 111 awsvpc target container
[ec2-user@ip-10-0-1-125 ~]$ docker logs -f 0ea26a38b729
2020-01-08 04:44:44.835  INFO 6 --- [nio-8080-exec-5] o.s.web.servlet.DispatcherServlet        : Completed initialization in 9 ms
1 systems.cloudlift.nbi.ApiController
2 systems.cloudlift.nbi.ApiController
3 systems.cloudlift.nbi.ApiController
4 systems.cloudlift.nbi.ApiController
5 systems.cloudlift.nbi.ApiController
6 systems.cloudlift.nbi.ApiController
7 systems.cloudlift.nbi.ApiController
8 systems.cloudlift.nbi.ApiController

# logs on the 125 non-awsvpc source container
[ec2-user@ip-10-0-1-125 ~]$ docker logs -f 6ffd98a23d73
2020-01-13 15:53:09.055  INFO 6 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Completed initialization in 10 ms
1 systems.cloudlift.nbi.ApiController
2 systems.cloudlift.nbi.ApiController
3 systems.cloudlift.nbi.ApiController

(remove)

20191222-11: Create CodeDeploy Application

Create a CodeDeploy application of compute platform ECS.

20191222-11: Create CodeDeploy DeploymentGroup

A CodeDeploy DeploymentGroup needs an existing ECS service definition

Create the VPC and 2 subnets

Put a bastion in the public subnet

Put a test instance in the private subnet

Verify private instance initiated web traffic

We need to verify that instances in the private subnet can reach github or any other public repos.

# better to tunnel - but for now scp your key to the bastion
$ scp ~/.ssh/rsa ubuntu@ecs-dev-bastion.cloudlift.systems:~/
rsa                                                                                                                                                                                                                        675    

# ssh into the bastion
$ ssh ubuntu@ecs-dev-bastion.cloudlift.systems
ubuntu@ip-10-0-0-48:~$ sudo chmod 400 rsa 
ubuntu@ip-10-0-0-48:~$ sudo cp rsa ~/.ssh/
ubuntu@ip-10-0-0-48:~$ sudo chown ubuntu:ubuntu ~/.ssh/rsa 
# test connectivity
ubuntu@ip-10-0-0-48:~$ curl www.google.com
<!doctype html><html itemscope=""...

# ssh from the bastion into a private test instance
ubuntu@ip-10-0-0-48:~$ ssh -i ~/.ssh/rsa ubuntu@10.0.1.233

# initiate web traffic
ubuntu@ip-10-0-1-233:~$ curl www.google.com
<!doctype html><html itemscope=""....
OK

Add ECS cluster to VPC with Auto Scaling

The ECS wizard can be used to create the cluster, select...


Type	Linux + EC2
Instance	spot, lowest price, t3a.large, $0.08, 3 instances, EBS storage=22 default, key-pair,
networking	vpc=ecs-dev-vpc, security-group=new, subnet=private, cidr-incoming=0.0.0.0/0 0-65535
container instance role	role= ecsInstanceRole (default)
spot fleet role	new role

20200107-1: Implement ECS awsvpc Task Networking

ECS-12 - Getting issue details... STATUS

Note: awsvpc simulates running containers in a single task - where they are all in the same private namespace.

“containers that belong to the same task can communicate over the localhost interface.” - this is similar to the pod in Kubernetes - even more so because the ENI is attached to the pause container.

Issues:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html and scenario 4 of https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/

enabled ECS awsVpcTrunking - this affects any new EC2 launched from now on in the cluster (essentially doubles the ENI's per instance - one extra for trunking) - see 4th checkbox for root accounts in https://ca-central-1.console.aws.amazon.com/ecs/home?region=ca-central-1#/settings
If there are issues with EC2 size limits we can managed this flag - we should be OK as we switched to default 16g xlarge vms which have a higher allocation.

awsVpcTrunking is an opt-in feature https://aws.amazon.com/blogs/compute/optimizing-amazon-ecs-task-density-using-awsvpc-network-mode/

NLB attachment only occurs during service creation.

Enable awsVpcTrunking in ECS

see https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_PutAccountSetting.html "When awsvpcTrunking is specified, the elastic network interface (ENI) limit for any new container instances that support the feature is changed. If awsvpcTrunking is enabled, any new container instances that support the feature are launched have the increased ENI limits available to them. For more information, see Elastic Network Interface Trunking in the Amazon Elastic Container Service Developer Guide."

follow https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-eni.html

# Check for existing role
$ aws2 iam list-roles | grep RoleName
...
            "RoleName": "AWSServiceRoleForElastiCache",

# already there
$ aws2 iam create-service-linked-role --aws-service-name ecs.amazonaws.com
An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation: Service role name AWSServiceRoleForECS has been taken in this account, please try a different suffix.

# enable the setting via AWS console | ECS settings | or use CLI
$ aws2 ecs put-account-setting-default --name awsvpcTrunking --value enabled --region us-east-1
{
    "setting": {
        "name": "awsvpcTrunking",
        "value": "enabled",
        "principalArn": "arn:aws:iam::2...:root"
    }
}

Terminate and Recreate ECS instances to enable awsvpc mode

Make sure the ASG has a minimum of 2 not 0

All gone - the ASG will restart them

Create awsvpc network mode task definition

View the task definition

Create the awsvpc ECS service

Verify network mode

check json task definition

{
  "ipcMode": null,
  "executionRoleArn": "arn:aws:iam::249302271888:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "logConfiguration": null,
      "entryPoint": null,
      "portMappings": [
        {
          "hostPort": 8080,
          "protocol": "tcp",
          "containerPort": 8080
        }
      ],
      "command": null,
      "linuxParameters": null,
      "cpu": 0,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "obrienlabs/ecs-app-source-nbi:0.0.1",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "ecs-app-target-nbi"
    }
  ],
  "placementConstraints": [],
  "memory": "2048",
  "taskRoleArn": "arn:aws:iam::249302271888:role/ecsTaskExecutionRole-ecs-app-source-nbi",
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "arn:aws:ecs:us-east-1:249302271888:task-definition/ecs-app-target-nbi:1",
  "family": "ecs-app-target-nbi",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "EC2"
  ],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "revision": 1,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}

Check for 2 ENI's on the ECS EC2 Instances

Private IPs
10.0.1.125, 10.0.1.111

Network interfaces
eth0
eth1

Check the EC2 instance that contains the task networking container

ec2-user@ip-10-0-1-125 ~]$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                 PORTS               NAMES
0ea26a38b729        obrienlabs/ecs-app-source-nbi:0.0.1   "/opt/app/bin/startS…"   15 minutes ago      Up 15 minutes                              ecs-ecs-app-target-nbi-1-ecs-app-target-nbi-f899a981d7c19b89d701
e2b4334fa7c3        amazon/amazon-ecs-pause:0.1.0         "./pause"                15 minutes ago      Up 15 minutes                              ecs-ecs-app-target-nbi-1-internalecspause-909cf3a19aa18ed1d801
dd183992dbaf        amazon/amazon-ecs-agent:latest        "/agent"                 2 hours ago         Up 2 hours (healthy)                       ecs-agent

# the pause container is used by awsvpc network mode as a private namespace to launch the application container into

# see the default docker0 network and eth0, notice ecs-bridge on the awsvpc/awsvpcTrunk enabled instance
# with a single ENI
[ec2-user@ip-10-0-1-116 ~]$ ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.0.1.116  netmask 255.255.255.0  broadcast 10.0.1.255

vethc4d90cc: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::90bb:8fff:fe15:c698  prefixlen 64  scopeid 0x20<link>

# with two ENI's - the 10.0.1.111 IP does not appear here
[ec2-user@ip-10-0-1-125 ~]$ ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255

ecs-bridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.172.1  netmask 255.255.252.0  broadcast 0.0.0.0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.0.1.125  netmask 255.255.255.0  broadcast 10.0.1.255

vetha420835f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::a413:8dff:feb4:e529  prefixlen 64  scopeid 0x20<link>

Switch first service to awsvpc network mode

Note: ssh sessions will get bounced as the 2nd ENI is attached via the pause container

[ec2-user@ip-10-0-1-125 ~]$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                 PORTS                    NAMES
6ffd98a23d73        obrienlabs/ecs-app-source-nbi:0.0.1   "/opt/app/bin/startS…"   40 seconds ago      Up 39 seconds          0.0.0.0:8080->8080/tcp   ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-eac6f2d08eb8c0fa6400
0ea26a38b729        obrienlabs/ecs-app-source-nbi:0.0.1   "/opt/app/bin/startS…"   About an hour ago   Up About an hour                                ecs-ecs-app-target-nbi-1-ecs-app-target-nbi-f899a981d7c19b89d701
e2b4334fa7c3        amazon/amazon-ecs-pause:0.1.0         "./pause"                About an hour ago   Up About an hour                                ecs-ecs-app-target-nbi-1-internalecspause-909cf3a19aa18ed1d801
dd183992dbaf        amazon/amazon-ecs-agent:latest        "/agent"                 2 hours ago         Up 2 hours (healthy)                            ecs-agent

We have 3 options 1= (all containers in a single task (same as a pod in kubernetes) - they use localhost because of the private namespace), - not recommended 2=route53 service discover (I have this enabled as a secondary option - the 2-level SRV/A record is maintained by the scheduler) and 3=awsvpc (essentially a simulated localhost namespace of a single task by launching the pause container (same thing as in Kubernetes) first - it gets associated with the 2nd trunk ENI and all other containers are launched into it's private namespace) - I have created the infrastructure around this locally and am testing connectivity from exec to exec in the containers

EC2-1
[ec2-user@ip-10-0-1-125 ~]$ docker ps
CONTAINER ID    IMAGE                 COMMAND         CREATED       STATUS         PORTS          NAMES
6ffd98a23d73    obrienlabs/ecs-app-source-nbi:0.0.1  "/opt/app/bin/startS…"  8 minutes ago    Up 8 minutes      0.0.0.0:8080->8080/tcp  ecs-ecs-app-source-nbi-1-ecs-app-source-nbi-eac6f2d08eb8c0fa6400
0ea26a38b729    obrienlabs/ecs-app-source-nbi:0.0.1  "/opt/app/bin/startS…"  About an hour ago  Up About an hour                ecs-ecs-app-target-nbi-1-ecs-app-target-nbi-f899a981d7c19b89d701
e2b4334fa7c3    amazon/amazon-ecs-pause:0.1.0     "./pause"        About an hour ago  Up About an hour                ecs-ecs-app-target-nbi-1-internalecspause-909cf3a19aa18ed1d801
dd183992dbaf    amazon/amazon-ecs-agent:latest    "/agent"         3 hours ago     Up 3 hours (healthy)              ecs-agent
[ec2-user@ip-10-0-1-125 ~]$ curl http://10.0.1.111:8080/nbi/api
{"id":1,"content":"PASS"}
EC2-2
[ec2-user@ip-10-0-1-116 ~]$ docker ps
CONTAINER ID    IMAGE              COMMAND       CREATED       STATUS         PORTS        NAMES
8f1d75f601ab    amazon/amazon-ecs-agent:latest  "/agent"      3 hours ago     Up 3 hours (healthy)            ecs-agent
[ec2-user@ip-10-0-1-116 ~]$ curl http://10.0.1.111:8080/nbi/api
{"id":2,"content":"PASS"}
the EC2 gets modified on the first awsvpc service launch with the 2nd ENI/IP - notice that the awsvpc container app-target has no port mappings - they are removed - it has the 2nd ENI ip 10.0.1.111Still need to test in-container and dynamically get the address from the service and also test using the nlb.

20200114-1: Verify Inter app REST calls through private NLB

# console 1: perform REST calls from one of the EC2 cluster nodes
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":6,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":6,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":7,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":8,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":9,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":10,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":11,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":7,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":12,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":13,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":8,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":9,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ curl http://ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com:8080/nbi/api
{"id":14,"content":"PASS remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com"}
[ec2-user@ip-10-0-1-116 ~]$ 


# console 2: logs from 1 of 2 target containers
6 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
7 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
8 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
9 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.192 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com

# console 3: logs for 2nd of 2 target containers
6 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
7 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
8 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
9 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
10 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
11 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
12 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
13 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com
14 systems.cloudlift.nbi.ApiController remoteAddr: 10.0.1.219 localAddr: 10.0.1.37 remoteHost: 10.0.1.219 serverName: ecs-dev-nlb-9599186434f660da.elb.us-east-1.amazonaws.com

FAQ

Why not route inter container calls through the API gateway

I might not have every detail but the main point is avoiding calls out through the NAT or IG to avoid public routing outside the VPC (the APIGW is outside the VPC RT) - so even though calls route back in through private VPC endpoints the request is public routed (so private traffic would need to be https - it is going to be anyway in the future). The 2nd trunk ENI's on the task networking enabled VMs are private RT routed, also the SG attached to the task using trunking can have granular per/service extra security than just using the host ENI sg.

We have 3 options 1= (all containers in a single task (same as a pod in kubernetes) - they use localhost because of the private namespace), - not recommended 2=route53 service discover (I have this enabled as a secondary option - the 2-level SRV/A record is maintained by the scheduler) and 3=awsvpc (essentially a simulated localhost namespace of a single task by launching the pause container (same thing as in Kubernetes) first - it gets associated with the 2nd trunk ENI and all other containers are launched into it's private namespace) - I have created the infrastructure around this locally and am testing connectivity from exec to exec in the containers

ECS awsvpc network mode compatible EC2 instance types

I have been using t3.xlarge but the list is officially more specific - switching to m5.xlarge - see https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-eni.html

Page tree

AWS ECS E2E Architecture