Page tree
Skip to end of metadata
Go to start of metadata

Code

https://github.com/obrienlabs/magellan

http://jenkins.obrienlabs.cloud/job/magellan-mvn-java-snapshot/

MAGELLAN-1 - Getting issue details... STATUS

There is something old school about knowing exactly where your packets are on the surface or below the ocean of the earth.  

I know this might sound weird but not to someone who lives science as a normal part each day.  I would like to make sure my REST calls propagate around the globe - not just to a server on the other side of the earth and back again like from us-east-1 to the Singapore ap-southeast-1 region but through us-west-2 through singapore - to europe and back to us-east-1.

    Sort of when I fly - in order to maximize the number of takeoffs and landings - I try to get the most number of stops - a direct flight is not the best way to get somewhere when testing latency - as in packets on the cable from Oregon to China are better routed through Hawaii.

    I liked Andrew Blum's "Tubes" book - which include details on the endpoints for sub-marine cables - https://www.amazon.com/Tubes-Journey-Internet-Andrew-Blum/dp/0061994952  - I also ran into a bit of this propagation delay working at Nokia.

This system will of course run a set of helm/kubernetes clusters (a cluster of 1 on each DC for now)

First principles will be install proxies at several regions in Amazon EC2 - https://aws.amazon.com/about-aws/global-infrastructure/

see https://www.submarinecablemap.com/

This is the path we will take (see the red circles on the Amazon AWS regions) - hopefully no backtracking and ideally it is roughly valid - as we do not actually cross the equator - I will also try through Sau Paulo.

through the following Amazon regions

AWS Regions

Propagation order

US East (N. Virginia)Canada (Central)US East (Ohio)US West (N. California)US West (Oregon)Asia Pacific (Tokyo)Asia Pacific (Seoul)
Asia Pacific (Singapore)
Asia Pacific (Sydney) - skip
Asia Pacific (Mumbai)EU (Frankfurt)EU (Paris)
EU (Stockholm) - skip
EU (London)EU (Ireland)South America (São Paulo) - skip






packet.globalDNS
*.packet.global




DCIdentifierLocation
RegionEIP
TZGEODistanceCity
00000US East (N. Virginia)
us-east-134.200.202.57p0/virginiaGMT-5

New York


Toronto
ca-central-1
p1/toronto


Toronto


US West Ohio
us-east-2
p2/ohio





California
us-west-1
p3/california





Oregon
us-west-2
p4/oregon







ap-northeast-1
p5


Tokyo




ap-northeast-2
p6/seoul


Seoul




ap-southeast-1

3.1.98.101

p7/singaporeGMT+7

Singapore




ap-south-1
p8/mumbai


Mumbai






p9


Sydney




eu-central-a
p10/frankfurt


Frankfurt




eu-west-3
p11/paris


Paris






p12


Stockholm




eu-west-2
p13/london


London




eu-west-1
p14/ireland


Ireland




sa-east-1
p15


Sao Paulo











Base Cases


Opposite sides of the earth - Singapore - ap-southeast-1 to New York - us-east-1

It will never be shorter that 160 ms - which is c/(radius * pi /2) = as it it takes 16% of a second for light to pass to the other side (this is without optical repeater and endpoint delays)

packets within us-east-1

ubuntu@ip-172-31-68-232:~$ ping rke.obrienlabs.cloud
PING rke.obrienlabs.cloud (3.216.18.230) 56(84) bytes of data.
64 bytes from ec2-3-216-18-230.compute-1.amazonaws.com (3.216.18.230): icmp_seq=1 ttl=63 time=0.786 ms

packets from singapore to us-east-1

they all take 225 ms except one 227
ubuntu@ip-172-31-46-121:~$ ping rke.obrienlabs.cloud
PING rke.obrienlabs.cloud (3.216.18.230) 56(84) bytes of data.
64 bytes from ec2-3-216-18-230.compute-1.amazonaws.com (3.216.18.230): icmp_seq=1 ttl=30 time=225 ms
64 bytes from ec2-3-216-18-230.compute-1.amazonaws.com (3.216.18.230): icmp_seq=2 ttl=30 time=225 ms
--- rke.obrienlabs.cloud ping statistics ---
58 packets transmitted, 58 received, 0% packet loss, time 57013ms
rtt min/avg/max/mdev = 225.138/225.248/225.545/0.188 ms

and back to singapore
ubuntu@ip-172-31-68-232:~$ ping singapore.obriensec.com
PING singapore.obriensec.com (18.138.217.106) 56(84) bytes of data.
64 bytes from ec2-18-138-217-106.ap-southeast-1.compute.amazonaws.com (18.138.217.106): icmp_seq=1 ttl=28 time=225 ms
64 bytes from ec2-18-138-217-106.ap-southeast-1.compute.amazonaws.com (18.138.217.106): icmp_seq=2 ttl=28 time=227 ms
--- singapore.obriensec.com ping statistics ---
86 packets transmitted, 86 received, 0% packet loss, time 85106ms
rtt min/avg/max/mdev = 225.188/225.350/227.678/0.687 ms


Architecture


Conceptual Architecture

Path Propagation

All paths are linear in sequence with the first bit reserved for off axis secondary routes.  For example 00 is us-east and 10 is us-east but 01 is ca-central (north of us-east).


DevOps Architecture

MAGELLAN-3 - Getting issue details... STATUS

https://hub.docker.com/repository/docker/obrienlabs/magellan-nbi


Deployment Diagram

Services Architecture

see

https://eclipsejpa.blogspot.com/2019/01/sending-packets-around-earth.html

Rest API

curl -X GET "http://127.0.0.1:8080/nbi/forward/health" -H "accept: */*"


Use Cases

Work Items

WITaskJira / Wiki
WI 1OpenAPI specification

Application Architecture

adjust readme

Deployment Architecture

adjust readme deployment guide

POC
1Terraform infrastructure




Helm/Kubernetes Charts















WI 1: Reflection Rest API

API is a simple reflection REST call that accepts a forwarding address (that may be the source address) from which we can configure any type of ring, tree topology.


Testing

Packet Reflection between 2 local systems

Packet Reflection between opposite sides of the Earth

us-east-1 in Virginia

ap-southeast-1 Singapore


Build/push the docker image to obrienlabs/magellan


Run the docker image in both endpoint servers

# on remote server
docker pull obrienlabs/magellan-nbi:0.0.2
docker run --name magellan-nbi -d -p 8080:8080 obrienlabs/magellan-nbi:0.0.2
docker logs -f magellan-nbi

# on local server
docker pull obrienlabs/magellan-nbi:0.0.2
docker run --name magellan-nbi -d -p 8080:8080 obrienlabs/magellan-nbi:0.0.2
docker logs -f magellan-nbi

check health

curl http://localhost:8080/nbi/api


Start a packet reflection on one server

http://127.0.0.1:8888/nbi/forward/packet?dnsFrom=192.168.100.148&dnsTo=192.168.100.145&from=8888&to=8080&delay=500

http://services.obrienlabs.cloud:8080/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1


On Apple Silicon M1-Max

Testing Results

A round trip between Singapore and Virgina and back to Singapore is 2021-09-26 04:39:52.116 -  2021-09-26 04:39:51.475 = 0.116 + (1 - .475) = .116 + .475 = .691 sec = 2 x 345 ms each way (packets may have followed different routes - or even in the same direction.  Next experiment is to increase the hops to force the packets to round the earth and not follow the same return path

how much of this is the "pure speed of light" - or the minimum lag time = 7927 * 3.141 = 24900km (both halves), therefore 24900/299792 = 80 ms (there is propagation delay in relay and 10gbe fiber/copper translation.

2nd run after optimizing the Java 11 HttpClient is 210ms round trip


http://127.0.0.1:8888/nbi/forward/packet?dnsFrom=host.docker.internal&dnsTo=host.docker.internal&from=8889&to=8888&delay=1

Response: 270
2021-09-26 04:39:51.457 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet        : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}
2021-09-26 04:39:51.458 DEBUG 1 --- [nio-8080-exec-2] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ForwardingController#getPacket(String, String, String, String, String)
2021-09-26 04:39:51.461 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2021-09-26 04:39:51.461 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Writing ["238"]
Request: http://singapore.packet.global:8080/nbi/forward/packet?dnsFrom=singapore.packet.global&dnsTo=services.obrienlabs.cloud&from=8080&to=8080&delay=1
2021-09-26 04:39:51.475 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
Response: 271
2021-09-26 04:39:52.116 DEBUG 1 --- [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet        : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}
2021-09-26 04:39:52.118 DEBUG 1 --- [nio-8080-exec-6] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ForwardingController#getPacket(String, String, String, String, String)
2021-09-26 04:39:52.121 DEBUG 1 --- [nio-8080-exec-6] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2021-09-26 04:39:52.122 DEBUG 1 --- [nio-8080-exec-6] m.m.a.RequestResponseBodyMethodProcessor : Writing ["239"]
2021-09-26 04:39:52.123 DEBUG 1 --- [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
Request: http://singapore.packet.global:8080/nbi/forward/packet?dnsFrom=singapore.packet.global&dnsTo=services.obrienlabs.cloud&from=8080&to=8080&delay=1
Response: 272
2021-09-26 04:39:52.773 DEBUG 1 --- [nio-8080-exec-9] o.s.web.servlet.DispatcherServlet        : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}


2nd run after moving httpclient builder out of the method and into the class - think michael
210 ms round trip

2021-09-26 15:22:37.738 DEBUG 1 --- [io-8080-exec-10] m.m.a.RequestResponseBodyMethodProcessor : Writing ["120"]
2021-09-26 15:22:37.747 DEBUG 1 --- [io-8080-exec-10] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
Response: 120
2021-09-26 15:22:37.957 DEBUG 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}
2021-09-26 15:22:37.958 DEBUG 1 --- [nio-8080-exec-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ForwardingController#getPacket(String, String, String, String, String)
Request: http://singapore.packet.global:8080/nbi/forward/packet?dnsFrom=singapore.packet.global&dnsTo=services.obrienlabs.cloud&from=8080&to=8080&delay=1
2021-09-26 15:22:37.960 DEBUG 1 --- [nio-8080-exec-1] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2021-09-26 15:22:37.960 DEBUG 1 --- [nio-8080-exec-1] m.m.a.RequestResponseBodyMethodProcessor : Writing ["121"]
2021-09-26 15:22:37.961 DEBUG 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
Response: 121
2021-09-26 15:22:38.209 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet        : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}
2021-09-26 15:22:38.211 DEBUG 1 --- [nio-8080-exec-2] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ForwardingController#getPacket(String, String, String, String, String)
Request: http://singapore.packet.global:8080/nbi/forward/packet?dnsFrom=singapore.packet.global&dnsTo=services.obrienlabs.cloud&from=8080&to=8080&delay=1
2021-09-26 15:22:38.213 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2021-09-26 15:22:38.214 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Writing ["122"]
2

Locally 64181 to in 60 sec 75851 = 195/sec

Training


  • No labels