Code
https://github.com/obrienlabs/magellan
http://jenkins.obrienlabs.cloud/job/magellan-mvn-java-snapshot/
MAGELLAN-1 - Getting issue details... STATUS
There is something old school about knowing exactly where your packets are on the surface or below the ocean of the earth.
I know this might sound weird but not to someone who lives science as a normal part each day. I would like to make sure my REST calls propagate around the globe - not just to a server on the other side of the earth and back again like from us-east-1 to the Singapore ap-southeast-1 region but through us-west-2 through singapore - to europe and back to us-east-1.
Sort of when I fly - in order to maximize the number of takeoffs and landings - I try to get the most number of stops - a direct flight is not the best way to get somewhere when testing latency - as in packets on the cable from Oregon to China are better routed through Hawaii.
I liked Andrew Blum's "Tubes" book - which include details on the endpoints for sub-marine cables - https://www.amazon.com/Tubes-Journey-Internet-Andrew-Blum/dp/0061994952 - I also ran into a bit of this propagation delay working at Nokia.
This system will of course run a set of helm/kubernetes clusters (a cluster of 1 on each DC for now)
First principles will be install proxies at several regions in Amazon EC2 - https://aws.amazon.com/about-aws/global-infrastructure/
see https://www.submarinecablemap.com/
This is the path we will take (see the red circles on the Amazon AWS regions) - hopefully no backtracking and ideally it is roughly valid - as we do not actually cross the equator - I will also try through Sau Paulo.
through the following Amazon regions
AWS Regions
Propagation order
US East (N. Virginia)Canada (Central)US East (Ohio)US West (N. California)US West (Oregon)Asia Pacific (Tokyo)Asia Pacific (Seoul)
Asia Pacific (Singapore)Asia Pacific (Sydney) - skip
Asia Pacific (Mumbai)EU (Frankfurt)EU (Paris)
EU (Stockholm) - skipEU (London)EU (Ireland)South America (São Paulo) - skip
| packet.global | DNS *.packet.global | |||||||||
| DC | Identifier | Location | Region | EIP | TZ | GEO | Distance | City | ||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0000 | US East (N. Virginia) | us-east-1 | 34.200.202.57 | p0/virginia | GMT-5 | New York | |||
| Toronto | ca-central-1 | p1/toronto | Toronto | |||||||
| US West Ohio | us-east-2 | p2/ohio | ||||||||
| California | us-west-1 | p3/california | ||||||||
| Oregon | us-west-2 | p4/oregon | ||||||||
| ap-northeast-1 | p5 | Tokyo | ||||||||
| ap-northeast-2 | p6/seoul | Seoul | ||||||||
| ap-southeast-1 | p7/singapore | GMT+7 | Singapore | |||||||
| ap-south-1 | p8/mumbai | Mumbai | ||||||||
| p9 | Sydney | |||||||||
| eu-central-a | p10/frankfurt | Frankfurt | ||||||||
| eu-west-3 | p11/paris | Paris | ||||||||
| p12 | Stockholm | |||||||||
| eu-west-2 | p13/london | London | ||||||||
| eu-west-1 | p14/ireland | Ireland | ||||||||
| sa-east-1 | p15 | Sao Paulo | ||||||||
Base Cases
Opposite sides of the earth - Singapore - ap-southeast-1 to New York - us-east-1
It will never be shorter that 160 ms - which is c/(radius * pi /2) = as it it takes 16% of a second for light to pass to the other side (this is without optical repeater and endpoint delays)
packets within us-east-1
ubuntu@ip-172-31-68-232:~$ ping rke.obrienlabs.cloud PING rke.obrienlabs.cloud (3.216.18.230) 56(84) bytes of data. 64 bytes from ec2-3-216-18-230.compute-1.amazonaws.com (3.216.18.230): icmp_seq=1 ttl=63 time=0.786 ms
packets from singapore to us-east-1
they all take 225 ms except one 227 ubuntu@ip-172-31-46-121:~$ ping rke.obrienlabs.cloud PING rke.obrienlabs.cloud (3.216.18.230) 56(84) bytes of data. 64 bytes from ec2-3-216-18-230.compute-1.amazonaws.com (3.216.18.230): icmp_seq=1 ttl=30 time=225 ms 64 bytes from ec2-3-216-18-230.compute-1.amazonaws.com (3.216.18.230): icmp_seq=2 ttl=30 time=225 ms --- rke.obrienlabs.cloud ping statistics --- 58 packets transmitted, 58 received, 0% packet loss, time 57013ms rtt min/avg/max/mdev = 225.138/225.248/225.545/0.188 ms and back to singapore ubuntu@ip-172-31-68-232:~$ ping singapore.obriensec.com PING singapore.obriensec.com (18.138.217.106) 56(84) bytes of data. 64 bytes from ec2-18-138-217-106.ap-southeast-1.compute.amazonaws.com (18.138.217.106): icmp_seq=1 ttl=28 time=225 ms 64 bytes from ec2-18-138-217-106.ap-southeast-1.compute.amazonaws.com (18.138.217.106): icmp_seq=2 ttl=28 time=227 ms --- singapore.obriensec.com ping statistics --- 86 packets transmitted, 86 received, 0% packet loss, time 85106ms rtt min/avg/max/mdev = 225.188/225.350/227.678/0.687 ms
Architecture
Conceptual Architecture
Path Propagation
All paths are linear in sequence with the first bit reserved for off axis secondary routes. For example 00 is us-east and 10 is us-east but 01 is ca-central (north of us-east).
DevOps Architecture
MAGELLAN-3 - Getting issue details... STATUS
https://hub.docker.com/repository/docker/obrienlabs/magellan-nbi
Deployment Diagram
Services Architecture
see
https://eclipsejpa.blogspot.com/2019/01/sending-packets-around-earth.html
Rest API
curl -X GET "http://127.0.0.1:8080/nbi/forward/health" -H "accept: */*"
Use Cases
Work Items
| WI | Task | Jira / Wiki |
|---|---|---|
| WI 1 | OpenAPI specification | |
| Application Architecture | ||
| adjust readme | ||
| Deployment Architecture | ||
| adjust readme deployment guide | ||
| POC | ||
| 1 | Terraform infrastructure | |
| Helm/Kubernetes Charts | ||
WI 1: Reflection Rest API
API is a simple reflection REST call that accepts a forwarding address (that may be the source address) from which we can configure any type of ring, tree topology.
Testing
Packet Reflection between 2 local systems
Packet Reflection between opposite sides of the Earth
us-east-1 in Virginia
ap-southeast-1 Singapore
Build/push the docker image to obrienlabs/magellan
Run the docker image in both endpoint servers
# on remote server docker pull obrienlabs/magellan-nbi:0.0.2 docker run --name magellan-nbi -d -p 8080:8080 obrienlabs/magellan-nbi:0.0.2 docker logs -f magellan-nbi # on local server docker pull obrienlabs/magellan-nbi:0.0.2 docker run --name magellan-nbi -d -p 8080:8080 obrienlabs/magellan-nbi:0.0.2 docker logs -f magellan-nbi check health curl http://localhost:8080/nbi/api
Start a packet reflection on one server
On Apple Silicon M1-Max
Testing Results
A round trip between Singapore and Virgina and back to Singapore is 2021-09-26 04:39:52.116 - 2021-09-26 04:39:51.475 = 0.116 + (1 - .475) = .116 + .475 = .691 sec = 2 x 345 ms each way (packets may have followed different routes - or even in the same direction. Next experiment is to increase the hops to force the packets to round the earth and not follow the same return path
how much of this is the "pure speed of light" - or the minimum lag time = 7927 * 3.141 = 24900km (both halves), therefore 24900/299792 = 80 ms (there is propagation delay in relay and 10gbe fiber/copper translation.
2nd run after optimizing the Java 11 HttpClient is 210ms round trip
Response: 270
2021-09-26 04:39:51.457 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}
2021-09-26 04:39:51.458 DEBUG 1 --- [nio-8080-exec-2] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ForwardingController#getPacket(String, String, String, String, String)
2021-09-26 04:39:51.461 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2021-09-26 04:39:51.461 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Writing ["238"]
Request: http://singapore.packet.global:8080/nbi/forward/packet?dnsFrom=singapore.packet.global&dnsTo=services.obrienlabs.cloud&from=8080&to=8080&delay=1
2021-09-26 04:39:51.475 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : Completed 200 OK
Response: 271
2021-09-26 04:39:52.116 DEBUG 1 --- [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}
2021-09-26 04:39:52.118 DEBUG 1 --- [nio-8080-exec-6] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ForwardingController#getPacket(String, String, String, String, String)
2021-09-26 04:39:52.121 DEBUG 1 --- [nio-8080-exec-6] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2021-09-26 04:39:52.122 DEBUG 1 --- [nio-8080-exec-6] m.m.a.RequestResponseBodyMethodProcessor : Writing ["239"]
2021-09-26 04:39:52.123 DEBUG 1 --- [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet : Completed 200 OK
Request: http://singapore.packet.global:8080/nbi/forward/packet?dnsFrom=singapore.packet.global&dnsTo=services.obrienlabs.cloud&from=8080&to=8080&delay=1
Response: 272
2021-09-26 04:39:52.773 DEBUG 1 --- [nio-8080-exec-9] o.s.web.servlet.DispatcherServlet : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}
2nd run after moving httpclient builder out of the method and into the class - think michael
210 ms round trip
2021-09-26 15:22:37.738 DEBUG 1 --- [io-8080-exec-10] m.m.a.RequestResponseBodyMethodProcessor : Writing ["120"]
2021-09-26 15:22:37.747 DEBUG 1 --- [io-8080-exec-10] o.s.web.servlet.DispatcherServlet : Completed 200 OK
Response: 120
2021-09-26 15:22:37.957 DEBUG 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}
2021-09-26 15:22:37.958 DEBUG 1 --- [nio-8080-exec-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ForwardingController#getPacket(String, String, String, String, String)
Request: http://singapore.packet.global:8080/nbi/forward/packet?dnsFrom=singapore.packet.global&dnsTo=services.obrienlabs.cloud&from=8080&to=8080&delay=1
2021-09-26 15:22:37.960 DEBUG 1 --- [nio-8080-exec-1] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2021-09-26 15:22:37.960 DEBUG 1 --- [nio-8080-exec-1] m.m.a.RequestResponseBodyMethodProcessor : Writing ["121"]
2021-09-26 15:22:37.961 DEBUG 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed 200 OK
Response: 121
2021-09-26 15:22:38.209 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : GET "/nbi/forward/packet?dnsFrom=services.obrienlabs.cloud&dnsTo=singapore.packet.global&from=8080&to=8080&delay=1", parameters={masked}
2021-09-26 15:22:38.211 DEBUG 1 --- [nio-8080-exec-2] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ForwardingController#getPacket(String, String, String, String, String)
Request: http://singapore.packet.global:8080/nbi/forward/packet?dnsFrom=singapore.packet.global&dnsTo=services.obrienlabs.cloud&from=8080&to=8080&delay=1
2021-09-26 15:22:38.213 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2021-09-26 15:22:38.214 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Writing ["122"]
2
Locally 64181 to in 60 sec 75851 = 195/sec
