bKubernetes Developer Guide | Helm Development Guide | Reference Architecture
Introduction
This page details and example cloud based microservice architecture.
Spring boot based reference architecture - Reference Architecture, Spring boot jar running the spring embedded tomcat container - Spring Boot Microservice
Architecture
Deployment Diagram
DevOps
Originally on github https://github.com/obrienlabs/refarch- moved to gitlab
site | url |
---|---|
git | |
docker hub | https://hub.docker.com/repository/docker/obrienlabs/reference-nbi |
gitlab CI/CD | |
jenkins CI | http://jenkins.obrienlabs.cloud/job/ref-arch-java-gitlab/ Backup |
base endpoint | http://services.obrienlabs.cloud:8888/nbi/api |
Jira | https://gitlab.com/refarch/reference/-/issues/1 |
Quickstart
Technology Coverage - Technology
Clone
# using your ssh key (o*_r*) :wse_gitlab $ git clone git@gitlab.com:refarch/reference
Build
:reference $ mvn clean install -U -T 16 -DskipTests=true [INFO] Reactor Build Order: [INFO] [INFO] reference-nbi [jar] [INFO] reference [pom] [INFO] [INFO] Using the MultiThreadedBuilder implementation with a thread count of 16
Commit/Push
#
Build/Run Docker endpoint on RKE EC2 VM
Locally cd reference-nbi/src/docker/ ./build.sh [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 6.412 s [INFO] Finished at: 2020-04-27T13:37:58-04:00 [INFO] ------------------------------------------------------------------------ Sending build context to Docker daemon 32.15MB Step 1/6 : FROM openjdk:8 ---> e890fe39c302 Step 2/6 : ARG USERVICE_HOME=/opt/app/ ---> Running in 146aa23cf6a8 Removing intermediate container 146aa23cf6a8 ---> b17a9e0d3ae6 Step 3/6 : RUN mkdir -p $USERVICE_HOME ---> Running in e3581d8e5d7a Removing intermediate container e3581d8e5d7a ---> 48e719469a9f Step 4/6 : ADD reference-nbi-*.jar $USERVICE_HOME/lib/reference-nbi.jar ---> bbae15d7b576 Step 5/6 : ADD startService.sh $USERVICE_HOME/bin/ ---> 7f3a12fbf15a Step 6/6 : CMD ["/opt/app/bin/startService.sh"] ---> Running in 6bddc74b8a77 Removing intermediate container 6bddc74b8a77 ---> 067ab19c41c9 [Warning] One or more build-args [build-id] were not consumed Successfully built 067ab19c41c9 Successfully tagged obrienlabs/reference-nbi:latest The push refers to repository [docker.io/obrienlabs/reference-nbi] 3ed4cee3cd68: Pushed 44ca53031496: Pushed 626aa2565d15: Pushed c601709dd5d2: Layer already exists 72ce39f2b7f6: Layer already exists 33783834b288: Layer already exists 5c813a85f7f0: Layer already exists bdca38f94ff0: Layer already exists faac394a1ad3: Layer already exists ce8168f12337: Layer already exists 0.0.1: digest: sha256:6dc5082fa5dea76439b7d72b73d442cdeb2bf257798d257c5853093b0165be08 size: 2420 reference-nbi reference-nbi starting: reference-nbi 0e9ab09b8c4082e2f0859cac462c921b186084b019699d348bca1aa0a7c858db On VM ubuntu@ip-172-31-81-46:~$ docker run --name reference-nbi -d -p 8888:8080 obrienlabs/reference-nbi:0.0.1 ubuntu@ip-172-31-81-46:~$ curl http://127.0.0.1:8888/nbi/api {"id":1,"content":"1 PASS cloud.containerization.reference.nbi.ApiController queryString: null decodedQueryString: "}
Design
Agenda:
Create initial spring boot maven project
Create a new project via the spring initializr https://start.spring.io/ see https://spring.io/guides/gs/serving-web-content/
pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.2.6.RELEASE</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>cloud.dev9</groupId> <artifactId>reference-sb-nbi</artifactId> <version>0.0.1-SNAPSHOT</version> <name>reference-sb-nbi</name> <description>Reference sb NBI</description> <properties> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-jersey</artifactId> </dependency> <!--dependency><!-- user:user --> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-security</artifactId> </dependency--> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams</artifactId> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-devtools</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <dependency> <groupId>com.oracle.ojdbc</groupId> <artifactId>ojdbc8</artifactId> <scope>runtime</scope> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> <exclusions> <exclusion> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.security</groupId> <artifactId>spring-security-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>
Add Swagger and OpenAPI 3
API - OpenAPI - Swagger#OpenAPI-Swagger-AddSwagger2toanexistingMavenJavaproject
Configure IDEs for development
Add Lombok APT IDE tool
Use Java 16 Records via https://openjdk.java.net/jeps/395 instead of Lombok as of March 2021
https://www.baeldung.com/lombok-ide and https://search.maven.org/artifact/org.projectlombok/lombok-maven-plugin/1.18.12.0/maven-plugin
Download and run the latest jar to adjust your eclipse.ini and eclipse/sts runtime - https://search.maven.org/remotecontent?filepath=org/projectlombok/lombok/1.18.12/lombok-1.18.12.jar
# add to ini file -javaagent:lombok-1.18.12 # copy jar to show packet contentsContents/MacOS// add to pom.xml<lombok.version>1.18.12</lombok.version><dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>${lombok.version}</version> </dependency> // add to your classes @Data
Lombok in IntelliJ
for 2020.03 -
2020.03 | File | Settings | Build, Execution, Deployment | Compiler | Build process VM options add : -Djps.track.ap.dependencies=false |
Add eclEmma code coverage
Add a default REST endpoint to spring boot
Add the context root to application.properties
server.servlet.context-path=/sb-nbi
Add the convenience annotation to the Application class
@SpringBootApplication
package cloud.containerization.reference.nbi; import org.springframework.web.bind.annotation.RestController; import org.springframework.web.bind.annotation.RequestMapping; @RestController public class DefaultRestController { @RequestMapping("/") public String index() { return "Greetings from Spring Boot!"; } }
Verify the endpoint
http://127.0.0.1:8180/sb-nbi/actuator/health
Add a JPA 2.2 Hibernate provider repository bean to spring boot
https://mvnrepository.com/artifact/org.eclipse.persistence/javax.persistence/2.2.0
pom.xml
# pom.xml changes <dependency> <groupId>org.eclipse.persistence</groupId> <artifactId>javax.persistence</artifactId> <version>2.2.0</version> </dependency> # mysql <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.42</version> <scope>runtime</scope> </dependency> # postgreSQL # oracle <oracle.driver.version>12.1.0.2</oracle.driver.version> <dependency> <groupId>com.oracle.ojdbc</groupId> <artifactId>ojdbc8</artifactId> <version>${oracle.driver.version}</version> </dependency>
remember to Databases#AddingtheOracleojdbc7.jarmanually only if you use ojdbc.jdbc7
Add spring jpa starter
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-jpa</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency>
Add JPA Schema to Entity code generation via hbm2java
Add JPA Entity to Schema DDL generation via hbm2ddl
Add Spring Profiles to spring boot
Add External configuration to spring boot
To reference an external properties file, put the file in the current dir, in a config subdir or reference directly with --spring.config.location
biometric:bin michaelobrien$ ls application.properties config field-0.0.1-SNAPSHOT.jar biometric:bin michaelobrien$ java -jar field-0.0.1-SNAPSHOT.jar
Add spring.xml XML config to spring boot project
The default spring boot template no longer comes with XML config as it relies solely on annotations. You can however add a mix of XML and annotation based config.
Add the following spring.xml - for example to reference external Thymeleaf templates out of the current spring boot jar classpath - trailing slash is required.
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd"> <bean id="templateResolver" class="org.thymeleaf.templateresolver.FileTemplateResolver"> <property name="prefix" value="/Users/michaelobrien/wse_github/obrienlabs/eventstream/deployment/routes/welcome/" /> <property name="suffix" value=".html" /> <property name="templateMode" value="HTML5" /> <property name="cacheable" value="false" /> </bean> </beans>
Add the following extra spring configuration bean to load the xml file
import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.ImportResource; @Configuration @ImportResource({"classpath*:spring.xml"}) public class XmlConfiguration { }
Add Environment Variables inside spring xml configuration
Add the following property-placeholder to enable environment variables from your application.properties overrides - with no hardcoded parameters.
Or use a PropertySourcesPlaceholderConfigurer - see https://github.com/obrienlabs/biometric/blob/master/biometric.web/src/main/webapp/WEB-INF/spring.xml#L50
spring.xml <context:property-placeholder />// usage <property name="prefix" value="${thymleaf.templates.prefix}" /> application.properties thymleaf.templates.prefix=/Users/michaelobrien/wse_github/obrienlabs/eventstream/deployment/routes/welcome/
Add Junit to spring boot
Add Sonar to spring boot
Add a spring aspect based logging framework to spring boot
Add spring security to your REST endpoints in spring boot
https://www.baeldung.com/spring-boot-https-self-signed-certificate
Create a Dockerfile framework
Build/Run Docker endpoint on RKE EC2 VM
see - REF-2Getting issue details... STATUS
see https://github.com/obrienlabs/refarch/tree/master/reference-nbi/src/docker
Locally cd reference-nbi/src/docker/ ./build.sh [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 6.412 s [INFO] Finished at: 2020-04-27T13:37:58-04:00 [INFO] ------------------------------------------------------------------------ Sending build context to Docker daemon 32.15MB Step 1/6 : FROM openjdk:8 ---> e890fe39c302 Step 2/6 : ARG USERVICE_HOME=/opt/app/ ---> Running in 146aa23cf6a8 Removing intermediate container 146aa23cf6a8 ---> b17a9e0d3ae6 Step 3/6 : RUN mkdir -p $USERVICE_HOME ---> Running in e3581d8e5d7a Removing intermediate container e3581d8e5d7a ---> 48e719469a9f Step 4/6 : ADD reference-nbi-*.jar $USERVICE_HOME/lib/reference-nbi.jar ---> bbae15d7b576 Step 5/6 : ADD startService.sh $USERVICE_HOME/bin/ ---> 7f3a12fbf15a Step 6/6 : CMD ["/opt/app/bin/startService.sh"] ---> Running in 6bddc74b8a77 Removing intermediate container 6bddc74b8a77 ---> 067ab19c41c9 [Warning] One or more build-args [build-id] were not consumed Successfully built 067ab19c41c9 Successfully tagged obrienlabs/reference-nbi:latest The push refers to repository [docker.io/obrienlabs/reference-nbi] 3ed4cee3cd68: Pushed 44ca53031496: Pushed 626aa2565d15: Pushed c601709dd5d2: Layer already exists 72ce39f2b7f6: Layer already exists 33783834b288: Layer already exists 5c813a85f7f0: Layer already exists bdca38f94ff0: Layer already exists faac394a1ad3: Layer already exists ce8168f12337: Layer already exists 0.0.1: digest: sha256:6dc5082fa5dea76439b7d72b73d442cdeb2bf257798d257c5853093b0165be08 size: 2420 reference-nbi reference-nbi starting: reference-nbi 0e9ab09b8c4082e2f0859cac462c921b186084b019699d348bca1aa0a7c858db On VM ubuntu@ip-172-31-81-46:~$ docker run --name reference-nbi -d -p 8888:8080 obrienlabs/reference-nbi:0.0.1 ubuntu@ip-172-31-81-46:~$ curl http://127.0.0.1:8888/nbi/api {"id":1,"content":"1 PASS cloud.containerization.reference.nbi.ApiController queryString: null decodedQueryString: "}
Create a Helm/Kubernetes deployment framework
see - REF-3Getting issue details... STATUS
see https://github.com/obrienlabs/refarch/commit/b255db3d2a7c6975f8a68d587617de7c2ef74411
Helm package and install chart
helm package reference-nbi helm install --name reference-nbi reference-nbi-0.1.0.tgz
Deployment
Kubernetes Cluster
Kubernetes on Docker Desktop OSX
Kubernetes on Docker Desktop Windows
Kubernetes on Minikube OSX VMware Fusion
Kubernetes on RKE Bare Metal Ubuntu 16
Kubernetes on RKE VMware OSX or Windows
Kubernetes on RKE on AWS EC2
Kubernetes Platform Services Infrastructure
Kubernetes Platform Services
service | Kubernetes | chart | artifacts | notes | links |
---|---|---|---|---|---|
dns | |||||
namespaces | |||||
storage | storageclass | ||||
security | certificate-manager | ||||
Network
API Gateway
An ingress/proxy that can act as an SSL terminator, load balancer, proxy, URL rewriter or ingress/egress traffic conditioner.
Storage
Default Storage Class
Security
Single Sign On
KeyCloak
MFA/2FA in KeyCloak - https://www.keycloak.org/docs/latest/server_admin/index.html#_webauthn
Certificate Management
Certificate Manager
The chart repo works at https://cert-manager.io/docs/installation/kubernetes/
there are issues with a couple of the docker images in https://github.com/jetstack/cert-manager defaulting to an older v0.1.0 - see a fix I posted to override with v0.15.2 in https://github.com/jetstack/cert-manager/issues/3104
git clone https://github.com/jetstack/cert-manager.git cd cert-manager/deploy/charts kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.2/cert-manager.crds.yaml sudo helm install --name cert-manager cert-manager/ --set installCRDs=true --set cainjector.image.tag=v0.15.1 --set webhook.image.tag=v0.15.1 --set image.tag=v0.15.1 --version v0.15.1 --namespace cert-manager cert-manager cert-manager-5c4c99cf54-cn7z7 1/1 Running 0 35s cert-manager cert-manager-cainjector-748895cb8f-84jc8 1/1 Running 0 35s cert-manager cert-manager-webhook-784bb44b6-lqldv 1/1 Running 0 35s :charts $ kubectl apply -f test-resources.yaml namespace/cert-manager-test created issuer.cert-manager.io/test-selfsigned created certificate.cert-manager.io/selfsigned-cert created
or an example without cloning the chart Example adding cert-manager to an RKE kubernetes cluster on VMware Fusion dont use the stable helm chart - it is deprecated kubectl create namespace cert-manager helm repo add jetstack https://charts.jetstack.io helm repo update for helm 2 add --name helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.1 # --set installCRDs=true or directly using kubernetes kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.yaml $ cat <<EOF > test-resources.yaml apiVersion: v1 kind: Namespace metadata: name: cert-manager-test --- apiVersion: cert-manager.io/v1alpha2 kind: Issuer metadata: name: test-selfsigned namespace: cert-manager-test spec: selfSigned: {} --- apiVersion: cert-manager.io/v1alpha2 kind: Certificate metadata: name: selfsigned-cert namespace: cert-manager-test spec: dnsNames: - example.com secretName: selfsigned-cert-tls issuerRef: name: test-selfsigned EOF $ kubectl apply -f test-resources.yaml namespace/cert-manager-test created issuer.cert-manager.io/test-selfsigned created certificate.cert-manager.io/selfsigned-cert created $ kubectl describe certificate -n cert-manager-test Name: selfsigned-cert Namespace: cert-manager-test Labels: <none> Annotations: API Version: cert-manager.io/v1alpha3 Kind: Certificate Metadata: Creation Timestamp: 2020-07-16T18:32:11Z Generation: 1 Managed Fields: API Version: cert-manager.io/v1alpha2 Fields Type: FieldsV1 fieldsV1: f:status: .: f:conditions: f:notAfter: Manager: controller Operation: Update Time: 2020-07-16T18:32:11Z API Version: cert-manager.io/v1alpha2 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:dnsNames: f:issuerRef: .: f:name: f:secretName: Manager: kubectl Operation: Update Time: 2020-07-16T18:32:11Z Resource Version: 297235 Self Link: /apis/cert-manager.io/v1alpha3/namespaces/cert-manager-test/certificates/selfsigned-cert UID: 507931ff-6f6a-46fc-8364-2582884029bc Spec: Dns Names: example.com Issuer Ref: Name: test-selfsigned Secret Name: selfsigned-cert-tls Status: Conditions: Last Transition Time: 2020-07-16T18:32:11Z Message: Certificate is up to date and has not expired Reason: Ready Status: True Type: Ready Not After: 2020-10-14T18:32:11Z Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal GeneratedKey 10s cert-manager Generated a new private key Normal Requested 10s cert-manager Created new CertificateRequest resource "selfsigned-cert-504566127" Normal Issued 10s cert-manager Certificate issued successfully
Monitoring
Prometheus
Grafana
Logging
https://platform9.com/blog/kubernetes-logging-comparing-fluentd-vs-logstash/
ElasticSearch
Logstash/FluentD
Kibana
Filebeat
Infrastructure
Gerrit
Jenkins
Sonar
Nexus / Repository
see
Confluence
Jira
Messaging
Kafka / Zookeeper Queue
Persistence
Relational Database: MariaDB / MySQL
Key Value Store : etcd
CI/CD
Gitlab CI/CD
Add a kubernetes cluster
I am running the gitlab service and not my own gitlab instance so I am limited to to the GKE instead of Azure AKS or AWS EKS.
https://gitlab.com/help/topics/autodevops/quick_start_guide
https://gitlab.com/refarch/reference/-/settings/ci_cd#autodevops-settings
Kubernetes RKE Cluster on 4 Intel NUC machines with 64G RAM
Work Items
WI | Task | Jira |
---|---|---|
Add terraform infrastructure | ||
Investigate Tech
neem
nodejs + arangoDB