dev
Page tree
Skip to end of metadata
Go to start of metadata

bKubernetes Developer Guide | Helm Development Guide | Reference Architecture

Introduction

This page details and example cloud based microservice architecture.

Spring boot based reference architecture - Reference Architecture, Spring boot jar running the spring embedded tomcat container - Spring Boot Microservice

Architecture

Deployment Diagram

Kubernetes Reference Architecture


DevOps

Originally on github https://github.com/obrienlabs/refarch- moved to gitlab

siteurl
git

https://github.com/obrienlabs/refarch

https://gitlab.com/refarch/reference

docker hubhttps://hub.docker.com/repository/docker/obrienlabs/reference-nbi 
gitlab  CI/CD
jenkins CI

http://jenkins.obrienlabs.cloud/job/ref-arch-java-gitlab/

Backup

http://jenkins.obrienlabs.cloud/job/ref-arch-java/

base endpointhttp://services.obrienlabs.cloud:8888/nbi/api 
Jira

https://gitlab.com/refarch/reference/-/issues/1

REF-1 - Getting issue details... STATUS

Quickstart

Technology Coverage - Technology

Clone

# using your ssh key (o*_r*)
:wse_gitlab $ git clone git@gitlab.com:refarch/reference

Build

:reference $ mvn clean install -U -T 16 -DskipTests=true
[INFO] Reactor Build Order:
[INFO] 
[INFO] reference-nbi                                                      [jar]
[INFO] reference                                                          [pom]
[INFO] 
[INFO] Using the MultiThreadedBuilder implementation with a thread count of 16

Commit/Push

#


Build/Run Docker endpoint on RKE EC2 VM


Locally
cd reference-nbi/src/docker/
./build.sh 
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.412 s
[INFO] Finished at: 2020-04-27T13:37:58-04:00
[INFO] ------------------------------------------------------------------------
Sending build context to Docker daemon  32.15MB
Step 1/6 : FROM openjdk:8
 ---> e890fe39c302
Step 2/6 : ARG USERVICE_HOME=/opt/app/
 ---> Running in 146aa23cf6a8
Removing intermediate container 146aa23cf6a8
 ---> b17a9e0d3ae6
Step 3/6 : RUN mkdir -p $USERVICE_HOME
 ---> Running in e3581d8e5d7a
Removing intermediate container e3581d8e5d7a
 ---> 48e719469a9f
Step 4/6 : ADD reference-nbi-*.jar $USERVICE_HOME/lib/reference-nbi.jar
 ---> bbae15d7b576
Step 5/6 : ADD startService.sh $USERVICE_HOME/bin/
 ---> 7f3a12fbf15a
Step 6/6 : CMD ["/opt/app/bin/startService.sh"]
 ---> Running in 6bddc74b8a77
Removing intermediate container 6bddc74b8a77
 ---> 067ab19c41c9
[Warning] One or more build-args [build-id] were not consumed
Successfully built 067ab19c41c9
Successfully tagged obrienlabs/reference-nbi:latest
The push refers to repository [docker.io/obrienlabs/reference-nbi]
3ed4cee3cd68: Pushed 
44ca53031496: Pushed 
626aa2565d15: Pushed 
c601709dd5d2: Layer already exists 
72ce39f2b7f6: Layer already exists 
33783834b288: Layer already exists 
5c813a85f7f0: Layer already exists 
bdca38f94ff0: Layer already exists 
faac394a1ad3: Layer already exists 
ce8168f12337: Layer already exists 
0.0.1: digest: sha256:6dc5082fa5dea76439b7d72b73d442cdeb2bf257798d257c5853093b0165be08 size: 2420
reference-nbi
reference-nbi
starting: reference-nbi
0e9ab09b8c4082e2f0859cac462c921b186084b019699d348bca1aa0a7c858db

On VM
ubuntu@ip-172-31-81-46:~$ docker run --name reference-nbi -d -p 8888:8080 obrienlabs/reference-nbi:0.0.1
ubuntu@ip-172-31-81-46:~$ curl http://127.0.0.1:8888/nbi/api
{"id":1,"content":"1 PASS cloud.containerization.reference.nbi.ApiController queryString: null decodedQueryString: "}

Design

Agenda: 

Create initial spring boot maven project

Create a new project via the spring initializr https://start.spring.io/ see https://spring.io/guides/gs/serving-web-content/


pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.2.6.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>
	<groupId>cloud.dev9</groupId>
	<artifactId>reference-sb-nbi</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>reference-sb-nbi</name>
	<description>Reference sb NBI</description>

	<properties>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-jersey</artifactId>
		</dependency>
		<!--dependency><!-- user:user -->
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-security</artifactId>
		</dependency-->
		<dependency>
			<groupId>org.apache.kafka</groupId>
			<artifactId>kafka-streams</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.kafka</groupId>
			<artifactId>spring-kafka</artifactId>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-devtools</artifactId>
			<scope>runtime</scope>
			<optional>true</optional>
		</dependency>
		<dependency>
			<groupId>com.oracle.ojdbc</groupId>
			<artifactId>ojdbc8</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<optional>true</optional>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
			<exclusions>
				<exclusion>
					<groupId>org.junit.vintage</groupId>
					<artifactId>junit-vintage-engine</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>org.springframework.kafka</groupId>
			<artifactId>spring-kafka-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

</project>

Add Swagger and OpenAPI 3 

API - OpenAPI - Swagger#OpenAPI-Swagger-AddSwagger2toanexistingMavenJavaproject

Configure IDEs for development

Add Lombok APT IDE tool

Use Java 16 Records via https://openjdk.java.net/jeps/395 instead of Lombok as of March 2021

https://www.baeldung.com/lombok-ide and https://search.maven.org/artifact/org.projectlombok/lombok-maven-plugin/1.18.12.0/maven-plugin

Download and run the latest jar to adjust your eclipse.ini and eclipse/sts runtime - https://search.maven.org/remotecontent?filepath=org/projectlombok/lombok/1.18.12/lombok-1.18.12.jar

# add to ini file
-javaagent:lombok-1.18.12

# copy jar to show packet contentsContents/MacOS// add to pom.xml<lombok.version>1.18.12</lombok.version><dependency>
  <groupId>org.projectlombok</groupId>
  <artifactId>lombok</artifactId>
  <version>${lombok.version}</version>
</dependency>

// add to your classes
@Data

Lombok in IntelliJ

for 2020.03 - 



2020.03

File | Settings | Build, Execution, Deployment | Compiler | Build process VM options add : -Djps.track.ap.dependencies=false
https://github.com/mplushnikov/lombok-intellij-plugin/issues/988

Add eclEmma code coverage



Add a default REST endpoint to spring boot

Add the context root to application.properties

server.servlet.context-path=/sb-nbi

Add the convenience annotation to the Application class

@SpringBootApplication

package cloud.containerization.reference.nbi;

import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMapping;

@RestController
public class DefaultRestController {

	@RequestMapping("/")
	public String index() {
		return "Greetings from Spring Boot!";
	}
}


Verify the endpoint

http://127.0.0.1:8180/sb-nbi/actuator/health


Add a JPA 2.2 Hibernate provider repository bean to spring boot

see Object Relational Mapping

https://mvnrepository.com/artifact/org.eclipse.persistence/javax.persistence/2.2.0

pom.xml

# pom.xml changes
	    <dependency>
            <groupId>org.eclipse.persistence</groupId>
            <artifactId>javax.persistence</artifactId>
            <version>2.2.0</version>
        </dependency>

# mysql
	    <groupId>mysql</groupId>
	    <artifactId>mysql-connector-java</artifactId>
	    <version>5.1.42</version>
	    <scope>runtime</scope>
	</dependency>

# postgreSQL

# oracle
<oracle.driver.version>12.1.0.2</oracle.driver.version>

        <dependency>
            <groupId>com.oracle.ojdbc</groupId>                                                                                                   
            <artifactId>ojdbc8</artifactId>    
            <version>${oracle.driver.version}</version>
        </dependency>

remember to Databases#AddingtheOracleojdbc7.jarmanually only if you use ojdbc.jdbc7


Add spring jpa starter


        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>


Add JPA Schema to Entity code generation via hbm2java

Add JPA Entity to Schema DDL generation via hbm2ddl


Add Spring Profiles to spring boot 

Add External configuration to spring boot

https://docs.spring.io/spring-boot/docs/2.4.3/reference/html/spring-boot-features.html#boot-features-external-config

https://docs-stage.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config

To reference an external properties file, put the file in the current dir, in a config subdir or reference directly with --spring.config.location

biometric:bin michaelobrien$ ls
application.properties		config				field-0.0.1-SNAPSHOT.jar
biometric:bin michaelobrien$ java -jar field-0.0.1-SNAPSHOT.jar


Add spring.xml XML config to spring boot project

The default spring boot template no longer comes with XML config as it relies solely on annotations.  You can however add a mix of XML and annotation based config.

Add the following spring.xml - for example to reference external Thymeleaf templates out of the current spring boot jar classpath - trailing slash is required.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/context
        http://www.springframework.org/schema/context/spring-context.xsd">

  <bean id="templateResolver" class="org.thymeleaf.templateresolver.FileTemplateResolver">
    <property name="prefix" value="/Users/michaelobrien/wse_github/obrienlabs/eventstream/deployment/routes/welcome/" />
    <property name="suffix" value=".html" />
    <property name="templateMode" value="HTML5" />
    <property name="cacheable" value="false" />
  </bean>
</beans>


Add the following extra spring configuration bean to load the xml file

import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.ImportResource;

@Configuration
@ImportResource({"classpath*:spring.xml"})
public class XmlConfiguration {
}

Add Environment Variables inside spring xml configuration

Add the following property-placeholder to enable environment variables from your application.properties overrides - with no hardcoded parameters.

Or use a PropertySourcesPlaceholderConfigurer - see https://github.com/obrienlabs/biometric/blob/master/biometric.web/src/main/webapp/WEB-INF/spring.xml#L50

spring.xml
<context:property-placeholder />// usage
<property name="prefix" value="${thymleaf.templates.prefix}" />

application.properties
thymleaf.templates.prefix=/Users/michaelobrien/wse_github/obrienlabs/eventstream/deployment/routes/welcome/


Add Junit to spring boot

Add Sonar to spring boot


Add a spring aspect based logging framework to spring boot

Add spring security to your REST endpoints in spring boot

https://www.baeldung.com/spring-boot-https-self-signed-certificate


Create a Dockerfile framework

Build/Run Docker endpoint on RKE EC2 VM

see  REF-2 - Getting issue details... STATUS

see https://github.com/obrienlabs/refarch/tree/master/reference-nbi/src/docker

Locally
cd reference-nbi/src/docker/
./build.sh 
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.412 s
[INFO] Finished at: 2020-04-27T13:37:58-04:00
[INFO] ------------------------------------------------------------------------
Sending build context to Docker daemon  32.15MB
Step 1/6 : FROM openjdk:8
 ---> e890fe39c302
Step 2/6 : ARG USERVICE_HOME=/opt/app/
 ---> Running in 146aa23cf6a8
Removing intermediate container 146aa23cf6a8
 ---> b17a9e0d3ae6
Step 3/6 : RUN mkdir -p $USERVICE_HOME
 ---> Running in e3581d8e5d7a
Removing intermediate container e3581d8e5d7a
 ---> 48e719469a9f
Step 4/6 : ADD reference-nbi-*.jar $USERVICE_HOME/lib/reference-nbi.jar
 ---> bbae15d7b576
Step 5/6 : ADD startService.sh $USERVICE_HOME/bin/
 ---> 7f3a12fbf15a
Step 6/6 : CMD ["/opt/app/bin/startService.sh"]
 ---> Running in 6bddc74b8a77
Removing intermediate container 6bddc74b8a77
 ---> 067ab19c41c9
[Warning] One or more build-args [build-id] were not consumed
Successfully built 067ab19c41c9
Successfully tagged obrienlabs/reference-nbi:latest
The push refers to repository [docker.io/obrienlabs/reference-nbi]
3ed4cee3cd68: Pushed 
44ca53031496: Pushed 
626aa2565d15: Pushed 
c601709dd5d2: Layer already exists 
72ce39f2b7f6: Layer already exists 
33783834b288: Layer already exists 
5c813a85f7f0: Layer already exists 
bdca38f94ff0: Layer already exists 
faac394a1ad3: Layer already exists 
ce8168f12337: Layer already exists 
0.0.1: digest: sha256:6dc5082fa5dea76439b7d72b73d442cdeb2bf257798d257c5853093b0165be08 size: 2420
reference-nbi
reference-nbi
starting: reference-nbi
0e9ab09b8c4082e2f0859cac462c921b186084b019699d348bca1aa0a7c858db

On VM
ubuntu@ip-172-31-81-46:~$ docker run --name reference-nbi -d -p 8888:8080 obrienlabs/reference-nbi:0.0.1
ubuntu@ip-172-31-81-46:~$ curl http://127.0.0.1:8888/nbi/api
{"id":1,"content":"1 PASS cloud.containerization.reference.nbi.ApiController queryString: null decodedQueryString: "}

Create a Helm/Kubernetes deployment framework

see  REF-3 - Getting issue details... STATUS

see https://github.com/obrienlabs/refarch/commit/b255db3d2a7c6975f8a68d587617de7c2ef74411

Helm package and install chart

helm package reference-nbi
helm install --name reference-nbi reference-nbi-0.1.0.tgz



Deployment

Kubernetes Cluster

Kubernetes on Docker Desktop OSX

Kubernetes on Docker Desktop Windows

Kubernetes on Minikube OSX VMware Fusion

Kubernetes on RKE Bare Metal Ubuntu 16

Kubernetes on RKE VMware OSX or Windows

Kubernetes on RKE on AWS EC2


Kubernetes Platform Services Infrastructure

Kubernetes Platform Services 

serviceKuberneteschartartifactsnoteslinks
dns




namespaces




storage
storageclass


security
certificate-manager





















Network

API Gateway

An ingress/proxy that can act as an SSL terminator, load balancer, proxy, URL rewriter or ingress/egress traffic conditioner.

Storage

Default Storage Class

Security

Single Sign On

KeyCloak

https://www.keycloak.org/

MFA/2FA in KeyCloak - https://www.keycloak.org/docs/latest/server_admin/index.html#_webauthn

Certificate Management

Certificate Manager

The chart repo works at https://cert-manager.io/docs/installation/kubernetes/

there are issues with a couple of the docker images in https://github.com/jetstack/cert-manager defaulting to an older v0.1.0 - see a fix I posted to override with v0.15.2 in https://github.com/jetstack/cert-manager/issues/3104

git clone https://github.com/jetstack/cert-manager.git
cd cert-manager/deploy/charts
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.2/cert-manager.crds.yaml
sudo helm install --name cert-manager cert-manager/ --set installCRDs=true --set cainjector.image.tag=v0.15.1 --set webhook.image.tag=v0.15.1 --set image.tag=v0.15.1 --version v0.15.1 --namespace cert-manager

cert-manager   cert-manager-5c4c99cf54-cn7z7              1/1     Running   0          35s
cert-manager   cert-manager-cainjector-748895cb8f-84jc8   1/1     Running   0          35s
cert-manager   cert-manager-webhook-784bb44b6-lqldv       1/1     Running   0          35s

:charts $ kubectl apply -f test-resources.yaml 
namespace/cert-manager-test created
issuer.cert-manager.io/test-selfsigned created
certificate.cert-manager.io/selfsigned-cert created


or an example without cloning the chart
Example adding cert-manager to an RKE kubernetes cluster on VMware Fusion
dont use the stable helm chart - it is deprecated

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update

for helm 2 add --name
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.1
  # --set installCRDs=true

or directly using kubernetes
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.yaml

$ cat <<EOF > test-resources.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-test
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: test-selfsigned
  namespace: cert-manager-test
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: selfsigned-cert
  namespace: cert-manager-test
spec:
  dnsNames:
    - example.com
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned
EOF

$ kubectl apply -f test-resources.yaml
namespace/cert-manager-test created
issuer.cert-manager.io/test-selfsigned created
certificate.cert-manager.io/selfsigned-cert created
$ kubectl describe certificate -n cert-manager-test
Name:         selfsigned-cert
Namespace:    cert-manager-test
Labels:       <none>
Annotations:  API Version:  cert-manager.io/v1alpha3
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-07-16T18:32:11Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1alpha2
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:notAfter:
    Manager:      controller
    Operation:    Update
    Time:         2020-07-16T18:32:11Z
    API Version:  cert-manager.io/v1alpha2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:dnsNames:
        f:issuerRef:
          .:
          f:name:
        f:secretName:
    Manager:         kubectl
    Operation:       Update
    Time:            2020-07-16T18:32:11Z
  Resource Version:  297235
  Self Link:         /apis/cert-manager.io/v1alpha3/namespaces/cert-manager-test/certificates/selfsigned-cert
  UID:               507931ff-6f6a-46fc-8364-2582884029bc
Spec:
  Dns Names:
    example.com
  Issuer Ref:
    Name:       test-selfsigned
  Secret Name:  selfsigned-cert-tls
Status:
  Conditions:
    Last Transition Time:  2020-07-16T18:32:11Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2020-10-14T18:32:11Z
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  GeneratedKey  10s   cert-manager  Generated a new private key
  Normal  Requested     10s   cert-manager  Created new CertificateRequest resource "selfsigned-cert-504566127"
  Normal  Issued        10s   cert-manager  Certificate issued successfully


Monitoring

Prometheus

Grafana

Logging

https://platform9.com/blog/kubernetes-logging-comparing-fluentd-vs-logstash/

ElasticSearch

Logstash/FluentD

Kibana

Filebeat

Infrastructure

Gerrit

Jenkins

Sonar

Nexus / Repository

see 

Confluence

Jira

Messaging

Kafka / Zookeeper Queue

Persistence

Relational Database: MariaDB / MySQL

Key Value Store : etcd


CI/CD

Gitlab CI/CD

Add a kubernetes cluster

I am running the gitlab service and not my own gitlab instance so I am limited to to the GKE instead of Azure AKS or AWS EKS.
 https://gitlab.com/help/topics/autodevops/quick_start_guide

https://gitlab.com/refarch/reference/-/settings/ci_cd#autodevops-settings

Kubernetes RKE Cluster on 4 Intel NUC machines with 64G RAM


Work Items

WITaskJira

Add terraform infrastructure 






Investigate Tech

neem

nodejs + arangoDB



  • No labels

http://jira.obrienlabs.cloud

https://github.com/obrienlabs

http://eclipsejpa.blogspot.ca/

https://www.linkedin.com/in/michaelobrien-developer/

https://twitter.com/_mikeobrien