Architecture

Decoupled microservices need an asynchronous publish/subscribe queue.

We will use Kafka for the messaging queue, Zookeeper for the key/value store and streaming processor.

Cloud Kafka as a Service

AWS: AWS MSK - Managed Streaming for Kafka

Azure: https://azure.microsoft.com/en-ca/blog/announcing-the-general-availability-of-azure-event-hubs-for-apache-kafka/

Kubernetes Cluster

Use any available Kubernetes cluster like RKE or Docker Desktop - as long as you have kubectl installed

Install helm

Kafka

Kafka Operator Charts

https://github.com/banzaicloud/kafka-operator

see Helm Development Guide

Kafka Installation using Helm Charts

see https://helm.sh/docs/helm/helm_install/ and the incubator chart https://github.com/helm/charts/tree/master/incubator/kafka

Run the current kafka 2.0 helm chart (5.0.1) consisting of a 3 node zookeeper stateful set and a 3 node kafka stateful set

# running Kubernetes 1.15.5 and Helm 3.0.3
:wse_helm $ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:16:51Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
:wse_helm $ helm version
version.BuildInfo{Version:"v3.0.3", GitCommit:"ac925eb7279f4a6955df663a0128044a8a6b7593", GitTreeState:"clean", GoVersion:"go1.13.7"}

# update the repo
helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator

:wse_helm $ helm install kafka incubator/kafka
or helm 2
helm install --name kafka incubator/kafka
NAME: kafka
LAST DEPLOYED: Sat Feb  8 19:28:41 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
### Connecting to Kafka from inside Kubernetes
You can connect to Kafka by running a simple pod in the K8s cluster like this with a configuration like this:

# don't use confluentinc/cp-kafka:5.0.1 - use solsson/kafka:0.11.0.0
apiVersion: v1
kind: Pod
metadata:
  name: testclient
  namespace: default
spec:
  containers:
  - name: kafka
    image: confluentinc/cp-kafka:5.0.1
    command:
      - sh
      - -c
      - "exec tail -f /dev/null"

Once you have the testclient pod above running, you can list all kafka
topics with:
  kubectl -n default exec testclient -- kafka-topics --zookeeper kafka-zookeeper:2181 --list

To create a new topic:
  kubectl -n default exec testclient -- kafka-topics --zookeeper kafka-zookeeper:2181 --topic test1 --create --partitions 1 --replication-factor 1

To listen for messages on a topic:
  kubectl -n default exec -ti testclient -- kafka-console-consumer --bootstrap-server kafka:9092 --topic test1 --from-beginning

To stop the listener session above press: Ctrl+C
To start an interactive message producer session:
  kubectl -n default exec -ti testclient -- kafka-console-producer --broker-list kafka-headless:9092 --topic test1

To create a message in the above session, simply type the message and press "enter"
To end the producer session try: Ctrl+C

If you specify "zookeeper.connect" in configurationOverrides, please replace "kafka-zookeeper:2181" with the value of "zookeeper.connect", or you will get error.

:main $ kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
default       kafka-0                                  0/1     Running   1          72s
default       kafka-zookeeper-0                        1/1     Running   0          72s
default       kafka-zookeeper-1                        0/1     Running   0          23s

:main $ kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
default       kafka-0                                  0/1     Running   1          89s
default       kafka-zookeeper-0                        1/1     Running   0          89s
default       kafka-zookeeper-1                        1/1     Running   0          40s
default       kafka-zookeeper-2                        0/1     Running   0          15s

:main $ kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
default       kafka-0                                  1/1     Running   1          2m14s
default       kafka-1                                  0/1     Running   0          32s
default       kafka-zookeeper-0                        1/1     Running   0          2m14s
default       kafka-zookeeper-1                        1/1     Running   0          85s
default       kafka-zookeeper-2                        1/1     Running   0          60s

:main $ kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
default       kafka-0                                  1/1     Running   1          2m35s
default       kafka-1                                  1/1     Running   0          53s
default       kafka-2                                  0/1     Running   0          16s
default       kafka-zookeeper-0                        1/1     Running   0          2m35s
default       kafka-zookeeper-1                        1/1     Running   0          106s
default       kafka-zookeeper-2                        1/1     Running   0          81s

:main $ kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
default       kafka-0                                  1/1     Running   1          2m58s
default       kafka-1                                  1/1     Running   0          76s
default       kafka-2                                  1/1     Running   0          39s
default       kafka-zookeeper-0                        1/1     Running   0          2m58s
default       kafka-zookeeper-1                        1/1     Running   0          2m9s
default       kafka-zookeeper-2                        1/1     Running   0          104s

:main $ kubectl get services --all-namespaces
NAMESPACE     NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default       kafka                      ClusterIP   10.100.190.143   <none>        9092/TCP                     135m
default       kafka-headless             ClusterIP   None             <none>        9092/TCP                     135m
default       kafka-zookeeper            ClusterIP   10.103.230.77    <none>        2181/TCP                     135m
default       kafka-zookeeper-headless   ClusterIP   None             <none>        2181/TCP,3888/TCP,2888/TCP   135m

Reference Helm Charts

rerun 20200928

navigate to the incubator repo, copy the kafka chart, copy the zookeeper chart to kafka/charts/zookeeper

https://github.com/helm/charts/tree/master/incubator/kafka

Kafka/Zookeeper on Docker Desktop OSX

biometric:reference michaelobrien$ git clone https://github.com/helm/charts.git
Cloning into 'charts'...
remote: Enumerating objects: 111933, done.

biometric:reference michaelobrien$ mkdir kafka
biometric:reference michaelobrien$ cp -R charts/incubator/kafka/ kafka
biometric:reference michaelobrien$ mkdir kafka/charts
biometric:reference michaelobrien$ cp -R charts/incubator/zookeeper/ kafka/charts/zookeeper
biometric:reference michaelobrien$ ls
_incubator	charts		kafka

biometric:reference michaelobrien$ helm3 install kafka kafka
NAME: kafka
LAST DEPLOYED: Mon Sep 28 20:08:22 2020
NAMESPACE: default

biometric:reference michaelobrien$ helm3 ls
NAME 	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART       	APP VERSION
kafka	default  	1       	2020-09-28 20:08:22.946823 -0400 EDT	deployed	kafka-0.21.2	5.0.1      
biometric:reference michaelobrien$ kubectl get pods --all-namespaces | grep kafka
default       kafka-0                                  0/1     Running   1          55s
default       kafka-zookeeper-0                        1/1     Running   0          55s
default       kafka-zookeeper-1                        0/1     Running   0          18s

fully up in 5 min
biometric:reference michaelobrien$ kubectl get pods --all-namespaces | grep kafka
default       kafka-0                                  1/1     Running   3          5m28s
default       kafka-1                                  1/1     Running   0          2m9s
default       kafka-2                                  1/1     Running   0          95s
default       kafka-zookeeper-0                        1/1     Running   2          5m28s
default       kafka-zookeeper-1                        1/1     Running   1          4m51s
default       kafka-zookeeper-2                        1/1     Running   0          3m51s

biometric:reference michaelobrien$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-kafka-0   Bound    pvc-ffbcd410-172d-45c6-b46f-85f923678b25   1Gi        RWO            hostpath       5m35s
datadir-kafka-1   Bound    pvc-88b53906-b026-4b97-bed2-29adb3c223c2   1Gi        RWO            hostpath       2m16s
datadir-kafka-2   Bound    pvc-90bf2c10-a6fb-478a-91fb-e544ca52a1c3   1Gi        RWO            hostpath       102s
biometric:reference michaelobrien$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-88b53906-b026-4b97-bed2-29adb3c223c2   1Gi        RWO            Delete           Bound    default/datadir-kafka-1   hostpath                2m21s
pvc-90bf2c10-a6fb-478a-91fb-e544ca52a1c3   1Gi        RWO            Delete           Bound    default/datadir-kafka-2   hostpath                107s
pvc-ffbcd410-172d-45c6-b46f-85f923678b25   1Gi        RWO            Delete           Bound    default/datadir-kafka-0   hostpath                5m40s

# delete kafka chart
helm3 delete kafka
# optionally delete pvc s which will delete pv s
kubectl delete pvc --all

Kafka/Zookeeper on Docker Desktop Windows

PS F:\wse_helm\reference> docker version
Client: Docker Engine - Community
 Azure integration  0.1.15
 Version:           19.03.12
PS F:\wse_helm\reference> kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
PS F:\wse_helm\reference> helm version
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}

PS F:\wse_helm\reference> git clone https://github.com/helm/charts.git
Cloning into 'charts'...
remote: Enumerating objects: 111933, done.

PS F:\wse_helm\reference> cp -r .\charts\incubator\kafka\ .
PS F:\wse_helm\reference> cp -r .\charts\incubator\zookeeper\ kafka/charts/zookeeper
PS F:\wse_helm\reference> helm install kafka kafka
NAME: kafka
LAST DEPLOYED: Mon Sep 28 22:23:28 2020

PS F:\wse_helm\reference> kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
default       kafka-0                                  1/1     Running   1          5m32s
default       kafka-1                                  1/1     Running   0          3m48s
default       kafka-2                                  1/1     Running   0          3m13s
default       kafka-zookeeper-0                        1/1     Running   1          5m32s
default       kafka-zookeeper-1                        1/1     Running   0          4m44s
default       kafka-zookeeper-2                        1/1     Running   0          4m17s

PS F:\wse_helm\reference> kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-3b498aa8-cb4d-4a50-96ec-7a2e462920e2   1Gi        RWO            Delete           Bound    default/datadir-kafka-0   hostpath                5m51s
pvc-76071f34-da92-469d-a34e-611211c144aa   1Gi        RWO            Delete           Bound    default/datadir-kafka-1   hostpath                4m7s
pvc-7ab7fea3-6600-4ac9-ae87-320d084f3a99   1Gi        RWO            Delete           Bound    default/datadir-kafka-2   hostpath                3m32s
PS F:\wse_helm\reference> kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-kafka-0   Bound    pvc-3b498aa8-cb4d-4a50-96ec-7a2e462920e2   1Gi        RWO            hostpath       5m54s
datadir-kafka-1   Bound    pvc-76071f34-da92-469d-a34e-611211c144aa   1Gi        RWO            hostpath       4m10s
datadir-kafka-2   Bound    pvc-7ab7fea3-6600-4ac9-ae87-320d084f3a99   1Gi        RWO            hostpath       3m35s
PS F:\wse_helm\reference> helm ls
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
kafka   default         1               2020-09-28 22:23:28.0147525 -0400 EDT   deployed        kafka-0.21.2    5.0.1

Running a Kafka client inside the default kubernetes namespace

:kafka $ vi kakfa-client.yaml
apiVersion: v1
kind: Pod
metadata:
  name: testclient
  namespace: default
spec:
  containers:
  - name: kafka
    image: solsson/kafka:0.11.0.0
    command:
      - sh
      - -c
      - "exec tail -f /dev/null"
                             
kubectl apply -f kakfa-client.yaml 
pod/testclient created
default       testclient                               1/1     Running   0          11s

# kubernetes 1.18
kubectl exec -it testclient -- bash
# kubernetes 1.16
kubectl exec -it testclient bash
root@testclient:/opt/kafka#cd bin 
root@testclient:/opt/kafka/bin# ./kafka-topics.sh --zookeeper kafka-zookeeper:2181 --list
root@testclient:/opt/kafka/bin# ./kafka-topics.sh --zookeeper kafka-zookeeper:2181 --topic test1 --create --partitions 1 --replication-factor 1
Created topic "test1".

Run a kafka producer and consumer across 2 kubernetes client pod shells

# producer
root@testclient:/opt/kafka/bin# ./kafka-console-producer.sh --broker-list kafka-headless:9092 --topic test1
>message1
>message2

# client
root@testclient:/opt/kafka/bin# ./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test1 --from-beginning
message1
message2
^C
Processed a total of 2 messages
root@testclient:/opt/kafka/bin# exit

Kafka on Kubernetes 1.16.6 over Docker 19.03.8 via Docker Desktop 2.3.0.2

20200513 rerun after Kubernetes 1.16 upgrade

# refresh helm charts
:charts $ kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

:charts $ helm repo list
NAME        	URL                                                      
incubator   	http://storage.googleapis.com/kubernetes-charts-incubator
stable      	https://kubernetes-charts.storage.googleapis.com         
oteemocharts	https://oteemo.github.io/charts

Troubleshoot PVC on Kubernetes Cluster without a default Storage Class

see: out of the box RKE does not ship with a default storageclass or provisioner - these need to be added post cluster install Kubernetes Developer Guide#AddLocalstorageclassforPVCprovisioningtoRKE

$ kubectl describe pod kafka-0
Name:           kafka-0

  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-kafka-0
    ReadOnly:   false
  default-token-d2z97:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-d2z97
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind.
$ kubectl get pv
No resources found in default namespace.
$ kubectl get pvc
NAME              STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
datadir-kafka-0   Pending                                      local-storage   98s
$ sudo helm delete kafka 
release "kafka" deleted
$ kubectl get pvc
NAME              STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
datadir-kafka-0   Pending                                      local-storage   7m35s
$ sudo helm del --purge kafka
release "kafka" deleted
$ kubectl delete pvc/datadir-kafka-0
persistentvolumeclaim "datadir-kafka-0" deleted

or delete pvc before deleting pv (s)
 kubectl delete pvc --all
 kubectl delete pv --all

Add Additional storage provisioner

see adding the Rancher Longhorn storageclass/provisioner in Kubernetes Developer Guide#AddRancherChartandcert-managerChart

Modifying/packaging Charts

Note: temporarily move the zookeeper chart off a created "charts" directory off kafka - if you are packaging it yourself and not using the repo

:incubator michaelobrien$ cd kafka
:kafka michaelobrien$ sudo helm dependency update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "oteemocharts" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading zookeeper from repo https://kubernetes-charts-incubator.storage.googleapis.com/
Deleting outdated charts
:kafka michaelobrien$ cd ..
:incubator michaelobrien$ sudo helm package kafka
Successfully packaged chart and saved it to: /Users/michaelobrien/wse_github/charts/incubator/kafka-0.20.9.tgz
:incubator michaelobrien$ ls *.tgz
kafka-0.20.9.tgz	zookeeper-2.1.3.tgz

Successfully packaged chart and saved it to: /Users/michaelobrien/wse_github/charts/incubator/kafka-0.20.9.tgz
:incubator michaelobrien$ ls -la /Users/michaelobrien/Library/Caches/helm/repository
total 20544
drwxr-xr-x  13 michaelobrien  staff      416 13 Jul 23:22 .
drwxr-xr-x   3 michaelobrien  staff       96  8 Feb 19:24 ..
-rw-r--r--   1 root           staff      898 13 Jul 23:05 incubator-charts.txt
-rw-r--r--   1 michaelobrien  staff   843920 13 Jul 23:05 incubator-index.yaml

:incubator michaelobrien$ sudo helm package kafka
:incubator michaelobrien$ sudo helm install kafka kafka-0.20.9.tgz 
NAME: kafka

check that pv/pvc s are coming in bound
:incubator michaelobrien$ kubectl get pvc --all-namespaces
NAMESPACE   NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default     datadir-kafka-0   Bound    pvc-f1712213-cd18-42f0-a78a-20a1b0acf357   1Gi        RWO            standard       8s
:incubator michaelobrien$ kubectl get pv --all-namespaces
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-f1712213-cd18-42f0-a78a-20a1b0acf357   1Gi        RWO            Delete           Bound    default/datadir-kafka-0   standard                20s

note that in constrained clusters like minikube - you will get some restarts/crashloopbackoffs initially

:incubator michaelobrien$ kubectl get pods --all-namespaces | grep kafka
default        kafka-0                                   0/1     Error     2          2m19s
default        kafka-zookeeper-0                         0/1     Running   2          2m19s
default        kafka-zookeeper-1                         0/1     Running   1          89s
default        kafka-zookeeper-2                         1/1     Running   0          46s
:incubator michaelobrien$ kubectl get pods --all-namespaces | grep kafka
default        kafka-0                                   0/1     CrashLoopBackOff   2          2m37s
default        kafka-zookeeper-0                         1/1     Running            2          2m37s
default        kafka-zookeeper-1                         1/1     Running            1          107s
default        kafka-zookeeper-2                         1/1     Running            0          64s

The will eventually start coming up
:incubator michaelobrien$ kubectl get pods --all-namespaces | grep kafka
default        kafka-0                                   1/1     Running             3          3m26s
default        kafka-1                                   0/1     ContainerCreating   0          2s
default        kafka-zookeeper-0                         1/1     Running             2          3m26s
default        kafka-zookeeper-1                         1/1     Running             1          2m36s
default        kafka-zookeeper-2                         1/1     Running             0          113s

check provisioner logs
iometric:incubator michaelobrien$ kubectl describe pvc datadir-kafka-0
  Normal  ExternalProvisioning   5m58s (x2 over 5m58s)  persistentvolume-controller                                    waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
  Normal  Provisioning           5m58s                  k8s.io/minikube-hostpath f31db6e7-c85e-11ea-9ea8-000c2969d6f9  External provisioner is provisioning volume for claim "default/datadir-kafka-0"
  Normal  ProvisioningSucceeded  5m58s                  k8s.io/minikube-hostpath f31db6e7-c85e-11ea-9ea8-000c2969d6f9  Successfully provisioned volume pvc-f1712213-cd18-42f0-a78a-20a1b0acf357

:incubator michaelobrien$ kubectl get pods --all-namespaces | grep kafka
default        kafka-0                                   1/1     Running   3          6m52s
default        kafka-1                                   1/1     Running   0          3m28s
default        kafka-2                                   1/1     Running   0          2m56s
default        kafka-zookeeper-0                         1/1     Running   2          6m52s
default        kafka-zookeeper-1                         1/1     Running   1          6m2s
default        kafka-zookeeper-2                         1/1     Running   0          5m19s

for reference - a faster vm

:incubator michaelobrien$ kubectl get pods --all-namespaces | grep kafka
default                kafka-0                                      1/1     Running   1          2m47s
default                kafka-1                                      1/1     Running   0          72s
default                kafka-2                                      1/1     Running   0          33s
default                kafka-zookeeper-0                            1/1     Running   0          2m47s
default                kafka-zookeeper-1                            1/1     Running   0          2m24s
default                kafka-zookeeper-2                            1/1     Running   0          114s

:incubator michaelobrien$ helm list
NAME 	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART       	APP VERSION
kafka	default  	1       	2020-07-13 23:24:05.547084 -0400 EDT	deployed	kafka-0.20.9	5.0.1 


Note: even though there are events in the log on failed bindings - all 3 kafka pods eventually bind to the PVC
you can see that kafka-0 had restarts, kafka-1 did not - but still has failure logs initially
:incubator michaelobrien$ kubectl describe pod kafka-1

  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "kafka-1": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "kafka-1": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled         <unknown>  default-scheduler  Successfully assigned default/kafka-1 to minikube
  Normal   Pulled            4m54s      kubelet, minikube  Container image "confluentinc/cp-kafka:5.0.1" already present on machine
  Normal   Created           4m54s      kubelet, minikube  Created container kafka-broker
  Normal   Started           4m54s      kubelet, minikube  Started container kafka-broker
:incubator michaelobrien$ kubectl get pv --all-namespaces
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-1f7ea6b6-b222-4b34-8124-ef4da0699c13   1Gi        RWO            Delete           Bound    default/datadir-kafka-2   standard                4m57s
pvc-b015ef5f-5c2a-4aec-80b1-248c1cd674d7   1Gi        RWO            Delete           Bound    default/datadir-kafka-1   standard                5m29s
pvc-f1712213-cd18-42f0-a78a-20a1b0acf357   1Gi        RWO            Delete           Bound    default/datadir-kafka-0   standard                8m53s
:incubator michaelobrien$ kubectl get pvc --all-namespaces
NAMESPACE   NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default     datadir-kafka-0   Bound    pvc-f1712213-cd18-42f0-a78a-20a1b0acf357   1Gi        RWO            standard       8m58s
default     datadir-kafka-1   Bound    pvc-b015ef5f-5c2a-4aec-80b1-248c1cd674d7   1Gi        RWO            standard       5m34s
default     datadir-kafka-2   Bound    pvc-1f7ea6b6-b222-4b34-8124-ef4da0699c13   1Gi        RWO            standard       5m2s

Deleting and fully removing the chart

Wait for all pods to stop
:incubator michaelobrien$ helm delete kafka
release "kafka" deleted
:incubator michaelobrien$ kubectl get pods --all-namespaces --watch
NAMESPACE      NAME                                      READY   STATUS        RESTARTS   AGE
cert-manager   cert-manager-7747db9d88-7zsqr             1/1     Running       1          115m
cert-manager   cert-manager-cainjector-87c85c6ff-6g499   1/1     Running       1          115m
cert-manager   cert-manager-webhook-64dc9fff44-rqhlr     1/1     Running       1          115m
default        kafka-1                                   1/1     Terminating   0          7m42s
default        kafka-2                                   1/1     Terminating   0          7m10s


:incubator michaelobrien$ kubectl get pvc --all-namespaces
NAMESPACE   NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default     datadir-kafka-0   Bound    pvc-6ea5527c-97a2-4ff4-b5a9-9b267a1b5f70   1Gi        RWO            standard       2d21h
default     datadir-kafka-1   Bound    pvc-07a5222d-a54d-4aae-b909-0858bd1024e0   1Gi        RWO            standard       2d21h
default     datadir-kafka-2   Bound    pvc-7eea5881-0dc2-4f4e-9b47-0b75e6d3541e   1Gi        RWO            standard       2d21h

:incubator michaelobrien$ kubectl get pv --all-namespaces
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-07a5222d-a54d-4aae-b909-0858bd1024e0   1Gi        RWO            Delete           Bound    default/datadir-kafka-1   standard                2d21h
pvc-6ea5527c-97a2-4ff4-b5a9-9b267a1b5f70   1Gi        RWO            Delete           Bound    default/datadir-kafka-0   standard                2d21h
pvc-7eea5881-0dc2-4f4e-9b47-0b75e6d3541e   1Gi        RWO            Delete           Bound    default/datadir-kafka-2   standard                2d21h

:incubator michaelobrien$ kubectl delete pvc/datadir-kafka-0
persistentvolumeclaim "datadir-kafka-0" deleted
:incubator michaelobrien$ kubectl delete pvc/datadir-kafka-1
persistentvolumeclaim "datadir-kafka-1" deleted
:incubator michaelobrien$ kubectl delete pvc/datadir-kafka-2
persistentvolumeclaim "datadir-kafka-2" deleted

:incubator michaelobrien$ kubectl delete pv/pvc-07a5222d-a54d-4aae-b909-0858bd1024e0
persistentvolume "pvc-07a5222d-a54d-4aae-b909-0858bd1024e0" deleted
:incubator michaelobrien$ kubectl delete pv/pvc-6ea5527c-97a2-4ff4-b5a9-9b267a1b5f70
persistentvolume "pvc-6ea5527c-97a2-4ff4-b5a9-9b267a1b5f70" deleted
:incubator michaelobrien$ kubectl delete pv/pvc-7eea5881-0dc2-4f4e-9b47-0b75e6d3541e
persistentvolume "pvc-7eea5881-0dc2-4f4e-9b47-0b75e6d3541e" deleted


or all at once
:incubator michaelobrien$ kubectl delete pvc --all
persistentvolumeclaim "datadir-kafka-0" deleted
persistentvolumeclaim "datadir-kafka-1" deleted
persistentvolumeclaim "datadir-kafka-2" deleted
:incubator michaelobrien$ kubectl delete pv --all
persistentvolume "pvc-1f7ea6b6-b222-4b34-8124-ef4da0699c13" deleted
persistentvolume "pvc-b015ef5f-5c2a-4aec-80b1-248c1cd674d7" deleted
persistentvolume "pvc-f1712213-cd18-42f0-a78a-20a1b0acf357" deleted

verify
:incubator michaelobrien$ kubectl get pv --all-namespaces
No resources found
:incubator michaelobrien$ kubectl get pvc --all-namespaces
No resources found
:incubator michaelobrien$ kubectl get storageclass --all-namespaces
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  3d2h

Kafka using Confluent Helm Charts

20200928

follow https://docs.confluent.io/5.1.2/installation/installing_cp/cp-helm-charts/docs/index.html

https://confluentinc.github.io/cp-helm-charts/

https://github.com/confluentinc/cp-helm-charts

# delete all existing kafka charts, pv, pvc
biometric:reference michaelobrien$ pwd
/Users/michaelobrien/wse_helm/reference
biometric:reference michaelobrien$ ls
kafka		zookeeper
biometric:reference michaelobrien$ ls kafka/charts/
zookeeper

biometric:reference michaelobrien$ kubectl get pods --all-namespaces | grep kafka
default       kafka-0                                  1/1     Running   2          7h53m
default       kafka-1                                  1/1     Running   0          7h51m
default       kafka-2                                  1/1     Running   0          7h51m
default       kafka-zookeeper-0                        1/1     Running   1          7h53m
default       kafka-zookeeper-1                        1/1     Running   1          7h53m
default       kafka-zookeeper-2                        1/1     Running   0          7h52m

biometric:reference michaelobrien$ helm3 delete kafka
release "kafka" uninstalled
biometric:reference michaelobrien$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-293e1fd4-c620-46e1-afc8-49a1996ed8c7   1Gi        RWO            Delete           Bound    default/datadir-kafka-0   hostpath                7h55m
pvc-e02602e9-9b57-4009-9487-1e34be1a5dd2   1Gi        RWO            Delete           Bound    default/datadir-kafka-1   hostpath                7h52m
pvc-f9d82f77-f1cd-440b-8a74-3a1103e38085   1Gi        RWO            Delete           Bound    default/datadir-kafka-2   hostpath                7h52m
biometric:reference michaelobrien$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-kafka-0   Bound    pvc-293e1fd4-c620-46e1-afc8-49a1996ed8c7   1Gi        RWO            hostpath       7h55m
datadir-kafka-1   Bound    pvc-e02602e9-9b57-4009-9487-1e34be1a5dd2   1Gi        RWO            hostpath       7h52m
datadir-kafka-2   Bound    pvc-f9d82f77-f1cd-440b-8a74-3a1103e38085   1Gi        RWO            hostpath       7h52m

biometric:reference michaelobrien$ kubectl delete pvc --all
persistentvolumeclaim "datadir-kafka-0" deleted
persistentvolumeclaim "datadir-kafka-1" deleted
persistentvolumeclaim "datadir-kafka-2" deleted
biometric:reference michaelobrien$ kubectl delete pv --all
No resources found
biometric:reference michaelobrien$ kubectl get pvc
No resources found in default namespace.
biometric:reference michaelobrien$ kubectl get pv
No resources found in default namespace.

# we are clean - reinstall
biometric:reference michaelobrien$ cd ..
biometric:wse_helm michaelobrien$ mkdir confluent
biometric:wse_helm michaelobrien$ ls
20200717_helm_git_update.rtf	cert-manager.yaml		charts				confluent			jetstack			kafka				reference
biometric:wse_helm michaelobrien$ cd confluent/
biometric:confluent michaelobrien$ git clone https://github.com/confluentinc/cp-helm-charts.git
Cloning into 'cp-helm-charts'...
remote: Enumerating objects: 45, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 2464 (delta 17), reused 8 (delta 1), pack-reused 2419
Receiving objects: 100% (2464/2464), 3.83 MiB | 8.09 MiB/s, done.
Resolving deltas: 100% (1716/1716), done.