dev
Page tree

Michael O'Brien

Skip to end of metadata
Go to start of metadata


Minikube Kubernetes Node

https://minikube.sigs.k8s.io/docs/commands/node/

Known Issues

No Filesystem retention on VM restart except for /mnt/sda1 affects saving PVC mounts


/dev/sda1       127380716    1441984 118582372   2% /mnt/sda1
$ ls /mnt/sda1
_16g  data  hostpath-provisioner  hostpath_pv  lost+found  var

/tmp is mounted into /dev/sda1 - so it is saved
$ findmnt | grep provisioner
| `-/tmp/hostpath-provisioner                                                                                                        /dev/sda1[/hostpath-provisioner] ext4             rw,relatime


Therefore if the default mountpath of /tmp/hostpath-provisioner is used as in https://github.com/kubernetes/minikube/blob/v0.30.0/pkg/storage/storage_provisioner.go#L49 we will loose the files after a minikube stop/start.  It would be advisable to modify the storageclass post VM start without going through a 2nd mount to /dev/sda1 - see https://github.com/kubernetes/minikube/issues/3318

Minikube on OSX VMware Fusion

https://kubernetes.io/docs/tasks/tools/install-minikube/

biometric:stable michaelobrien$ brew install minikube
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).

==> Downloading https://homebrew.bintray.com/bottles/kubernetes-cli-1.18.5.catalina.bottle.1.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/23c3217506d45d24de95a35c5faf1ebbea2de5f0808b5f9188d4b0e8478b7193?response-content-disposition=attachment%3Bfilename%3D%22kubernetes-cli-1.18.5.catalina.bottle.1.tar.gz%22&Policy=eyJTdGF0ZW1l
######################################################################## 100.0%
==> Downloading https://homebrew.bintray.com/bottles/minikube-1.12.0.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/bfa00eb2a001481ecd9439978bc8b7b4bbfa88a5ccc2cdd0f09a3892eae399c9?response-content-disposition=attachment%3Bfilename%3D%22minikube-1.12.0.catalina.bottle.tar.gz%22&Policy=eyJTdGF0ZW1lbnQiOiBb
######################################################################## 100.0%
==> Installing dependencies for minikube: kubernetes-cli
==> Installing minikube dependency: kubernetes-cli
==> Pouring kubernetes-cli-1.18.5.catalina.bottle.1.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/kubectl
Target /usr/local/bin/kubectl
already exists. You may want to remove it:
  rm '/usr/local/bin/kubectl'

To force the link and overwrite all conflicting files:
  brew link --overwrite kubernetes-cli

To list all files that would be deleted:
  brew link --overwrite --dry-run kubernetes-cli

Possible conflicting files are:
/usr/local/bin/kubectl -> /Applications/Docker.app/Contents/Resources/bin/kubectl
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> Summary
  /usr/local/Cellar/kubernetes-cli/1.18.5: 232 files, 49.2MB
==> Installing minikube
==> Pouring minikube-1.12.0.catalina.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> Summary
  /usr/local/Cellar/minikube/1.12.0: 8 files, 63.5MB
==> `brew cleanup` has not been run in 30 days, running now...
Removing: /Users/michaelobrien/Library/Caches/Homebrew/gdbm--1.18.1.catalina.bottle.1.tar.gz... (196.7KB)
Removing: /Users/michaelobrien/Library/Caches/Homebrew/helm--3.2.1.catalina.bottle.tar.gz... (13.0MB)
Removing: /Users/michaelobrien/Library/Caches/Homebrew/readline--8.0.4.catalina.bottle.tar.gz... (517.7KB)
Removing: /Users/michaelobrien/Library/Caches/Homebrew/sqlite--3.31.1.catalina.bottle.tar.gz... (1.9MB)
==> Caveats
==> kubernetes-cli
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> minikube
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh completions have been installed to:
  /usr/local/share/zsh/site-functions

Start minikube

biometric:charts michaelobrien$ minikube start --driver=vmwarefusion
  minikube v1.12.0 on Darwin 10.15.5
  Using the vmwarefusion driver based on user configuration
  Downloading VM boot image ...
    > minikube-v1.12.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
    > minikube-v1.12.0.iso: 173.57 MiB / 173.57 MiB [] 100.00% 57.33 MiB p/s 3s
  Starting control plane node minikube in cluster minikube
  Downloading Kubernetes v1.18.3 preload ...
    > preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4: 526.27 MiB
  The vmwarefusion driver is deprecated and support for it will be removed in a future release.
			Please consider switching to the new vmware unified driver, which is intended to replace the vmwarefusion driver.
			See https://minikube.sigs.k8s.io/docs/reference/drivers/vmware/ for more information.
			To disable this message, run [minikube config set ShowDriverDeprecationNotification false]
  Creating vmwarefusion VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
  Preparing Kubernetes v1.18.3 on Docker 19.03.12 ...
E0714 13:28:56.032108   94504 start.go:96] Unable to get host IP: HostIP not yet implemented for "vmwarefusion" driver
  failed to start node: startup failed: Failed to setup kubeconfig: HostIP not yet implemented for "vmwarefusion" driver
  minikube is exiting due to an error. If the above message is not useful, open an issue:
  https://github.com/kubernetes/minikube/issues/new/choose

see https://github.com/kubernetes/minikube/issues/8374

Install newer docker-machine-driver-vmware

https://minikube.sigs.k8s.io/docs/drivers/vmware/

biometric:charts michaelobrien$ brew install docker-machine-driver-vmware
==> Downloading https://homebrew.bintray.com/bottles/docker-machine-0.16.2.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/99b50d9809a0aa881e01686e3356fbd17fa61e5a5e8cb937a2a9e9ff103be097?response-content-disposition=attachment%3Bfilename%3D%22docker-machine-0.16.2.catalina.bottle.tar.gz%22&Policy=eyJTdGF0ZW1lbn
######################################################################## 100.0%
==> Downloading https://homebrew.bintray.com/bottles/docker-machine-driver-vmware-0.1.1.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/d73c0be19fba7a7166c65202133b18c63367b5758d04ec19e23d3fd9406a8a7a?response-content-disposition=attachment%3Bfilename%3D%22docker-machine-driver-vmware-0.1.1.catalina.bottle.tar.gz%22&Policy=e
######################################################################## 100.0%
==> Installing dependencies for docker-machine-driver-vmware: docker-machine
==> Installing docker-machine-driver-vmware dependency: docker-machine
==> Pouring docker-machine-0.16.2.catalina.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh completions have been installed to:
  /usr/local/share/zsh/site-functions

To have launchd start docker-machine now and restart at login:
  brew services start docker-machine
Or, if you don't want/need a background service you can just run:
  docker-machine start
==> Summary
  /usr/local/Cellar/docker-machine/0.16.2: 11 files, 36MB
==> Installing docker-machine-driver-vmware
==> Pouring docker-machine-driver-vmware-0.1.1.catalina.bottle.tar.gz
  /usr/local/Cellar/docker-machine-driver-vmware/0.1.1: 5 files, 11.6MB
==> Caveats
==> docker-machine
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh completions have been installed to:
  /usr/local/share/zsh/site-functions

To have launchd start docker-machine now and restart at login:
  brew services start docker-machine
Or, if you don't want/need a background service you can just run:
  docker-machine start

biometric:charts michaelobrien$ minikube delete
  Deleting "minikube" in vmwarefusion ...
  Removed all traces of the "minikube" cluster.

biometric:charts michaelobrien$ minikube start --driver=vmware
  minikube v1.12.0 on Darwin 10.15.5
  Using the vmware driver based on user configuration
  Starting control plane node minikube in cluster minikube
  Creating vmware VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
  Preparing Kubernetes v1.18.3 on Docker 19.03.12 ...
  Verifying Kubernetes components...
  Enabled addons: default-storageclass, storage-provisioner
  Done! kubectl is now configured to use "minikube"

  /usr/local/bin/kubectl is version 1.16.6-beta.0, which may be incompatible with Kubernetes 1.18.3.
  You can also use 'minikube kubectl -- get pods' to invoke a matching version
biometric:charts michaelobrien$ kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-66bff467f8-sv4sq           1/1     Running   0          53s
kube-system   etcd-minikube                      1/1     Running   0          53s
kube-system   kube-apiserver-minikube            1/1     Running   0          53s
kube-system   kube-controller-manager-minikube   1/1     Running   0          53s
kube-system   kube-proxy-2987v                   1/1     Running   0          53s
kube-system   kube-scheduler-minikube            1/1     Running   0          53s
kube-system   storage-provisioner                1/1     Running   1          59s
biometric:charts michaelobrien$ kubectl get nodes
NAME       STATUS   ROLES    AGE   VERSION
minikube   Ready    master   77s   v1.18.3
biometric:charts michaelobrien$ kubectl get services --all-namespaces
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  91s
kube-system   kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   90s

biometric:charts michaelobrien$ kubectl get storageclass
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  114s
biometric:charts michaelobrien$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
biometric:charts michaelobrien$ 

I0714 13:56:14.560252   95756 kubeadm.go:143] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.58.140
  bindPort: 8443
bootstrapTokens:
  - groups:
      - system:bootstrappers:kubeadm:default-node-token
    ttl: 24h0m0s
    usages:
      - signing
      - authentication
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: "minikube"
  kubeletExtraArgs:
    node-ip: 192.168.58.140
  taints: []
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  certSANs: ["127.0.0.1", "localhost", "192.168.58.140"]
  extraArgs:
    enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
certificatesDir: /var/lib/minikube/certs
clusterName: mk
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/minikube/etcd
controllerManager:
  extraArgs:
    "leader-elect": "false"
scheduler:
  extraArgs:
    "leader-elect": "false"
kubernetesVersion: v1.18.3
networking:
  dnsDomain: cluster.local
  podSubnet: ""
  serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
  nodefs.available: "0%"
  nodefs.inodesFree: "0%"
  imagefs.available: "0%"
failSwapOn: false
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: ""
metricsBindAddress: 192.168.58.140:10249

Increase resources of Minikube VM

$ minikube stop
$ minikube delete
$ minikube start --driver=vmware --disk-size=140GB --alsologtostderr --cpus 14 --memory 20480 --kubernetes-version=v1.17.4

 minikube v1.12.0 on Darwin 10.15.5
 Using the vmware driver based on user configuration
 Starting control plane node minikube in cluster minikube
 Creating vmware VM (CPUs=2, Memory=6000MB, Disk=101376MB) ...
 Preparing Kubernetes v1.18.3 on Docker 19.03.12 ...
 Enabled addons: default-storageclass, storage-provisioner

$ free
              total        used        free      shared  buff/cache   available
Mem:       20547020      759724    17922684      507804     1864612    18966332
Swap:             0           0           0
$ df
Filesystem      1K-blocks       Used Available Use% Mounted on
tmpfs            18492320     498212  17994108   3% /
devtmpfs         10189140          0  10189140   0% /dev
tmpfs            10273508          0  10273508   0% /dev/shm
tmpfs            10273508       9544  10263964   1% /run
tmpfs            10273508          0  10273508   0% /sys/fs/cgroup
tmpfs            10273508         12  10273496   1% /tmp
/dev/sda1        90293332    1445796  83640756   2% /mnt/sda1
vmhgfs-fuse    1953902876 1618325232 335577644  83% /mnt/hgfs

Storage class in Minikube

$ sudo cat /etc/kubernetes/addons/storageclass.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  namespace: kube-system
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists

provisioner: k8s.io/minikube-hostpath
$ sudo cat /etc/kubernetes/addons/storage           
storage-provisioner.yaml  storageclass.yaml         
$ sudo cat /etc/kubernetes/addons/storage-provisioner.yaml 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: storage-provisioner
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: storage-provisioner
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:persistent-volume-provisioner
subjects:
  - kind: ServiceAccount
    name: storage-provisioner
    namespace: kube-system

---
apiVersion: v1
kind: Pod
metadata:
  name: storage-provisioner
  namespace: kube-system
  labels:
    integration-test: storage-provisioner
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  serviceAccountName: storage-provisioner
  hostNetwork: true
  containers:
  - name: storage-provisioner
    image: gcr.io/k8s-minikube/storage-provisioner:v1.8.1
    command: ["/storage-provisioner"]
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /tmp
      name: tmp
  volumes:
  - name: tmp
    hostPath:
      path: /tmp
      type: Directory

Location of VMware VMs

$ vmrun list
Total running VMs: 1
/Users/mi...ien/.minikube/machines/minikube/minikube.vmx


SCP into Minikube

scp -i $(minikube ssh-key) README.md docker@192.168.58.140:~/


Minikube DevOps

Kubernetes version 

--kubernetes-version=v1.17.4



Issues

Watch mounts that get root:root - https://github.com/kubernetes/minikube/issues/2290

Backup/Restore/Move VMs

minikube vms are located at ~/.minikube/machines

Proxy Minikube Docker in Host

There is a need to build a docker image or pre-pull docker images into the minikube vm.  We can use the docker instance from the osx host by following https://github.com/kubernetes/minikube/blob/0c616a6b42b28a1aab8397f5a9061f8ebbd9f3d9/README.md#reusing-the-docker-daemon

before we get docker desktop
biometric:charts michaelobrien$ docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
obrienlabs/reference-nbi   0.0.1               7eeb1e531e9c        41 hours ago        542MB
obrienlabs/reference-nbi   latest              7eeb1e531e9c        41 hours ago        542MB
<none>                     <none>              a848368c9114        41 hours ago        542MB
openjdk                    8                   b190ad78b520        5 weeks ago         510MB

switch to minikube
eval $(minikube docker-env)

now inside the vm
biometric:charts michaelobrien$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
nginx                                     stable              8fcf4560562d        6 days ago          132MB
gcr.io/kubernetes-helm/tiller             v2.16.9             fd3fd9ab134a        4 weeks ago         88MB
kubernetesui/dashboard                    v2.0.1              85d666cddd04        8 weeks ago         223MB
k8s.gcr.io/kube-proxy                     v1.18.3             3439b7546f29        8 weeks ago         117MB
k8s.gcr.io/kube-apiserver                 v1.18.3             7e28efa976bd        8 weeks ago         173MB
k8s.gcr.io/kube-controller-manager        v1.18.3             da26705ccb4b        8 weeks ago         162MB
k8s.gcr.io/kube-scheduler                 v1.18.3             76216c34ed0c        8 weeks ago         95.3MB
kubernetesui/metrics-scraper              v1.0.4              86262685d9ab        3 months ago        36.9MB
k8s.gcr.io/pause                          3.2                 80d28bedfe5d        5 months ago        683kB
k8s.gcr.io/coredns                        1.6.7               67da37a9a360        5 months ago        43.8MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        8 months ago        288MB
zookeeper                                 3.5.5               5a65375c17a4        9 months ago        225MB
confluentinc/cp-kafka                     5.0.1               5467234daea9        20 months ago       557MB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB


Switch back to the host -  https://github.com/kubernetes/minikube/issues/348
not
eval "$(docker-machine env -u)"

use
eval $(minikube docker-env --unset)

Links

https://thenewstack.io/tutorial-dynamic-provisioning-of-persistent-storage-in-kubernetes-with-minikube/

  • No labels

1 Comment

  1. Michael O'Brien

    Automation

    see Docker Development Guide#Dockerfullsystemprune

    biometric:wse_helm michaelobrien$ minikube start --driver=vmware --disk-size=140GB --alsologtostderr --cpus 10 --memory 24576

$ df
Filesystem      1K-blocks       Used Available Use% Mounted on
tmpfs            22207876     498212  21709664   3% /
/dev/sda1       127380716    1449920 118574436   2% /mnt/sda1
vmhgfs-fuse    1953902876 1673764340 280138536  86% /mnt/hgfs
$ free
              total        used        free      shared  buff/cache   available
Mem:       24675416      770208    22038216      507816     1866992    23055120

biometric:wse_helm michaelobrien$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
biometric:wse_helm michaelobrien$ eval $(minikube docker-env )
biometric:wse_helm michaelobrien$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
kubernetesui/dashboard                    v2.0.1              85d666cddd04        8 weeks ago         223MB
k8s.gcr.io/kube-proxy                     v1.18.3             3439b7546f29        8 weeks ago         117MB
k8s.gcr.io/kube-scheduler                 v1.18.3             76216c34ed0c        8 weeks ago         95.3MB
k8s.gcr.io/kube-controller-manager        v1.18.3             da26705ccb4b        8 weeks ago         162MB
k8s.gcr.io/kube-apiserver                 v1.18.3             7e28efa976bd        8 weeks ago         173MB
kubernetesui/metrics-scraper              v1.0.4              86262685d9ab        3 months ago        36.9MB
k8s.gcr.io/pause                          3.2                 80d28bedfe5d        5 months ago        683kB
k8s.gcr.io/coredns                        1.6.7               67da37a9a360        5 months ago        43.8MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        8 months ago        288MB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB

Stop and snapshot the vm in ./minikube
biometric:wse_helm michaelobrien$ minikube stop
  Stopping "minikube" in vmware ...
  Node "minikube" stopped.

restart
minikube start

switch docker context to minikube (use --unset to return)
eval $(minikube docker-env )

preload any docker images tar if required (check pull policy later)
docker load < images.tar 
e4009cf2b566: Loading layer [==================================================>]  58.75MB/58.75MB

obrienlabs