See the Kubernetes Developer Guide for kubernetes and helm installations on any type of bare metal, virtual or cloud VM.
Gone are the days of setting up your own Java EE distributed cluster implementing ForkJoin through remote EJB. We now just add machines to a kubernetes cluster.
Hardware
For Intel NUC computers (two i7, one i5, one i3 - all with 16G ram and 128-512 nVMe SSD drives) connected by a private layer 3 switch on a 192.168.15.0/24 subnet.
Software
Kubernetes 1.14.6, Helm 2.14.3, Docker 19.03.2, Ubuntu 16.04
20200517: RKE 1.0.8: Kubernetes 1.17.5 on Ubuntu 16.04
https://github.com/rancher/rke/releases/tag/v1.0.8
4 Node Deployment
Prepare Nodes
Add ssh key Kubernetes Developer Guide#SingleNodeKubernetesclusterrunningRKEonAWSEC2withHelm
Kubernetes Developer Guide#InstallDocker
Prepare Master
Kubernetes Developer Guide#InstallDocker
ubuntu@kub0:~$ wget https://github.com/rancher/rke/releases/download/v1.0.8/rke_linux-amd64
ubuntu@kub0:~$ cp rke_linux-amd64 rke
ubuntu@kub0:~$ sudo chmod 777 rke
[sudo] password for ubuntu:
ubuntu@kub0:~$ ./rke --version
rke version v1.0.8
ubuntu@kub0:~$ ./rke config --name cluster.yml
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 4
[+] SSH Address of host (1) [none]: 192.168.0.200
[+] SSH Port of host (1) [22]:
[+] SSH Private Key Path of host (192.168.0.200) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (192.168.0.200) [ubuntu]:
[+] Is host (192.168.0.200) a Control Plane host (y/n)? [y]: y
[+] Is host (192.168.0.200) a Worker host (y/n)? [n]: y
[+] Is host (192.168.0.200) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (192.168.0.200) [none]:
[+] Internal IP of host (192.168.0.200) [none]:
[+] Docker socket path on host (192.168.0.200) [/var/run/docker.sock]:
[+] SSH Address of host (2) [none]: 192.168.0.201
[+] SSH Port of host (2) [22]:
[+] SSH Private Key Path of host (192.168.0.201) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (192.168.0.201) [ubuntu]:
[+] Is host (192.168.0.201) a Control Plane host (y/n)? [y]: y
[+] Is host (192.168.0.201) a Worker host (y/n)? [n]: y
[+] Is host (192.168.0.201) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (192.168.0.201) [none]:
[+] Internal IP of host (192.168.0.201) [none]: 192.168.0.201
[+] Docker socket path on host (192.168.0.201) [/var/run/docker.sock]:
[+] SSH Address of host (3) [none]: 192.168.0.203
[+] SSH Port of host (3) [22]:
[+] SSH Private Key Path of host (192.168.0.203) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (192.168.0.203) [ubuntu]:
[+] Is host (192.168.0.203) a Control Plane host (y/n)? [y]: y
[+] Is host (192.168.0.203) a Worker host (y/n)? [n]: y
[+] Is host (192.168.0.203) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (192.168.0.203) [none]:
[+] Internal IP of host (192.168.0.203) [none]: 192.168.0.203
[+] Docker socket path on host (192.168.0.203) [/var/run/docker.sock]:
[+] SSH Address of host (4) [none]: 192.168.0.203
[+] SSH Port of host (4) [22]:
[+] SSH Private Key Path of host (192.168.0.203) [none]:
[-] You have entered empty SSH key path, trying fetch from SSH key parameter
[+] SSH Private Key of host (192.168.0.203) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (192.168.0.203) [ubuntu]:
[+] Is host (192.168.0.203) a Control Plane host (y/n)? [y]: y
[+] Is host (192.168.0.203) a Worker host (y/n)? [n]: y
[+] Is host (192.168.0.203) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (192.168.0.203) [none]:
[+] Internal IP of host (192.168.0.203) [none]: 192.168.0.203
[+] Docker socket path on host (192.168.0.203) [/var/run/docker.sock]:
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]:
[+] Authentication Strategy [x509]:
[+] Authorization Mode (rbac, none) [rbac]:
[+] Kubernetes Docker image [rancher/hyperkube:v1.17.5-rancher1]:
[+] Cluster domain [cluster.local]:
[+] Service Cluster IP Range [10.43.0.0/16]:
[+] Enable PodSecurityPolicy [n]:
[+] Cluster Network CIDR [10.42.0.0/16]:
[+] Cluster DNS Service IP [10.43.0.10]:
[+] Add addon manifest URLs or YAML files [no]:
editedubuntu@kub0:~$ cat cluster.yml
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: 192.168.0.200
port: "22"
internal_address: 192.168.0.200
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: ubuntu
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/onap_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
- address: 192.168.0.201
port: "22"
internal_address: 192.168.0.201
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: ubuntu
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/onap_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
- address: 192.168.0.202
port: "22"
internal_address: 192.168.0.202
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: ubuntu
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/onap_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
- address: 192.168.0.203
port: "22"
internal_address: 192.168.0.203
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: ubuntu
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/onap_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
services:
etcd:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
external_urls: []
ca_cert: ""
cert: ""
key: ""
path: ""
uid: 0
gid: 0
snapshot: null
retention: ""
creation: ""
backup_config: null
kube-api:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
service_cluster_ip_range: 10.43.0.0/16
service_node_port_range: ""
pod_security_policy: false
always_pull_images: false
secrets_encryption_config: null
audit_log: null
admission_configuration: null
event_rate_limit: null
kube-controller:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
scheduler:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
kubelet:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
cluster_domain: cluster.local
infra_container_image: ""
cluster_dns_server: 10.43.0.10
fail_swap_on: false
generate_serving_certificate: false
kubeproxy:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
network:
plugin: canal
options: {}
mtu: 0
node_selector: {}
authentication:
strategy: x509
sans: []
webhook: null
addons: ""
addons_include: []
system_images:
etcd: rancher/coreos-etcd:v3.4.3-rancher1
alpine: rancher/rke-tools:v0.1.56
nginx_proxy: rancher/rke-tools:v0.1.56
cert_downloader: rancher/rke-tools:v0.1.56
kubernetes_services_sidecar: rancher/rke-tools:v0.1.56
kubedns: rancher/k8s-dns-kube-dns:1.15.0
dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.0
kubedns_sidecar: rancher/k8s-dns-sidecar:1.15.0
kubedns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
coredns: rancher/coredns-coredns:1.6.5
coredns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
nodelocal: rancher/k8s-dns-node-cache:1.15.7
kubernetes: rancher/hyperkube:v1.17.5-rancher1
flannel: rancher/coreos-flannel:v0.11.0-rancher1
flannel_cni: rancher/flannel-cni:v0.3.0-rancher5
calico_node: rancher/calico-node:v3.13.0
calico_cni: rancher/calico-cni:v3.13.0
calico_controllers: rancher/calico-kube-controllers:v3.13.0
calico_ctl: rancher/calico-ctl:v2.0.0
calico_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.0
canal_node: rancher/calico-node:v3.13.0
canal_cni: rancher/calico-cni:v3.13.0
canal_flannel: rancher/coreos-flannel:v0.11.0
canal_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.0
weave_node: weaveworks/weave-kube:2.5.2
weave_cni: weaveworks/weave-npc:2.5.2
pod_infra_container: rancher/pause:3.1
ingress: rancher/nginx-ingress-controller:nginx-0.25.1-rancher1
ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1
metrics_server: rancher/metrics-server:v0.3.6
windows_pod_infra_container: rancher/kubelet-pause:v0.1.3
ssh_key_path: ~/.ssh/onap_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
mode: rbac
options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
provider: ""
options: {}
node_selector: {}
extra_args: {}
dns_policy: ""
extra_envs: []
extra_volumes: []
extra_volume_mounts: []
cluster_name: ""
cloud_provider:
name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
address: ""
port: ""
user: ""
ssh_key: ""
ssh_key_path: ""
ssh_cert: ""
ssh_cert_path: ""
monitoring:
provider: ""
options: {}
node_selector: {}
restore:
restore: false
snapshot_name: ""
dns: null
ubuntu@kub0:~$ curl https://releases.rancher.com/install-docker/19.03.sh | sh
and 1,2,3
rke up
ubuntu@kub3:~$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rancher/rke-tools v0.1.56 8c8e0533fa43 7 weeks ago 132MB
ubuntu@kub3:~$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rancher/hyperkube v1.17.5-rancher1 ff99d966b0ee 4 weeks ago 1.56GB
rancher/rke-tools v0.1.56 8c8e0533fa43 7 weeks ago 132MB
ubuntu@kub0:~$ rke --version
rke version v1.0.8
ubuntu@kub0:~$ sudo rke up
INFO[0000] Running RKE version: v1.0.8
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [192.168.0.203]
INFO[0000] [dialer] Setup tunnel for host [192.168.0.200]
INFO[0000] [dialer] Setup tunnel for host [192.168.0.201]
INFO[0000] [dialer] Setup tunnel for host [192.168.0.202]
INFO[0000] Checking if container [cluster-state-deployer] is running on host [192.168.0.200], try #1
INFO[0000] Pulling image [rancher/rke-tools:v0.1.56] on host [192.168.0.200], try #1
INFO[0006] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0006] Starting container [cluster-state-deployer] on host [192.168.0.200], try #1
INFO[0006] [state] Successfully started [cluster-state-deployer] container on host [192.168.0.200]
INFO[0006] Checking if container [cluster-state-deployer] is running on host [192.168.0.201], try #1
INFO[0006] Pulling image [rancher/rke-tools:v0.1.56] on host [192.168.0.201], try #1
INFO[0013] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0013] Starting container [cluster-state-deployer] on host [192.168.0.201], try #1
INFO[0013] [state] Successfully started [cluster-state-deployer] container on host [192.168.0.201]
INFO[0013] Checking if container [cluster-state-deployer] is running on host [192.168.0.202], try #1
INFO[0013] Pulling image [rancher/rke-tools:v0.1.56] on host [192.168.0.202], try #1
INFO[0020] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0020] Starting container [cluster-state-deployer] on host [192.168.0.202], try #1
INFO[0021] [state] Successfully started [cluster-state-deployer] container on host [192.168.0.202]
INFO[0021] Checking if container [cluster-state-deployer] is running on host [192.168.0.203], try #1
INFO[0021] Pulling image [rancher/rke-tools:v0.1.56] on host [192.168.0.203], try #1
INFO[0029] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0029] Starting container [cluster-state-deployer] on host [192.168.0.203], try #1
INFO[0029] [state] Successfully started [cluster-state-deployer] container on host [192.168.0.203]
INFO[0029] [certificates] Generating CA kubernetes certificates
INFO[0029] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates
INFO[0029] [certificates] Generating Kubernetes API server certificates
INFO[0030] [certificates] Generating Service account token key
INFO[0030] [certificates] Generating Kube Controller certificates
INFO[0030] [certificates] Generating Kube Scheduler certificates
INFO[0030] [certificates] Generating Kube Proxy certificates
INFO[0030] [certificates] Generating Node certificate
INFO[0030] [certificates] Generating admin certificates and kubeconfig
INFO[0030] [certificates] Generating Kubernetes API server proxy client certificates
INFO[0031] [certificates] Generating kube-etcd-192-168-0-200 certificate and key
INFO[0031] [certificates] Generating kube-etcd-192-168-0-201 certificate and key
INFO[0031] [certificates] Generating kube-etcd-192-168-0-202 certificate and key
INFO[0031] [certificates] Generating kube-etcd-192-168-0-203 certificate and key
INFO[0031] Successfully Deployed state file at [./cluster.rkestate]
INFO[0031] Building Kubernetes cluster
INFO[0031] [dialer] Setup tunnel for host [192.168.0.203]
INFO[0031] [dialer] Setup tunnel for host [192.168.0.202]
INFO[0031] [dialer] Setup tunnel for host [192.168.0.200]
INFO[0031] [dialer] Setup tunnel for host [192.168.0.201]
INFO[0031] [network] Deploying port listener containers
INFO[0031] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0031] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0031] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0031] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0031] Starting container [rke-etcd-port-listener] on host [192.168.0.201], try #1
INFO[0032] Starting container [rke-etcd-port-listener] on host [192.168.0.203], try #1
INFO[0032] Starting container [rke-etcd-port-listener] on host [192.168.0.202], try #1
INFO[0032] Starting container [rke-etcd-port-listener] on host [192.168.0.200], try #1
INFO[0032] [network] Successfully started [rke-etcd-port-listener] container on host [192.168.0.201]
INFO[0032] [network] Successfully started [rke-etcd-port-listener] container on host [192.168.0.203]
INFO[0032] [network] Successfully started [rke-etcd-port-listener] container on host [192.168.0.200]
INFO[0032] [network] Successfully started [rke-etcd-port-listener] container on host [192.168.0.202]
INFO[0032] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0032] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0032] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0032] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0032] Starting container [rke-cp-port-listener] on host [192.168.0.201], try #1
INFO[0032] Starting container [rke-cp-port-listener] on host [192.168.0.200], try #1
INFO[0032] Starting container [rke-cp-port-listener] on host [192.168.0.203], try #1
INFO[0032] Starting container [rke-cp-port-listener] on host [192.168.0.202], try #1
INFO[0033] [network] Successfully started [rke-cp-port-listener] container on host [192.168.0.201]
INFO[0033] [network] Successfully started [rke-cp-port-listener] container on host [192.168.0.200]
INFO[0033] [network] Successfully started [rke-cp-port-listener] container on host [192.168.0.203]
INFO[0033] [network] Successfully started [rke-cp-port-listener] container on host [192.168.0.202]
INFO[0033] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0033] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0033] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0033] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0033] Starting container [rke-worker-port-listener] on host [192.168.0.201], try #1
INFO[0033] Starting container [rke-worker-port-listener] on host [192.168.0.200], try #1
INFO[0033] Starting container [rke-worker-port-listener] on host [192.168.0.203], try #1
INFO[0033] Starting container [rke-worker-port-listener] on host [192.168.0.202], try #1
INFO[0033] [network] Successfully started [rke-worker-port-listener] container on host [192.168.0.201]
INFO[0033] [network] Successfully started [rke-worker-port-listener] container on host [192.168.0.200]
INFO[0033] [network] Successfully started [rke-worker-port-listener] container on host [192.168.0.203]
INFO[0033] [network] Successfully started [rke-worker-port-listener] container on host [192.168.0.202]
INFO[0033] [network] Port listener containers deployed successfully
INFO[0033] [network] Running etcd <-> etcd port checks
INFO[0033] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0033] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0033] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0033] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0033] Starting container [rke-port-checker] on host [192.168.0.201], try #1
INFO[0033] Starting container [rke-port-checker] on host [192.168.0.200], try #1
INFO[0033] Starting container [rke-port-checker] on host [192.168.0.203], try #1
INFO[0034] Starting container [rke-port-checker] on host [192.168.0.202], try #1
INFO[0034] [network] Successfully started [rke-port-checker] container on host [192.168.0.200]
INFO[0034] [network] Successfully started [rke-port-checker] container on host [192.168.0.201]
INFO[0034] Removing container [rke-port-checker] on host [192.168.0.200], try #1
INFO[0034] Removing container [rke-port-checker] on host [192.168.0.201], try #1
INFO[0034] [network] Successfully started [rke-port-checker] container on host [192.168.0.203]
INFO[0034] [network] Successfully started [rke-port-checker] container on host [192.168.0.202]
INFO[0034] Removing container [rke-port-checker] on host [192.168.0.203], try #1
INFO[0034] Removing container [rke-port-checker] on host [192.168.0.202], try #1
INFO[0034] [network] Running control plane -> etcd port checks
INFO[0034] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0034] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0034] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0034] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0034] Starting container [rke-port-checker] on host [192.168.0.201], try #1
INFO[0034] Starting container [rke-port-checker] on host [192.168.0.203], try #1
INFO[0034] Starting container [rke-port-checker] on host [192.168.0.200], try #1
INFO[0034] Starting container [rke-port-checker] on host [192.168.0.202], try #1
INFO[0034] [network] Successfully started [rke-port-checker] container on host [192.168.0.201]
INFO[0035] [network] Successfully started [rke-port-checker] container on host [192.168.0.200]
INFO[0035] Removing container [rke-port-checker] on host [192.168.0.201], try #1
INFO[0035] Removing container [rke-port-checker] on host [192.168.0.200], try #1
INFO[0035] [network] Successfully started [rke-port-checker] container on host [192.168.0.203]
INFO[0035] [network] Successfully started [rke-port-checker] container on host [192.168.0.202]
INFO[0035] Removing container [rke-port-checker] on host [192.168.0.203], try #1
INFO[0035] Removing container [rke-port-checker] on host [192.168.0.202], try #1
INFO[0035] [network] Running control plane -> worker port checks
INFO[0035] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0035] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0035] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0035] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0035] Starting container [rke-port-checker] on host [192.168.0.200], try #1
INFO[0035] Starting container [rke-port-checker] on host [192.168.0.203], try #1
INFO[0035] Starting container [rke-port-checker] on host [192.168.0.201], try #1
INFO[0035] Starting container [rke-port-checker] on host [192.168.0.202], try #1
INFO[0035] [network] Successfully started [rke-port-checker] container on host [192.168.0.200]
INFO[0035] [network] Successfully started [rke-port-checker] container on host [192.168.0.201]
INFO[0035] Removing container [rke-port-checker] on host [192.168.0.200], try #1
INFO[0035] Removing container [rke-port-checker] on host [192.168.0.201], try #1
INFO[0035] [network] Successfully started [rke-port-checker] container on host [192.168.0.203]
INFO[0035] [network] Successfully started [rke-port-checker] container on host [192.168.0.202]
INFO[0035] Removing container [rke-port-checker] on host [192.168.0.203], try #1
INFO[0035] Removing container [rke-port-checker] on host [192.168.0.202], try #1
INFO[0036] [network] Running workers -> control plane port checks
INFO[0036] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0036] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0036] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0036] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0036] Starting container [rke-port-checker] on host [192.168.0.200], try #1
INFO[0036] Starting container [rke-port-checker] on host [192.168.0.203], try #1
INFO[0036] Starting container [rke-port-checker] on host [192.168.0.201], try #1
INFO[0036] Starting container [rke-port-checker] on host [192.168.0.202], try #1
INFO[0036] [network] Successfully started [rke-port-checker] container on host [192.168.0.200]
INFO[0036] [network] Successfully started [rke-port-checker] container on host [192.168.0.201]
INFO[0036] Removing container [rke-port-checker] on host [192.168.0.200], try #1
INFO[0036] Removing container [rke-port-checker] on host [192.168.0.201], try #1
INFO[0036] [network] Successfully started [rke-port-checker] container on host [192.168.0.203]
INFO[0036] [network] Successfully started [rke-port-checker] container on host [192.168.0.202]
INFO[0036] Removing container [rke-port-checker] on host [192.168.0.203], try #1
INFO[0036] Removing container [rke-port-checker] on host [192.168.0.202], try #1
INFO[0036] [network] Checking KubeAPI port Control Plane hosts
INFO[0036] [network] Removing port listener containers
INFO[0036] Removing container [rke-etcd-port-listener] on host [192.168.0.203], try #1
INFO[0036] Removing container [rke-etcd-port-listener] on host [192.168.0.200], try #1
INFO[0036] Removing container [rke-etcd-port-listener] on host [192.168.0.201], try #1
INFO[0036] Removing container [rke-etcd-port-listener] on host [192.168.0.202], try #1
INFO[0036] [remove/rke-etcd-port-listener] Successfully removed container on host [192.168.0.200]
INFO[0036] [remove/rke-etcd-port-listener] Successfully removed container on host [192.168.0.201]
INFO[0036] [remove/rke-etcd-port-listener] Successfully removed container on host [192.168.0.203]
INFO[0037] [remove/rke-etcd-port-listener] Successfully removed container on host [192.168.0.202]
INFO[0037] Removing container [rke-cp-port-listener] on host [192.168.0.200], try #1
INFO[0037] Removing container [rke-cp-port-listener] on host [192.168.0.201], try #1
INFO[0037] Removing container [rke-cp-port-listener] on host [192.168.0.202], try #1
INFO[0037] Removing container [rke-cp-port-listener] on host [192.168.0.203], try #1
INFO[0037] [remove/rke-cp-port-listener] Successfully removed container on host [192.168.0.201]
INFO[0037] [remove/rke-cp-port-listener] Successfully removed container on host [192.168.0.200]
INFO[0037] [remove/rke-cp-port-listener] Successfully removed container on host [192.168.0.203]
INFO[0037] [remove/rke-cp-port-listener] Successfully removed container on host [192.168.0.202]
INFO[0037] Removing container [rke-worker-port-listener] on host [192.168.0.200], try #1
INFO[0037] Removing container [rke-worker-port-listener] on host [192.168.0.201], try #1
INFO[0037] Removing container [rke-worker-port-listener] on host [192.168.0.203], try #1
INFO[0037] Removing container [rke-worker-port-listener] on host [192.168.0.202], try #1
INFO[0037] [remove/rke-worker-port-listener] Successfully removed container on host [192.168.0.200]
INFO[0037] [remove/rke-worker-port-listener] Successfully removed container on host [192.168.0.201]
INFO[0037] [remove/rke-worker-port-listener] Successfully removed container on host [192.168.0.203]
INFO[0037] [remove/rke-worker-port-listener] Successfully removed container on host [192.168.0.202]
INFO[0037] [network] Port listener containers removed successfully
INFO[0037] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0037] Checking if container [cert-deployer] is running on host [192.168.0.201], try #1
INFO[0037] Checking if container [cert-deployer] is running on host [192.168.0.202], try #1
INFO[0037] Checking if container [cert-deployer] is running on host [192.168.0.200], try #1
INFO[0037] Checking if container [cert-deployer] is running on host [192.168.0.203], try #1
INFO[0037] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0037] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0037] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0037] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0037] Starting container [cert-deployer] on host [192.168.0.201], try #1
INFO[0037] Starting container [cert-deployer] on host [192.168.0.200], try #1
INFO[0037] Starting container [cert-deployer] on host [192.168.0.203], try #1
INFO[0037] Starting container [cert-deployer] on host [192.168.0.202], try #1
INFO[0037] Checking if container [cert-deployer] is running on host [192.168.0.200], try #1
INFO[0037] Checking if container [cert-deployer] is running on host [192.168.0.201], try #1
INFO[0038] Checking if container [cert-deployer] is running on host [192.168.0.203], try #1
INFO[0038] Checking if container [cert-deployer] is running on host [192.168.0.202], try #1
INFO[0042] Checking if container [cert-deployer] is running on host [192.168.0.200], try #1
INFO[0042] Removing container [cert-deployer] on host [192.168.0.200], try #1
INFO[0042] Checking if container [cert-deployer] is running on host [192.168.0.201], try #1
INFO[0042] Removing container [cert-deployer] on host [192.168.0.201], try #1
INFO[0043] Checking if container [cert-deployer] is running on host [192.168.0.203], try #1
INFO[0043] Removing container [cert-deployer] on host [192.168.0.203], try #1
INFO[0043] Checking if container [cert-deployer] is running on host [192.168.0.202], try #1
INFO[0043] Removing container [cert-deployer] on host [192.168.0.202], try #1
INFO[0043] [reconcile] Rebuilding and updating local kube config
INFO[0043] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0043] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0043] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0043] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0043] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0043] [reconcile] Reconciling cluster state
INFO[0043] [reconcile] This is newly generated cluster
INFO[0043] Pre-pulling kubernetes images
INFO[0043] Pulling image [rancher/hyperkube:v1.17.5-rancher1] on host [192.168.0.200], try #1
INFO[0043] Pulling image [rancher/hyperkube:v1.17.5-rancher1] on host [192.168.0.201], try #1
INFO[0043] Pulling image [rancher/hyperkube:v1.17.5-rancher1] on host [192.168.0.203], try #1
INFO[0043] Pulling image [rancher/hyperkube:v1.17.5-rancher1] on host [192.168.0.202], try #1
INFO[0188] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.200]
INFO[0197] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.201]
INFO[0202] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.202]
INFO[0204] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.203]
INFO[0204] Kubernetes images pulled successfully
INFO[0204] [etcd] Building up etcd plane..
INFO[0204] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0204] Starting container [etcd-fix-perm] on host [192.168.0.200], try #1
INFO[0205] Successfully started [etcd-fix-perm] container on host [192.168.0.200]
INFO[0205] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.200]
INFO[0205] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.200]
INFO[0205] Container [etcd-fix-perm] is still running on host [192.168.0.200]
INFO[0206] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.200]
INFO[0206] Removing container [etcd-fix-perm] on host [192.168.0.200], try #1
INFO[0206] [remove/etcd-fix-perm] Successfully removed container on host [192.168.0.200]
INFO[0206] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [192.168.0.200], try #1
INFO[0211] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [192.168.0.200]
INFO[0211] Starting container [etcd] on host [192.168.0.200], try #1
INFO[0211] [etcd] Successfully started [etcd] container on host [192.168.0.200]
INFO[0211] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [192.168.0.200]
INFO[0211] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0211] Starting container [etcd-rolling-snapshots] on host [192.168.0.200], try #1
INFO[0212] [etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.0.200]
INFO[0217] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0217] Starting container [rke-bundle-cert] on host [192.168.0.200], try #1
INFO[0217] [certificates] Successfully started [rke-bundle-cert] container on host [192.168.0.200]
INFO[0217] Waiting for [rke-bundle-cert] container to exit on host [192.168.0.200]
INFO[0217] Container [rke-bundle-cert] is still running on host [192.168.0.200]
INFO[0218] Waiting for [rke-bundle-cert] container to exit on host [192.168.0.200]
INFO[0218] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.0.200]
INFO[0218] Removing container [rke-bundle-cert] on host [192.168.0.200], try #1
INFO[0218] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0219] Starting container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0219] [etcd] Successfully started [rke-log-linker] container on host [192.168.0.200]
INFO[0219] Removing container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0219] [remove/rke-log-linker] Successfully removed container on host [192.168.0.200]
INFO[0219] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0219] Starting container [etcd-fix-perm] on host [192.168.0.201], try #1
INFO[0219] Successfully started [etcd-fix-perm] container on host [192.168.0.201]
INFO[0219] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.201]
INFO[0219] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.201]
INFO[0219] Container [etcd-fix-perm] is still running on host [192.168.0.201]
INFO[0220] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.201]
INFO[0220] Removing container [etcd-fix-perm] on host [192.168.0.201], try #1
INFO[0220] [remove/etcd-fix-perm] Successfully removed container on host [192.168.0.201]
INFO[0220] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [192.168.0.201], try #1
INFO[0226] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [192.168.0.201]
INFO[0226] Starting container [etcd] on host [192.168.0.201], try #1
INFO[0228] [etcd] Successfully started [etcd] container on host [192.168.0.201]
INFO[0228] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [192.168.0.201]
INFO[0228] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0229] Starting container [etcd-rolling-snapshots] on host [192.168.0.201], try #1
INFO[0229] [etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.0.201]
INFO[0234] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0234] Starting container [rke-bundle-cert] on host [192.168.0.201], try #1
INFO[0235] [certificates] Successfully started [rke-bundle-cert] container on host [192.168.0.201]
INFO[0235] Waiting for [rke-bundle-cert] container to exit on host [192.168.0.201]
INFO[0235] Container [rke-bundle-cert] is still running on host [192.168.0.201]
INFO[0236] Waiting for [rke-bundle-cert] container to exit on host [192.168.0.201]
INFO[0236] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.0.201]
INFO[0236] Removing container [rke-bundle-cert] on host [192.168.0.201], try #1
INFO[0236] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0236] Starting container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0236] [etcd] Successfully started [rke-log-linker] container on host [192.168.0.201]
INFO[0236] Removing container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0236] [remove/rke-log-linker] Successfully removed container on host [192.168.0.201]
INFO[0236] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0236] Starting container [etcd-fix-perm] on host [192.168.0.202], try #1
INFO[0237] Successfully started [etcd-fix-perm] container on host [192.168.0.202]
INFO[0237] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.202]
INFO[0237] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.202]
INFO[0237] Container [etcd-fix-perm] is still running on host [192.168.0.202]
INFO[0238] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.202]
INFO[0238] Removing container [etcd-fix-perm] on host [192.168.0.202], try #1
INFO[0238] [remove/etcd-fix-perm] Successfully removed container on host [192.168.0.202]
INFO[0238] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [192.168.0.202], try #1
INFO[0244] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [192.168.0.202]
INFO[0244] Starting container [etcd] on host [192.168.0.202], try #1
INFO[0244] [etcd] Successfully started [etcd] container on host [192.168.0.202]
INFO[0244] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [192.168.0.202]
INFO[0244] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0245] Starting container [etcd-rolling-snapshots] on host [192.168.0.202], try #1
INFO[0245] [etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.0.202]
INFO[0250] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0250] Starting container [rke-bundle-cert] on host [192.168.0.202], try #1
INFO[0250] [certificates] Successfully started [rke-bundle-cert] container on host [192.168.0.202]
INFO[0250] Waiting for [rke-bundle-cert] container to exit on host [192.168.0.202]
INFO[0250] Container [rke-bundle-cert] is still running on host [192.168.0.202]
INFO[0251] Waiting for [rke-bundle-cert] container to exit on host [192.168.0.202]
INFO[0251] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.0.202]
INFO[0251] Removing container [rke-bundle-cert] on host [192.168.0.202], try #1
INFO[0251] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0252] Starting container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0252] [etcd] Successfully started [rke-log-linker] container on host [192.168.0.202]
INFO[0252] Removing container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0252] [remove/rke-log-linker] Successfully removed container on host [192.168.0.202]
INFO[0252] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0252] Starting container [etcd-fix-perm] on host [192.168.0.203], try #1
INFO[0253] Successfully started [etcd-fix-perm] container on host [192.168.0.203]
INFO[0253] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.203]
INFO[0253] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.203]
INFO[0253] Container [etcd-fix-perm] is still running on host [192.168.0.203]
INFO[0254] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.203]
INFO[0254] Removing container [etcd-fix-perm] on host [192.168.0.203], try #1
INFO[0254] [remove/etcd-fix-perm] Successfully removed container on host [192.168.0.203]
INFO[0254] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [192.168.0.203], try #1
INFO[0260] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [192.168.0.203]
INFO[0260] Starting container [etcd] on host [192.168.0.203], try #1
INFO[0260] [etcd] Successfully started [etcd] container on host [192.168.0.203]
INFO[0260] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [192.168.0.203]
INFO[0260] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0260] Starting container [etcd-rolling-snapshots] on host [192.168.0.203], try #1
INFO[0261] [etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.0.203]
INFO[0266] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0266] Starting container [rke-bundle-cert] on host [192.168.0.203], try #1
INFO[0266] [certificates] Successfully started [rke-bundle-cert] container on host [192.168.0.203]
INFO[0266] Waiting for [rke-bundle-cert] container to exit on host [192.168.0.203]
INFO[0266] Container [rke-bundle-cert] is still running on host [192.168.0.203]
INFO[0267] Waiting for [rke-bundle-cert] container to exit on host [192.168.0.203]
INFO[0267] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.0.203]
INFO[0267] Removing container [rke-bundle-cert] on host [192.168.0.203], try #1
INFO[0267] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0267] Starting container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0267] [etcd] Successfully started [rke-log-linker] container on host [192.168.0.203]
INFO[0267] Removing container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0267] [remove/rke-log-linker] Successfully removed container on host [192.168.0.203]
INFO[0267] [etcd] Successfully started etcd plane.. Checking etcd cluster health
INFO[0268] [controlplane] Building up Controller Plane..
INFO[0268] Checking if container [service-sidekick] is running on host [192.168.0.200], try #1
INFO[0268] Checking if container [service-sidekick] is running on host [192.168.0.202], try #1
INFO[0268] Checking if container [service-sidekick] is running on host [192.168.0.201], try #1
INFO[0268] Checking if container [service-sidekick] is running on host [192.168.0.203], try #1
INFO[0268] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0268] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0268] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0268] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0268] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.200]
INFO[0268] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.203]
INFO[0268] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.201]
INFO[0268] Starting container [kube-apiserver] on host [192.168.0.200], try #1
INFO[0268] Starting container [kube-apiserver] on host [192.168.0.203], try #1
INFO[0268] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.202]
INFO[0268] Starting container [kube-apiserver] on host [192.168.0.201], try #1
INFO[0268] Starting container [kube-apiserver] on host [192.168.0.202], try #1
INFO[0268] [controlplane] Successfully started [kube-apiserver] container on host [192.168.0.200]
INFO[0268] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.0.200]
INFO[0268] [controlplane] Successfully started [kube-apiserver] container on host [192.168.0.201]
INFO[0268] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.0.201]
INFO[0268] [controlplane] Successfully started [kube-apiserver] container on host [192.168.0.203]
INFO[0268] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.0.203]
INFO[0268] [controlplane] Successfully started [kube-apiserver] container on host [192.168.0.202]
INFO[0268] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.0.202]
INFO[0276] [healthcheck] service [kube-apiserver] on host [192.168.0.201] is healthy
INFO[0276] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0276] [healthcheck] service [kube-apiserver] on host [192.168.0.200] is healthy
INFO[0276] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0276] Starting container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0276] Starting container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0276] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.201]
INFO[0276] Removing container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0276] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.200]
INFO[0276] Removing container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0276] [remove/rke-log-linker] Successfully removed container on host [192.168.0.201]
INFO[0276] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.201]
INFO[0276] Starting container [kube-controller-manager] on host [192.168.0.201], try #1
INFO[0276] [remove/rke-log-linker] Successfully removed container on host [192.168.0.200]
INFO[0276] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.200]
INFO[0276] Starting container [kube-controller-manager] on host [192.168.0.200], try #1
INFO[0277] [controlplane] Successfully started [kube-controller-manager] container on host [192.168.0.201]
INFO[0277] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.0.201]
INFO[0277] [controlplane] Successfully started [kube-controller-manager] container on host [192.168.0.200]
INFO[0277] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.0.200]
INFO[0277] [healthcheck] service [kube-apiserver] on host [192.168.0.202] is healthy
INFO[0277] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0277] Starting container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0277] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.202]
INFO[0277] Removing container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0278] [remove/rke-log-linker] Successfully removed container on host [192.168.0.202]
INFO[0278] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.202]
INFO[0278] Starting container [kube-controller-manager] on host [192.168.0.202], try #1
INFO[0278] [healthcheck] service [kube-apiserver] on host [192.168.0.203] is healthy
INFO[0278] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0278] Starting container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0278] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.203]
INFO[0278] [controlplane] Successfully started [kube-controller-manager] container on host [192.168.0.202]
INFO[0278] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.0.202]
INFO[0278] Removing container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0278] [remove/rke-log-linker] Successfully removed container on host [192.168.0.203]
INFO[0278] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.203]
INFO[0279] Starting container [kube-controller-manager] on host [192.168.0.203], try #1
INFO[0279] [controlplane] Successfully started [kube-controller-manager] container on host [192.168.0.203]
INFO[0279] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.0.203]
INFO[0282] [healthcheck] service [kube-controller-manager] on host [192.168.0.201] is healthy
INFO[0282] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0282] [healthcheck] service [kube-controller-manager] on host [192.168.0.200] is healthy
INFO[0282] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0282] Starting container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0282] Starting container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0282] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.201]
INFO[0282] Removing container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0282] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.200]
INFO[0282] Removing container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0282] [remove/rke-log-linker] Successfully removed container on host [192.168.0.201]
INFO[0282] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.201]
INFO[0282] Starting container [kube-scheduler] on host [192.168.0.201], try #1
INFO[0282] [remove/rke-log-linker] Successfully removed container on host [192.168.0.200]
INFO[0282] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.200]
INFO[0282] Starting container [kube-scheduler] on host [192.168.0.200], try #1
INFO[0283] [controlplane] Successfully started [kube-scheduler] container on host [192.168.0.201]
INFO[0283] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.0.201]
INFO[0283] [controlplane] Successfully started [kube-scheduler] container on host [192.168.0.200]
INFO[0283] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.0.200]
INFO[0284] [healthcheck] service [kube-controller-manager] on host [192.168.0.202] is healthy
INFO[0284] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0284] Starting container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0284] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.202]
INFO[0284] Removing container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0284] [remove/rke-log-linker] Successfully removed container on host [192.168.0.202]
INFO[0284] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.202]
INFO[0284] [healthcheck] service [kube-controller-manager] on host [192.168.0.203] is healthy
INFO[0284] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0284] Starting container [kube-scheduler] on host [192.168.0.202], try #1
INFO[0284] Starting container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0285] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.203]
INFO[0285] Removing container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0285] [controlplane] Successfully started [kube-scheduler] container on host [192.168.0.202]
INFO[0285] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.0.202]
INFO[0285] [remove/rke-log-linker] Successfully removed container on host [192.168.0.203]
INFO[0285] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.203]
INFO[0285] Starting container [kube-scheduler] on host [192.168.0.203], try #1
INFO[0285] [controlplane] Successfully started [kube-scheduler] container on host [192.168.0.203]
INFO[0285] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.0.203]
INFO[0288] [healthcheck] service [kube-scheduler] on host [192.168.0.201] is healthy
INFO[0288] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0288] [healthcheck] service [kube-scheduler] on host [192.168.0.200] is healthy
INFO[0288] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0288] Starting container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0288] Starting container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0288] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.201]
INFO[0288] Removing container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0288] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.200]
INFO[0288] Removing container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0288] [remove/rke-log-linker] Successfully removed container on host [192.168.0.201]
INFO[0288] [remove/rke-log-linker] Successfully removed container on host [192.168.0.200]
INFO[0290] [healthcheck] service [kube-scheduler] on host [192.168.0.202] is healthy
INFO[0290] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0290] Starting container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0290] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.202]
INFO[0290] Removing container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0291] [healthcheck] service [kube-scheduler] on host [192.168.0.203] is healthy
INFO[0291] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0291] Starting container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0291] [remove/rke-log-linker] Successfully removed container on host [192.168.0.202]
INFO[0291] [controlplane] Successfully started [rke-log-linker] container on host [192.168.0.203]
INFO[0291] Removing container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0291] [remove/rke-log-linker] Successfully removed container on host [192.168.0.203]
INFO[0291] [controlplane] Successfully started Controller Plane..
INFO[0291] [authz] Creating rke-job-deployer ServiceAccount
INFO[0291] [authz] rke-job-deployer ServiceAccount created successfully
INFO[0291] [authz] Creating system:node ClusterRoleBinding
INFO[0291] [authz] system:node ClusterRoleBinding created successfully
INFO[0291] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding
INFO[0291] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully
INFO[0291] Successfully Deployed state file at [./cluster.rkestate]
INFO[0291] [state] Saving full cluster state to Kubernetes
INFO[0291] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: cluster-state
INFO[0291] [worker] Building up Worker Plane..
INFO[0291] Checking if container [service-sidekick] is running on host [192.168.0.201], try #1
INFO[0291] Checking if container [service-sidekick] is running on host [192.168.0.200], try #1
INFO[0291] Checking if container [service-sidekick] is running on host [192.168.0.202], try #1
INFO[0291] Checking if container [service-sidekick] is running on host [192.168.0.203], try #1
INFO[0291] [sidekick] Sidekick container already created on host [192.168.0.200]
INFO[0291] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.200]
INFO[0291] [sidekick] Sidekick container already created on host [192.168.0.201]
INFO[0291] [sidekick] Sidekick container already created on host [192.168.0.202]
INFO[0291] [sidekick] Sidekick container already created on host [192.168.0.203]
INFO[0291] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.201]
INFO[0291] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.203]
INFO[0291] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.202]
INFO[0291] Starting container [kubelet] on host [192.168.0.203], try #1
INFO[0291] Starting container [kubelet] on host [192.168.0.201], try #1
INFO[0291] Starting container [kubelet] on host [192.168.0.200], try #1
INFO[0291] Starting container [kubelet] on host [192.168.0.202], try #1
INFO[0291] [worker] Successfully started [kubelet] container on host [192.168.0.200]
INFO[0291] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.0.200]
INFO[0291] [worker] Successfully started [kubelet] container on host [192.168.0.201]
INFO[0291] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.0.201]
INFO[0291] [worker] Successfully started [kubelet] container on host [192.168.0.203]
INFO[0291] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.0.203]
INFO[0291] [worker] Successfully started [kubelet] container on host [192.168.0.202]
INFO[0291] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.0.202]
INFO[0297] [healthcheck] service [kubelet] on host [192.168.0.200] is healthy
INFO[0297] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0297] Starting container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0297] [worker] Successfully started [rke-log-linker] container on host [192.168.0.200]
INFO[0297] Removing container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0297] [remove/rke-log-linker] Successfully removed container on host [192.168.0.200]
INFO[0297] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.200]
INFO[0297] Starting container [kube-proxy] on host [192.168.0.200], try #1
INFO[0297] [worker] Successfully started [kube-proxy] container on host [192.168.0.200]
INFO[0297] [healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.0.200]
INFO[0297] [healthcheck] service [kube-proxy] on host [192.168.0.200] is healthy
INFO[0297] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0297] Starting container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0297] [worker] Successfully started [rke-log-linker] container on host [192.168.0.200]
INFO[0298] Removing container [rke-log-linker] on host [192.168.0.200], try #1
INFO[0298] [remove/rke-log-linker] Successfully removed container on host [192.168.0.200]
INFO[0317] [healthcheck] service [kubelet] on host [192.168.0.201] is healthy
INFO[0317] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0317] Starting container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0317] [healthcheck] service [kubelet] on host [192.168.0.203] is healthy
INFO[0317] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0317] [worker] Successfully started [rke-log-linker] container on host [192.168.0.201]
INFO[0317] Removing container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0317] [healthcheck] service [kubelet] on host [192.168.0.202] is healthy
INFO[0317] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0317] Starting container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0317] Starting container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0317] [remove/rke-log-linker] Successfully removed container on host [192.168.0.201]
INFO[0317] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.201]
INFO[0318] Starting container [kube-proxy] on host [192.168.0.201], try #1
INFO[0318] [worker] Successfully started [rke-log-linker] container on host [192.168.0.203]
INFO[0318] Removing container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0318] [worker] Successfully started [kube-proxy] container on host [192.168.0.201]
INFO[0318] [healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.0.201]
INFO[0318] [healthcheck] service [kube-proxy] on host [192.168.0.201] is healthy
INFO[0318] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0318] [worker] Successfully started [rke-log-linker] container on host [192.168.0.202]
INFO[0318] Removing container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0318] Starting container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0318] [remove/rke-log-linker] Successfully removed container on host [192.168.0.203]
INFO[0318] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.203]
INFO[0318] Starting container [kube-proxy] on host [192.168.0.203], try #1
INFO[0318] [worker] Successfully started [kube-proxy] container on host [192.168.0.203]
INFO[0318] [healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.0.203]
INFO[0318] [worker] Successfully started [rke-log-linker] container on host [192.168.0.201]
INFO[0318] Removing container [rke-log-linker] on host [192.168.0.201], try #1
INFO[0318] [remove/rke-log-linker] Successfully removed container on host [192.168.0.202]
INFO[0318] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.202]
INFO[0318] Starting container [kube-proxy] on host [192.168.0.202], try #1
INFO[0318] [healthcheck] service [kube-proxy] on host [192.168.0.203] is healthy
INFO[0318] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0318] Starting container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0318] [remove/rke-log-linker] Successfully removed container on host [192.168.0.201]
INFO[0318] [worker] Successfully started [kube-proxy] container on host [192.168.0.202]
INFO[0318] [healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.0.202]
INFO[0318] [worker] Successfully started [rke-log-linker] container on host [192.168.0.203]
INFO[0318] Removing container [rke-log-linker] on host [192.168.0.203], try #1
INFO[0318] [healthcheck] service [kube-proxy] on host [192.168.0.202] is healthy
INFO[0318] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0319] Starting container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0319] [remove/rke-log-linker] Successfully removed container on host [192.168.0.203]
INFO[0319] [worker] Successfully started [rke-log-linker] container on host [192.168.0.202]
INFO[0319] Removing container [rke-log-linker] on host [192.168.0.202], try #1
INFO[0319] [remove/rke-log-linker] Successfully removed container on host [192.168.0.202]
INFO[0319] [worker] Successfully started Worker Plane..
INFO[0319] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.200]
INFO[0319] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.201]
INFO[0319] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.203]
INFO[0319] Image [rancher/rke-tools:v0.1.56] exists on host [192.168.0.202]
INFO[0319] Starting container [rke-log-cleaner] on host [192.168.0.200], try #1
INFO[0319] Starting container [rke-log-cleaner] on host [192.168.0.203], try #1
INFO[0319] Starting container [rke-log-cleaner] on host [192.168.0.201], try #1
INFO[0319] Starting container [rke-log-cleaner] on host [192.168.0.202], try #1
INFO[0319] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.0.200]
INFO[0319] Removing container [rke-log-cleaner] on host [192.168.0.200], try #1
INFO[0319] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.0.201]
INFO[0319] Removing container [rke-log-cleaner] on host [192.168.0.201], try #1
INFO[0319] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.0.203]
INFO[0319] Removing container [rke-log-cleaner] on host [192.168.0.203], try #1
INFO[0320] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.0.202]
INFO[0320] Removing container [rke-log-cleaner] on host [192.168.0.202], try #1
INFO[0320] [remove/rke-log-cleaner] Successfully removed container on host [192.168.0.200]
INFO[0320] [remove/rke-log-cleaner] Successfully removed container on host [192.168.0.201]
INFO[0320] [remove/rke-log-cleaner] Successfully removed container on host [192.168.0.203]
INFO[0320] [remove/rke-log-cleaner] Successfully removed container on host [192.168.0.202]
INFO[0320] [sync] Syncing nodes Labels and Taints
INFO[0320] [sync] Successfully synced nodes Labels and Taints
INFO[0320] [network] Setting up network plugin: canal
INFO[0320] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes
INFO[0320] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes
INFO[0320] [addons] Executing deploy job rke-network-plugin
INFO[0330] [addons] Setting up coredns
INFO[0330] [addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes
INFO[0330] [addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes
INFO[0330] [addons] Executing deploy job rke-coredns-addon
INFO[0335] [addons] CoreDNS deployed successfully
INFO[0335] [dns] DNS provider coredns deployed successfully
INFO[0335] [addons] Setting up Metrics Server
INFO[0335] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes
INFO[0335] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes
INFO[0335] [addons] Executing deploy job rke-metrics-addon
INFO[0340] [addons] Metrics Server deployed successfully
INFO[0340] [ingress] Setting up nginx ingress controller
INFO[0340] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes
INFO[0340] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes
INFO[0340] [addons] Executing deploy job rke-ingress-controller
INFO[0345] [ingress] ingress controller nginx deployed successfully
INFO[0345] [addons] Setting up user addons
INFO[0345] [addons] no user addons defined
INFO[0345] Finished building Kubernetes cluster successfully
ubuntu@kub0:~$ sudo curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.17.5/bin/linux/amd64/kubectl
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 41.4M 100 41.4M 0 0 9913k 0 0:00:04 0:00:04 --:--:-- 9915k
ubuntu@kub0:~$ sudo chmod +x ./kubectl
ubuntu@kub0:~$ sudo mv ./kubectl /usr/local/bin/kubectl
ubuntu@kub0:~$ sudo mkdir ~/.kube
ubuntu@kub0:~$ wget http://storage.googleapis.com/kubernetes-helm/helm-v3.2.1-linux-amd64.tar.gz
--2020-05-18 16:07:22-- http://storage.googleapis.com/kubernetes-helm/helm-v3.2.1-linux-amd64.tar.gz
Resolving storage.googleapis.com (storage.googleapis.com)... 2607:f8b0:400b:809::2010, 172.217.1.176
Connecting to storage.googleapis.com (storage.googleapis.com)|2607:f8b0:400b:809::2010|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-05-18 16:07:22 ERROR 404: Not Found.
ubuntu@kub0:~$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
ubuntu@kub0:~$ chmod 700 get_helm.sh
ubuntu@kub0:~$ ./get_helm.sh
Downloading https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
ubuntu@kub0:~$ sudo cp kube_config_cluster.yml ~/.kube/config
ubuntu@kub0:~$ sudo chmod 777 ~/.kube/config
ubuntu@kub0:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx default-http-backend-67cf578fc4-b2rh8 1/1 Running 0 21m 10.42.2.2 192.168.0.202 <none> <none>
ingress-nginx nginx-ingress-controller-8gh7x 1/1 Running 0 21m 192.168.0.202 192.168.0.202 <none> <none>
ingress-nginx nginx-ingress-controller-8wgkl 1/1 Running 0 21m 192.168.0.201 192.168.0.201 <none> <none>
ingress-nginx nginx-ingress-controller-hwmjs 1/1 Running 0 21m 192.168.0.203 192.168.0.203 <none> <none>
ingress-nginx nginx-ingress-controller-r677w 1/1 Running 0 21m 192.168.0.200 192.168.0.200 <none> <none>
kube-system canal-6wgdm 2/2 Running 0 21m 192.168.0.201 192.168.0.201 <none> <none>
kube-system canal-c44qz 2/2 Running 0 21m 192.168.0.200 192.168.0.200 <none> <none>
kube-system canal-hq5ks 2/2 Running 0 21m 192.168.0.203 192.168.0.203 <none> <none>
kube-system canal-lzhhs 2/2 Running 0 21m 192.168.0.202 192.168.0.202 <none> <none>
kube-system coredns-7c5566588d-j4bzm 1/1 Running 0 19m 10.42.0.2 192.168.0.200 <none> <none>
kube-system coredns-7c5566588d-msmzh 1/1 Running 0 21m 10.42.1.2 192.168.0.201 <none> <none>
kube-system coredns-autoscaler-65bfc8d47d-8tbbh 1/1 Running 0 21m 10.42.2.3 192.168.0.202 <none> <none>
kube-system metrics-server-6b55c64f86-l2f99 1/1 Running 0 21m 10.42.3.2 192.168.0.203 <none> <none>
kube-system rke-coredns-addon-deploy-job-8h7xh 0/1 Completed 0 21m 192.168.0.200 192.168.0.200 <none> <none>
kube-system rke-ingress-controller-deploy-job-dqp9q 0/1 Completed 0 21m 192.168.0.200 192.168.0.200 <none> <none>
kube-system rke-metrics-addon-deploy-job-nrvfv 0/1 Completed 0 21m 192.168.0.200 192.168.0.200 <none> <none>
kube-system rke-network-plugin-deploy-job-s9nh6 0/1 Completed 0 22m 192.168.0.200 192.168.0.200 <none> <none>
ubuntu@kub0:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.0.200 Ready controlplane,etcd,worker 22m v1.17.5
192.168.0.201 Ready controlplane,etcd,worker 22m v1.17.5
192.168.0.202 Ready controlplane,etcd,worker 22m v1.17.5
192.168.0.203 Ready controlplane,etcd,worker 22m v1.17.5
ubuntu@kub0:~$
ubuntu@kub0:~$ kubectl get services --all-namespaces -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 33m <none>
ingress-nginx default-http-backend ClusterIP 10.43.72.129 <none> 80/TCP 32m app=default-http-backend
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 32m k8s-app=kube-dns
kube-system metrics-server ClusterIP 10.43.105.202 <none> 443/TCP 32m k8s-app=metrics-server
ubuntu@kub0:~$ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
192.168.0.200 95m 2% 2612Mi 16%
192.168.0.201 93m 2% 2195Mi 13%
192.168.0.202 122m 3% 2225Mi 14%
192.168.0.203 131m 3% 2172Mi 13%
Install Tomcat helm chart on 4 node kubernetes cluster
ubuntu@kub0:~$ sudo helm repo add stable https://kubernetes-charts.storage.googleapis.com
"stable" has been added to your repositories
ubuntu@kub0:~$ helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
ubuntu@kub0:~$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
ubuntu@kub0:~$ sudo helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈
ubuntu@kub0:~$ sudo helm install tomcat-dev stable/tomcat
[sudo] password for ubuntu:
NAME: tomcat-dev
LAST DEPLOYED: Sat May 23 14:59:58 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w tomcat-dev'
export SERVICE_IP=$(kubectl get svc --namespace default tomcat-dev -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo http://$SERVICE_IP:
# get the node IP - however any IP on the cluster will work as it is a LoadBalancer
ubuntu@kub0:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default tomcat-dev-64d5c484b8-rn6rd 1/1 Running 0 3m49s 10.42.2.5 192.168.0.202 <none> <none>
# get the port
ubuntu@kub0:~$ kubectl get services --all-namespaces -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 4d23h <none>
default tomcat-dev LoadBalancer 10.43.133.127 <pending> 80:32353/TCP 6m57s app=tomcat,release=tomcat-dev
# check server
ubuntu@kub0:~$ kubectl exec -it tomcat-dev-64d5c484b8-rn6rd bash
root@tomcat-dev-64d5c484b8-rn6rd:/usr/local/tomcat# cat logs/catalina.2020-05-23.log
INFO: Deployment of web application archive [/usr/local/tomcat/webapps/sample.war] has finished in [275] ms
May 23, 2020 8:00:01 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 342 ms
# check the site
ubuntu@kub0:~$ wget http://192.168.0.202:32353/sample
--2020-05-23 15:08:52-- http://192.168.0.202:32353/sample
Connecting to 192.168.0.202:32353... connected.
HTTP request sent, awaiting response... 302 Found
Location: /sample/ [following]
HTTP request sent, awaiting response... 200 OK
Length: 636 [text/html]
2020-05-23 15:08:52 (186 MB/s) - ‘sample’ saved [636/636]
Add 5th osx node
Move key to master node
biometric:wse_go user$ scp ~/__devops/macmini1/id_rsa ubuntu@192.168.0.200:~/macmini1
# rerun rke yaml
biometric:wse_go user$ ssh ubuntu@192.168.0.200
ubuntu@kub0:~$ vi cluster.yml
- address: 192.168.0.53
port: "22"
internal_address: 192.168.0.53
role:
- worker
hostname_override: ""
user: user
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/macmini1/id_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
ubuntu@kub0:~$ sudo rke up --update-only
top
PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPRS PGRP PPID STATE BOOSTS %CPU_ME %CPU_OTHRS UID FAULTS COW MSGSENT MSGRECV SYSBSD SYSMACH CSW PAGEIN IDLEW POWER INSTRS CYCLES
971 com.docker.h 144.7 03:57.99 19/5 0 43 11G 0B 8436K 959 963 running *0[1] 0.00000 0.00000 501 2866956+ 422 364 176 5648246+ 415 3237189+ 0 248811+ 154.4 1409125945 4390646170
961 com.docker.v 46.0 00:15.77 10/1 0 27 53M- 0B 848K 959 959 running *0[1] 0.00000 0.00000 501 41585+ 1461 62 29 885359+ 2021+ 162290+ 0 926+ 46.3 958398813 1407087845
INFO[0036] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.200]
INFO[0036] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.201]
INFO[0036] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.203]
INFO[0036] Image [rancher/hyperkube:v1.17.5-rancher1] exists on host [192.168.0.202]
INFO[0036] Pulling image [rancher/hyperkube:v1.17.5-rancher1] on host [192.168.0.53], try #1
macmini1:~ user$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1f8f32661b82 rancher/rke-tools:v0.1.56 "nginx-proxy CP_HOST…" 28 seconds ago Up 5 seconds nginx-proxy
# fs issue
INFO[0180] Starting container [kubelet] on host [192.168.0.53], try #1
WARN[0180] Can't start Docker container [kubelet] on host [192.168.0.53]: Error response from daemon: Mounts denied:
The paths /var/log/pods and /var/log/containers and /etc/ceph and /var/lib/kubelet and /var/lib/calico and /opt/cni
are not shared from OS X and are not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
.
INFO[0180] Starting container [kubelet] on host [192.168.0.53], try #2
WARN[0180] Can't start Docker container [kubelet] on host [192.168.0.53]: Error response from daemon: Mounts denied:
The paths /var/log/pods and /var/log/containers and /etc/ceph and /var/lib/kubelet and /var/lib/calico and /opt/cni
are not shared from OS X and are not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
.
INFO[0180] Starting container [kubelet] on host [192.168.0.53], try #3
WARN[0180] Can't start Docker container [kubelet] on host [192.168.0.53]: Error response from daemon: Mounts denied:
The paths /var/log/pods and /var/log/containers and /etc/ceph and /var/lib/kubelet and /var/lib/calico and /opt/cni
are not shared from OS X and are not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
.
FATA[0180] [workerPlane] Failed to bring up Worker Plane: [Failed to start [kubelet] container on host [192.168.0.53]: Error response from daemon: Mounts denied:
The paths /var/log/pods and /var/log/containers and /etc/ceph and /var/lib/kubelet and /var/lib/calico and /opt/cni
are not shared from OS X and are not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
.]
# fixing
508 sudo mkdir /var/log/pods
509 sudo mkdir /var/log/containers
510 sudo mkdir /var/lib/kubelet
511 sudo mkdir /etc/ceph
512 sudo mkdir /var/lib/calico
513 sudo mkdir /opt/cni
how to handle the proxy /var and /etc dirs into /private - manually add them - don't use the fs picker in (-)
rke up
bettermacmini1:~user$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6e033ff09d39 rancher/hyperkube:v1.17.5-rancher1 "/opt/rke-tools/entr…" 25 minutes ago Up 11 seconds kubelet
1f8f32661b82 rancher/rke-tools:v0.1.56 "nginx-proxy CP_HOST…" 26 minutes ago Up About a minute nginx-proxy
NFO[0053] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.0.53]
FATA[0106] [workerPlane] Failed to bring up Worker Plane: [Failed to verify healthcheck: Failed to check http://localhost:10248/healthz for service [kubelet] on host [192.168.0.53]: Get http://localhost:10248/healthz: Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: F0621 20:48:23.458119 4084 server.go:253] mkdir /var/lib/kubelet/pki: permission denied]
because of a slow machine (2012 mac min - waiting on 2018 version)macmini1:~ user$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6e033ff09d39 rancher/hyperkube:v1.17.5-rancher1 "/opt/rke-tools/entr…" 26 minutes ago Restarting (255) 12 seconds ago kubeletmacmini1:~ user$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6e033ff09d39 rancher/hyperkube:v1.17.5-rancher1 "/opt/rke-tools/entr…" 27 minutes ago Restarting (255) 34 seconds ago kubelet
1f8f32661b82 rancher/rke-tools:v0.1.56 "nginx-proxy CP_HOST…" 27 minutes ago Up 3 minutes nginx-proxy
macmini1:~ user$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6e033ff09d39 rancher/hyperkube:v1.17.5-rancher1 "/opt/rke-tools/entr…" 27 minutes ago Up 1 second kubelet
1f8f32661b82 rancher/rke-tools:v0.1.56 "nginx-proxy CP_HOST…" 28 minutes ago Up 3 minutes nginx-proxyrestarting6e033ff09d39 rancher/hyperkube:v1.17.5-rancher1 "/opt/rke-tools/entr…" 28 minutes ago Restarting (255) 47 seconds ago kubelet
add role etcd
macmini1:~ user$ sudo mkdir /var/lib/etcd
INFO[0078] Waiting for [etcd-fix-perm] container to exit on host [192.168.0.53]
FATA[0078] [etcd] Failed to bring up Etcd Plane: Container [etcd-fix-perm] exited with non-zero exit code [1] on host [192.168.0.53]: stdout: , stderr: chmod: /var/lib/rancher/etcd/: Operation not permitted
chmod 777 on all 7 dirs
further
INFO[0049] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.0.53]
FATA[0104] [workerPlane] Failed to bring up Worker Plane: [Failed to verify healthcheck: Failed to check http://localhost:10248/healthz for service [kubelet] on host [192.168.0.53]: Get http://localhost:10248/healthz: Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: F0621 21:09:07.155830 20503 kubelet.go:1380] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/var/lib/kubelet": could not find device with major: 0, minor: 105 in cached partitions map]
trying etcd role againCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5d43fef301a7 rancher/coreos-etcd:v3.4.3-rancher1 "/usr/local/bin/etcd…" 12 minutes ago Up 3 minutes etcd
6e033ff09d39 rancher/hyperkube:v1.17.5-rancher1 "/opt/rke-tools/entr…" 48 minutes ago Restarting (255) 10 seconds ago kubelet
1f8f32661b82 rancher/rke-tools:v0.1.56 "nginx-proxy CP_HOST…" 49 minutes ago Up 13 minutes nginx-proxy
FATA[0006] [[network] Host [192.168.0.201] is not able to connect to the following ports: [192.168.0.53:2380, 192.168.0.53:2379]. Please check network policies and firewall rules]
flipping kubernetes on/off in docker desktop
1 Comment
Michael O'Brien
RKE 1.1.3 ubuntu 16 VM on OSX