dev
Page tree
Skip to end of metadata
Go to start of metadata

Quickstart

Azure Developer Guide

Azure Arc GA

https://portal.azure.com/#blade/HubsExtension/Resources/resourceType/Microsoft.Resources%2Fresources

https://portal.azure.com/#blade/HubsExtension/BrowseResourceBlade/resourceType/Microsoft.HybridCompute%2Fmachines

Install Azure CLI

Azure Developer Guide#AzureCLIInstallation

Install Azure CLI on OSX

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-macos?view=azure-cli-latest

$ brew install azure-cli
==> Installing dependencies for azure-cli: openssl@1.1, gdbm, readline, sqlite, xz and python@3.8
==> Installing azure-cli dependency: openssl@1.1
  /usr/local/Cellar/python@3.8/3.8.1: 4,094 files, 62.8MB
==> Installing azure-cli
==> Downloading https://homebrew.bintray.com/bottles/azure-cli-2.1.0.catalina.bottle.tar.gz

# upgrade
brew upgrade azure-cli


Create Azure Jumpbox on Ubuntu

Follow https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest

Single 

Login to Azure CLI

https://portal.azure.com/#blade/HubsExtension/Resources/resourceType/Microsoft.Resources%2Fresources

$ az login
You have logged in. Now let us find all the subscriptions to which you have access...
[{
    "cloudName": "AzureCloud",
    "homeTenantId": "bcb.4f",
    "id": "f4b7..70e8b",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "bcb..f4f",
    "user": {
      "name": "mic...sg",
      "type": "user"
    }}]

Kubernetes on Azure

see Kubernetes Developer Guide#SwitchingcontextfromAzureAKSbacktolocalDockerDesktopKubernetes

Kubernetes on Azure AKS - Managed

https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough

https://docs.microsoft.com/en-us/azure/aks/kubernetes-dashboard

Create AKS Cluster


Console and ARM version





template
{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "resourceName": {
            "type": "string",
            "metadata": {
                "description": "The name of the Managed Cluster resource."
            }
        },
        "location": {
            "type": "string",
            "metadata": {
                "description": "The location of AKS resource."
            }
        },
        "dnsPrefix": {
            "type": "string",
            "metadata": {
                "description": "Optional DNS prefix to use with hosted Kubernetes API server FQDN."
            }
        },
        "osDiskSizeGB": {
            "type": "int",
            "defaultValue": 0,
            "metadata": {
                "description": "Disk size (in GiB) to provision for each of the agent pool nodes. This value ranges from 0 to 1023. Specifying 0 will apply the default disk size for that agentVMSize."
            },
            "minValue": 0,
            "maxValue": 1023
        },
        "kubernetesVersion": {
            "type": "string",
            "defaultValue": "1.7.7",
            "metadata": {
                "description": "The version of Kubernetes."
            }
        },
        "networkPlugin": {
            "type": "string",
            "allowedValues": [
                "azure",
                "kubenet"
            ],
            "metadata": {
                "description": "Network plugin used for building Kubernetes network."
            }
        },
        "enableRBAC": {
            "type": "bool",
            "defaultValue": true,
            "metadata": {
                "description": "Boolean flag to turn on and off of RBAC."
            }
        },
        "vmssNodePool": {
            "type": "bool",
            "defaultValue": false,
            "metadata": {
                "description": "Boolean flag to turn on and off of virtual machine scale sets"
            }
        },
        "windowsProfile": {
            "type": "bool",
            "defaultValue": false,
            "metadata": {
                "description": "Boolean flag to turn on and off of virtual machine scale sets"
            }
        },
        "servicePrincipalClientId": {
            "metadata": {
                "description": "Client ID (used by cloudprovider)."
            },
            "type": "securestring"
        },
        "servicePrincipalClientSecret": {
            "metadata": {
                "description": "The Service Principal Client Secret."
            },
            "type": "securestring"
        },
        "enablePrivateCluster": {
            "type": "bool",
            "defaultValue": false,
            "metadata": {
                "description": "Enable private network access to the Kubernetes cluster."
            }
        },
        "aadSessionKey": {
            "type": "securestring",
            "defaultValue": ""
        },
        "enableHttpApplicationRouting": {
            "type": "bool",
            "defaultValue": true,
            "metadata": {
                "description": "Boolean flag to turn on and off http application routing."
            }
        },
        "enableAzurePolicy": {
            "type": "bool",
            "defaultValue": false,
            "metadata": {
                "description": "Boolean flag to turn on and off Azure Policy addon."
            }
        },
        "principalId": {
            "type": "string",
            "metadata": {
                "description": "The objectId of service principal."
            }
        },
        "enableOmsAgent": {
            "type": "bool",
            "defaultValue": true,
            "metadata": {
                "description": "Boolean flag to turn on and off omsagent addon."
            }
        },
        "workspaceRegion": {
            "type": "string",
            "defaultValue": "East US",
            "metadata": {
                "description": "Specify the region for your OMS workspace."
            }
        },
        "workspaceName": {
            "type": "string",
            "metadata": {
                "description": "Specify the name of the OMS workspace."
            }
        },
        "omsWorkspaceId": {
            "type": "string",
            "metadata": {
                "description": "Specify the resource id of the OMS workspace."
            }
        },
        "omsSku": {
            "type": "string",
            "defaultValue": "standalone",
            "allowedValues": [
                "free",
                "standalone",
                "pernode"
            ],
            "metadata": {
                "description": "Select the SKU for your workspace."
            }
        }
    },
    "resources": [
        {
            "apiVersion": "2020-09-01",
            "dependsOn": [
                "[concat('Microsoft.Resources/deployments/', 'WorkspaceDeployment-20210114120627')]"
            ],
            "type": "Microsoft.ContainerService/managedClusters",
            "location": "[parameters('location')]",
            "name": "[parameters('resourceName')]",
            "properties": {
                "kubernetesVersion": "[parameters('kubernetesVersion')]",
                "enableRBAC": "[parameters('enableRBAC')]",
                "dnsPrefix": "[parameters('dnsPrefix')]",
                "agentPoolProfiles": [
                    {
                        "name": "agentpool",
                        "osDiskSizeGB": "[parameters('osDiskSizeGB')]",
                        "count": 1,
                        "vmSize": "Standard_DS2_v2",
                        "osType": "Linux",
                        "storageProfile": "ManagedDisks",
                        "type": "VirtualMachineScaleSets",
                        "mode": "System",
                        "maxPods": 110,
                        "availabilityZones": [
                            "1",
                            "2",
                            "3"
                        ]
                    }
                ],
                "networkProfile": {
                    "loadBalancerSku": "standard",
                    "networkPlugin": "[parameters('networkPlugin')]"
                },
                "servicePrincipalProfile": {
                    "ClientId": "[parameters('servicePrincipalClientId')]",
                    "Secret": "[parameters('servicePrincipalClientSecret')]",
                    "aadSessionKey": "[parameters('aadSessionKey')]"
                },
                "apiServerAccessProfile": {
                    "enablePrivateCluster": "[parameters('enablePrivateCluster')]"
                },
                "addonProfiles": {
                    "httpApplicationRouting": {
                        "enabled": "[parameters('enableHttpApplicationRouting')]"
                    },
                    "azurePolicy": {
                        "enabled": "[parameters('enableAzurePolicy')]"
                    },
                    "omsAgent": {
                        "enabled": "[parameters('enableOmsAgent')]",
                        "config": {
                            "logAnalyticsWorkspaceResourceID": "[parameters('omsWorkspaceId')]"
                        }
                    }
                }
            },
            "tags": {
                "env": "stg"
            }
        },
        {
            "type": "Microsoft.Resources/deployments",
            "name": "SolutionDeployment-20210114120627",
            "apiVersion": "2017-05-10",
            "resourceGroup": "[split(parameters('omsWorkspaceId'),'/')[4]]",
            "subscriptionId": "[split(parameters('omsWorkspaceId'),'/')[2]]",
            "properties": {
                "mode": "Incremental",
                "template": {
                    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
                    "contentVersion": "1.0.0.0",
                    "parameters": {},
                    "variables": {},
                    "resources": [
                        {
                            "apiVersion": "2015-11-01-preview",
                            "type": "Microsoft.OperationsManagement/solutions",
                            "location": "[parameters('workspaceRegion')]",
                            "name": "[concat('ContainerInsights', '(', split(parameters('omsWorkspaceId'),'/')[8], ')')]",
                            "properties": {
                                "workspaceResourceId": "[parameters('omsWorkspaceId')]"
                            },
                            "plan": {
                                "name": "[concat('ContainerInsights', '(', split(parameters('omsWorkspaceId'),'/')[8], ')')]",
                                "product": "[concat('OMSGallery/', 'ContainerInsights')]",
                                "promotionCode": "",
                                "publisher": "Microsoft"
                            }
                        }
                    ]
                }
            },
            "dependsOn": [
                "[concat('Microsoft.Resources/deployments/', 'WorkspaceDeployment-20210114120627')]"
            ]
        },
        {
            "type": "Microsoft.Resources/deployments",
            "name": "WorkspaceDeployment-20210114120627",
            "apiVersion": "2017-05-10",
            "resourceGroup": "[split(parameters('omsWorkspaceId'),'/')[4]]",
            "subscriptionId": "[split(parameters('omsWorkspaceId'),'/')[2]]",
            "properties": {
                "mode": "Incremental",
                "template": {
                    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
                    "contentVersion": "1.0.0.0",
                    "parameters": {},
                    "variables": {},
                    "resources": [
                        {
                            "apiVersion": "2015-11-01-preview",
                            "type": "Microsoft.OperationalInsights/workspaces",
                            "location": "[parameters('workspaceRegion')]",
                            "name": "[parameters('workspaceName')]",
                            "properties": {
                                "sku": {
                                    "name": "[parameters('omsSku')]"
                                }
                            }
                        }
                    ]
                }
            }
        },
        {
            "type": "Microsoft.Resources/deployments",
            "name": "ClusterMonitoringMetricPulisherRoleAssignmentDepl-20210114120627",
            "apiVersion": "2017-05-10",
            "resourceGroup": "aks-rg",
            "subscriptionId": "f4b79fca...e8b",
            "properties": {
                "mode": "Incremental",
                "template": {
                    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
                    "contentVersion": "1.0.0.0",
                    "parameters": {},
                    "variables": {},
                    "resources": [
                        {
                            "type": "Microsoft.ContainerService/managedClusters/providers/roleAssignments",
                            "apiVersion": "2018-01-01-preview",
                            "name": "biometric/Microsoft.Authorization/9be768bc-1810-4a0b-9898-d3b077d36f70",
                            "properties": {
                                "roleDefinitionId": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/', '3913510d-42f4-4e42-8a64-420c390055eb')]",
                                "principalId": "[parameters('principalId')]",
                                "scope": "/subscriptions/f4b79fca-5d87-4814-b97e-ec5012170e8b/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/biometric"
                            }
                        }
                    ]
                }
            },
            "dependsOn": [
                "/subscriptions/f4b79fca-5d87-4814-b97e-ec5012170e8b/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/biometric"
            ]
        }
    ],
    "outputs": {
        "controlPlaneFQDN": {
            "type": "string",
            "value": "[reference(concat('Microsoft.ContainerService/managedClusters/', parameters('resourceName'))).fqdn]"
        }
    }
}
parameters
{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "resourceName": {
            "value": "biometric"
        },
        "location": {
            "value": "eastus"
        },
        "dnsPrefix": {
            "value": "biometric-dns"
        },
        "kubernetesVersion": {
            "value": "1.18.10"
        },
        "networkPlugin": {
            "value": "kubenet"
        },
        "enableRBAC": {
            "value": true
        },
        "servicePrincipalClientId": {
            "value": null
        },
        "servicePrincipalClientSecret": {
            "value": null
        },
        "enablePrivateCluster": {
            "value": false
        },
        "aadSessionKey": {
            "value": null
        },
        "enableHttpApplicationRouting": {
            "value": false
        },
        "enableAzurePolicy": {
            "value": false
        },
        "vmssNodePool": {
            "value": true
        },
        "workspaceName": {
            "value": "DefaultWorkspace-f4b79fca-5d87-4814-b97e-ec5012170e8b-EUS"
        },
        "omsWorkspaceId": {
            "value": "/subscriptions/f4b79fca-5d87-4814-b97e-ec5012170e8b/resourceGroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-f4b79fca-5d87-4814-b97e-ec5012170e8b-EUS"
        },
        "workspaceRegion": {
            "value": "eastus"
        },
        "principalId": {
            "value": "baa06276-b73c-4aef-a538-bec11f437eb7"
        }
    }
}

Connect to AKS Cluster

// windows 2021
PS C:\Windows\system32> az aks get-credentials --resource-group aks-rg --name biometric
Merged "biometric" as current context in C:\Users\michaelobrien\.kube\config
PS C:\Windows\system32> kubectl get nodes
NAME                                STATUS   ROLES   AGE   VERSION
aks-agentpool-21090939-vmss000000   Ready    agent   74m   v1.18.10
PS C:\Windows\system32> kubectl get pods --all-namespaces
NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE
kube-system   coredns-748cdb7bf4-r6tkc              1/1     Running   0          74m
kube-system   coredns-748cdb7bf4-vgzjd              1/1     Running   0          75m
kube-system   coredns-autoscaler-868b684fd4-xjrpr   1/1     Running   0          75m
kube-system   kube-proxy-dhctc                      1/1     Running   0          74m
kube-system   metrics-server-58fdc875d5-7p4kx       1/1     Running   0          75m
kube-system   omsagent-qvgbv                        1/1     Running   0          74m
kube-system   omsagent-rs-564f494847-hh975          1/1     Running   0          75m
kube-system   tunnelfront-8569f8cb64-txphv          1/1     Running   0          75m
PS C:\Windows\system32> kubectl top nodes
NAME                                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
aks-agentpool-21090939-vmss000000   168m         8%     995Mi           21%

// osx
biometric:azure michaelobrien$ az aks install-cli
Downloading client to "/usr/local/bin/kubectl" from "https://storage.googleapis.com/kubernetes-release/release/v1.17.3/bin/darwin/amd64/kubectl"

# will not conflict with a local docker desktop kubernetes install
biometric:azure michaelobrien$ az aks get-credentials --resource-group obl_dev_aks --name obl-dev
Merged "obl-dev" as current context in /Users/michaelobrien/.kube/config

# add clusterrolebinding to avoid RBAC errors on the dashboard
biometric:azure michaelobrien$ kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

# check the cluster for the pod
kube-system   kube-proxy-h89tp                        1/1     Running   0          172m

# view the kubernetes dashboard
biometric:azure michaelobrien$ az aks browse --resource-group obl_dev_aks --name obl-dev
Merged "obl-dev" as current context in /var/folders/vv/d6dvwfmx0cgd19qs2yw51p1m0000gn/T/tmphexar_yl
Proxy running on http://127.0.0.1:8001/
Press CTRL+C to close the tunnel...


AKS Costing Model

It looks like Microsoft Azure is not charging for the control plane - just the underlying VMs - I ran both an unmanaged RKE K8S cluster on a Standard_D2s_v3 and an AKS Standard_DS2_v2 for a day and the costs were about the same for both VMs at .81 and .53 for 8h.

Kubernetes on Azure VM - Direct Unmanaged via RKE

Provision one or more VMs via ARM


Create Resource Group

git clone git@github.com:obrienlabs/infrastructure.git
cd azure
biometric:azure michaelobrien$ az group create --name obl_dev_kubernetes_eastus --location eastus


Run ARM template

biometric:azure michaelobrien$ az group deployment create --resource-group obl_dev_kubernetes_eastus --template-file arm_deploy_obl_dev_kubernetes.json --parameters @arm_deploy_obl_dev_kubernetes_parameters.json

Wait for Rancher/Kubernetes install

The entrypoint.sh script will be run as a cloud-init script on the VM

# on your laptop
biometric:opt michaelobrien$ scp ~/.ssh/onap_rsa ubuntu@1.82.174.1:~/
# on the host
biometric:opt michaelobrien$ ssh ubuntu@1.82.174.1
chmod 400 onap_rsa 
sudo mkdir ~/.ssh
cp onap_rsa ~/.ssh
sudo chown ubuntu:ubuntu ~/.ssh/onap_rsa 
git clone --recurse-submodules https://github.com/obrienlabs/magellan.git
cd magellan/kubernetes/
sudo ./rke_setup.sh -b master -s 1.82.174.1 -e obl -k onap_rsa -l ubuntu

ubuntu@obl-dev-kubernetes:~$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE
ingress-nginx   default-http-backend-5954bd5d8c-28lhq     1/1     Running     0          13m
ingress-nginx   nginx-ingress-controller-t9h5g            1/1     Running     0          13m
kube-system     canal-6zgp7                               2/2     Running     0          14m
kube-system     coredns-autoscaler-5d5d49b8ff-9lgkd       1/1     Running     0          13m
kube-system     coredns-bdffbc666-dvqff                   1/1     Running     0          14m
kube-system     metrics-server-7f6bd4c888-gtgl6           1/1     Running     0          13m
kube-system     rke-coredns-addon-deploy-job-9jwsq        0/1     Completed   0          14m
kube-system     rke-ingress-controller-deploy-job-pzs6h   0/1     Completed   0          13m
kube-system     rke-metrics-addon-deploy-job-hng52        0/1     Completed   0          13m
kube-system     rke-network-plugin-deploy-job-nnh4j       0/1     Completed   0          14m
kube-system     tiller-deploy-7f4d76c4b6-j6rgm            1/1     Running     0          11m
ubuntu@obl-dev-kubernetes:~$ helm version
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}


Training

https://www.microsoft.com/en-us/learning/azure-training.aspx

https://learning.oreilly.com/library/view/exam-ref-az-300/9780135881477/ch02.xhtml

https://www.exitcertified.com/training/microsoft/azure/azure-fundamentals/introduction-to-microsoft-azure-for-it-professionals-51272-detail.html

Certification

https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE2PjDI

Azure SA Expert : https://docs.microsoft.com/en-us/learn/certifications/azure-solutions-architect?wt.mc_id=learningredirect_certs-web-wwl

Outages



linkRoot cause
2003036hhttps://www.zdnet.com/article/microsofts-march-3-azure-east-us-outage-what-went-wrong-or-right/https://status.azure.com/en-us/status/history/




Troubleshooting

AKS CLI behind a VPN

to fix the timeout

PS F:\> az aks install-cli
The command failed with an unexpected error. Here is the traceback:

<urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\urllib\request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))


Links

https://www.microsoft.com/en-us/learning/browse-all-certifications.aspx?jobrole=solutions%20architect


  • No labels

1 Comment

  1. Michael O'Brien

    AKS accounts secured with Azure AD Managed Identity MI
    AKS authentication via AAD as the IDM provider (app level) - OAuth 2.0
    token: access + refresh

    mobile clean OAuth 2.0 auth code flow

    app: single tenant AAD v2A

http://jira.obrienlabs.cloud

https://github.com/obrienlabs

http://eclipsejpa.blogspot.ca/

https://www.linkedin.com/in/michaelobrien-developer/

https://twitter.com/_mikeobrien