Deploying containerized apps to a Azure Kubernetes Service (AKS) cluster using the default settings in Azure isn’t that much tough as the tools handle most of the hard work for us. I recently discovered a scenario where I needed to connect my AKS cluster back to the on-prem resources. So I went ahead and created a AKS cluster via the Azure portal. In the creation step, I chose Azure VNet and this automatically set my cluster to use Azure Networking Interface (CNI).
This is what’s documented in the Microsoft docs:
With Azure Container Networking Interface (CNI), every pod gets an IP address from the subnet and can be accessed directly. These IP addresses must be unique across your network space, and must be planned in advance. Each node has a configuration parameter for the maximum number of pods that it supports. The equivalent number of IP addresses per node are then reserved up front for that node. This approach requires more planning, and often leads to IP address exhaustion or the need to rebuild clusters in a larger subnet as your application demands grow.
This isn’t what I want as my cluster will join the on-prem resources. Doing so is a waste of IP addresses as I only have a mouthful of pods to run. When it comes to routing the traffic, CNI makes it easier for us such as using ingresses but this requires more IPs. I’ll write another post regarding this approach using this network type shortly . For now, we just want to configure our cluster to use kubenet instead, using an automation tool called Ansible.
If you are interested in deploying to a local cluster, please refer to my other blog talking about Minikube.
Just a brief overview of the two types of networking in AKS, I won’t dive into it as this is quite a complicated topic so you may want to do a bit of research on these.
Basic networking (kubenet)
Advanced networking (ANI)
Prerequisites
- Azure subscription
- Azure service principal
- Ansible installed on a Linux computer
Generate a SSH key
You need a SSH key to establish a connection between your computer and Azure:
ssh-keygen -t rsa -C "your_email@example.com"If you hit enter all the way, then you should see a folder being created:
cd ~/.sshMake a playbook folder, you need this to store all of your yml files required later:
mkdir playbook
cd playbookGet your Azure VNet and its Subnet Id
Since we are deploying our containerized apps to AKS on an existing Azure VNet, we need its Id to associate the AKS cluster with existing Azure Virtual Network, later in this guide. An easy way to do is to use Azure CLI:
az network vnet show --resource-group your_resourcegroup --name your_vnetname --query id -o tsv
/subscriptions/4e914e0c-d188-437d-8f8d-01c09781ea71/resourceGroups/your_resourcegroup/providers/Microsoft.Network/virtualNetworks/your_vnetname
az network vnet subnet show --resource-group your_resourcegroup --vnet-name your_vnetname --name your_aks_subnet --query id -o tsv
/subscriptions/4e914e0c-d088-457d-8f8d-01c09781ea71/resourceGroups/your_resourcegroup/providers/Microsoft.Network/virtualNetworks/DTGVNet/subnets/your_aks_subnet
Create an Azure Kubernetes cluster
Create a yml file within your playbook folder, aks.yml:
- name: List supported kubernetes version from Azure
azure_rm_aksversion_info:
location: "{{ location }}"
register: versions
- name: Create AKS cluster with vnet
azure_rm_aks:
resource_group: "{{ resource_group }}"
name: "{{ name }}"
dns_prefix: "{{ name }}"
kubernetes_version: "{{ versions.azure_aks_versions[-1] }}"
agent_pool_profiles:
- count: 1
name: nodepool1
vm_size: Standard_D2_v2
vnet_subnet_id: "{{ vnet_subnet_id }}"
linux_profile:
admin_username: gpham
ssh_key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
service_principal:
client_id: '1e2726f1-3b55-4813-af83-112c3be598ce'
client_secret: '2de00b61-c661-43b2-84bb-16b6bfd5feb5'
network_profile:
network_plugin: kubenet
pod_cidr: 192.168.4.0/24
docker_bridge_cidr: 172.17.0.1/16
dns_service_ip: 192.168.3.10
service_cidr: 192.168.3.0/24
register: aksA couple things to note:
vnet_subnet_idis the id we retrieved previously.service_principalis where we place our service principle credentials. If you don’t how to get this information, please check this link. This is the account AKS uses to create the required Kubernetes resources under this user. Basically, it’s just an identity, just like when we register our apps via the App Registration portal.linux_profileis where we create an user so we can login to our AKS cluster via its UI dashboard. It basically defines a way to ssh to our cluster node. A default Linux node is required for every Kubernetes cluster. In case of adding a windows node, this can change towindows_profileinstead.network_profileis where we define our AKS networking. Note that we are using kubenet here. This is because we don’t want to assign our VNet IP to every single pod within our cluster. If you want to learn more about the different types of networking in Azure Kubernetes Service, please check this out for the kubenet type, and this one for Azure-CNI.
Associate the cluster with your existing Azure VNet
Add another yml file within your playbook, associate.yml:
- name: Get route table
azure_rm_routetable_facts:
resource_group: "{{ node_resource_group }}"
register: routetable
- name: Get network security group
azure_rm_securitygroup_facts:
resource_group: "{{ node_resource_group }}"
register: nsg
- name: Parse subnet id
set_fact:
subnet_name: "{{ vnet_subnet_id | regex_search(subnet_regex, '\\1') }}"
subnet_rg: "{{ vnet_subnet_id | regex_search(rg_regex, '\\1') }}"
subnet_vn: "{{ vnet_subnet_id | regex_search(vn_regex, '\\1') }}"
vars:
subnet_regex: '/subnets/(.+)'
rg_regex: '/resourceGroups/(.+?)/'
vn_regex: '/virtualNetworks/(.+?)/'
- name: Associate network resources with the node subnet
azure_rm_subnet:
name: "{{ subnet_name[0] }}"
resource_group: "{{ subnet_rg[0] }}"
virtual_network_name: "{{ subnet_vn[0] }}"
security_group: "{{ nsg.ansible_facts.azure_securitygroups[0].id }}"
route_table: "{{ routetable.route_tables[0].id }}"Run the complete playbook
Now we have our playbooks defined, let’s add a complete playbook so we can create the cluster, we name it as aks_complete.yml:
---
- hosts: localhost
vars:
resource_group: your_resourcegroup
name: your-kube-cluster
location: australiaeast
tasks:
- name: Ensure resource group exist
azure_rm_resourcegroup:
name: "{{ resource_group }}"
location: "{{ location }}"
- name: Create AKS
vars:
vnet_subnet_id: "/subscriptions/4e914e0c-d088-457d-8f8d-01c09781ea71/resourceGroups/DIT_VNet/providers/Microsoft.Network/virtualNetworks/DTGVNet/subnets/aks-subnet"
include_tasks: aks.yml
- name: Associate network resources with the node subnet
vars:
vnet_subnet_id: "/subscriptions/4e914e0c-d088-457d-8f8d-01c09781ea71/resourceGroups/DIT_VNet/providers/Microsoft.Network/virtualNetworks/DTGVNet/subnets/aks-subnet"
node_resource_group: "{{ aks.node_resource_group }}"
include_tasks: associate.yml
- name: Get details of the AKS
azure_rm_aks_facts:
name: "{{ name }}"
resource_group: "{{ resource_group }}"
show_kubeconfig: user
register: output
- name: Show AKS cluster detail
debug:
var: output.aks[0]Execute the playbook:
ansible-playbook aks_complete.ymlAnd here is our final result:
PLAY [localhost] ********************************************************************************************************************
TASK [Gathering Facts] **************************************************************************************************************
ok: [localhost]
TASK [Ensure resource group exist] **************************************************************************************************
ok: [localhost]
TASK [Create AKS] *******************************************************************************************************************
included: /home/gpham/aks_playbook/aks.yml for localhost
TASK [List supported kubernetes version from Azure] *********************************************************************************
[WARNING]: Azure API profile latest does not define an entry for ContainerServiceClient
ok: [localhost]
TASK [Create AKS cluster with vnet] *************************************************************************************************
changed: [localhost]
TASK [Associate network resources with the node subnet] *****************************************************************************
included: /home/gpham/aks_playbook/associate.yml for localhost
TASK [Get route table] **************************************************************************************************************
[DEPRECATION WARNING]: The 'azure_rm_routetable_facts' module has been renamed to 'azure_rm_routetable_info'. This feature will be
removed in version 2.13. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [localhost]
TASK [Get network security group] ***************************************************************************************************
[DEPRECATION WARNING]: The 'azure_rm_securitygroup_facts' module has been renamed to 'azure_rm_securitygroup_info'. This feature
will be removed in version 2.13. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [localhost]
TASK [Parse subnet id] **************************************************************************************************************
ok: [localhost]
TASK [Associate network resources with the node subnet] *****************************************************************************
changed: [localhost]
TASK [Get details of the AKS] *******************************************************************************************************
[DEPRECATION WARNING]: The 'azure_rm_aks_facts' module has been renamed to 'azure_rm_aks_info'. This feature will be removed in
version 2.13. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [localhost]
TASK [Show AKS cluster detail] ******************************************************************************************************
ok: [localhost] => {
"output.aks[0]": {
"id": "/subscriptions/4e914e0c-d088-457d-8f8d-01c09781ea41/resourcegroups/your_resoucegroup/providers/Microsoft.ContainerService/managedClusters/your-kube-cluster",
"kube_config": "apiVersion: v1\nclusters:\n- cluster:\n certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0t .... 27d2159b63f56164b346d49\n",
"location": "australiaeast",
"name": "your-kube-cluster",
"properties": {
"agentPoolProfiles": [
{
"count": 1,
"maxPods": 110,
"name": "nodepool1",
"osDiskSizeGB": 100,
"osType": "Linux",
"storageProfile": "ManagedDisks",
"vmSize": "Standard_D2_v2",
"vnetSubnetID": "/subscriptions/4e914e0c-d088-457d-8f8d-01c09783ea71/resourceGroups/your_resourcegroup/providers/Microsoft.Network/virtualNetworks/YourVNet/subnets/aks-subnet"
}
],
"dnsPrefix": "your-kube-cluster",
"enableRBAC": false,
"fqdn": "your-kube-cluster-e34f51e3.hcp.australiaeast.azmk8s.io",
"kubernetesVersion": "1.16.7",
"linuxProfile": {
"adminUsername": "gpham",
"ssh": {
"publicKeys": [
{
"keyData": "ssh-rsa AAAAB3NzaC .... A5BGu1 your_email@example.com"
}
]
}
},
"networkProfile": {
"dnsServiceIP": "192.168.3.10",
"dockerBridgeCidr": "172.17.0.1/16",
"networkPlugin": "kubenet",
"podCidr": "192.168.4.0/24",
"serviceCidr": "192.168.3.0/24"
},
"nodeResourceGroup": "MC_your_resourcegroup_your-kube-cluster_australiaeast",
"provisioningState": "Succeeded",
"servicePrincipalProfile": {
"clientId": "1e2726f1-3b55-2213-af43-912c5be598ce"
}
},
"type": "Microsoft.ContainerService/ManagedClusters"
}
}
PLAY RECAP **************************************************************************************************************************
localhost : ok=12 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Hi there, love you post. I’m trying something similar but for some reason the “vnet_subnet_id” variable is not set for the “agent_pool_profiles” which creates AKS environment without a subnet and cannot be altered.
Did you run into something similar?
It’s unlikely you can alter the cluster’s networking after it’s being created since the cluster routing is based on the subnet is was assigned. Changes are your cluster may have been created with a new virtual network. A couple of things you can try is to make sure that the service principle has the right to modify the subnet, in this case adding the cluster’s ip to the subnet.