Tanzu Kubernetes Grid air gapped installation (TKG v1.1.2) on vSphere v6.7- offline environment

Recently,I have done a POC for a client with the latest TKG v1.1.2 ( Latest updated: July’20) and faced a couple of challenges on air- gapped environment. Installing and configuring TKG management and worker clusters on air-gapped (No Internet/offline) environment is nightmare. You need to plan properly and download all required docker images of TKG and related technology stacks and libraries on your private image registry first. I have used Harbor open source image registry in this blog.

This blog is not replacement of official doc. It’s quick references to join all the dots, tips like how to manually download, tag, push and change K8s manifest files images, prerequisite and other quick references to save time and have everything on a single pager.

I have followed this deploying TKG instructions on an air-gapped environment (Deploy Tanzu Kubernetes Grid to vSphere in an Air-Gapped Environment), there are some more steps required to complete successful installation. This blog will cover TKG v1.1.2 on vSphere 6.7 in air gapped environment.

Note: I have used latest Ubuntu v20.04 LTS on bootstrap VM which will have Internet connectivity. You can sue CentOS are any other Linux flavour.

Note: I have used latest Ubuntu v20.04 LTS on bootstrap VM which will have Internet connectivity. You can use CentOS are any other Linux flavour.

I have used TKG dev plan:

Prerequisite for Bootstrap Env – Ubuntu/CentOS LinuxPackages/URLs
DHCP Should be Enabledhttps://www.tecmint.com/install-dhcp-server-in-ubuntu-debian/Mandatory: DHCP- DHCP installation on Ubuntu
DNS Enabled Mandatory- Public or private DNS muste be enabled on subnet IP tange 
Ubuntu OS Core server installhttps://ubuntu.com/download/alternative-downloadsLatest version 20.04 LTS/ 18.04 LTS
HomeBrew ( if not available)Linux/MAcOS-https://docs.brew.sh/Homebrew-on-Linux
Ubuntu- https://brew.sh/
Ubuntu- https://medium.com/@smartsplash/using-homebrew-on-ubuntu-1089f70c8aa7
Optional – It’s good to install CLIs and other required libaries. Node advisable for air-gapped env installation on K8s clusters.
TKG CLIhttps://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.1/vmware-tanzu-kubernetes-grid-11/GUID-install-tkg-set-up-tkg.htmlMandatory 
Kuebctlhttps://v1-17.docs.kubernetes.io/docs/tasks/tools/install-kubectl/

Ref docs:
https://docs.docker.com/engine/install/ubuntu/
Mandatory – For K8s
$ brew install kubectl
$ kubectl version
Docker Desktop and CLI Installation and Setup: (Ubuntu)Ubuntu:

https://gist.github.com/rstacruz/297fc799f094f55d062b982f7dac9e41
Mandatory : Ubuntu- docker.io is available from the Ubuntu repositories (as of Xenial).
# Install Docker
sudo apt install docker.io
sudo apt install docker-compose

# Start /stop
sudo systemctl start docker
sudo systemctl stop docker

#Verify
sudo docker ps -a
sudo docker rm -f <PID>
docker info
Harborhttps://goharbor.io/docs/1.10/install-config/Mandatory- Need to setup DNS servers for Harbor to resolve domaion name.  
Follow TKG v1.1.2 installation steps on air gapped env for vSphere v6.7https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.1/vmware-tanzu-kubernetes-grid-11/GUID-install-tkg-vsphere.html 
   
Download TKG binaries:  Linuxhttps://my.vmware.com/web/vmware/details?downloadGroup=TKG-110&productId=988&rPId=46507 
VMware Tanzu Kubernetes Grid 1.1.0 Kubernetes v1.18.2 OVAPhoton v3 Kubernetes 1.18.2 OVA 
VMware Tanzu Kubernetes Grid 1.1 CLIVMware Tanzu Kubernetes Grid CLI 1.1 Linux 
VMware Tanzu Kubernetes Grid 1.1 Load Balancer OVA Photon v3 capv haproxy v1.2.4 OVA 
clusterawsadm Account Preparation Tool v0.5.3ClusterAdmin AWS v0.5.3 Linux 
VMware Tanzu Kubernetes Grid 1.1 Extension manifestsVMware Tanzu Kubernetes Grid Extensions Manifest 1.1 
Crash Diagnostics v0.2.2Crash Recovery and Diagnostics for Kubernetes 0.2.2 Linux 

Step-1: Setup all prerequisite, install Ubuntu OS on bootstrap VM

Step:2 Download all binaries with your VMware credentials and push/copy all compressed tar files to bootstrap VM machine.

Step:3 Make sure Internet is available on the bootstrap VM from where you need to initiate installation of TKG and other binaries.

Step:4 Install Docker Desktop and CLI. Make sure that the internet-connected machine has Docker installed and running.

Step:5 Install Harbor and create certificate using OpenSSL and https config. Also, add harbor certificates in docker config file in .harbor/harbor.yml

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /root/harbor/data/cert/harbor.vmwaredc.com.crt
  private_key: /root/harbor/data/cert/harbor.vmwaredc.com.key
#edit /etc/hosts and add namespace entry of DNS server
10.109.19.13 harbor.vmwaredc.com

$ systemd-resolve --status


UI: https://harbor.vmwaredc.com
Verify: Make sure that you can connect to the private registry from the internet-connected machine.

$ docker login -u admin -p <password> harbor.vmwaredc.com
/library

Step:6 Install kubectl CLI

Step:7 Install tkg CLI on same bootstrap VM with external internet connection, and follow the instructions in Download and Install the Tanzu Kubernetes Grid CLI to download, unpack, and install the Tanzu Kubernetes Grid CLI binary on your internet-connected system.

Step:8 Follow all the steps as mentioned in the installation doc. Open vSphere UI console and provide all vCenter v6.7 server details, vLAN, resource configuration etc. It will create configuration file config.yaml in .tkg folder which will have all main TKG installation configuration.

Note: vCenter server should be an IP or FQDN in only small letters.

Step:9 Upload TKG and HAProxy OVA to vSphere UI console.

Step:10 Add this export before initiating TKG installation –

$ export TKG_CUSTOM_IMAGE_REPOSITORY="harbor.vmwaredc.com/library"

Step:11 Download all required docker images for TKG installation and push to Harbor and follow these steps –

Note:

Note: TKG Repo pull all docker images from public image registry- https://registry.tkg.vmware.run/v2/

  • On the bootstrap machine with an internet connection on which you have performed the initial setup tasks and installed the Tanzu Kubernetes Grid CLI, install yq 2.x. NOTE: You must use yq version 2.x. Version 3.x does not work with this script.
  • Run the $ tkg get management-cluster command.
  • Running a tkg command for the first time installs the necessary Tanzu Kubernetes
  • Grid configuration files in the ~/.tkg folder on your system. The script that you create and run in subsequent steps requires the files in the ~/.tkg/bom folder to be present on your machine. Note: TKG v1.1.2 picks bom/bom-1.1.2+vmware.1.yaml image file.
  • Set the IP address or FQDN of your local registry as an environment variable.In the following command example, replace custom-image-repository.io with the address of your private Docker registry.
  • Copy and paste the following script in a text editor, and save it as gen-publish-images.sh
#!/usr/bin/env bash
# Copyright 2020 The TKG Contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
BOM_DIR=${HOME}/.tkg/bom
if [ -z "$TKG_CUSTOM_IMAGE_REPOSITORY" ]; then
    echo "TKG_CUSTOM_IMAGE_REPOSITORY variable is not defined"
    exit 1
fi
for TKG_BOM_FILE in "$BOM_DIR"/*.yaml; do
    # Get actual image repository from BoM file
    actualImageRepository=$(yq .imageConfig.imageRepository "$TKG_BOM_FILE" | tr -d '"')
    # Iterate through BoM file to create the complete Image name
    # and then pull, retag and push image to custom registry
    yq .images "$TKG_BOM_FILE" | jq -c '.[]' | while read -r i; do
        # Get imagePath and imageTag
        imagePath=$(jq .imagePath <<<"$i" | tr -d '"')
        imageTag=$(jq .tag <<<"$i" | tr -d '"')
        # create complete image names
        actualImage=$actualImageRepository/$imagePath:$imageTag
        customImage=$TKG_CUSTOM_IMAGE_REPOSITORY/$imagePath:$imageTag
        echo "docker pull $actualImage"
        echo "docker tag $actualImage $customImage"
        echo "docker push $customImage"
        echo ""
    done
done 
  • Make the script executable .chmod +x gen-publish-images.sh
  • Generate a new version of the script that is populated with the address of your private Docker registry ../gen-publish-images.sh > publish-images.sh
  • Verify that the generated version of the script contains the correct registry address cat publish-images.sh
  • Make the script executable .chmod +x publish-images.sh
  • Log in to your local private registry. docker login ${TKG_CUSTOM_IMAGE_REPOSITORY}
  • Run the script to pull the required images from the public Tanzu Kubernetes Grid registry, retag them, and push them to your private registry ../publish-images.sh
  • When the script finishes, turn off your internet connection. (Optional) After that Internet is not required for TKG.
  • Modify TKG dev installation plan. Run these following commands on the home directory one level up (outside of .tkg folder location) :
$ export REGISTRY="harbor.vmwaredc.com"
$ export NAMESERVER="10.109.19.5"
$ export DOMAIN="vmwaredc.com"
$ cat > /tmp/harbor.sh <<EOF
echo "nameserver $NAMESERVER" > /usr/lib/systemd/resolv.conf
echo "domain $DOMAIN" >> /usr/lib/systemd/resolv.conf
rm /etc/resolv.conf
ln -s /usr/lib/systemd/resolv.conf /etc/resolv.conf
mkdir -p /etc/containerd
echo "" > /etc/containerd/config.toml
sed -i '1 i\# Use config version 2 to enable new configuration fields.' /etc/containerd/config.toml
sed -i '2 i\# Config file is parsed as version 1 by default.' /etc/containerd/config.toml
sed -i '3 i\version = 2' /etc/containerd/config.toml
sed -i '4 i\ ' /etc/containerd/config.toml
sed -i '5 i\[plugins]' /etc/containerd/config.toml
sed -i '6 i\  [plugins."io.containerd.grpc.v1.cri"]' /etc/containerd/config.toml
sed -i '7 i\    sandbox_image = "registry.tkg.vmware.run/pause:3.2"' /etc/containerd/config.toml
sed -i '8 i\    [plugins."io.containerd.grpc.v1.cri".registry]' /etc/containerd/config.toml
sed -i '9 i\      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]' /etc/containerd/config.toml
sed -i '10 i\        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."$REGISTRY"]' /etc/containerd/config.toml
sed -i '11 i\          endpoint = ["https://$REGISTRY"]' /etc/containerd/config.toml
sed -i '12 i\      [plugins."io.containerd.grpc.v1.cri".registry.configs]' /etc/containerd/config.toml
sed -i '13 i\        [plugins."io.containerd.grpc.v1.cri".registry.configs."$REGISTRY"]' /etc/containerd/config.toml
sed -i '14 i\          [plugins."io.containerd.grpc.v1.cri".registry.configs."$REGISTRY".tls]' /etc/containerd/config.toml
sed -i '15 i\            insecure_skip_verify = true' /etc/containerd/config.toml
systemctl restart containerd
EOF
 
$ awk '{print "    -", $0}' /tmp/harbor.sh > /tmp/harbor1.yaml
$ awk '{print "      -", $0}' /tmp/harbor.sh > /tmp/harbor2.yaml
$ sed -i '197 e cat /tmp/harbor1.yaml\n' ~/.tkg/providers/infrastructure-vsphere/v0.6.5/cluster-template-dev.yaml
$ sed -i '249 e cat /tmp/harbor2.yaml\n' ~/.tkg/providers/infrastructure-vsphere/v0.6.5/cluster-template-dev.yaml
 
$ rm /tmp/harbor1.yaml /tmp/harbor2.yaml /tmp/harbor.sh

Step:12 Run this on terminal to initiate installation process, it will create .tkg folder and required config file. In v1.1.2 .bom folder has all image repositiories

$ sudo tkg init --ui -v 6

Step:13  As soon as kind container is up . Run this exec steps into KIND cluster and ran below script.

Identify KIND docker image by :

$ docker ps -a
$ docker exec -it <KIND docker image id> /bin/sh
echo '# explicitly use v2 config format
version = 2
# set default runtime handler to v2, which has a per-pod shim
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "registry.tkg.vmware.run/pause:3.2"
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
        runtime_type = "io.containerd.runc.v2"
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.vmwaredc.com"]
          endpoint = ["https://harbor.vmwaredc.com"]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.vmwaredc.com"]
          [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.vmwaredc.com".tls]
            insecure_skip_verify = true' > /etc/containerd/config.toml

Step:14 At this step, management cluster is created. Now, you can create work load clusters as per installation instructions.

Step:15 To visualize, monitor and inspect TKG Kubernetes clusters. Install Octant UI dashboard. Octant should immediately launch your default web browser on http://127.0.0.1:7777/#/cluster-overview

$ octant

Note: Or to run it on a specific host and fixed port:

OCTANT_LISTENER_ADDR=0.0.0.0:8900 octant

 

Important Trick:

Pull and push docker images in Air gapped environment

Now, your K8s cluster is ready, next you would like to install K8s deployment or any other K8s images which pulls dependent images from public Internet. Your Kubernetes cluster running on air-gapped environment can’t download any image from public repository (dockerhub, docker.io, gcr etc).

Refer my short blog for how to do operation: Pull and push docker images in Air gapped (No Internet) environment

Published by

Rajiv Srivastava

Cloud Native Architect with VMware

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s