Install Tanzu Build Service (TBS v1.0 GA) on Kubernetes – build docker image and store in DockerHub image registry

In this blog I will cover these following scope-

  1. How to install TBS on local KIND/TKGI/TKG and other Kubernetes clusters.
  2. How to auto build portable OCI docker image of a SpringBoot (Java) project using an automated build tool VMware Tanzu Build Service which auto detects Git source code repository commit and inject required dependencies and OS base image based on the source code languages and its configuration files. e.g: application.yaml and Maven’s pom.xml for Java/Spring app.
  3. Push this docker image automatically to Docker Hub image registry. You can use any image registry like onprem Harbor, AWS ECR, GCR, Azure ACR etc.
  4. Test Build Image by downloading from Docker-hub image registry and run using Docker
  5. How to build Dot Net application using TBS (Appendix)
  6. FAQ

Currently, Tanzu Build Service (TBS) ships with the following Buildpacks:

  • Java
  • NodeJS
  • .NET Core (supports Windows DotNet App)
  • Python
  • Golang
  • PHP
  • HTTPD
  • NGINX

Why Tanzu Build Service( TBS)?

  1. Save time to re-build, re-test and re-deploy during patching hundreds of containers.
  2. It auto scans source code configuration and language and inject dependencies into docker image that will be required to compile and/or run the app on container/K8s.
  3. Faster build and patching on hundreds of containers.
  4. Faster developer productivity by setting local build on developer’s machine and sync with source code repo like GitHub etc.
  5. Manage common project dependencies for dev teams and sync all developers code with single git branch to avoid any code conflict/sync issues.
  6. Maintain latest image in image registry.
  7. OCI docker image support, build and run anywhere!

Please refer this official documentation page for more detail

https://docs.pivotal.io/build-service/1-0/

Tanzu Build Service v1.0 GA can be installed on any Kubernetes private and public cloud clusters (v1.14 or later) including local machine using Kubernetes shipped with Docker Desktop, MiniKube and managed K8s like TKGI, TKG, GKE, and AKS clusters etc. 

Build Service Components Tanzu Build Service ships with the following components:

  1. kpack
  2. CNB lifecycle

Prerequisite:

  1. Create Pivnet account. Refer to the official docs for more details on obtaining a Pivotal Network API token. You can create a free account and try on your local machine.
  2. Install Pivnet CLI – https://github.com/pivotal-cf/pivnet-cli/releases/tag/v2.0.1
  3. Install Docker Desktop (Mac, Windows) – Optional if you are trying to install on your local single node K8s cluster.
  4. Docker Hub account
  5. Install Kubernetes. I have used TKGI K8s cluster on GCP, you can also use KIND(K8s in Docker) with Docker Desktop.
  6. Install the TKGI CLI or kubectl CLI
  7. Install these three Carvel CLIs for your operating system. These can be found on their respective Tanzu Network pages:
  • kapp is a deployment tool that allows users to manage Kubernetes resources in bulk.
  • ytt is a templating tool that understands YAML structure.
  • kbld is tool that builds, pushes, and relocates container images.

How to Install & Configure TBS:

You can download TBS from VMware Tanzu Network (formerly the Pivotal Network, or PivNet) or install using Pivnet CLI command. 

Note: I have used Pivnet CLI for all the downloads from VMware PivNet. Refer this official installation guide and advance level configuration:
#Pivnet login using secret token
$ pivnet login --api-token='my-api-token'

$ pivnet download-product-files --product-slug='build-service' --release-version='1.0.2' --product-file-id=773503

#Unarchive the Build Service Bundle file:
$ tar xvf build-service-<version>.tar -C /tmp

#Login to docker-hub. This step will save docker-hub credentials to your K8s cluster. Note: You can use Harbor's url also.
$ docker login index.docker.io

#Login to VMware registry Docker-Hub thru Docker CLI (downloaded with Docker Desktop client)
$ docker login ≈

#Relocate the images for DockerHub with the Carvel tool kbld by running:
# Syntax: kbld relocate -f /tmp/images.lock --lock-output /tmp/images-relocated.lock --repository <IMAGE-REPOSITORY>

$kbld relocate -f /tmp/images.lock --lock-output /tmp/images-relocated.lock --repository itsrajivsrivastava/tanzu-build-service

Connect with your K8s cluster where you want to install TBS:

$ kubectl config use-context <K8s-cluster-name>

Now, install TBS on K8s. You can run these commands from home folder

Then use ytt to push the bundle to the image registry to DockerHub/image registry. It will upload all laungauge buildpacks and other supporting images to Docker-Hub/Harbor.

Note: It will take good time to upload bunch of images. If it fails then you need to re-run the command after deleting failed build of TBS. You can delete “kpack” and “build-service” Kubernetes namespaces.

$ ytt -f /tmp/values.yaml \
    -f /tmp/manifests/ \
    -v docker_repository="<IMAGE-REPOSITORY>" \
    -v docker_username="<REGISTRY-USERNAME>" \
    -v docker_password="<REGISTRY-PASSWORD>" \
    | kbld -f /tmp/images-relocated.lock -f- \
    | kapp deploy -a tanzu-build-service -f- -y

#Example:
ytt -f /tmp/values.yaml \
    -f /tmp/manifests/ \
    -v docker_repository=“itsrajivsrivastava” \
    -v docker_username="itsrajivsrivastava" \
    -v docker_password=‘******’ \
    | kbld -f /tmp/images-relocated.lock -f- \
    | kapp deploy -a tanzu-build-service -f- -y

Where:

  • IMAGE-REPOSITORY is the image repository where Tanzu Build Service images exist.
  • REGISTRY-USERNAME is the username you use to access the registry. gcr.io expects _json_key as the username when using JSON key file authentication.
  • REGISTRY-PASSWORD is the password you use to access the registry

Install KP CLI:

kp CLI, is used for interacting with your Tanzu Build Service (TBS) installation on K8s cluster. Download the kp binary from the Tanzu Build Service page on Tanzu Network.

Import Tanzu Build Service Dependencies

The Tanzu Build Service Dependencies (Stacks, Buildpacks, Builders, etc.) are used to build applications and keep them patched.

  1. Run this command on CLI  “docker login registry.pivotal.io” 
  2. Accept all these EULA agreements online.

Note: Successfully performing a kp import command requires that your Tanzu Network account has access to the images specified in the Dependency Descriptor file. Currently, users can only access these images if they agree to the EULA for each dependency. Users must navigate to each of the dependency product pages in Tanzu Network and accept the EULA highlighted in yellow underneath the Releases dropdown.

Here are the links to each Tanzu Network page in which users must accept the EULA:

  1. Tanzu Build Service Dependencies
  2. Java Buildpack for VMware Tanzu
  3. Java Native Image Buildpack for VMware Tanzu
  4. Node.js Buildpack for VMware Tanzu
  5. Go Buildpack for VMware Tanzu

Note: `kp import` will fail if it cannot access the images in all of the above Tanzu Network pages.

Note: You must be logged in locally to the registry used for `IMAGE-REGISTRY` during relocation and the Tanzu Network registry `registry.pivotal.io`.

These must be imported with the kp cli and the Dependency Descriptor (descriptor-<version>.yaml) file from the Tanzu Build Service Dependencies page:

$ kp import -f /tmp/descriptor-<version>.yaml

Verify kp Installation

List the custom cluster builders available in your installation:

You should see an output that looks as follows:

$  kp clusterbuilder list
NAME       READY    STACK                          IMAGE
base       true     io.buildpacks.stacks.bionic    itsrajivsrivastava/base@sha256:b3062df93d2da25aeff827605790c508570446e53daa8afe05ed2ab4157d1c02
default    true     io.buildpacks.stacks.bionic    itsrajivsrivastava/default@sha256:f16ed5de160ca9c13a0080d67280d0b2b843c926595c4d171568d75f96479247
full       true     io.buildpacks.stacks.bionic    itsrajivsrivastava/full@sha256:f16ed5de160ca9c13a0080d67280d0b2b843c926595c4d171568d75f96479247
tiny       true     io.paketo.stacks.tiny          itsrajivsrivastava/tiny@sha256:abc4879c03512a072623a7dcb18621d68122b5e608b452f411d8bc552386b8c5

# List the custom cluster builders available in your installation:

$ kp clusterstack list
NAME       READY    ID
base       True     io.buildpacks.stacks.bionic
default    True     io.buildpacks.stacks.bionic
full       True     io.buildpacks.stacks.bionic
tiny       True     io.paketo.stacks.tiny

Create Git and Image registry secrets in your K8s cluster:

# Docker Hub secret
$ kp secret create docker-creds --dockerhub itsrajivsrivastava -n tbs-demo
  dockerhub password:
  "docker-creds" created

# GitHub Secret
$ kp secret create github-creds --git https://github.com --git-user rajivmca2004 -n tbs-demo

#Verify and list secret
$ kp secret list -n tbs-demo

NAME                   TARGET
default-token-fqwvj
docker-creds           https://index.docker.io/v1/
github-creds           https://github.com

#Delete secret
$ kp secret delete <SECRET-NAME> -n tbs-demo

Create and manage docker image using kp CLI commands:

# Create TBS image: (--tag is mandatory)

$ kp image create <name> \
  --tag <tag> \
  [--builder <builder> or --cluster-builder <cluster-builder>] \
  --namespace <namespace> \
  --env <env> \
  --wait \
  --git <git-repo> \
  --git-revision <git-revision>

#Syntax:

$ kp image create spring-petclinic \
  --tag index.docker.io/itsrajivsrivastava/spring-petclinic:latest \
  --namespace tbs-demo \
  --git https://github.com/rajivmca2004/spring-petclinic \
  --git-revision master
"spring-petclinic" created

#  Verify image status
$ kp image status spring-petclinic -n tbs-demo
Status:         Ready
Message:        --
LatestImage:    index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:4e712dec26810f026357281be4ba49ff8da3b45698700fd5dc470b7914c0d13d

Last Successful Build
Id:        1
Reason:    CONFIG

Last Failed Build
Id:        --
Reason:    --

# List the project(s):
  
$ kp image list -n tbs-demo
NAME                READY    LATEST IMAGE
spring-petclinic    True     index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:d96fdd60633cd5582a9c28edceff80762ee79ef2985cd18f518bc1503563b7ef

# To check image list of any specific project 

$ kp image list spring-petclinic -n tbs-demo
NAME                READY    LATEST IMAGE
spring-petclinic    True     index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:d96fdd60633cd5582a9c28edceff80762ee79ef2985cd18f518bc1503563b7ef

Build docker images:

Here, TBS will sync up with two systems:
1. GitHub – TBS will be in sync with GitHub repo for any commit and triggered after every commit. It also builds docker images.
2. Image Registry/Docker Hub – TBS will automatically push this image which has been created in the last step to image registry Docker-Hub and keep it refreshed all the time for developers and CI/CD build and deployment to K8s containers.
There are two ways to build the image:

  1. Auto Build
  2. Manual Build

Note: If you want to build source code directory which is inside sub-directories or if you have a parent project repo and multiple child project inside this

--sub-path DotNetBuild

#Example:
$ kp image create dotnetbuildtest index.docker.io/itsrajivsrivastava/dotnetbuildtest \
  --sub-path DotNetBuild \
  --namespace tbs-dotnet-demo \
  --git https://github.com/rajivmca2004/DotNetBuild.git \
  --git-revision master

1. Auto Build:

Note: First build will be triggered automatically when you create image at the first time.

To check auto build, just make some code changes in your Git branch and check the build progress using this command, It takes a few seconds to trigger. You can watch build logs locally –

# Use, this command, to check build status, also can be used for the first time to build. It will also show build revisions

$ kp build status spring-petclinic -n tbs-demo
Image:            index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:e09bc84c0287d8d0985244876a68ae4b3963a96707622d81f6d9b9efa581be92
Status:           SUCCESS
Build Reasons:    COMMIT

Pod Name:    spring-petclinic-build-2-5csx8-build-pod

Builder:      itsrajivsrivastava/default@sha256:f16ed5de160ca9c13a0080d67280d0b2b843c926595c4d171568d75f96479247
Run Image:    index.docker.io/itsrajivsrivastava/run@sha256:ca460a285b00d8f25ca3734b8f783af240771eb10974d90e26501fd52c0271b8

Source:      GitUrl
Url:         https://github.com/rajivmca2004/spring-petclinic
Revision:    c5b4f7f717a1dc239c002c993c178f75283a7751

BUILDPACK ID                           BUILDPACK VERSION
paketo-buildpacks/bellsoft-liberica    4.0.0
paketo-buildpacks/maven                3.1.1
paketo-buildpacks/executable-jar       3.1.1
paketo-buildpacks/apache-tomcat        2.3.0
paketo-buildpacks/dist-zip             2.2.0
paketo-buildpacks/spring-boot          3.2.1

$ kp build list spring-petclinic -n tbs-demo

BUILD    STATUS     IMAGE                                                                                                                          STARTED                FINISHED               REASON
1        SUCCESS    index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:b6d6a0944600be3552c0b33e0a0759a12b422168484ffff55f9f9e0be4c93217    2020-10-10 00:39:10    2020-10-10 00:59:40    CONFIG
2        SUCCESS    index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:e09bc84c0287d8d0985244876a68ae4b3963a96707622d81f6d9b9efa581be92    2020-10-10 01:07:35    2020-10-10 01:09:48    COMMIT

2. Manual Build:

$ kp image trigger spring-petclinic -n tbs-demo

Check Build Logs:

# To view running logs for a build:

 $ kp build logs spring-petclinic -n tbs-demo

#Check logs for the given release:

 $ kp build logs spring-petclinic --build 1 -n tbs-demo

Note: First build will take some time to download all the dependent libraries. Subsequent build will be super fast!

Applied source code related build packages are mentioned in this following build logs. Since, its a Java app, these build-packs have been applied automatically –

[INFO] Building jar: /workspace/target/spring-petclinic-2.3.0.BUILD-SNAPSHOT.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.3.0.RELEASE:repackage (repackage) @ spring-petclinic ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  17:44 min
[INFO] Finished at: 2020-10-09T19:28:07Z
[INFO] ------------------------------------------------------------------------
  Removing source code

Paketo Executable JAR Buildpack 3.1.1
  https://github.com/paketo-buildpacks/executable-jar
  Process types:
    executable-jar: java org.springframework.boot.loader.JarLauncher
    task:           java org.springframework.boot.loader.JarLauncher
    web:            java org.springframework.boot.loader.JarLauncher

Paketo Spring Boot Buildpack 3.2.1
  https://github.com/paketo-buildpacks/spring-boot
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
    Writing profile.d/helper
  Web Application Type: Contributing to layer
    Servlet web application detected
    Writing env.launch/BPL_JVM_THREAD_COUNT.default
  Spring Cloud Bindings 1.6.0: Contributing to layer
    Reusing cached download from buildpack
    Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
  Image labels:
    org.opencontainers.image.title
    org.opencontainers.image.version
    org.springframework.boot.spring-configuration-metadata.json
    org.springframework.boot.version
===> EXPORT
Reusing layers from image 'index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:129f1e4231095c7504e01a4f233487b056eb28975b93f4dbb6195534bee4220e'
Adding layer 'paketo-buildpacks/bellsoft-liberica:helper'
Adding layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
Adding layer 'paketo-buildpacks/bellsoft-liberica:jre'
Adding layer 'paketo-buildpacks/bellsoft-liberica:jvmkill'
Reusing layer 'paketo-buildpacks/executable-jar:class-path'
Adding layer 'paketo-buildpacks/spring-boot:helper'
Adding layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Adding layer 'paketo-buildpacks/spring-boot:web-application-type'
Adding 1/1 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.opencontainers.image.title'
Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.spring-configuration-metadata.json'
Adding label 'org.springframework.boot.version'
*** Images (sha256:b6d6a0944600be3552c0b33e0a0759a12b422168484ffff55f9f9e0be4c93217):
      index.docker.io/itsrajivsrivastava/spring-petclinic:latest
      index.docker.io/itsrajivsrivastava/spring-petclinic:b1.20201009.190910
Adding cache layer 'paketo-buildpacks/bellsoft-liberica:jdk'
Adding cache layer 'paketo-buildpacks/maven:application'
Adding cache layer 'paketo-buildpacks/maven:cache'
===> COMPLETION
Build successful

Test Build Image

Pull image from Docker-hub:

docker pull itsrajivsrivastava/spring-petclinic

Run this pulled image on docker:

docker run -p 8080:8080 itsrajivsrivastava/spring-petclinic

Now , test on this browser: http://localhost:8080

How to build Dot Net application using TBS

TBS installation, setup yaml configuration, Github and DockerHub secret are same for Dot NET also.Deployment process is also same as Java on docker. There is a separate DotNet buildpack which will be injected to ASP.Net source code project.

Now, we will create TBS Dot Net image:

$ kp image create dotnetbuildtest \
  --tag index.docker.io/itsrajivsrivastava/dotnetbuildtest:latest \
  --namespace tbs-demo \
  --git https://github.com/rajivmca2004/DotNetBuild \
  --git-revision master

#Verify
$ kp image status dotnetbuildtest -n tbs-demo

If you face this in .Net apps:

Unable to start Kestrel.
System.Net.Sockets.SocketException (13): Permission denied
at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress

kestrel will bind to both ipv4 & ipv6. maybe  k8s env doesn’t let it bind to ipv6. Change env port 8080 to force it to only bind to ipv4

Fix– Add this in K8s deployment manifests.

Add this in Ku8s deployment manifest file:
         env:
         – name: PORT
           value: “8080”

FAQ

QuestionsAnswers
Where exactly build happens on local machine or on K8s cluster Builds happen in pods on the cluster. Basically everything with the service happens in the K8s cluster. You can interact with the service with the kp cli (now kp cli)
Where it stores all dependent libraries K8s or machine from where kp build starts?
App dependent libraries live in the Stack/buildpacks which would be on the k8s cluster.
Does it keep an image copy locallyIt does not keep a copy of the app image, it only uploads to a registry
Can we cover all CI job for build configuration pipeline and deploy with CD tools/pluginTBS is meant to be a solution that works well in a CI/CD setting. Currently have an integration with concourse CI via https://github.com/pivotal/concourse-kpack-resource
It can also integrated with Jenkin with additional configuration
Can build-pack modified or created new custom buildYes, follow this – https://buildpacks.io/docs/operator-guide/create-a-builder/

Published by

Rajiv Srivastava

Principal Architect with Wells Fargo

6 thoughts on “Install Tanzu Build Service (TBS v1.0 GA) on Kubernetes – build docker image and store in DockerHub image registry”

  1. follow your steps on minikube, hit the issue on Verify kp Installation.
    ###########
    kp clusterbuilder list
    Error: no clusterbuilders found
    ##############

    Liked by 1 person

    Reply

  2. also hit the issue
    kp import -f descriptor-100.0.23.yaml
    Importing Cluster Store ‘default’…
    Uploading ‘index.docker.io/284946040/build-service/tanzu-buildpacks_go@sha256:8b0664b7644a2dca2cd7dd318f356eec4738bbe2ded20b18730d262504f22acd’
    Error: invalid credentials, ensure registry credentials for ‘index.docker.io/284946040/build-service/tanzu-buildpacks_go’ are available locally
    ✘ yanglu@yanglu-a01  ~/Downloads/minikube/tbs  kp clusterbuilder list
    Error: no clusterbuilders found

    Like

    Reply

  3. Hello Rajeev, we’re trying to install TBS on AWS EKS with ECR. Getting below error. are we missing anything?
    While executing below ytt command, I got below error:
    : waiting on reconcile deployment/controller-manager (apps/v1) namespace: stacks-operator-system:
    Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet “controller-manager-66df884b46” has timed out progressing.))

    Like

    Reply

  4. Hello Rajiv, getting below are while installing TBS on EKS with ECR registry, with yet command:
    Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message:Replicaset “controller-manager-xxxx. Has timed out progressing.))

    Like

    Reply

Leave a comment