In this blog I will cover these following scope-
- How to install TBS on local KIND/TKGI/TKG and other Kubernetes clusters.
- How to auto build portable OCI docker image of a SpringBoot (Java) project using an automated build tool VMware Tanzu Build Service which auto detects Git source code repository commit and inject required dependencies and OS base image based on the source code languages and its configuration files. e.g: application.yaml and Maven’s pom.xml for Java/Spring app.
- Push this docker image automatically to Docker Hub image registry. You can use any image registry like onprem Harbor, AWS ECR, GCR, Azure ACR etc.
- Test Build Image by downloading from Docker-hub image registry and run using Docker
- How to build Dot Net application using TBS (Appendix)
- FAQ
Currently, Tanzu Build Service (TBS) ships with the following Buildpacks:
- Java
- NodeJS
- .NET Core (supports Windows DotNet App)
- Python
- Golang
- PHP
- HTTPD
- NGINX
Why Tanzu Build Service( TBS)?
- Save time to re-build, re-test and re-deploy during patching hundreds of containers.
- It auto scans source code configuration and language and inject dependencies into docker image that will be required to compile and/or run the app on container/K8s.
- Faster build and patching on hundreds of containers.
- Faster developer productivity by setting local build on developer’s machine and sync with source code repo like GitHub etc.
- Manage common project dependencies for dev teams and sync all developers code with single git branch to avoid any code conflict/sync issues.
- Maintain latest image in image registry.
- OCI docker image support, build and run anywhere!
Please refer this official documentation page for more detail
https://docs.pivotal.io/build-service/1-0/
Tanzu Build Service v1.0 GA can be installed on any Kubernetes private and public cloud clusters (v1.14 or later) including local machine using Kubernetes shipped with Docker Desktop, MiniKube and managed K8s like TKGI, TKG, GKE, and AKS clusters etc.
Build Service Components Tanzu Build Service ships with the following components:
- kpack
- CNB lifecycle
Prerequisite:
- Create Pivnet account. Refer to the official docs for more details on obtaining a Pivotal Network API token. You can create a free account and try on your local machine.
- Install Pivnet CLI – https://github.com/pivotal-cf/pivnet-cli/releases/tag/v2.0.1
- Install Docker Desktop (Mac, Windows) – Optional if you are trying to install on your local single node K8s cluster.
- Docker Hub account
- Install Kubernetes. I have used TKGI K8s cluster on GCP, you can also use KIND(K8s in Docker) with Docker Desktop.
- Install the TKGI CLI or kubectl CLI
- Install these three Carvel CLIs for your operating system. These can be found on their respective Tanzu Network pages:
- kapp is a deployment tool that allows users to manage Kubernetes resources in bulk.
- ytt is a templating tool that understands YAML structure.
- kbld is tool that builds, pushes, and relocates container images.
How to Install & Configure TBS:
You can download TBS from VMware Tanzu Network (formerly the Pivotal Network, or PivNet) or install using Pivnet CLI command.
Note: I have used Pivnet CLI for all the downloads from VMware PivNet. Refer this official installation guide and advance level configuration:
#Pivnet login using secret token
$ pivnet login --api-token='my-api-token'
$ pivnet download-product-files --product-slug='build-service' --release-version='1.0.2' --product-file-id=773503
#Unarchive the Build Service Bundle file:
$ tar xvf build-service-<version>.tar -C /tmp
#Login to docker-hub. This step will save docker-hub credentials to your K8s cluster. Note: You can use Harbor's url also.
$ docker login index.docker.io
#Login to VMware registry Docker-Hub thru Docker CLI (downloaded with Docker Desktop client)
$ docker login ≈
#Relocate the images for DockerHub with the Carvel tool kbld by running:
# Syntax: kbld relocate -f /tmp/images.lock --lock-output /tmp/images-relocated.lock --repository <IMAGE-REPOSITORY>
$kbld relocate -f /tmp/images.lock --lock-output /tmp/images-relocated.lock --repository itsrajivsrivastava/tanzu-build-service
Connect with your K8s cluster where you want to install TBS:
$ kubectl config use-context <K8s-cluster-name>
Now, install TBS on K8s. You can run these commands from home folder
Then use ytt to push the bundle to the image registry to DockerHub/image registry. It will upload all laungauge buildpacks and other supporting images to Docker-Hub/Harbor.
Note: It will take good time to upload bunch of images. If it fails then you need to re-run the command after deleting failed build of TBS. You can delete “kpack” and “build-service” Kubernetes namespaces.
$ ytt -f /tmp/values.yaml \
-f /tmp/manifests/ \
-v docker_repository="<IMAGE-REPOSITORY>" \
-v docker_username="<REGISTRY-USERNAME>" \
-v docker_password="<REGISTRY-PASSWORD>" \
| kbld -f /tmp/images-relocated.lock -f- \
| kapp deploy -a tanzu-build-service -f- -y
#Example:
ytt -f /tmp/values.yaml \
-f /tmp/manifests/ \
-v docker_repository=“itsrajivsrivastava” \
-v docker_username="itsrajivsrivastava" \
-v docker_password=‘******’ \
| kbld -f /tmp/images-relocated.lock -f- \
| kapp deploy -a tanzu-build-service -f- -y
Where:
IMAGE-REPOSITORY
is the image repository where Tanzu Build Service images exist.REGISTRY-USERNAME
is the username you use to access the registry.gcr.io
expects_json_key
as the username when using JSON key file authentication.REGISTRY-PASSWORD
is the password you use to access the registry
Install KP CLI:
kp CLI, is used for interacting with your Tanzu Build Service (TBS) installation on K8s cluster. Download the kp
binary from the Tanzu Build Service page on Tanzu Network.
Import Tanzu Build Service Dependencies
The Tanzu Build Service Dependencies (Stacks, Buildpacks, Builders, etc.) are used to build applications and keep them patched.
- Run this command on CLI “docker login registry.pivotal.io”
- Accept all these EULA agreements online.
Note: Successfully performing a kp import command requires that your Tanzu Network account has access to the images specified in the Dependency Descriptor file. Currently, users can only access these images if they agree to the EULA for each dependency. Users must navigate to each of the dependency product pages in Tanzu Network and accept the EULA highlighted in yellow underneath the Releases dropdown.
Here are the links to each Tanzu Network page in which users must accept the EULA:
- Tanzu Build Service Dependencies
- Java Buildpack for VMware Tanzu
- Java Native Image Buildpack for VMware Tanzu
- Node.js Buildpack for VMware Tanzu
- Go Buildpack for VMware Tanzu
Note: `kp import` will fail if it cannot access the images in all of the above Tanzu Network pages.
Note: You must be logged in locally to the registry used for `IMAGE-REGISTRY` during relocation and the Tanzu Network registry `registry.pivotal.io`.
These must be imported with the kp
cli and the Dependency Descriptor (descriptor-<version>.yaml
) file from the Tanzu Build Service Dependencies page:
$ kp import -f /tmp/descriptor-<version>.yaml
Verify kp Installation
List the custom cluster builders available in your installation:
You should see an output that looks as follows:
$ kp clusterbuilder list
NAME READY STACK IMAGE
base true io.buildpacks.stacks.bionic itsrajivsrivastava/base@sha256:b3062df93d2da25aeff827605790c508570446e53daa8afe05ed2ab4157d1c02
default true io.buildpacks.stacks.bionic itsrajivsrivastava/default@sha256:f16ed5de160ca9c13a0080d67280d0b2b843c926595c4d171568d75f96479247
full true io.buildpacks.stacks.bionic itsrajivsrivastava/full@sha256:f16ed5de160ca9c13a0080d67280d0b2b843c926595c4d171568d75f96479247
tiny true io.paketo.stacks.tiny itsrajivsrivastava/tiny@sha256:abc4879c03512a072623a7dcb18621d68122b5e608b452f411d8bc552386b8c5
# List the custom cluster builders available in your installation:
$ kp clusterstack list
NAME READY ID
base True io.buildpacks.stacks.bionic
default True io.buildpacks.stacks.bionic
full True io.buildpacks.stacks.bionic
tiny True io.paketo.stacks.tiny
Create Git and Image registry secrets in your K8s cluster:
# Docker Hub secret
$ kp secret create docker-creds --dockerhub itsrajivsrivastava -n tbs-demo
dockerhub password:
"docker-creds" created
# GitHub Secret
$ kp secret create github-creds --git https://github.com --git-user rajivmca2004 -n tbs-demo
#Verify and list secret
$ kp secret list -n tbs-demo
NAME TARGET
default-token-fqwvj
docker-creds https://index.docker.io/v1/
github-creds https://github.com
#Delete secret
$ kp secret delete <SECRET-NAME> -n tbs-demo
Create and manage docker image using kp CLI commands:
# Create TBS image: (--tag is mandatory)
$ kp image create <name> \
--tag <tag> \
[--builder <builder> or --cluster-builder <cluster-builder>] \
--namespace <namespace> \
--env <env> \
--wait \
--git <git-repo> \
--git-revision <git-revision>
#Syntax:
$ kp image create spring-petclinic \
--tag index.docker.io/itsrajivsrivastava/spring-petclinic:latest \
--namespace tbs-demo \
--git https://github.com/rajivmca2004/spring-petclinic \
--git-revision master
"spring-petclinic" created
# Verify image status
$ kp image status spring-petclinic -n tbs-demo
Status: Ready
Message: --
LatestImage: index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:4e712dec26810f026357281be4ba49ff8da3b45698700fd5dc470b7914c0d13d
Last Successful Build
Id: 1
Reason: CONFIG
Last Failed Build
Id: --
Reason: --
# List the project(s):
$ kp image list -n tbs-demo
NAME READY LATEST IMAGE
spring-petclinic True index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:d96fdd60633cd5582a9c28edceff80762ee79ef2985cd18f518bc1503563b7ef
# To check image list of any specific project
$ kp image list spring-petclinic -n tbs-demo
NAME READY LATEST IMAGE
spring-petclinic True index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:d96fdd60633cd5582a9c28edceff80762ee79ef2985cd18f518bc1503563b7ef
Build docker images:
Here, TBS will sync up with two systems:
1. GitHub – TBS will be in sync with GitHub repo for any commit and triggered after every commit. It also builds docker images.
2. Image Registry/Docker Hub – TBS will automatically push this image which has been created in the last step to image registry Docker-Hub and keep it refreshed all the time for developers and CI/CD build and deployment to K8s containers.
There are two ways to build the image:
- Auto Build
- Manual Build
Note: If you want to build source code directory which is inside sub-directories or if you have a parent project repo and multiple child project inside this
--sub-path DotNetBuild
#Example:
$ kp image create dotnetbuildtest index.docker.io/itsrajivsrivastava/dotnetbuildtest \
--sub-path DotNetBuild \
--namespace tbs-dotnet-demo \
--git https://github.com/rajivmca2004/DotNetBuild.git \
--git-revision master
1. Auto Build:
Note: First build will be triggered automatically when you create image at the first time.
To check auto build, just make some code changes in your Git branch and check the build progress using this command, It takes a few seconds to trigger. You can watch build logs locally –
# Use, this command, to check build status, also can be used for the first time to build. It will also show build revisions
$ kp build status spring-petclinic -n tbs-demo
Image: index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:e09bc84c0287d8d0985244876a68ae4b3963a96707622d81f6d9b9efa581be92
Status: SUCCESS
Build Reasons: COMMIT
Pod Name: spring-petclinic-build-2-5csx8-build-pod
Builder: itsrajivsrivastava/default@sha256:f16ed5de160ca9c13a0080d67280d0b2b843c926595c4d171568d75f96479247
Run Image: index.docker.io/itsrajivsrivastava/run@sha256:ca460a285b00d8f25ca3734b8f783af240771eb10974d90e26501fd52c0271b8
Source: GitUrl
Url: https://github.com/rajivmca2004/spring-petclinic
Revision: c5b4f7f717a1dc239c002c993c178f75283a7751
BUILDPACK ID BUILDPACK VERSION
paketo-buildpacks/bellsoft-liberica 4.0.0
paketo-buildpacks/maven 3.1.1
paketo-buildpacks/executable-jar 3.1.1
paketo-buildpacks/apache-tomcat 2.3.0
paketo-buildpacks/dist-zip 2.2.0
paketo-buildpacks/spring-boot 3.2.1
$ kp build list spring-petclinic -n tbs-demo
BUILD STATUS IMAGE STARTED FINISHED REASON
1 SUCCESS index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:b6d6a0944600be3552c0b33e0a0759a12b422168484ffff55f9f9e0be4c93217 2020-10-10 00:39:10 2020-10-10 00:59:40 CONFIG
2 SUCCESS index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:e09bc84c0287d8d0985244876a68ae4b3963a96707622d81f6d9b9efa581be92 2020-10-10 01:07:35 2020-10-10 01:09:48 COMMIT
2. Manual Build:
$ kp image trigger spring-petclinic -n tbs-demo
Check Build Logs:
# To view running logs for a build:
$ kp build logs spring-petclinic -n tbs-demo
#Check logs for the given release:
$ kp build logs spring-petclinic --build 1 -n tbs-demo
Note: First build will take some time to download all the dependent libraries. Subsequent build will be super fast!
Applied source code related build packages are mentioned in this following build logs. Since, its a Java app, these build-packs have been applied automatically –
[INFO] Building jar: /workspace/target/spring-petclinic-2.3.0.BUILD-SNAPSHOT.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.3.0.RELEASE:repackage (repackage) @ spring-petclinic ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:44 min
[INFO] Finished at: 2020-10-09T19:28:07Z
[INFO] ------------------------------------------------------------------------
Removing source code
Paketo Executable JAR Buildpack 3.1.1
https://github.com/paketo-buildpacks/executable-jar
Process types:
executable-jar: java org.springframework.boot.loader.JarLauncher
task: java org.springframework.boot.loader.JarLauncher
web: java org.springframework.boot.loader.JarLauncher
Paketo Spring Boot Buildpack 3.2.1
https://github.com/paketo-buildpacks/spring-boot
Launch Helper: Contributing to layer
Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
Writing profile.d/helper
Web Application Type: Contributing to layer
Servlet web application detected
Writing env.launch/BPL_JVM_THREAD_COUNT.default
Spring Cloud Bindings 1.6.0: Contributing to layer
Reusing cached download from buildpack
Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
Image labels:
org.opencontainers.image.title
org.opencontainers.image.version
org.springframework.boot.spring-configuration-metadata.json
org.springframework.boot.version
===> EXPORT
Reusing layers from image 'index.docker.io/itsrajivsrivastava/spring-petclinic@sha256:129f1e4231095c7504e01a4f233487b056eb28975b93f4dbb6195534bee4220e'
Adding layer 'paketo-buildpacks/bellsoft-liberica:helper'
Adding layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
Adding layer 'paketo-buildpacks/bellsoft-liberica:jre'
Adding layer 'paketo-buildpacks/bellsoft-liberica:jvmkill'
Reusing layer 'paketo-buildpacks/executable-jar:class-path'
Adding layer 'paketo-buildpacks/spring-boot:helper'
Adding layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Adding layer 'paketo-buildpacks/spring-boot:web-application-type'
Adding 1/1 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.opencontainers.image.title'
Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.spring-configuration-metadata.json'
Adding label 'org.springframework.boot.version'
*** Images (sha256:b6d6a0944600be3552c0b33e0a0759a12b422168484ffff55f9f9e0be4c93217):
index.docker.io/itsrajivsrivastava/spring-petclinic:latest
index.docker.io/itsrajivsrivastava/spring-petclinic:b1.20201009.190910
Adding cache layer 'paketo-buildpacks/bellsoft-liberica:jdk'
Adding cache layer 'paketo-buildpacks/maven:application'
Adding cache layer 'paketo-buildpacks/maven:cache'
===> COMPLETION
Build successful
Test Build Image
Pull image from Docker-hub:
docker pull itsrajivsrivastava/spring-petclinic
Run this pulled image on docker:
docker run -p 8080:8080 itsrajivsrivastava/spring-petclinic
Now , test on this browser: http://localhost:8080
How to build Dot Net application using TBS
TBS installation, setup yaml configuration, Github and DockerHub secret are same for Dot NET also.Deployment process is also same as Java on docker. There is a separate DotNet buildpack which will be injected to ASP.Net source code project.
Now, we will create TBS Dot Net image:
$ kp image create dotnetbuildtest \
--tag index.docker.io/itsrajivsrivastava/dotnetbuildtest:latest \
--namespace tbs-demo \
--git https://github.com/rajivmca2004/DotNetBuild \
--git-revision master
#Verify
$ kp image status dotnetbuildtest -n tbs-demo
If you face this in .Net apps:
Unable to start Kestrel.
System.Net.Sockets.SocketException (13): Permission denied
at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress
kestrel will bind to both ipv4 & ipv6. maybe k8s env doesn’t let it bind to ipv6. Change env port 8080 to force it to only bind to ipv4
Fix– Add this in K8s deployment manifests.
Add this in Ku8s deployment manifest file:
env:
– name: PORT
value: “8080”
FAQ
Questions | Answers |
Where exactly build happens on local machine or on K8s cluster | Builds happen in pods on the cluster. Basically everything with the service happens in the K8s cluster. You can interact with the service with the kp cli (now kp cli) |
Where it stores all dependent libraries K8s or machine from where kp build starts? | App dependent libraries live in the Stack/buildpacks which would be on the k8s cluster. |
Does it keep an image copy locally | It does not keep a copy of the app image, it only uploads to a registry |
Can we cover all CI job for build configuration pipeline and deploy with CD tools/plugin | TBS is meant to be a solution that works well in a CI/CD setting. Currently have an integration with concourse CI via https://github.com/pivotal/concourse-kpack-resource It can also integrated with Jenkin with additional configuration |
Can build-pack modified or created new custom build | Yes, follow this – https://buildpacks.io/docs/operator-guide/create-a-builder/ |
follow your steps on minikube, hit the issue on Verify kp Installation.
###########
kp clusterbuilder list
Error: no clusterbuilders found
##############
LikeLiked by 1 person
Thanks Yang for feedback. I have added missing steps of relocating all required TBS, build-packs and other OS images to image registry. Please go thru those steps and try now.
LikeLike
also hit the issue
kp import -f descriptor-100.0.23.yaml
Importing Cluster Store ‘default’…
Uploading ‘index.docker.io/284946040/build-service/tanzu-buildpacks_go@sha256:8b0664b7644a2dca2cd7dd318f356eec4738bbe2ded20b18730d262504f22acd’
Error: invalid credentials, ensure registry credentials for ‘index.docker.io/284946040/build-service/tanzu-buildpacks_go’ are available locally
✘ yanglu@yanglu-a01 ~/Downloads/minikube/tbs kp clusterbuilder list
Error: no clusterbuilders found
LikeLike
Check your credentials
LikeLike
Hello Rajeev, we’re trying to install TBS on AWS EKS with ECR. Getting below error. are we missing anything?
While executing below ytt command, I got below error:
: waiting on reconcile deployment/controller-manager (apps/v1) namespace: stacks-operator-system:
Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet “controller-manager-66df884b46” has timed out progressing.))
LikeLike
Hello Rajiv, getting below are while installing TBS on EKS with ECR registry, with yet command:
Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message:Replicaset “controller-manager-xxxx. Has timed out progressing.))
LikeLike