Thursday, September 14, 2017

Docker Java Example Part 5: Kubernetes

Now that I've got my project getting packaged up in a docker image, the next step in this POC is to look at platforms for running docker. The only PaaS that I am familiar with now is Pivotal Cloud Foundry, which we used at my last job to deploy Spring Boot executable jars. PCF was working on a docker story, not sure how far that got. It looks like they are pretty bought into Kubernetes these days. In fact it seems like the whole cloud world is moving in that direction, with the likes of Pivotal, VMware, Amazon, Microsoft, Dell, Alibaba, and Mesosphere joining the Cloud Native Computing Foundation. So, I set out to learn more about Kubernetes.

I started out by following the excellent Hello Minikube tutorial provided in the kubernetes docs. It steps you through installing local kubernetes (a.k.a. minikube), creating a docker image, deploying it to kubernetes, making it accessible outside the cluster, and live updating the running image. I followed the tutorial as written first, then applied it to my demo java project. Of course, I ran into some issues.

Making Minikube Aware of your Docker Image

Minikube runs its own Docker daemon. As outlined here, you have a few options for getting your docker images into minikube. Part of the hello minikube tutorial is to point your local docker client at the minikube docker daemon, and build your image there:
$ eval $(minikube docker-env)
$ env | grep DOCKER
DOCKER_HOST=tcp://192.168.64.2:2376
DOCKER_API_VERSION=1.23
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/ryanmckay/.minikube/certs
That works fine in the tutorial, because they are using the docker cli tool, which respects those env variables. Unfortunately, the bmuschko gradle docker plugin does not. But it can be configured to relatively easily. java-docker-example v0.5.1 adds:
So now you can build the docker image into kubernetes:
$ ./gradlew buildImage
And you can stop pointing at kubernetes' docker instance with:
$ eval $(minikube docker-env -u)
I'm not sure this is the long term strategy for local dev, but at least it makes gradle and docker cli work the same way, which seems appropriate.

Kubernetes Concepts

It's worth looking over the kubernetes concepts docs to understand the domain language.  A deployment is a declaration of how you want your container deployed.  It specifies things like which image to deploy, how many instances it should have, ports to expose, etc.  A deployment is mutable. The configuration of a live deployment can be modified to, e.g. target a new docker image, change number of replicas, etc.  

A deployment manages one or more replica sets.  Each replica set corresponds to a distinct configuration of the deployment.  So if the docker image config is changed on the deployment, a new replica set representing the new config is created.  The deployment remembers the mapping from configuration to replica set, so if it sees the same configuration again, it will reuse an existing replica set. Replica sets managed by deployments should not be modified directly, even though the api allows it.

A replica set manages one or more pods, depending on the number of desired replicas.  In most cases, a pod runs a single container, though it can be configured to run multiple containers that need to be colocated on the same cluster node.

Create a Deployment

A complete deployment spec is a lengthy document, but kubernetes provides a quick and easy way to create one with minimal input:
$ kubectl run java-docker-example --image=ryanmckay/java-docker-example:0.0.1-SNAPSHOT --port=8080
deployment "java-docker-example” created
Then you can look at the deployment on the cli with:
$ kubectl get deployment java-docker-example
NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
java-docker-example   1         1         1            1           1d

$ kubectl describe deployment java-docker-example
Name:            java-docker-example
Namespace:        default
CreationTimestamp:    Thu, 07 Sep 2017 00:00:37 -0500
Labels:            run=java-docker-example
Annotations:        deployment.kubernetes.io/revision=1
Selector:        run=java-docker-example
Replicas:        1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:        RollingUpdate
MinReadySeconds:    0
RollingUpdateStrategy:    1 max unavailable, 1 max surge
Pod Template<:
  Labels:    run=java-docker-example
  Containers:
   java-docker-example:
    Image:        ryanmckay/java-docker-example:0.0.1-SNAPSHOT
    Port:        8080/TCP
    Environment:    <none>
    Mounts:        <none>
  Volumes:        <none>
Conditions:
  Type        Status    Reason
  ----        ------    ------
  Available     True    MinimumReplicasAvailable
OldReplicaSets:    <none>
NewReplicaSet:    java-docker-example-3948992014 (1/1 replicas created)
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  1d        1d        1    deployment-controller            Normal        ScalingReplicaSet    Scaled up replica set java-docker-example-3948992014 to 1
Notice the "Pod Template" section that describes the type of pod that will be managed by this deployment (through a replica set). At any given time, a deployment may be managing multiple active replica sets, which may in turn be managing multiple pods. In this example, there is only one replica set, and it is only managing one pod. But if you configured higher replication and rolling update, then during a change to the deployment spec, it will be managing spinning down the old replica set while spinning up the new replica set, at a minimum. If the spec changes faster than kubernetes can apply it, it could be more than that.

The ownership relationship can be traversed at the command line. You can see the new and old replica set in the deployment description above. Replica set details can be obtained in similar fashion:
$ kubectl describe replicaset java-docker-example-3948992014
Name:  java-docker-example-3948992014
Namespace: default
Selector: pod-template-hash=3948992014,run=java-docker-example
Labels:  pod-template-hash=3948992014
  run=java-docker-example
Annotations: deployment.kubernetes.io/desired-replicas=1
  deployment.kubernetes.io/max-replicas=2
  deployment.kubernetes.io/revision=1
Controlled By: Deployment/java-docker-example
Replicas: 1 current / 1 desired
Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels: pod-template-hash=3948992014
  run=java-docker-example
  Containers:
   java-docker-example:
    Image:  ryanmckay/java-docker-example:0.0.1-SNAPSHOT
    Port:  8080/TCP
    Environment: <none>
    Mounts:  <none>
  Volumes:  <none>
Events:
  FirstSeen LastSeen Count From   SubObjectPath Type  Reason   Message
  --------- -------- ----- ----   ------------- -------- ------   -------
  24m  24m  1 replicaset-controller   Normal  SuccessfulCreate Created pod: java-docker-example-3948992014-h1c0l

You can see the created pods in the replica set's Events log. It is worth noting that the "kubectl describe" command output is intended for human consumption. To get details in a machine readable format, use "kubectl get -o json".

Minikube Dashboard

Its good to know the cli, but there is also the very nice
$ minikube dashboard

That will launch your browser pointed at the minikube dashboard app. The information we saw at the cli is available and hyperlinked.


Internal Access to Container

At this point, the deployed container is running, and you can see logs with:
$ kubectl logs deployment/java-docker-example
$ kubectl logs java-docker-example-3948992014-h1c0l
You can access it from within the cluster. Note the pod's IP address from the Pod image above. The following will start another pod running busybox.
$ kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
/ # telnet 172.17.0.4:8080
GET /greeting
{"id":4,"content":"Hello, World!"}
There are some issues here. We had to know the IP address of the pod. Also, if we were running more replicas, we wouldn't want to be reaching out to one specific instance.  The way to expose pods in kubernetes is through a service. First, note the busybox pod's environment:

/ # env | sort
HOME=/root
HOSTNAME=busybox
KUBERNETES_PORT=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
TERM=xterm
Now we launch a service for our deployment:
$ kubectl expose deployment java-docker-example
service "java-docker-example" exposed

$ kubectl describe service java-docker-example
Name:   java-docker-example
Namespace:  default
Labels:   run=java-docker-example
Annotations:  <none>
Selector:  run=java-docker-example
Type:   ClusterIP
IP:   10.0.0.32
Port:   <unset> 8080/TCP
Endpoints:  172.17.0.4:8080
Session Affinity: None
Events:   <none>
Now if we restart our busybox pod, we will have some new env variables related to the new service.
/ # env | sort
HOME=/root
HOSTNAME=busybox
JAVA_DOCKER_EXAMPLE_PORT=tcp://10.0.0.32:8080
JAVA_DOCKER_EXAMPLE_PORT_8080_TCP=tcp://10.0.0.32:8080
JAVA_DOCKER_EXAMPLE_PORT_8080_TCP_ADDR=10.0.0.32
JAVA_DOCKER_EXAMPLE_PORT_8080_TCP_PORT=8080
JAVA_DOCKER_EXAMPLE_PORT_8080_TCP_PROTO=tcp
JAVA_DOCKER_EXAMPLE_SERVICE_HOST=10.0.0.32
JAVA_DOCKER_EXAMPLE_SERVICE_PORT=8080
KUBERNETES_PORT=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
TERM=xterm

/ # telnet $JAVA_DOCKER_EXAMPLE_SERVICE_HOST:$JAVA_DOCKER_EXAMPLE_SERVICE_PORT
GET /greeting
{"id":5,"content":"Hello, World!"}
There are a couple of points to make here.  First, if you plan for pods within the cluster to use a service, and you want to use the env variables for discovery, the service needs to be created before those consuming pods. Second, there are several different service types.  Since we didn't specify a type, we got the default, ClusterIP. This exposes the service only within the cluster.

External Access to Container

At some point you're going to want to expose your containers outside the cluster.  The service types build on each other.

NodePort Service Type

NodePort exposes the service externally on each node's IP at a static port.  This supports managing your own load balancer in front of the nodes. Notice that it also set up a ClusterIP at 10.0.0.131.

$ kubectl expose deployment java-docker-example --type=NodePort
service "java-docker-example" exposed

$ kubectl describe service java-docker-example
Name:   java-docker-example
Namespace:  default
Labels:   run=java-docker-example
Annotations:  <none>
Selector:  run=java-docker-example
Type:   NodePort
IP:   10.0.0.131
Port:   <unset> 8080/TCP
NodePort:  <unset> 32478/TCP
Endpoints:  172.17.0.4:8080
Session Affinity: None
Events:   <none>

$ kubectl get node minikube -o jsonpath='{.status.addresses[].address}'
192.168.99.100

$ curl 192.168.99.100:32478/greeting
{"id":6,"content":"Hello, World!"}

LoadBalancer Service Type

This type will configure a cloud-based load balancer for you.  I need to learn more about this, as I did all these exercises on minikube only. Even on minikube though, LoadBalancer type makes your life easier.
$ kubectl expose deployment java-docker-example --type=LoadBalancer
service "java-docker-example" exposed

$ kubectl describe service java-docker-example
Name:   java-docker-example
Namespace:  default
Labels:   run=java-docker-example
Annotations:  <none>
Selector:  run=java-docker-example
Type:   LoadBalancer
IP:   10.0.0.193
Port:   <unset> 8080/TCP
NodePort:  <unset> 32535/TCP
Endpoints:  172.17.0.4:8080
Session Affinity: None
Events:   <none>

$ minikube service java-docker-example
Opening kubernetes service default/java-docker-example in default browser...
This saves you from having to track down and piece together the node ip and port.

Friday, September 1, 2017

Docker Java Example Part 4: Bmuschko and Nebula Gradle Docker Plugins

Converting from the transmode gradle plugin to the bmuschko remote api gradle plugin was pretty straightforward. Other than importing and applying the plugin, the code to get local docker image creation working is as follows:

Note that bmuschko does support multiple image tags, and I took advantage of that to get the versioned tag as well as the "latest" tag.
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
ryanmckay/java-docker-example   0.0.1-SNAPSHOT      7fd01d5b247f        6 seconds ago       115MB
ryanmckay/java-docker-example   latest              7fd01d5b247f        6 seconds ago       115MB
I tagged the code repo at this point v0.4.1

Java Application plugin

In addition to the low-level remote api plugin, bmuschko offers an opinionated docker-java-application plugin based on the application gradle plugin. Using the opinionated plugin cuts down dramatically on the boilerplate in the build.gradle:


Unfortunately, this task only supports one tag. By default, you get the versioned one.
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
ryanmckay/java-docker-example   0.0.1-snapshot      415a9e4b201d        3 seconds ago       115MB
The generated Dockerfile looks like this:

As an interesting side note, the ADD Dockerfile directive has special behavior when the file being added is a tar file. In that case, it unpacks it to the destination.

The application gradle plugin is a more generic method of packaging up a java application than that offered by the spring boot plugin. It creates a tar file containing the application jar and all the dependency jars. It also contains a shell script for launching the application, which has OS detection and some OS-specific config.
$ tar tf build/distributions/java-docker-example-0.0.1-SNAPSHOT.tar 
java-docker-example-0.0.1-SNAPSHOT/
java-docker-example-0.0.1-SNAPSHOT/lib/
java-docker-example-0.0.1-SNAPSHOT/lib/java-docker-example-0.0.1-SNAPSHOT.jar
java-docker-example-0.0.1-SNAPSHOT/lib/spring-boot-starter-1.5.4.RELEASE.jar
java-docker-example-0.0.1-SNAPSHOT/lib/spring-boot-starter-web-1.5.4.RELEASE.jar
...
java-docker-example-0.0.1-SNAPSHOT/bin/
java-docker-example-0.0.1-SNAPSHOT/bin/java-docker-example
java-docker-example-0.0.1-SNAPSHOT/bin/java-docker-example.bat

I started using gradle about the same time I started using spring boot (which has its own gradle plugin with executable jar packaging), so wasn't familiar with the application plugin. It makes sense that bmuschko would base the opinionated plugin on that, so it can support all types of java applications, not just spring boot.  However, since I plan to exclusively use spring boot for the foreseeable future, and can completely specify the execution environment in Docker (so don't need the OS-related functionality provided by the application plugin), I want to stick with Spring Boot application packaging and running.
I left the modifications in a branch tagged as v0.4.2

Nebula docker gradle plugin

Netflix publishes a set of plugins for gradle called Nebula. The nebula-docker-plugin is another opinionated plugin built on top of the bmuschko and application plugins.  It doesn't seem to add a lot beyond the bmuschko application plugin, other than the concept of separate test and production repositories for publishing docker images.  I'm going to look into docker deployment models next so it might come into play there.