Vamshi Krishna Santhapuri

Experienced Operations Engineer with a demonstrated history of working in the computer software industry. Skilled in Openshift, Kuberntes, Docker, Devops practices, Linux System Administration, Strong engineering professional with a Bachelor of Technology (B.Tech.) focused in Computer Science from JNTUH.

Various states of a docker containers

Docker container lifecycle

The containers has the following stages from the moment it is created from an image till it’s removed from the docker engine  running, restarting, removing, paused, dead and exited.
All statuses apart from running and created are not going to serve a live purpose and tend to use us the system resources. unless the are brought back to the action through the use of docker start command

We can easily perform filtering operation on the containers using the status flag:

# docker ps -f status=[created | dead | exited | running | restarting | removing]

Then the docker also allows removal of individual containers using the rm command we can use the docker container rm to have the docker delete container

[vamshi@node01 ~]$ docker container rm <container-id | container-name>
[vamshi@node01 ~]$ docker rm <container-id | container-name>

You can also use the -f flag to forcefully remove it.

# docker rm -f <container-id | container-name>

On large docker farms containing 100’s of containers, Its often a practical approach to continually keep scanning for the stale containers and cleaning them up.

Clean up the docker containers which are in exited state.

# docker ps -q -f status=exited | xargs -I {} docker rm {}
# docker container ls --filter "status=exited" | grep 'weeks ago' | awk '{print $1}' | xargs --no-run-if-empty -I {} docker rm {}

 

List docker containers currently running:

[vamshi@node01 ~]$ docker container ls -f status=running

The docker subsystem also does offer some internal system commands to get the job done using the docker’s garbage collection mechanism.

The docker system build also results in leaving some reminiscences of older build data which has to be cleaned up at regular intervals of time on the docker engine host.

 

How to print out the docker container pid’s

docker ps -qa -f status=running | xargs docker inspect --format='{{ .State.Pid }}'

A docker one liner to clear up some docker stale cache:

[root@node01 ~]# docker container ls --filter "status=exited" --filter=status="dead" | awk '{print $1}' | xargs --no-run-if-empty -I {} docker rm {}

How to identify if a docker container files have been modified

The docker container is simply a run time copy of a docker image resources, The docker container utilizes the filesystem structure originally packed into it via the union filesystem packaged from various image layers during the docker image creation.

The docker provides a standard diff command which compares the filesystem data in docker image with the container.

Syntax:

# docker diff [CONTAINER ID | CONTAINER NAME]

Before jumping in lets examine a docker container below and take a look at filesystem by logging into it.

root@node03:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
e3de85aaf61c        9a0b6e4f0956        "sh"                2 months ago        Up 3 minutes                            jovial_hertz

We have a running container with a random name jovial_hertz, and we login to the container as follows

root@node03:~# docker exec -it jovial_hertz bash

We are now inside the container and now will create a directory linuxcent and the also create a ASCII text file test and then exit out from the container.

root@e3de85aaf61c:~# mkdir linuxcent
root@e3de85aaf61c:~# cd linuxcent
root@e3de85aaf61c:~# touch test
root@e3de85aaf61c:~# exit

Created a directory called linuxcent and a touched a test file and now logged out from the container with the exit command.
The docker diff command will run against the container should result in the modified data and we contemplate the results

root@node03:~# docker diff jovial_hertz 
C /root
A /root/linuxcent
A /root/linuxcent/test
C /root/.bash_history

The Flags A in front of /root/linuxcent and /root/linuxcent/test indicate that these are directory and file that were the new additions to the container and Flag C indicates that the other 2 files were changed.
Thus it helps us to compare and contrast the new changes to a container filesystem for better auditing.

Docker login to private registry

STEP 1: Docker login to private registry

Lets see the syntax of docker login command followed by the authorized username and the repository URL.
Syntax:

[root@docker03:~]# docker login [DOCKER-REGISTRY-SERVER] -u <username> [-p][your password will be seen here]

The -p is the option for password which can be given along with the docker command or you can type it in the password prompt after hitting enter on the docker login command.

Example given:

[root@docker03:~]# docker login nexusreg.linuxcent.com:5000 -u vamshi
Password:
Login Succeeded

Once the docker login is succeeded a json file will be generated under your home directory at the following path which contains the auth metadata information.

[root@docker03:~]# cat $HOME/.docker/config.json
{
	"auths": {
		"nexusreg.linuxcent.com:5000": {
			"auth": "1234W46TmV0ZW5yaWNoMjAxOQ=="
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/18.09.1 (linux)"
	}
}

The docker login repository URL can be found out from your docker client machine using docker info command if you had previously logged in, as we see below:

[root@docker03:~]# docker info | grep Registry
Registry: https://index.docker.io/v1/

How to logout from the specific docker registry use the docker logout command.
The syntax is shown as below:
docker logout [DOCKER-REGISTRY-SERVER]

Example given:

[root@docker03:~]# docker logout 
Removing login credentials for https://index.docker.io/v1/

Build context in Dockerfile; Best practices

Best practices while building the Dockerfile.

The context in Dockerfile is relative to the current working directory of the Dockerfile and that the location where Dockerfile is present becomes its context.

Which means we can create a Directory with some content and place our Dockerfile inside it and then traverse a number of directories away from the directory and can still execute the build command

Here is an example of out general approach to building an image from a Dockerfile with . context:

# docker build --tag nginx-linuxcent .

And we list the image as follows:

[vamshi@docker01 ~]$ docker images
REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
nginx-linuxcent                              latest              0b0a4ea4d48a        3 minutes ago      210.1 MB

The build context is a . dot and the Dockerfile is present in the same directory.
If You are working locally you don’t really need a repository name and specifying just the image name is sufficient and then adding a tag is considered optional, in such cases a latest tag is appended to the end of the newly build image

As a standard practice that the Dockerfile doesn’t traverse back from the current working directory. Lets see an demonstration of of building a Dockerfile by giving the relative path from its Dockerfile.

Example given:

[vamshi@node01 ~]$ ls nginx/
default.conf Dockerfile-nginx index.html nginx.conf Portal.tar.gz

Here is our nginx/Dockerfile-nginx.

cat Dockerfile-nginx
FROM nginx:1.17.2-alpine
COPY index.html /usr/share/nginx/html/
ADD Portal.tar.gz /tmp/new1/portal
CMD ["/usr/sbin/nginx"]

Our command to build this Dockerfile-nginx now becomes:

[vamshi@docker01 ~]$ docker build -t nginx-linuxcent -f nginx/Dockerfile-nginx nginx/
Sending build context to Docker daemon 155.4 MB
Step 1/4 : FROM nginx:1.17.2-alpine
---> 55ceb2abad47
Step 2/4 : COPY index.html /usr/share/nginx/html/
---> cc652d0fc2b7
Removing intermediate container 11f195a0e2ac
Step 3/4 : ADD Portal.tar.gz /tmp/new1/portal1/
---> b18a86545c47
Removing intermediate container 1e1849be08b4
Step 4/4 : CMD /usr/sbin/nginx
---> Running in fdac087b636b
---> 02e2795eab12
Removing intermediate container fdac087b636b
Successfully built 02e2795eab12

Or you can also mention the absolute path as shown below.

[vamshi@docker01 ~]$  # docker build -t nginx-linuxcent -f /home/vamshi/nginx/Dockerfile-nginx /home/vamshi/nginx/

The above example successfully builds a docker image. The Directory nginx/ is its build context as nginx/Dockerfile-nginx is the relative path of the input Dockerfile-nginx to docker build command.

Dockerfile and the context being different

Placing the Dockerfile-nginx inside the nginx directory and context placed one directory above the Dockerfile-nginx.

We now need to modify and carefully place the ADD/COPY commands relative to its directory in order for it to work properly, The context being one directory ahead, they should be prefixed with the directory name as we see below:

FROM nginx:1.17.2-alpine
COPY nginx/index.html /usr/share/nginx/html/
ADD nginx/Portal.tar.gz /tmp/new1/portal1/
CMD ["/usr/sbin/nginx"]

Now our docker build command takes the following syntax:

[vamshi@docker01 ~]$ docker build -t nginx-linuxcent -f nginx/Dockerfile-nginx .
Sending build context to Docker daemon 155.4 MB
Step 1/4 : FROM nginx:1.17.2-alpine
 ---> 55ceb2abad47
Step 2/4 : COPY nginx/index.html /usr/share/nginx/html/
 ---> Using cache
 ---> cc652d0fc2b7
Step 3/4 : ADD nginx/Portal.tar.gz /tmp/new1/portal1/
 ---> Using cache
 ---> b18a86545c47
Step 4/4 : CMD /usr/sbin/nginx
 ---> Using cache
 ---> 02e2795eab12
Successfully built 02e2795eab12

Here the context remains outside the directory and the Dockerfile is present inside the subdirectory, the ADD/COPY commands are prefixed with the relative path of the dubirectory

Common errors Encountered with context mismatch:

unable to prepare context: The Dockerfile must be within the build context

How to tag a Docker image with a repository name during build process?

You can name your Dockerfile anything and it doesnt matter to the build process as long as you refer it with the -f

The standard naming convention is as shown below.

# docker build -t <DOCKER_IMAGE-NAME>:<TAG> -f Dockerfile .

Syntax:

# docker build -t <REPOSITORY/REGISTRY NAME>/<DOCKER_IMAGE-NAME>:<TAG> -f Dockerfile .
# docker build --tag mydocker-registry-name/nginx-linuxcent:version1.0 -f Dockerfile .

Here the Dockerfile need not be explicitly mentioned with -f as the name of the file is Dockerfile and the context being .

# docker build --tag mydocker-registry-name/nginx-linuxcent:version1.0 - .

The Build context . at the end is important because it signifies the current context and the context cannot span backward.

The tag name is a must for best practices and helps in identifying the newly build images and tagging enables visible versioning and better identification of images.

Docker build with no-cache

Creating Docker images with the --no-cache option when you do not use cache when building the image, The default option for this is set to false and can be used explicitly to enforce no-cache..

It can be at times important when building container images which are dependent upon downloading latest libraries from the internet or practically from your on-premise code repository which contains the freshly compiled code artifacts.

Build the Docker image with no cache:

# docker build --no-cache -t mydocker-registry-name/nginx:version0.1 -f Dockerfile .

Once the docker container is successfully built, we can take a look at the newly created image as below.

# docker images
REPOSITORY                                       TAG                 IMAGE ID            CREATED             SIZE
mydocker-registry-name/nginx:                    version0.1          bcdd25553d01        3 minutes ago       298 MB

Conclusion:

The docker build context becomes the present path of the Dockerfile.The docker image build is a simple process if things are neatly organized and the context can be quiet tricky if you are managing multiple Docker builds. You have the flexibility to give the absolute of relative path to the docker build.
Its always advised to implement the relative path and use the . dot as context being in the same directory where your Dockerfile is present to run the Docker builds.

Ensure to use the no-cache option

And have a proper tagging in place to enable better version identity of your docker images.

If at all you need to build an image being in different context then always write the Dockerfile relative to the directory path of current context

How to Inject Variables into Docker image in build time without modifying the Dockerfile

How to Inject Variables into Docker image in build time without modifying the Dockerfile

 

We have a requirement wherein we need to modify the specific Dockerfile with the build information and we make use of string replace operations like SED to modify the data in Dockerfile, But we can make use of the docker build time arguments to achieve the results efficiently.

Here’s a snippet of Dockerfile

FROM ubuntu:17.04
LABEL maintainer="vamshi" version="1.0.0" description="JRUBY Docker image"
WORKDIR /app
ARG JRUBY_VER 9.2.11.1
ENV JRUBY_VER ${JRUBY_VER}
ADD https://repo1.maven.org/maven2/org/jruby/jruby-dist/${JRUBY_VER}/jruby-dist-${JRUBY_VER}-bin.tar.gz .

We now build this Dockerfile as follows passing the –build-arg:

# docker build --build-arg JRUBY_VER=9.2.8.0 -t jruby:v2  -f ./Dockerfile .

Here we have passed in the build arguments of –build-arg the JRUBY_VER=9.2.8.0 and this then assigns the Argument to the ENV and passed it to ADD command which downloads tar.gz.

When run the command without any build-args it readsup the predefined ARG JRUBY_VER 9.2.11.1 from the Dockerfile.

Conclusion:
Another best usecase is while you have a continuous Docker build pipeline system and want to pass the build time arguments on user input, Its best to use ARG statements inside while writing your Dockerfile

Environmental Variables in Dockerfile.

The Docker Environment variables can be declared during the (1) Docker image creation inside the Dockerfile and (2) During the docker container run time.

The Dockerfile ENV syntax is shown as follows:

ENV VAR_NAME value

We will look at the Dockerfile syntax in the following snippet:

# Exporting the Environment Variables
ENV MAVEN_HOME /usr/share/maven/
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
# Added as per Java team
ENV JAVA_OPTS -Xms256m -Xmx768m

We can even reference the ENV variable after the declaration at the next statement.

RUN ${JAVA_HOME}/bin/java -version

Conclusion:

Setting the ENV is a lot better than Hard coding the values into the Dockerfile, as it makes maintaining the Dockerfile a simpler task.

Its a good practice for storing the path names, version of packages which are set to be modified over a longer course of time.

Best practices in creating a Dockerfile – build docker images

The Dockerfile is a very simple to understand format containing of Statements often referred to as Docker DSL(Domain Specific Language), It tends to become quiet complex and difficult to understand over the time.

[vamshi@docker01 ~]$ cat Dockerfile
# Dockerfile which runs a Latest Ubuntu image and sleeps for 100 seconds
FROM ubuntu:18.04
LABEL maintainer="vamshi" version="1.0.0" description="My First Docker Image"
RUN apt-get update && apt-get dist-upgrade -y && apt-get autoremove -y && apt-get install -y tomcat
RUN apt-get remove --purge -y $(apt-mark showauto) && rm -rf /var/lib/apt/lists/*
WORKDIR /data
ENTRYPOINT ["sleep", "100"]

Let’s examine the Dockerfile and the Statements as follows:

The First line starting with # is a comment.
The Second line with FROM tag determines the image and the latest tag; .
Eg:

FROM ubuntu:latest FROM ubuntu:18.04 or FROM centos:7

The LABEL is a Descriptive tag and contains the information about the original author credits
RUN command simulates a shell command and the subsequent statements are executed as a shell command inside the container.
WORKDIR determines the  directory context inside the running container.
ENTRYPOINT is the invocation of the container main process which runs when the docker container runs and its failure to run means the termination or the end of the particular container.
From the above Dockerfile you would have noticed 2 formats of Instruction, Now lets discuss them in details:
Shell Form
The instructions are written as shell commands

RUN apt-get update , this in turn is formatted as bin sh -c “apt-get update” and enables for command expansion, inclusion of the special characters and it enables combining of multiple commands.

Exec Form
It is JSON array style
These instructions are also shell commands but they are represented in the form of elements in a list.

["command-name","arg1","arg2"]

This format has the following drawbacks
Here the shell is not provided
No scope for variable expansion and
also this format doesn’t allow the special characters like (&&, ||, >….) to be included into the command expression statements.

While running the docker container, the CMD takes the run time arguments and the JSON list format works as a preventive measure.

Advantage of CMD and ENTRYPOINT using square bracket JSON array notation

The most advantageous point with CMD or ENTRYPOINT being written in JSON list format is the during the container runtime, the CMD can take certain arguments which can alter the main container process.. and Thus we can shield against variable expansion, Injecting special characters and not providing a shell as a counter measure security practice.

How to build a Dockerfile?

From the current working directory navigate to the location where Dockerfile is present and run the below command.

# docker build --tag first-docker-image -f Dockerfile .

How to extract Build description from docker image?

We are able to extract the docker LABEL description and MAINTAINER information from the docker command which will help in identifying its purpose when have some hundreds of docker images.

[vamshi@node01 ~]$ docker image inspect first-image --format='{{.Config.Labels}}'
map[description:My First Docker Image maintainer:vamshi version:1.0.0]

Best Practices while building Docker images.

  • While writing the Dockerfile, Its a good practice to include the version information.
  • Include the description of the image which can be easily understood while inspecting the docker image from the inspect command.
  • Grouping RUN shell commands together with && which are inter-dependent and relevant. The Docker by design stores a single STATEMENT as One Layer Image. This will enable better compressing and storing of docker image layers. This technique is called cache-busting.
    RUN apt-get update && apt-get dist-upgrade -y
  • Its a good practice to cleanup the installed package sources and build files which lightens up the size of docker image layer.
    RUN apt-get autoremove -y && rm -rf /var/lib/apt/lists/*
  • The Docker statements can be grouped together that have require less to no modification to save the network bandwidth and increase the docker build time.

How to Create Endpoints for external services in Kubernetes

The endpoints in kubernetes are the mechanism that directly interact and implement the Kubernetes Service

The Endpoints are underlying mechanism which are created in the background and enable us to talk to the kubernetes Services.
As we know that by creating a Kubernetes service we automatically generate the FQDN with the help of core-dns services

There was a requirement for me to setup the specific endpoint and create a service to convert a outside IP into the kubernetes FQDN [svc-name.namespace.cluster.local]
I managed to work around it by creating an endpoint of my external IP which was running mySQL.

[root@master01 ~]# cat Mysql-ep.yaml 
---
kind: Endpoints
apiVersion: v1
metadata:
 name: mysql-svc
 namespace: actoneye
subsets:
 - addresses:
     - ip: 172.22.110.130
   ports:
     - port: 3306

Lets take a look at Kubernetes Service yaml file.

[root@master01 ~]# cat Mysql-svc.yaml
---
kind: Service
apiVersion: v1
metadata:
 name: mysql-svc
spec:
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306

The mysql FQDN mysq-svc.dev.cluster.local was used in my application code for my cluster which the mysql resource was outside the kubernetes cluster.

Please try this out and let me know if you had any similar experiences to share.

Generate SSL certificates using openssl

Generate SSL certificates using openssl

The file ca.key and ca.crt are the Certificate Authority

We will be genrating the .key and .csr (Certificate Signing Request) files from the below command.

[root@node01 ssl]# openssl req -new -sha256 -newkey rsa:2048 -nodes -keyout linuxcent.com.key -days 365 -out linuxcent.com.csr -sha256 -subj "/C=IN/ST=TG/L=My Location/O=Company Ltd./OU=IT/CN=linuxcent.com/subjectAltName=DNS.1=linuxcent.com"

Verify the .csr file that is generated as shown below:

[root@node01 ssl]# openssl req -in linuxcent.com.csr -noout -text
Certificate Request:
Data:
Version: 0 (0x0)
Subject: C=IN, ST=TG, L=MY Location, O=Company Ltd., OU=IT, CN=linuxcent.com/subjectAltName=DNS.1=linuxcent.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:00:e4:b4:24:d7:22:ec:5d:c1:37:8c:d1:a0:62:17:
96:24:77:8d:75:4e:d5:74:15:4d:61:e0:8b:66:d6:
                Exponent: 65537 (0x10001)
        Attributes:
            a0:00
    Signature Algorithm: sha256WithRSAEncryption
         87:ef:83:b2:a6:f5:3a:f3:6f:1c:e4:02:ec:bf:5d:75:64:1d:
-- OUTPUT TRUNCATED --

Now we will using the root ca.key and [/code]ca.crt[/code] to digitally sign this .csr and generate a .crt

[root@node01 ssl]# openssl x509 -req -in linuxcent.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out linuxcent.com.crt -days 365 -sha256
Signature ok
subject=/C=IN/ST=TG/L=My Location/O=Company Ltd./OU=IT/CN=linuxcent.com/subjectAltName=DNS.1=linuxcent.com

We have generated the .crt file from the .csr

[root@node01 ssl]# ls linuxcent.com.crt linuxcent.com.key 
linuxcent.com.crt linuxcent.com.key

Therefore we have successfully generated the linuxcent.com.key file and linuxcent.com.crt, and digitally self signed with the root CA key and certificates.

Create a new kubernetes user with custom kubeconfig auth

The important directories in the reckoning are /etc/kubernetes/pki/
The file ca.key and ca.crt are the Certificate Authority key and certificate respectively.

STEP 1: Generating the key and .csr(Certificate Signing Request)

Lets now generate the .key and .csr. certificates for 1 year with openssl:

[root@node01 ssl]# openssl req -new -sha256 -newkey rsa:2048 -nodes -keyout builduser01.key -days 365 -out builduser01.csr -sha256 -subj "/C=IN/ST=TG/L=/O=/OU=/CN=/subjectAltName=DNS.1="

Verification of the CSR:

[root@node01 ssl]# openssl req -in linuxcent.com.csr -noout -text
Certificate Request:
Data:
Version: 0 (0x0)
Subject: C=IN, ST=TG/subjectAltName=DNS.1= -- INFORMATION RETRACTED --
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:00:e4:b4:24:d7:22:ec:5d:c1:37:8c:d1:a0:62:17:
96:24:77:8d:75:4e:d5:74:15:4d:61:e0:8b:66:d6:
                Exponent: 65537 (0x10001)
        Attributes:
            a0:00
    Signature Algorithm: sha256WithRSAEncryption
         87:ef:83:b2:a6:f5:3a:f3:6f:1c:e4:02:ec:bf:5d:75:64:1d:

STEP 2: Digitally Signing .csr and generating .crt using root CA files.

Now we will using the root ca.key and ca.crt to digitally sign this csr and generate a .crt

[root@node01 ssl]# openssl x509 -req -in builduser01.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out builduser01.crt -days 365 -sha256
Signature ok
subject=/C=IN/ST=TG/subjectAltName=DNS.1=
Getting CA Private Key

We have successfully generated the .crt file from the .csr along with the .key file from STEP1 with the below names. builduser01.crt and builduser01.key

How to create user accounts on kubernetes

We now will create a builduser-config to create a kubeconfig for new user.

Injecting the cluster and the API server information into the kubernetes config file:

[root@node01 ssl]# kubectl config --kubeconfig=builduser-config set-cluster kubernetes --server=https://10.100.0.10:6443 --insecure-skip-tls-verify

We are now injecting the CA certificate information into the config file:

[root@node01 ssl]# kubectl config --kubeconfig=builduser-config set-cluster kubernetes --server=https://10.100.0.10:6443 --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true

Injecting the credentials key and cert file data into the config file.

[root@node01 ssl]# kubectl config --kubeconfig=builduser-config set-credentials builduser01 --client-certificate=linuxcent.com.crt --client-key=linuxcent.com.key --embed-certs=true

Using –embed-certs=true, we can dump the cert and key file data into the config file instead of writing the path names

[root@node01 ssl]# kubectl config --kubeconfig=builduser-config set-credentials builduser01 --username=builduser01 --password=password123

Using the username and password is not explicitly required while the keys are being used.
Now copy the builduser-config to the $HOME/.kube/config and connect to the kubernetes cluster.

kubernetes bash completion

Wouldn’t it be great when we are using kubernetes and just tab to get a list of possible subcommands?

We have the extended facility to automatically tab and get the kubectl command suggestions, This feature is inbuilt into the kubectl and kubeadmin, All we need to just extract the script and enable it to the .bash_profile and source it. Lets do it as show below

[vamshi@workstation ~]$ kubectl completion bash > ~/.kube/k8s_bash_completion.sh
[vamshi@workstation ~]$ echo -e "\n#kubectl shell completion\nsource '$HOME/.kube/k8s_bash_completion.sh'\n" >> $HOME/.bash_profile
[vamshi@workstation ~]$ source $HOME/.bash_profile

Once you have successfully sourced the .bash_profie file.. type kubectl commands and keep tabbing all the way to get the suitable suggestions.