Docker Monitoring with top and stats

The docker provides some in-house monitoring tools to extract the runtime information.

docker top is one such command.

[root@node01 ~]# docker top [container-id | container-name]

Here is the output of the docker top command.

[root@node01 ~]# docker top admiring_torvalds 
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
centos              13169               13153               0                   10:43               ?                   00:00:00            /bin/tini -- /usr/local/bin/jenkins.sh
centos              13191               13169               9                   10:43               ?                   00:00:21            java -jar /usr/share/jenkins/jenkins.war

When you try to see the stats of an exited container it shows up with an error as follows as the container is not running

[vamshi@node01 ~]$ docker top exited_container
Error response from daemon: Container 31b8ab84a175ce79b03ba1c9608e9d358db45f74914407212b0d2c7744324806 is not running

The docker also provides a stat command which prints out the current docker image stats
The syntax:
# sudo docker stats [CONTAINER ID | CONTAINER NAME]

This is demonstrated as follows:

CONTAINER ID        NAME                   CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
73caf780c813        do-nothing-container   0.00%               996KiB / 471.5MiB   0.21%               2.08kB / 0B         4.81MB / 0B         1

How to build a docker image from scratch using its root file system?

How to build a docker image from scratch using its root file system?

We will be demonstrating the custom docker image build by using the linux debian Distribution’s rootfs.

Our workstation environment is Debian Buster, which will be used to build a debian docker Image from a same distro linux rootfs.

Lets get our environment upto speed and download the necessary build tool for our project.

We need the command debootstrap which gives the necessary build tools to go ahead.

$ wget http://ftp.debian.org/debian/pool/main/d/debootstrap/debootstrap_1.0.116_all.deb

from your debian/ubuntu apt repository you can install as shown below:

# apt-get install debootstrap

Install the debootstrap package on the Host workstation.

$ sudo dpkg --install debootstrap_1.0.116_all.deb

To Neatly organize our build directory we can create a directory where we can download the rootfs.

$ sudo mkdir /tmp/debian-build

Downloading the debian rootfile system

Now we begin our process by looking up the Debian latest rootfs and running it against the debootstrap as shown below.

$ sudo debootstrap stable [BUILD-DIR] http://deb.debian.org/debian/

And also you can really explicit and filter some specifics as follows and choose a specific variant of debian OS as shown below.

$ sudo debootstrap --variant=minbase --components=main,contrib --include=dirmngr,apt-transport-https --arch=amd64 buster debian-buster http://deb.debian.org/debian/

Now you can also append the –verbose flag to print the background information.

Once you have successfully downloaded the rootfs, you can then verify the directory, which is /tmp/debian-build build directory in our case.

$ ls /tmp/debian-build

chroot-ing to the build directory

Now we need to chroot into the [BUILD-DIR] as follows

$ sudo chroot /tmp/debian-build

Now please note how the PS! prompt changes are you are chrooted to the latest build directory.

root@node03:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@node03:/# pwd
/
Now we are chroot-ed into our Debian build directory and our main goal is to remove as many unneeded packages as possible to downsize the image.

We now need to follow some best practices to implement and configure the build.

Firstly we create a file /etc/apt/apt.conf.d/10-assume_yes, as shown below which auto assumes the apt commands.
And run the below commands to remove the packages.
# echo "APT::Get::Assume-Yes \"true\";" | tee /etc/apt/apt.conf.d/10-assume_yes
# apt-get remove --purge -y $(apt-mark showauto) && rm -rf /var/lib/apt/lists/*
# apt-get autoremove -y
# apt-get clean
# apt-get remove --allow-remove-essential e2fsprogs e2fslibs nano pinentry-curses whiptail kmod iptables iproute2 dmidecode
# apt-get clean

We should now exit of the chroot environment by typing exit

With this you will be left with the total size of around 160MB.

root@node03:~# du -sh /tmp/debian-build
164M /tmp/debian-build

Once we are satisfied with the rootfs contents we proceed to the next steps to accomplish our goal of building a docker image.

Processing the rootfs directory into a tar file.

# sudo tar --verbose --create --file archive-name.tar --directory [BUILD-DIR] .

In our case the [BUILD-DIR] is debian-build directory.

# sudo tar --verbose --create --file debian-vamshi.tar --directory debian-build .

Creating a Docker image from the tar file

# cat archive.tar | sudo docker import - [Docker-image-name]
# cat debian-vamshi.tar | sudo docker import - debian-vamshi

Once the import operation is successfully completed we are left with a pure debian docker image to be used up.

By Successfully Stripping down the debian Image, I finally ended up with 178MB size of of debian image.

$ docker images
docker-repo/debian-vamshi        latest                  f57963009dd8        About a minute ago        178MB

Thus the process of debian image stripdown and creation of a docker image.
The same process applies to the Ubuntu linux distro, which will be discussed in another post.

How to restart start a docker container from commandline?

How to restart a docker container from commandline?

The docker command provides many options and one of them enables us to restart a container.

How to restart a running docker container?

[vamshi@node01 ~]$ docker restart jenkins_master01
jenkins_master01

We can set a wait timer using the -t[number in seconds] flag:

[vamshi@node01 ~]$ docker  restart -t20 jenkins_master01
[root@node01 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                 NAMES
c86791fad19c        jenkins             "/bin/tini -- /usr..."   42 minutes ago      Up 6 seconds        8080/tcp, 50000/tcp   jenkins_master01

How to start a docker container?

# docker start <container-id | container-name>

We look at a practical example, listing out the exited container as follows:

[vamshi@node01 ~]$ docker ps -f status="exited"  | grep jenkins -B1
CONTAINER ID        IMAGE                                        COMMAND                  CREATED             STATUS                     PORTS               NAMES
b511234ebe31        rrskris/jenkins                            "/bin/tini -- /usr..."   3 weeks ago         Exited (143) 1 day ago                       jenkins_master01

We now will start the container back live as follows:

[vamshi@node01 ~]$ docker start b511234ebe31 
b511234ebe31
It did not complain and lets check the running status again now:
[vamshi@node01 ~]$ docker ps -f status="running"  | grep jenkins -B1
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                 NAMES
b511234ebe31        rrskris/jenkins   "/bin/tini -- /usr..."   3 weeks ago         Up 9 minutes        8080/tcp, 50000/tcp   jenkins_master01

It did start alright and carried on the same old name and preserved its old state

What is Docker and what comprises inside the docker image

A Docker Container is an isolated independent instance of kernel space, which means any number of docker instances can run independent applications.

The Docker Containers by design are isolated application runtime environment using the common host system resources exposed through cgroups and the host filesystem through the tarball filesystem obtained from generating a docker image.

All because of the Kernel namespaces.. Which provisions the pids and manages its port ranges, filesystem partitions, networking, and the most astonishing feature of having root privileges inside of the container but not outside of the container all by the help of chroot functionality.
The Docker storage implements the concept of the copy-on-write (COW) layered filesystems.

Each container gets its own network isolation.

Thus Containers are Lightweight than a VM.On the back end this functions by using the chroot filesystem much like its predecessor like LXC’s, with its own hierarchy.

It also controls group resources(cgroups), groups together resources and then applies the limits on Block i/o, memory, CPU.
Namespace: It takes the system wide resources, wraps them and provides those resources as a isolated environment to the instances.

By Using a container you don’t really have to install an OS, enabling no repetition of similar workforce and you are not using the whole disk space repetitively for the similar OS files..

There’s only a single kernel which will be shared by multiple docker containers.

In this post we will explain some of the practical Docker use cases and commands :

There are two parts to the Docker Engine interns of user interaction:
One being the docker Daemon and the other being the docker client which send commands to interact with the Docker Daemon

How to build a Docker Image?:

# docker build -t <name>:<version-number> -f Dockerfile <.>

The . at the end is important because it signifies the current context and the context cannot span backward.

The option of --no-cache is important when building container images which are dependent upon downloading latest libraries from the internet or practically from your on-premise code repository which contains the freshly compiled code artifacts.

Build the Docker image with no caching:

# docker build --no-cache -t frontend-centos-lc:dev0.1 -f Dockerfile .

Once the docker container is successfully built, we can take a look at the newly created image:
Creating a docker image from scratch rootfilesystem is also a better option to create a base docker image, which gives you the freedom to package the libraries you wish and have complete control over it.

List docker images command

# docker images
# docker image ls

What are present inside the Docker Image?

The images are composed of multiple layers which form a auto union filesystem by bringing the various layers of docker image with each stages of build command creating interdependent layers.. The base image being the minimal rootfs in most cases comprised of a stripped down version of linux rootfilesystem. You can find more details from here, Building a Docker image from rootfilesystem
We run the docker inspect command on the docker image to describe various build related details.

# docker image inspect <image-name | image-id >

Example given:

root@node03:/home/vamshi# docker images nexusreg.linuxcent.com:8123/ubuntu-vamshi:v1 --no-trunc
REPOSITORY                                       TAG                 IMAGE ID                                                                  CREATED             SIZE
nexusreg.netenrichcloud.com:8088/ubuntu-vamshi   v1                  sha256:9a0b6e4f09562a0e515bb2a0ef2eca193437373fb3941c4956e13a281fe457d7   6 months ago        354MB
root@node03:/home/vamshi# 

Can be listed by the –tree option

# docker container inspect 73caf780c813
# docker images --tree

This –tree option is deprecated and history command is used to provide the image layer details.

# docker history <image layer id>

The images are stored under /var/lib/docker/<storage driver> and can be viewed, the filesystem container the container hash name followed by the docker container filesystem organized in the sub-directories.

Using the docker Tag command to tag the existing docker images to match a meaningful Repository name and append a version tag.

Example given for docker tag command.

# docker tag frontend-centos-nginx:dev0.1 my-repo:8123/frontend-nginx:v0.1

Run the docker command again to check the images, and see the newly tagged image present.

We use the docker push choose to upload the image to the docker registry which is a remote docker repository using the docker push command.

# docker push <docker Registry_name>/<image-name>:<version>
# docker push my-repo:8123/frontend-nginx:v0.1

 

Reset Gitlab password from cli

How to change the user password on gitlab?

Here is the demonstration to reset the gitlab password:

We connect to the gitlab-rails console to reset the password of the user,

In this demonstration we are going to reset the root password of the user called root whose uid is 1

[vamshi@gitlab ~]$ sudo gitlab-rails console -e production
-------------------------------------------------------------------------------------
GitLab: 11.10.4-ee (88a3c791734)
GitLab Shell: 9.0.0
PostgreSQL: 9.6.11
-------------------------------------------------------------------------------------
Loading production environment (Rails 5.0.7.2)
Loading production environment (Rails 5.0.7.2)
irb(main):001:0> User.where(id:1).first
=> #<User id:1 @root>

Now we can confirm the Uid 1 Belongs to the root user and this is the account whose password we want to reset

irb(main):001:0> user=User.where(id: 1).first
=> #<User id:1 @root>

Now enter your new password and followup with the password confirmation:

irb(main):003:0> user.password = 'AlphanumericPassword'
=> "AlphanumericPassword"
irb(main):004:0> user.password_confirmation = 'AlphanumericPassword'
=> "AlphanumericPassword"

Now save the password

irb(main):005:0> user.save
Enqueued ActionMailer::DeliveryJob (Job ID: 961d30a2b-df21-45c8-83e6-1993c85e6030) to Sidekiq(mailers) with arguments: "DeviseMailer", "password_change", "deliver_now", #<GlobalID:0x10007feafe3d64e0 @uri=#<URI::GID gid://gitlab/User/1>>
=> true

Then we exit out of the interactive ruby shell only after saving the changes

Jenkins API Copy Jenkins Job

Using the Jenkins API is perhaps the most effective way of managing your Devops Tasks, Be it getting the job information, Finding the users information, Copying the jobs,  Creating new jobs, triggering the build on multiple jobs, Clearing the Queues, Creating Credentials, Installing a Plugin, safe restart of Jenkins and many more through the API for ease of Jenkins Administration.

How to Copy a Jenkins job from Jenkins API?

A job can be referenced in order to be copied and replicated any number of times from the Jenkins UI dashboard.

First up we navigate to the Jenkins dashboard and then navigate to New Item

Enter an item name Copy jenkins job name
-> Go to the end of the page -> Enter the job name to be copied from.

The pattern name will be automatically matched from your typing and Jenkins auto suggests the source reference job name and click ok to proceed and then save in the next screen.

Using the Jenkins API:

We can use the jenkins API more effectively copy the job and create another job exact same preferences.
The syntax for the Jenkins API call will be as follows:

$ java -jar /tmp/jenkins-cli.jar -s http://jenkins.linuxcent.com:8080 -auth admin:admin copy-job SRC-JOB-NAME DST-JOB-NAME

You can also update the passwords to a text file and then pass the credentials to -auth with @<filename>
Know the Jenkins server version

$ java -jar /tmp/jenkins-cli.jar -s http://jenkins.linuxcent.com:8080 -auth @passfile -version
Version: 2.222.1

How to Download the Jenkins jarfile ?

If you are Unable to access jarfile jenkins-cli.jar on your jenkins server, Then you can download it from this URL path on your Jenkins Instance: http://YOUR-Jenkins-IP/jnlpJars/jenkins-cli.jar

$ wget http://YOUR-Jenkins-IP:8080/jnlpJars/jenkins-cli.jar -O /tmp/jenkins-cli.jar

 

Connect to remote Docker server on tcp port

The docker master is where the docker server/engine daemon exists. There is strategic importance of maintaining a unique docker server in Build and Deployment during continuous release cycles, The docker clients such as the jenkins CICD server and other docker hosts connect to this master ensuring credibility and atomicity of the docker build process, And most of the times the Dynamic Docker agent from the jenkins build can connect to it and execute the docker builds.
The Docker master is the server where the build images are initially created when you run the docker build command during continuous build process.

To make a docker instance as the docker master you need to identify the following things.

Have an up to date docker daemon running with good amount of disk space for mount point /var/lib/docker.

Next up, In the file /etc/sysconfig/docker add the line OPTIONS="-H tcp://0.0.0.0:4243" at the end of the file.
As this docker master is running on a Centos machine we have the filepath /etc/sysconfig/docker.
But on Ubuntu/Debian the filepath location could be /etc/default/docker
And then restart docker daemon as follows:

[vamshi@docker-master01 ~]$ sudo systemctl restart docker

Confirm the changes with the ps command as follows:

[vamshi@docker-master01 ~]$  ps -ef | grep docker
root 2556 1 0 16:09 ? 00:00:05 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json -H tcp://0.0.0.0:4243 --storage-driver overlay2

Connecting to Docker master from client on TCP

Now the changes we got to make on the docker client are as follows:

Make sure the docker daemon on client is stopped and disabled, the following command does them both at once:

[vamshi@jenkins01 ~]$ sudo systemctl disable docker --now

From the docker client, we should test and establish the connection to the docker server through tcp ip port 4243

[vamshi@jenkins01 ~]$ docker -H tcp://10.100.0.10:4243 version
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: cccb291/1.13.1
Built: Tue Mar 3 17:21:24 2020
OS/Arch: linux/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: b2f74b2/1.13.1
Built: Wed May 1 14:55:20 2019
OS/Arch: linux/amd64
Experimental: false

Now that we have confirmed the successful connection from the client to the docker master server we can make the changes permanent, we shall export the  DOCKER_HOST to the system user profile.
Now on the docker client(here: our Jenkins server) with export of DOCKER_HOST as the environment variables.

[vamshi@jenkins01 ~]$ sudo sh -c 'echo "export DOCKER_HOST=\"tcp://10.100.0.10:4243\"" > /etc/profile.d/docker.sh'

Now we see the results as our docker client is able to connect to the master.

[vamshi@jenkins01 ~]$ docker version

Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: cccb291/1.13.1
Built: Tue Mar 3 17:21:24 2020
OS/Arch: linux/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: b2f74b2/1.13.1
Built: Wed May 1 14:55:20 2019
OS/Arch: linux/amd64
Experimental: false

You might generally face an error saying : Cannot connect to the Docker daemon at unix:///var/run/docker.sock
This is generally caused by not having privileges to access the /var/run/docker.sock and the socket attributes being owned by the docker group is must. See https://linuxcent.com/cannot-connect-to-the-docker-daemon-at-unix-var-run-docker-sock-is-the-docker-daemon-running/ on changing the group ownership for unix:///var/run/docker.sock
The solution is to add your user to the docker group
# useradd -aG docker <username>

The best way to identify this issue is to run the docker info and docker version commands.

# docker version

The docker version command output has two sections.
The first section is describes Client information; which is your workstation.

The second part of the output describes about the server side information.
And here you can list out the

# docker version
Client:
Version: 1.13.1
API version: 1.26
Package version:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
(or)
Cannot connect to the Docker daemon at tcp://<docker-server-ip>:4243. Is the docker daemon running?

Either of them can mean that the destined server is not running.
Ensure by running ps -ef | grep docker

# docker info

This presents the complete information about the docker system.
In case of a tcp connection outage or if the server is not running, this command doesn’t yield any output and the output describes the error details

Its is a best practice to have a docker group created on the server and have the user part of the docker group.

# sudo groupadd docker

And add the curent user as part of the docker group.

# sudo usermod -aG docker $USER