Connect to remote Docker server on tcp port

The docker master is where the docker server/engine daemon exists. There is strategic importance of maintaining a unique docker server in Build and Deployment during continuous release cycles, The docker clients such as the jenkins CICD server and other docker hosts connect to this master ensuring credibility and atomicity of the docker build process, And most of the times the Dynamic Docker agent from the jenkins build can connect to it and execute the docker builds.
The Docker master is the server where the build images are initially created when you run the docker build command during continuous build process.

To make a docker instance as the docker master you need to identify the following things.

Have an up to date docker daemon running with good amount of disk space for mount point /var/lib/docker.

Next up, In the file /etc/sysconfig/docker add the line OPTIONS="-H tcp://0.0.0.0:4243" at the end of the file.
As this docker master is running on a Centos machine we have the filepath /etc/sysconfig/docker.
But on Ubuntu/Debian the filepath location could be /etc/default/docker
And then restart docker daemon as follows:

[vamshi@docker-master01 ~]$ sudo systemctl restart docker

Confirm the changes with the ps command as follows:

[vamshi@docker-master01 ~]$  ps -ef | grep docker
root 2556 1 0 16:09 ? 00:00:05 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json -H tcp://0.0.0.0:4243 --storage-driver overlay2

Connecting to Docker master from client on TCP

Now the changes we got to make on the docker client are as follows:

Make sure the docker daemon on client is stopped and disabled, the following command does them both at once:

[vamshi@jenkins01 ~]$ sudo systemctl disable docker --now

From the docker client, we should test and establish the connection to the docker server through tcp ip port 4243

[vamshi@jenkins01 ~]$ docker -H tcp://10.100.0.10:4243 version
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: cccb291/1.13.1
Built: Tue Mar 3 17:21:24 2020
OS/Arch: linux/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: b2f74b2/1.13.1
Built: Wed May 1 14:55:20 2019
OS/Arch: linux/amd64
Experimental: false

Now that we have confirmed the successful connection from the client to the docker master server we can make the changes permanent, we shall export theĀ  DOCKER_HOST to the system user profile.
Now on the docker client(here: our Jenkins server) with export of DOCKER_HOST as the environment variables.

[vamshi@jenkins01 ~]$ sudo sh -c 'echo "export DOCKER_HOST=\"tcp://10.100.0.10:4243\"" > /etc/profile.d/docker.sh'

Now we see the results as our docker client is able to connect to the master.

[vamshi@jenkins01 ~]$ docker version

Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: cccb291/1.13.1
Built: Tue Mar 3 17:21:24 2020
OS/Arch: linux/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: b2f74b2/1.13.1
Built: Wed May 1 14:55:20 2019
OS/Arch: linux/amd64
Experimental: false

You might generally face an error saying : Cannot connect to the Docker daemon at unix:///var/run/docker.sock
This is generally caused by not having privileges to access the /var/run/docker.sock and the socket attributes being owned by the docker group is must. See https://linuxcent.com/cannot-connect-to-the-docker-daemon-at-unix-var-run-docker-sock-is-the-docker-daemon-running/ on changing the group ownership for unix:///var/run/docker.sock
The solution is to add your user to the docker group
# useradd -aG docker <username>

The best way to identify this issue is to run the docker info and docker version commands.

# docker version

The docker version command output has two sections.
The first section is describes Client information; which is your workstation.

The second part of the output describes about the server side information.
And here you can list out the

# docker version
Client:
Version: 1.13.1
API version: 1.26
Package version:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
(or)
Cannot connect to the Docker daemon at tcp://<docker-server-ip>:4243. Is the docker daemon running?

Either of them can mean that the destined server is not running.
Ensure by running ps -ef | grep docker

# docker info

This presents the complete information about the docker system.
In case of a tcp connection outage or if the server is not running, this command doesn’t yield any output and the output describes the error details

Its is a best practice to have a docker group created on the server and have the user part of the docker group.

# sudo groupadd docker

And add the curent user as part of the docker group.

# sudo usermod -aG docker $USER

Create a new User in Jenkins

Login to the Jenkins and navigate to the Manage Jenkins -> Manage Users. Now you will be able to see the current users on the Jenkins.

Create-new-user-in-jenkins-securityRealm
You should be able to land on the URL similar to this: http://jenkins.linuxcent.com:8080/securityRealm/

On the Left hand side menu Click on Create User and follow the process. .
Create-user-Jenkins-own-user-database
With this process you will be able to create a new user using the Jenkins’s own user database.

You can also ensure the Jenkins system user details from the jenkins server by navigating to the path $JENKINS_HOME and from the file users/users.xml

[root@node02 jenkins]# grep -w builduser * users/users.xml 2>/dev/null 
users/users.xml: <string>builduser</string>

How to start Jenkins on a different port

The jenkins Build and Integration application based on java programming language, tends to have its roots stuck to the typical Tomcat web server
Out of the box the jenkins starts up on the default port 8080 and this can be customized on the system or a new proxy routing can be setup which might add a slight overhead on the jenkins inbound requests,

How to change the Jenkins default Port?

We have the Jenkins default home located at /var/lib/jenkins all the configuration files are present in this location.
The important configuration although is /etc/sysconfig/jenkins which is present outside $JENKINS_HOME
The Jenkins Default JENKINS_PORT parameter is set to 8080

[root@node02 jenkins]# grep JENKINS_PORT /etc/sysconfig/jenkins
JENKINS_PORT="8080"

If you are on Debian or Ubuntu linux, then the relevant file will be /etc/default/jenkins
We will be now changing the variable $JENKINS_HOME to listen to the port 9090 and restarting the jenkins service.

[root@node02 jenkins]# systemctl restart jenkins.service

You can check for the port information from the ps -ef | grep jenkins output as follows or checkout from the netstat command

[root@node02 jenkins]# ps -ef |grep jenkins
jenkins 5110 1 90 23:04 ? 00:00:02 /etc/alternatives/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=9090 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

 

Jenkins admin password reset

The config file contains the XML tags to enforce the security, you have to disable this option by changing it to false from true.

If you happen to forget your admin account jenkins password, Then you need to have the shell acccess to your jenkins setup
and modify config.xml and restart the jenkins service..
The passwords in jenkins are one way hash and they are encrypted using jbcrypt.

Lets look at a couple of ways to reset the password, first is disable to global security on jenkins server by modifying the /var/lib/jenkins/config.xml as shown below:
First method is to completely disable the security on Jenkins so that you can access the Jenkins without any password prompt, which means any body on the network can do so if they have the knowledge about it. Please advise caution.

[vamshi@linuxcent jenkins]$ sudo grep useSecurity -C1 /var/lib/jenkins/config.xml
<mode>NORMAL</mode>
<useSecurity>false</useSecurity>
<authorizationStrategy class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">

Proceed to restart the jenkins service from commandline.

Login to the GUI and then update the password. These files contain the jenkins login information.

How to reset the user password of jenkins ?

The second method being directly modifying the jbcrypt hashed password present in the file /var/lib/jenkins/users/<Your UserName>/config.xml
For example:

[root@linuxcent jenkins]# grep password /var/lib/jenkins/users/admin_353942241645223362/config.xml
<passwordHash>#jbcrypt:$2a$10$razdYOUR-OLD-HASH.IVrFydsxkcQCcLmujmFQz12345678</passwordHash>

Now replace the jbcrypt hash code line with this
<passwordHash>#jbcrypt:$2a$10$uEbq9yc/UErlbO0BjqcaNutxTXueFcTkbad1jR9iOeOiNx5koRFi6</passwordHash>
and save the file. The decrypted password is admin and use this password for the next time login after restating jenkins service.

This process will come in real handy in case you forgot your jenkins login password
Alternately you can use this hash
#jbcrypt:$2a$10$razd3L1aXndFfBNHO95aj.IVrFydsxkcQCcLmujmFQzll3hcUrY7S — for a decrypted password called test

NOTE: This is only a hash is a simple temporary password to login to the jenkins and needs to be change to a stronger password

Now save the file /var/lib/jenkins/users/admin_353942241645223362/config.xml and restart the Jenkins server:

$ sudo systemctl restart jenkins

Now navigate to your Jenkins UI, enter your username and the password “admin” to login.
Once you are logged in make sure to navigate to http://jenkins.linuxcent.com:8080/user/admin/configure
eg: http://jenkins.linuxcent.com:8080/user/<Your Username>/configure

Changing admin password in Jenkinsand Now enter a preferred new strong password and save the changes, You will be automatically logged out and then logback in with the new password.

How to restart Jenkins safely

Jenkins provides the Frontend User interface and the API to access the jenkins servers and API calls also can be sent from the URL

Process to restart jenkins server safely

Here is our jenkins server hosted on our url: http://jenkins.linuxcent.com:8080

And the API request to restart Jenkins safely is to run http://YourJenkins-url-or-ip/safeRestart
http://jenkins.linuxcent.com:8080/safeRestart
See the below screenshot for more information.

This option is reliable as the restart operation will wait for the currently running jobs to complete and then proceed with restart

Safe Restart jenkins from UI API

Force restart option in jenkins

http://jenkins.linuxcent.com:8080/restart
This option will restart the Jenkins forcefully and the currently running jobs will be subjected for force termination.
Forcefully Restart jenkins from UI API

Restart jenkins server from commandline

Through the command you can initiate the restart command, but this will be a forceful restart of Jenkins server.

It will be stopping and starting the jenkins server from commandline, although you can run the stop and then start with same results.

[vamshi@jenkins jenkins]$ sudo systemctl restart jenkins

On older systemv servers you can also initiate the restart using service command

[vamshi@jenkins jenkins]$ sudo service jenkins restart