git config system vs global vs local

Setting up your environment while working with GIT is essential as part of user authentication to the remote code base.

The git config can be applied at 3 levels:

  • First is at system level scope:

git config --system : Resulting in the file /etc/gitconfig getting modified.

  • Second is at the Global level scope:

git config --global : The resultant configuration is present at /home/<username>/.gitconfig

  • Third is the local level scope

git config --local : The resultant configuration is present under .git/config of the current repo workspace

Lets examine the changes on various levels with the git config.

The git config at system level requires the root level permissions and provides the unique user identify to the whole system.
Lets see the demonstration as below:

[vamshi@linuxcent ~]$ git config --system user.name vamshi
error: could not lock config file /etc/gitconfig: Permission denied

We need to run the git config –system with sudo permissions.

[vamshi@linuxcent ~]$ sudo git config --system user.name vamshi
[vamshi@linuxcent ~]$ sudo git config --system user.email [email protected]
[vamshi@linuxcent ~]$ cat /etc/gitconfig 
[user]
	name = vamshi
	email = [email protected]

In the case of –global, The config is limited to your home directory and not on a system level and It is best advised as if you have multiple users using the system.

git config --global
[vamshi@linuxcent ~]$  git config --global user.name vamshi
[vamshi@linuxcent ~]$  git config --global user.email [email protected]

To ensure the changes are made at the user level on the home directory of a user, permanently updates users’ /home/username/.gitconfig file as below:

With this the configuration at global level in the user home directory is generated and stored into the file ~/.gitconfig as shown below:

[vamshi@linuxcent ~]$ cat ~/.gitconfig 
[user]
name = vamshi
email = [email protected]

The other entries can be added to the git config to accept connections from Insecure https site as shown below.

# git config --global http.sslVerify false

Resulting in modification of the global ~/.gitconfig file

[vamshi@workstation ~]$ cat ~/.gitconfig
[user]
	name = vamshi
	email = [email protected]
[http]
	sslVerify = false

With these settings in place now we should be well placed and would have no issues in accessing our gitlab repos.

Adding the changes to with git config –local which only reflect the specific git repository, we will see a practical demonstration as below.

[vamshi@node02 pipeline-101]$ git config --local user.name vamshi
[vamshi@node02 pipeline-101]$ git config --local user.email [email protected]
[vamshi@node02 pipeline-101]$ cat .git/config 
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
[user]
	name = vamshi
	email = [email protected]

Could not lock config file config file exist?

This will happen, if the user home directory is not writable by the user. git config –global needs to create a “lock” file ( ~/. gitconfig. … If a user has no permission of creating this file, You must check and change permissions for the home directory.

Where is Gitconfig?

The system level configuration file lives in a gitconfig file off the system root path. $(prefix)/etc/gitconfig on unix systems. On windows this file can be found at C:\Documents and Settings\All Users\Application Data\Git\config on Windows XP, and in C:\ProgramData\Git\config on Windows Vista and newer.

How do I change .gitconfig location?

Changing . gitconfig location on Windows

  1. Move your . gitconfig from user home directory, to the directory where you want.
  2. Run command line as Administrator.
  3. Go to your user home directory.
  4. Enter mklink .gitconfig \PathForNewLocationOfConfig.gitconfig.

How do I add a config file to github?

In the working directory of your git repo, you would do the following:

  • Create a version of the config file called “config-sample. …
  • Create a symbolic link between “config. …
  • Update your .gitignore to prevent the “config.ini” from being stored. …
    (Optional, but highly recommended) Create a . …
  • Do some coding and deploy…

How do I open Gitconfig?

Open a terminal of your choice. You can also use the shortcut key Ctrl+Alt+T to open a terminal.
In your terminal type : git config –global –edit It will open your global configuration file of git in your default editor.
Change the Settings that you want.

What is Gitconfig file?

gitconfig is used to store a per-user configuration as fallback values for the . git/config file. The file /etc/gitconfig can be used to store a system-wide default configuration. The configuration variables are used by both the Git plumbing and the porcelains.

How do I change my git config file?

The global git config is simply a text file, so it can be edited with whatever text editor you choose. Open, edit global git config, save and close, and the changes will take effect the next time you issue a git command. It’s that easy.

Where is my Gitconfig file Mac?

The global Git configuration file is stored at $HOME/. gitconfig on all platforms. However, you can simply open a terminal and execute git config , which will write the appropriate changes to this file.

How do I change git config in Windows?

gitconfig on my Windows machine and found this neat trick using git. Do a: git config –global -e and then, if you are lucky, you will get a text editor loaded with your global . gitconfig file.

How do I change my git config name?

  1. In your terminal, navigate to the repo you want to make the changes in.
  2. Execute git config –list to check current username & email in your local repo.
  3. Change username & email as desired. Make it a global change or specific to the local repo: git config [–global] –replace-all user.name “Full Name” …
  4. Done!

Jenkins admin password reset

The config file contains the XML tags to enforce the security, you have to disable this option by changing it to false from true.

If you happen to forget your admin account jenkins password, Then you need to have the shell acccess to your jenkins setup
and modify config.xml and restart the jenkins service..
The passwords in jenkins are one way hash and they are encrypted using jbcrypt.

Lets look at a couple of ways to reset the password, first is disable to global security on jenkins server by modifying the /var/lib/jenkins/config.xml as shown below:
First method is to completely disable the security on Jenkins so that you can access the Jenkins without any password prompt, which means any body on the network can do so if they have the knowledge about it. Please advise caution.

[vamshi@linuxcent jenkins]$ sudo grep useSecurity -C1 /var/lib/jenkins/config.xml
<mode>NORMAL</mode>
<useSecurity>false</useSecurity>
<authorizationStrategy class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">

Proceed to restart the jenkins service from commandline.

Login to the GUI and then update the password. These files contain the jenkins login information.

How to reset the user password of jenkins ?

The second method being directly modifying the jbcrypt hashed password present in the file /var/lib/jenkins/users/<Your UserName>/config.xml
For example:

[root@linuxcent jenkins]# grep password /var/lib/jenkins/users/admin_353942241645223362/config.xml
<passwordHash>#jbcrypt:$2a$10$razdYOUR-OLD-HASH.IVrFydsxkcQCcLmujmFQz12345678</passwordHash>

Now replace the jbcrypt hash code line with this
<passwordHash>#jbcrypt:$2a$10$uEbq9yc/UErlbO0BjqcaNutxTXueFcTkbad1jR9iOeOiNx5koRFi6</passwordHash>
and save the file. The decrypted password is admin and use this password for the next time login after restating jenkins service.

This process will come in real handy in case you forgot your jenkins login password
Alternately you can use this hash
#jbcrypt:$2a$10$razd3L1aXndFfBNHO95aj.IVrFydsxkcQCcLmujmFQzll3hcUrY7S — for a decrypted password called test

NOTE: This is only a hash is a simple temporary password to login to the jenkins and needs to be change to a stronger password

Now save the file /var/lib/jenkins/users/admin_353942241645223362/config.xml and restart the Jenkins server:

$ sudo systemctl restart jenkins

Now navigate to your Jenkins UI, enter your username and the password “admin” to login.
Once you are logged in make sure to navigate to http://jenkins.linuxcent.com:8080/user/admin/configure
eg: http://jenkins.linuxcent.com:8080/user/<Your Username>/configure

Changing admin password in Jenkinsand Now enter a preferred new strong password and save the changes, You will be automatically logged out and then logback in with the new password.

How do I find my Jenkins admin password?

For this the Username is admin. Password should be located in: $JENKINS_HOME/secrets/initialAdminPassword.
You can view the password using: cat /var/lib/jenkins/secrets/initialAdminPassword.
cat $JENKINS_HOME/secrets/initialAdminPassword.

What is Jenkins default admin password?

Initializing Jenkins. The first time you start Jenkins, the configuration is created along with the administrator user and password. The default login is admin/password.

How do I find my username and password for Jenkins Windows?

  • Copy the initialAdminPassword in Specified path.
  • Login with following Credentials User Name : admin Password : <da12906084fd405090a9fabfd66342f0>
  • Once you login into the jenkins application you can click on admin profile and reset the password.

How do I log into Jenkins without a password?

How to Reset Jenkins Admin User Password

  • Enable “Enable Security’ option.
  • In Security Realm option select “Jenkins own database”
  • Make sure to uncheck “Allow users to sign up” option under the “Jenkins own database” option.
  • Save Changes – Save all changes made above.

How do I change my Jenkins UI password?

Reset Jenkins Admin Password

Select the “Security Realm” (e.g. “Jenkins’ own user database” ) and click on “Save” Go to “People” -> Click on a username for which you want to change the password (e.g. admin ) -> “Configure” -> Enter a new password in the “Password” and “Confirm password” fields and click on “Save”

How do I create a Jenkins user and password?

  • Steps to create a new user
  • Login into Jenkins.
  • Go to Manage Jenkins.
  • Go to Create Users.
  • Enter all the details – Username, Password, Confirm Pwd, FullName, Email.
  • Select Create User.

How do I reset Jenkins to default?

  • In config. xml , set disableSignup to false .
  • Restart Jenkins.
  • Go to the Jenkins web page and sign up with a new user.
  • In config. …
  • If it’s a private server, set disableSignup back to true in config. …
  • Restart Jenkins.
  • Go to the Jenkins web page and log in as the new user.
  • Reset the password of the original user.

How do I restart Jenkins?

  • Go to the Jenkins installation, open the cmd and run:
  • To stop: jenkins.exe stop.
  • To start: jenkins.exe start.
  • To restart: jenkins.exe restart.

How do I unlock Jenkins?

  • Unlocking Jenkins
  • From the Jenkins console log output, copy the automatically-generated alphanumeric password (or) open the file located in /var/Jenkins_home/secrets/initialAdminPassword .
  • On the Unlock Jenkins page, paste this password into the Administrator password field and click Continue.

How do I reset my Jenkins password on Mac?

  • To Reset individual password:
  • Go to /opt/bitnami/apps/jenkins/jenkins_home/users/gmhawash.
  • Edit config. xml file.
  • Remove the passwordHash tag (you can now log in without password)

How to access Tar files in Linux/Unix

The Linux tar command abbreviation is “tar archive” released under POSIX standards initially and It now follows the GNU standards

Often when you download the files from the internet they are in the .tar, tar.gz. Or tar.bz2 compressed format using either bz2 or gz compression algorithm.

If you are familiar with the Opensource then there’s a good chance that you would have come across the extensions like Package-name.tar, tar.gz, tar.bz, tar.xz which are standard.

Well most open source software use tarballs to distribute programs/source codes in this format as it offers efficient compression and better organized way of grouping files.

It supports a vast range of compression programs such as gzip, bzip2, xz, lzip, lzma, lzop.

In the following tutorial we will show how to Compress/Extract the files into tar.gz or tgz.

How to specify the format POSIX Or GNU while using tar Command?

[vamshi@linuxcent ]$ tar --format=posix -cf posix.tar *
[vamshi@linuxcent ]$  tar --format=gnu -cf gnu.tar *
[vamshi@linuxcent ]$  file posix.tar gnu.tar 
posix.tar: POSIX tar archive
gnu.tar: POSIX tar archive (GNU)

GNU is based on an older POSIX format, So that’s why it says POSIX for both.

To print the contents information of an archived directory, use the directory name as a file name argument in conjunction with --list (-t). To find out file attributes, include the --verbose (-v) option.

Firstly start off by long listing ls in the present directory we have here:

vamshi@LinuxCent:~/Linux> ls
Debian-Distro/  OpenSuse-Distro/ README  Redhat-Distro/

We will now compress the present working directory using Linux GNU’s tar command line utility.. Creating a tar of the current directory mentioned by Asterisk *. The Options -c creates, -v: Verbose mode, -z: Uses GZIP algorithm
Demonstration shown below:
Either one of the following commands can be used

vamshi@LinuxCent:~/Linux> tar --gzip -cvf Linux-flavours.tar.gz *
vamshi@LinuxCent:~/Linux> tar -cvzf Linux-flavours.tar.gz *
Debian-Distro/
Debian-Distro/ubuntu.txt
Debian-Distro/debian.txt
Debian-Distro/README-Debian-Distro
Linux-flavours.tar.gz
OpenSuse-Distro/
OpenSuse-Distro/README-Opensuse-Distro
OpenSuse-Distro/opensuse.txt
README
Redhat-Distro/
Redhat-Distro/Fedora/
Redhat-Distro/Fedora/fedora.txt
Redhat-Distro/Centos/
Redhat-Distro/Centos/centos.txt
Redhat-Distro/Centos/CentOS-versions/
Redhat-Distro/Centos/CentOS-versions/centos7.txt
Redhat-Distro/Centos/CentOS-versions/centos5.5.txt
Redhat-Distro/Centos/CentOS-versions/centos6.9.txt
Redhat-Distro/Centos/CentOS-versions/centos5.8.txt
Redhat-Distro/Centos/CentOS-versions/centos6.1.txt
Redhat-Distro/Centos/README-CentOS
Redhat-Distro/README-Redhat-Distro
Redhat-Distro/RHEL-Versions/
Redhat-Distro/RHEL-Versions/redhat5.txt
Redhat-Distro/RHEL-Versions/redhat7.txt
Redhat-Distro/RHEL-Versions/redhat6.txt
Redhat-Distro/RHEL-Versions/redhat8.txt
Redhat-Distro/redhat.txt

How to List/view Archives

The option-t does a Dry-run of extract operation but only to print the contents of the mentioned compression.

vamshi@LinuxCent:~/Linux> tar -tvf Linux-flavours.tar.gz
Debian-Distro/
Debian-Distro/ubuntu.txt
Debian-Distro/debian.txt
Debian-Distro/README-Debian-Distro
…

How to Extracting tar.gz File?

Extracting the tar file using the options -x works out just fine as -x option chooses the tye of decompression based on the compression file type, and the content will be extracted to current working directory.

Here are various extraction options:

$ tar -xvf Linux-flavours.tar.gz
$ tar -zxvf Linux-flavours.tar.gz
$ tar --gzip -xvf Linux-flavours.tar.gz

The Filter / Options for compression types:
-z or --gzip : Used for Archival operation for .gzip type
-j -r --bzip2: Used for archival operation for .bzip2 type
[/code]-J[/code] or --xz: User for Archival for .xz type

How to Extract Only Specific Files from a compressed tar archive (tar) File

This doesn’t require any special option but you have to name the exact file and directory that you want to extract

[vamshi@linuxcent ~]$ tar -zxvf redmine-4.0.6.tar.gz redmine-4.0.6/bin/about
redmine-4.0.6/bin/about

Successful in extracting only the redmine-4.0.6/bin/about file from the redmine-4.0.6.tar.gz archive.

[vamshi@linuxcent ~]$ ls -l redmine-4.0.6/bin/about 
-rwxrwxr-x. 1 vamshi vamshi 167 Dec 20 11:46 redmine-4.0.6/bin/about

How to Extract specific files or Directories from the archive

vamshi@LinuxCent:/tmp/linux-test> tar -zxvf Linux-flavours.tar.gz Redhat-Distro/Centos/
Redhat-Distro/Centos/
Redhat-Distro/Centos/centos.txt
Redhat-Distro/Centos/CentOS-versions/
Redhat-Distro/Centos/CentOS-versions/centos7.txt
Redhat-Distro/Centos/CentOS-versions/centos5.5.txt
Redhat-Distro/Centos/CentOS-versions/centos6.9.txt
Redhat-Distro/Centos/CentOS-versions/centos5.8.txt
Redhat-Distro/Centos/CentOS-versions/centos6.1.txt
Redhat-Distro/Centos/README-CentOS

 

This way we are able to extract only the specific directories pertaining to Centos from Redhat-Distro.
Now we would do the extraction of the contents on Redhat-Distro and its sub directories.
Extracting Sub-directories inside the compressed tar.gz

How to Extract TAR file Contents to a New directory?

The tar program by default, extracts the archive contents to the present working directory.
By specifying the option --directory (-C) You can extract archive files in a Target directory.

vamshi.santhapuri@LinuxCent:/tmp/linux-test1> tar -xzvf ~/Linux/jdk-8u101-linux-x64.tar.gz  -C /usr/local/

The above operation extracts the java bundle to /usr/local/ directory.

How to extract only the specific directory from the Compression to a Target directory?

vamshi.santhapuri@LinuxCent:/tmp/linux-test1> tar -xzvf ~/Linux/Linux-flavours.tar.gz Redhat-Distro/RHEL-Versions/ -C /tmp/linux-test
Redhat-Distro/RHEL-Versions/
Redhat-Distro/RHEL-Versions/redhat5.txt
Redhat-Distro/RHEL-Versions/redhat7.txt
Redhat-Distro/RHEL-Versions/redhat6.txt
Redhat-Distro/RHEL-Versions/redhat8.txt

How to tar compress the specific Files and Directory and Sub-directories / Multiple Directories using tar command?

Below is the Demonstration of compression program using gz compression.

vamshi@LinuxCent:~/Linux> tar -cvzf Redhat-Distro/ Linux-flavours.tar.gz centos.txt fedora.txt opensuse.txt redhat.txt ubuntu.txt
centos.txt
fedora.txt
opensuse.txt
redhat.txt
Redhat-Distro/centos.txt
Redhat-Distro/fedora.txt
Redhat-Distro/redhat.txt
Redhat-Distro/Redhat-Versions/
Redhat-Distro/Redhat-Versions/redhat5.txt
Redhat-Distro/Redhat-Versions/redhat7.txt
Redhat-Distro/Redhat-Versions/redhat6.txt
Redhat-Distro/Redhat-Versions/redhat8.txt

You can also compress multiple directories and files into a single tar.gz file as demonstrated below

How to exclude particular directories and file from the compression using tar command?

We can make use of the --exclude="DontIncludethisPath" in Linux tar command, where in the base directory of mentioned pattern are excluded.. Lets run the tar command and see the results

vamshi@linuxCent:~/Linux/OSes> tar -czvf exclude-flavours.tar.gz --exclude="Redhat-Distro" .
./
./OpenSuse-Distro/
./OpenSuse-Distro/README-Opensuse-Distro
./OpenSuse-Distro/opensuse.txt
./Debian-Distro/
./Debian-Distro/ubuntu.txt
./Debian-Distro/debian.txt
./Debian-Distro/README-Debian-Distro
tar: .: file changed as we read it

The compression successfully completed and now lets list out the files

vamshi@linuxCent:~/Linux/OSes> ls -lthr
total 16K
drwxr-xr-x 2 vamshi users 4.0K Apr  8 14:09 OpenSuse-Distro
drwxr-xr-x 2 vamshi users 4.0K Apr  8 14:09 Debian-Distro
drwxr-xr-x 5 vamshi users 4.0K Apr  8 14:09 Redhat-Distro
-rw-r--r-- 1 vamshi users  718 Apr 8 19:18 exclude-flavours.tar.gz

From the dry-run extract output of the compression does not contain the Directory Redhat-Distro

vamshi@linuxCent:~/Linux/OSes> tar -tvf exclude-flavours.tar.gz
drwxr-xr-x vamshi/users 0 2020-04-08 19:18 ./
drwxr-xr-x vamshi/users 0 2020-04-08 14:09 ./OpenSuse-Distro/
-rw-r--r-- vamshi/users 0 2020-04-08 14:09 ./OpenSuse-Distro/README-Opensuse-Distro
-rw-r--r-- vamshi/users 9 2020-04-08 13:22 ./OpenSuse-Distro/opensuse.txt
drwxr-xr-x vamshi/users 0 2020-04-08 14:09 ./Debian-Distro/
-rw-r--r-- vamshi/users 7 2020-04-08 13:22 ./Debian-Distro/ubuntu.txt
-rw-r--r-- vamshi/users 7 2020-04-08 13:50 ./Debian-Distro/debian.txt
-rw-r--r-- vamshi/users 0 2020-04-08 14:09 ./Debian-Distro/README-Debian-Distro

How do I open a tar file in Linux?

tar Utility

  1. Let’s consider that we want to extract and open a doc file, and then we can use the below command to unzip the
  2. file on Linux:
  3. tar –xvzf doc.tar.gz. Remember that the tar. …
  4. tar –cvzf docs.tar.gz ~/Documents. …
  5. tar -cvf documents.tar ~/Documents. …
  6. tar –xvf docs.tar. …
  7. gzip xyz.txt. …
  8. gunzip test.txt. …
  9. gzip *.txt.

How do I view the contents of a tar file in Unix?

tar File Content. Use -t switch with tar command to list content of a archive. tar file without actually extracting.

How do I view a tar file?

How to open TAR files

  • Download and save the TAR file to your computer. …
  • Launch WinZip and open the compressed file by clicking File > Open. …
  • Select all of the files in the compressed folder or select only the files you want to extract by holding the CTRL key and left-clicking on them.

Where can I find tar files in Linux?

Combining find and tar commands so that we can find and tar files into a tarball

  1. -name “*. doc” : Find file as per given pattern/criteria. In this case find all *. doc files in $HOME.
  2. -exec tar … : Execute tar command on all files found by the find command.

How do I extract a tar file?

To extract (unzip) a tar. gz file simply right-click on the file you want to extract and select “Extract”. Windows users will need a tool named 7zip to extract tar.

How do I unzip a tar file in Terminal?

The most common uses of the tar command are to create and extract a tar archive. To extract an archive, use the tar -xf command followed by the archive name, and to create a new one use tar -czf followed by the archive name and the files and directories you want to add to the archive.

How do I extract a tar file from a directory in Linux?

Syntax For Tar Command To Extract Tar Files To a Different Directory

  1. x : Extract files.
  2. f : Tar archive name.
  3. –directory : Set directory name to extract files.
  4. -C : Set dir name to extract files.
  5. -z : Work on . tar. …
  6. -j : Work on . tar. …
  7. -J (capital J ) : Work on . tar. …
  8. -v : Verbose output i.e. show progress on screen.

Can 7 Zip open tar files?

7-Zip can also be used to unpack many other formats and to create tar files (amongst others). Download and install 7-Zip from 7-zip.org. … Move the tar file to the directory you wish to unpack into (usually the tar file will put everything into a directory inside this directory).

What is tar file in Linux?

The Linux ‘tar’ stands for tape archive, is used to create Archive and extract the Archive files. tar command in Linux is one of the important command which provides archiving functionality in Linux. We can use Linux tar command to create compressed or uncompressed Archive files and also maintain and modify them.

Build context in Dockerfile; Best practices

Best practices while building the Dockerfile.

The context in Dockerfile is relative to the current working directory of the Dockerfile and that the location where Dockerfile is present becomes its context.

Which means we can create a Directory with some content and place our Dockerfile inside it and then traverse a number of directories away from the directory and can still execute the build command

Here is an example of out general approach to building an image from a Dockerfile with . context:

# docker build --tag nginx-linuxcent .

And we list the image as follows:

[vamshi@docker01 ~]$ docker images
REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
nginx-linuxcent                              latest              0b0a4ea4d48a        3 minutes ago      210.1 MB

The build context is a . dot and the Dockerfile is present in the same directory.
If You are working locally you don’t really need a repository name and specifying just the image name is sufficient and then adding a tag is considered optional, in such cases a latest tag is appended to the end of the newly build image

As a standard practice that the Dockerfile doesn’t traverse back from the current working directory. Lets see an demonstration of of building a Dockerfile by giving the relative path from its Dockerfile.

Example given:

[vamshi@node01 ~]$ ls nginx/
default.conf Dockerfile-nginx index.html nginx.conf Portal.tar.gz

Here is our nginx/Dockerfile-nginx.

cat Dockerfile-nginx
FROM nginx:1.17.2-alpine
COPY index.html /usr/share/nginx/html/
ADD Portal.tar.gz /tmp/new1/portal
CMD ["/usr/sbin/nginx"]

Our command to build this Dockerfile-nginx now becomes:

[vamshi@docker01 ~]$ docker build -t nginx-linuxcent -f nginx/Dockerfile-nginx nginx/
Sending build context to Docker daemon 155.4 MB
Step 1/4 : FROM nginx:1.17.2-alpine
---> 55ceb2abad47
Step 2/4 : COPY index.html /usr/share/nginx/html/
---> cc652d0fc2b7
Removing intermediate container 11f195a0e2ac
Step 3/4 : ADD Portal.tar.gz /tmp/new1/portal1/
---> b18a86545c47
Removing intermediate container 1e1849be08b4
Step 4/4 : CMD /usr/sbin/nginx
---> Running in fdac087b636b
---> 02e2795eab12
Removing intermediate container fdac087b636b
Successfully built 02e2795eab12

Or you can also mention the absolute path as shown below.

[vamshi@docker01 ~]$  # docker build -t nginx-linuxcent -f /home/vamshi/nginx/Dockerfile-nginx /home/vamshi/nginx/

The above example successfully builds a docker image. The Directory nginx/ is its build context as nginx/Dockerfile-nginx is the relative path of the input Dockerfile-nginx to docker build command.

Dockerfile and the context being different

Placing the Dockerfile-nginx inside the nginx directory and context placed one directory above the Dockerfile-nginx.

We now need to modify and carefully place the ADD/COPY commands relative to its directory in order for it to work properly, The context being one directory ahead, they should be prefixed with the directory name as we see below:

FROM nginx:1.17.2-alpine
COPY nginx/index.html /usr/share/nginx/html/
ADD nginx/Portal.tar.gz /tmp/new1/portal1/
CMD ["/usr/sbin/nginx"]

Now our docker build command takes the following syntax:

[vamshi@docker01 ~]$ docker build -t nginx-linuxcent -f nginx/Dockerfile-nginx .
Sending build context to Docker daemon 155.4 MB
Step 1/4 : FROM nginx:1.17.2-alpine
 ---> 55ceb2abad47
Step 2/4 : COPY nginx/index.html /usr/share/nginx/html/
 ---> Using cache
 ---> cc652d0fc2b7
Step 3/4 : ADD nginx/Portal.tar.gz /tmp/new1/portal1/
 ---> Using cache
 ---> b18a86545c47
Step 4/4 : CMD /usr/sbin/nginx
 ---> Using cache
 ---> 02e2795eab12
Successfully built 02e2795eab12

Here the context remains outside the directory and the Dockerfile is present inside the subdirectory, the ADD/COPY commands are prefixed with the relative path of the dubirectory

Common errors Encountered with context mismatch:

unable to prepare context: The Dockerfile must be within the build context

How to tag a Docker image with a repository name during build process?

You can name your Dockerfile anything and it doesnt matter to the build process as long as you refer it with the -f

The standard naming convention is as shown below.

# docker build -t <DOCKER_IMAGE-NAME>:<TAG> -f Dockerfile .

Syntax:

# docker build -t <REPOSITORY/REGISTRY NAME>/<DOCKER_IMAGE-NAME>:<TAG> -f Dockerfile .
# docker build --tag mydocker-registry-name/nginx-linuxcent:version1.0 -f Dockerfile .

Here the Dockerfile need not be explicitly mentioned with -f as the name of the file is Dockerfile and the context being .

# docker build --tag mydocker-registry-name/nginx-linuxcent:version1.0 - .

The Build context . at the end is important because it signifies the current context and the context cannot span backward.

The tag name is a must for best practices and helps in identifying the newly build images and tagging enables visible versioning and better identification of images.

Docker build with no-cache

Creating Docker images with the --no-cache option when you do not use cache when building the image, The default option for this is set to false and can be used explicitly to enforce no-cache..

It can be at times important when building container images which are dependent upon downloading latest libraries from the internet or practically from your on-premise code repository which contains the freshly compiled code artifacts.

Build the Docker image with no cache:

# docker build --no-cache -t mydocker-registry-name/nginx:version0.1 -f Dockerfile .

Once the docker container is successfully built, we can take a look at the newly created image as below.

# docker images
REPOSITORY                                       TAG                 IMAGE ID            CREATED             SIZE
mydocker-registry-name/nginx:                    version0.1          bcdd25553d01        3 minutes ago       298 MB

Conclusion:

The docker build context becomes the present path of the Dockerfile.The docker image build is a simple process if things are neatly organized and the context can be quiet tricky if you are managing multiple Docker builds. You have the flexibility to give the absolute of relative path to the docker build.
Its always advised to implement the relative path and use the . dot as context being in the same directory where your Dockerfile is present to run the Docker builds.

Ensure to use the no-cache option

And have a proper tagging in place to enable better version identity of your docker images.

If at all you need to build an image being in different context then always write the Dockerfile relative to the directory path of current context

What is build context Docker daemon?

The build context is the set of files located at the specified PATH or URL. Those files are sent to the Docker daemon during the build so it can use them in the filesystem of the image.

How do I put files outside Docker build context?

The best way to work around this is to specify the Dockerfile independently of the build context, using -f. For instance, this command will give the ADD command access to anything in your current directory. docker build -f docker-files/Dockerfile .

What is Docker context file?

The docker context command makes it easy to export and import contexts on different machines with the Docker client installed. You can use the docker context export command to export an existing context to a file. This file can later be imported on another machine that has the docker client installed.

What is Docker compose context?

Either a path to a directory containing a Dockerfile, or a url to a git repository. When the value supplied is a relative path, it is interpreted as relative to the location of the Compose file. This directory is also the build context that is sent to the Docker daemon.

What can you do with Docker compose Yml?

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

How do I launch a Docker daemon?

On MacOS go to the whale in the taskbar > Preferences > Daemon > Advanced. You can also start the Docker daemon manually and configure it using flags. This can be useful for troubleshooting problems. Many specific configuration options are discussed throughout the Docker documentation.

Can you have multiple Dockerfiles?

Docker Compose is the most common way to build an application that uses multiple Dockerfiles. This requires a YAML file to create the container based on a series of commands.

What is difference between ADD and copy in Dockerfile?

COPY takes in a src and destruction. It only lets you copy in a local or directory from your host (the machine-building the Docker image) into the Docker image itself. ADD lets you do that too, but it also supports 2 other sources. First, you can use a URL instead of a local file/directory.

Does Docker copy follow symlinks?

That is not possible and will not be implemented. Please have a look at the discussion on github issue #1676: A symlink on your machine is the not the same as my machine and the same Dockerfile would produce two different results.

What is the Docker daemon?

The Docker daemon ( dockerd ) listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. A daemon can also communicate with other daemons to manage Docker services.

cannot connect to the docker daemon at unix:///var/run/docker.sock. is the docker daemon running?

Cannot connect to the docker daemon at unix:///var/run/docker.sock. is the docker daemon running?

The above error is very common in docker as there any many factors that cause this and we will see the practical use cases of then and the explanation with solutions.

Firstly the socket file is the main Unix/Linux pipe that is used by the dockerd which provides a self-sufficient runtime environment for the docker containers.

We generally come across the issues with the following error stating "Docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock” as the aprticular user executing docker cli is not authorized

We have a simple fix for this issue, As a Best practice on linux server running docker daemon, we need to update the following file /etc/sysconfig/docker with relevant docker group as shown below:

# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false -G dockerroot'

We have added the new Value of -G dockerroot to the OPTIONS= in /etc/sysconfig/docker
-G is the Flag to add the linux group and dockerroot is the privileged docker linux group for the docker daemon on centos/RHEL systems.
We now need to save the file and restart the docker daemon as we have modified the configuration file for the Docker daemon.

[vamshi@node01 ~]$ sudo systemctl restart docker

We now need to ensure the user accessing the docker command is part of the group dockerroot using usermod command as shown below.

[root@node01 ~]# sudo usermod -aG dockerroot vamshi

We now verify the user group information and verify the user is part of the group dockerroot.

[root@node01 ~]# id vamshi
uid=1001(vamshi) gid=1001(vamshi) groups=1001(vamshi),10(wheel),994(dockerroot)

You need relogin back if you are using the username to apply the linux group changes and once you login back you will be able to access the docker with the user account.

This process works for centos systems as we have modified the file /etc/sysconfig/docker and
The same approach can be implemented on Debian/Ubuntu systems by modifying /etc/default/docker which we will look at another post.

Now lets look at the second approach to the problem which is more of getting your hands dirty..
You will now modify the socket file by Hand as we will demonstrate in the following demonstration.

For us to be able to access this, we have to be have a read permission to this socket.
The permissions on /var/run/docker.sock will be as follows:

[vamshi@node02 ~]$ ls -l /var/run/docker.sock
srw-rw----. 1 root root 0 Apr 07 14:02 /var/run/docker.sock

As I mentioned earlier this being a socket file, as the symbol s is indicated at the start of the file permission attributes in the above output.

To overcome this issue, we have to create a docker group on the docker server as follows:

[vamshi@node02 ~]$ sudo groupadd docker

We check the group entity with the following command:

[vamshi@node02 ~]$ sudo getent group docker
docker:x:1009:

Now applying the docker group ownership to /var/run/docker.sock

[vamshi@node02 ~]$ sudo chgrp docker /var/run/docker.sock

And Here is the socket file attributes with updated group ownership:

[vamshi@node01 ~]$ ls -l /var/run/docker.sock 
srw-rw----. 1 root docker 0 Apr 07 14:09 /var/run/docker.sock

Now we have to add our user to the docker group to gain the rw permissions.
We do that in the following steps:

# sudo usermod -aG docker vamshi

Now we confirm the group members as below:

[vamshi@node02 ~]$ sudo getent group docker
docker:x:1009:vamshi

Now we need to logout and login back to this system and we will be able to access the docker command with the user.

The Other issues you might face while accessing the docker cli is probably accessing the docker server over a network where in the connection is established over the TCP connection with the docker server.

The docker server is also capable of handling the requests over the network provided it is exposed over a TCP port.
In most of the docker client server architecture, The Docker server listens on a tcp port over the network and This will be the second scenario where you have to access the docker server over the tcp socket and have to enable to the DOCKER_HOST with the docker server details over tcp.

You can try to connect to the docker server as follows

[user@docker-client ~]$ docker -H tcp://<Your-Docker-Server-IP>:<Port> version

Below is a practical command example:

[vamshi@jenkins-slave01 ~]$ docker -H tcp://10.100.0.10:4243 version

This scenario is most common with build agents like the jenkins slave and containers which run the post build and perform deployments. and is more common in the jenkins build environment..

The third most probable cause being that the actual docker server is not running if you have a local only setup and you can confirm by running the simple troubleshooting steps below by checking the process list and grep for docker process
ps -ef | grep docker and ensure that you have the docker services is started sudo systemctl status docker and ensure that its enabled on startup with sudo systemctl enable docker

Please refer our other Devops documents and do share your best practices in comments.

How do you fix Docker Cannot connect to the Docker daemon at Unix VAR run Docker sock is the Docker daemon running?

How to Resolve the “cannot connect to the Docker daemon” Error

  1. Method 1: Check the Docker Engine.
  2. Method 2: Assign Ownership to the Docker Unix Socket.
  3. Method 3: Check the Ownership of Used Files.
  4. Method 4: Add Your User to the Docker Group.
  5. Method 5: Add Environment Tables on OS X.

How do I connect to a Docker daemon?

How to Connect to a Remote Docker Daemon

  1. sudo mkdir -p /etc/systemd/system/docker.service.d.
  2. sudo nano /etc/systemd/system/docker.service.d/options.conf.
  3. [Service] ExecStart= ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2375.
  4. # Reload the systemd daemon.

How do I make sure Docker daemon is running?

Another way to check for a running Docker daemon is by inspecting its process ID file. The daemon writes its process ID to /var/run/docker. pid each time it starts up. When this file exists, Docker should be running and ready for CLI connections.

Is the Docker daemon running Docker in Docker?

Docker daemon by default listens to docker. sock. If you are on the same host where Docker daemon is running, you can use the / var/run/docker. … To run docker inside docker, all you have to do it just run docker with the default Unix socket docker.

Is the Docker daemon running MacOS?

On macOS the docker binary is only a client and you cannot use it to run the docker daemon, because Docker daemon uses Linux-specific kernel features, therefore you can’t run Docker natively in OS X.

What is Unix VAR run?

A new TMPFS-mounted file system, /var/run , is the repository for temporary system files that are not needed across system reboots in this Solaris release and future releases. For security reasons, /var/run is owned by root.

What is Unix VAR run Docker sock?

sock is basically the Unix socket the Docker daemon listens on by default. It is also a tool used to communicate with the Docker daemon from within a container. The image shows how container can access the features of docker daemon without being part of the interface itself.

What is Docker daemon and Docker?

Docker Daemon is simply a part of Docker Engine. Quoting the Docker engine overview page: Docker Engine is an open source containerization technology for building and containerizing your applications. Docker Engine acts as a client-server application with: A server with a long-running daemon process dockerd.

What is Docker daemon?

Docker daemon is a persistent background process that manages the containers on a single host. It is a self-sufficient runtime that manages Docker objects such as images, containers, network, and storage. … By default, Docker daemon creates a non-networked Unix domain socket at /var/run/docker.

How do I shutdown a docker daemon?

To stop the docker process that is already running you can just press ctrl + c or send a kill signal to it.

Where is Docker daemon port?

Check your Docker daemon. After restarting docker service, you can see the port in the output of systemctl status docker. service like /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock

Time and date setup in Linux systemd with timedatectl

The Linux command timedatectl is a systemd linux command that enables the Linux system admin to configure the date and time effectively.

The timedatectl is an important and a handy tool in linux to configure the time and date of the system, especially in a cluster setup where the synchronization is essential.

[vamshi@node01 .ssh]$ sudo timedatectl
Local time: Wed 2020-04-01 13:53:03 UTC
Universal time: Wed 2020-04-01 13:53:03 UTC
RTC time: Sun 2020-04-26 14:26:09
Time zone: UTC (UTC, +0000)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
List the timezones from commandline

[vamshi@node01 ~]$ sudo timedatectl list-timezones

How to sync the server to with ntp time sync using timedatectl ?

You have to ensure that the ntp in installed and running and then automatically the time sync will be enforced with the following command.

# sudo timedatectl set-ntp true

The manual time set can be performed using timedatectl as follows:

[vamshi@node01 ~]$ sudo timedatectl set-time 17:53:03
[vamshi@node01 ~]$ date
Wed Apr 1 17:53:04 UTC 2020

The best practice and practical approach is to set the timesync on with ntpd process enabled and in running state as show below:

# sudo systemctl enable ntpd --now

How to restart start a docker container from commandline?

How to restart a docker container from commandline?

The docker command provides many options and one of them enables us to restart a container.

How to restart a running docker container?

[vamshi@node01 ~]$ docker restart jenkins_master01
jenkins_master01

We can set a wait timer using the -t[number in seconds] flag:

[vamshi@node01 ~]$ docker  restart -t20 jenkins_master01
[root@node01 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                 NAMES
c86791fad19c        jenkins             "/bin/tini -- /usr..."   42 minutes ago      Up 6 seconds        8080/tcp, 50000/tcp   jenkins_master01

How to start a docker container?

# docker start <container-id | container-name>

We look at a practical example, listing out the exited container as follows:

[vamshi@node01 ~]$ docker ps -f status="exited"  | grep jenkins -B1
CONTAINER ID        IMAGE                                        COMMAND                  CREATED             STATUS                     PORTS               NAMES
b511234ebe31        rrskris/jenkins                            "/bin/tini -- /usr..."   3 weeks ago         Exited (143) 1 day ago                       jenkins_master01

We now will start the container back live as follows:

[vamshi@node01 ~]$ docker start b511234ebe31 
b511234ebe31

It did not complain and lets check the running status again now:

[vamshi@node01 ~]$ docker ps -f status="running"  | grep jenkins -B1
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                 NAMES
b511234ebe31        rrskris/jenkins   "/bin/tini -- /usr..."   3 weeks ago         Up 9 minutes        8080/tcp, 50000/tcp   jenkins_master01

It did start alright and carried on the same old name and preserved its old state

Buildah – An alternate Docker build tool

What is Buildah? Is it a better Docker alternate?

Used to build container images as per OCI(Open Container Initiative) standard which provides API to build Images in docker format using golang.

The Buildah project provides a command line tool that be used to create an OCI or traditional Docker image format image and to then build a working container from the image. The container can be mounted and modified and then an image can be saved based on the updated container.

Podman – Used to manage the pulling, tagging the images similar to Docker based on OCI standard.
To be used for Production grade deployments. It also allows you to create, run, and maintain containers created from those images.

Buildah Concepts:

  • Buildah’s commands replicate all of the commands that are found in a Dockerfile.
  • Also has the feasibility to build docker images with Dockerfile while not requiring root privileges while providing lowerlevel coreutils interface to build images
  • Buildah uses simple fork-exec model and does not run as a daemon but it is based on a comprehensive API in golang.
  • Buildah uses runcrun commands when buildah run used, or when buildah build-using-dockerfile encounters a RUN instruction. And it is namespace dependent on the Host System
  • Buildah relies on the CNI library and plugins to set up interfaces and routing for network namespaces
  • Buildah on Red Hat Enterprise Linux or CentOS, version 7.4 or higher is required. On other Linux distributions Buildah requires a kernel version of 4.0 or higher in order to support the OverlayFS filesystem – https://github.com/containers/buildah/blob/master/install.md
  • The dependencies required as btree filesystem, containernetworking-cni, golang and bz2 and other common-container tools.

Buildah Configuration Files and its Directory Structure:

# /usr/share/containers/

# registries.conf
# mounts.conf
# seccomp.json
# policy.json

How to build a docker image from scratch using its root file system?

How to build a docker image from scratch using its root file system?

We will be demonstrating the custom docker image build by using the linux debian Distribution’s rootfs.

Our workstation environment is Debian Buster, which will be used to build a debian docker Image from a same distro linux rootfs.

Lets get our environment upto speed and download the necessary build tool for our project.

We need the command debootstrap which gives the necessary build tools to go ahead.

$ wget http://ftp.debian.org/debian/pool/main/d/debootstrap/debootstrap_1.0.116_all.deb

from your debian/ubuntu apt repository you can install as shown below:

# apt-get install debootstrap

Install the debootstrap package on the Host workstation.

$ sudo dpkg --install debootstrap_1.0.116_all.deb

To Neatly organize our build directory we can create a directory where we can download the rootfs.

$ sudo mkdir /tmp/debian-build

Downloading the debian rootfile system

Now we begin our process by looking up the Debian latest rootfs and running it against the debootstrap as shown below.

$ sudo debootstrap stable [BUILD-DIR] http://deb.debian.org/debian/

And also you can really explicit and filter some specifics as follows and choose a specific variant of debian OS as shown below.

$ sudo debootstrap --variant=minbase --components=main,contrib --include=dirmngr,apt-transport-https --arch=amd64 buster debian-buster http://deb.debian.org/debian/

Now you can also append the –verbose flag to print the background information.

Once you have successfully downloaded the rootfs, you can then verify the directory, which is /tmp/debian-build build directory in our case.

$ ls /tmp/debian-build

chroot-ing to the build directory

Now we need to chroot into the [BUILD-DIR] as follows

$ sudo chroot /tmp/debian-build

Now please note how the PS! prompt changes are you are chrooted to the latest build directory.

root@node03:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@node03:/# pwd
/

Now we are chroot-ed into our Debian build directory and our main goal is to remove as many unneeded packages as possible to downsize the image.

We now need to follow some best practices to implement and configure the build.

Firstly we create a file /etc/apt/apt.conf.d/10-assume_yes, as shown below which auto assumes the apt commands.
And run the below commands to remove the packages.

# echo "APT::Get::Assume-Yes \"true\";" | tee /etc/apt/apt.conf.d/10-assume_yes
# apt-get remove --purge -y $(apt-mark showauto) && rm -rf /var/lib/apt/lists/*
# apt-get autoremove -y
# apt-get clean
# apt-get remove --allow-remove-essential e2fsprogs e2fslibs nano pinentry-curses whiptail kmod iptables iproute2 dmidecode
# apt-get clean

We should now exit of the chroot environment by typing exit

With this you will be left with the total size of around 160MB.

root@node03:~# du -sh /tmp/debian-build
164M /tmp/debian-build

Once we are satisfied with the rootfs contents we proceed to the next steps to accomplish our goal of building a docker image.

Processing the rootfs directory into a tar file.

# sudo tar --verbose --create --file archive-name.tar --directory [BUILD-DIR] .

In our case the [BUILD-DIR] is debian-build directory.

# sudo tar --verbose --create --file debian-vamshi.tar --directory debian-build .

Creating a Docker image from the tar file

# cat archive.tar | sudo docker import - [Docker-image-name]
# cat debian-vamshi.tar | sudo docker import - debian-vamshi

Once the import operation is successfully completed we are left with a pure debian docker image to be used up.

By Successfully Stripping down the debian Image, I finally ended up with 178MB size of of debian image.

$ docker images
docker-repo/debian-vamshi        latest                  f57963009dd8        About a minute ago        178MB

Thus the process of debian image stripdown and creation of a docker image.
The same process applies to the Ubuntu linux distro, which will be discussed in another post.

Docker Monitoring with top and stats

The docker provides some in-house monitoring tools to extract the runtime information.

docker top is one such command.

[root@node01 ~]# docker top [container-id | container-name]

Here is the output of the docker top command.

[root@node01 ~]# docker top admiring_torvalds 
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
centos              13169               13153               0                   10:43               ?                   00:00:00            /bin/tini -- /usr/local/bin/jenkins.sh
centos              13191               13169               9                   10:43               ?                   00:00:21            java -jar /usr/share/jenkins/jenkins.war

When you try to see the stats of an exited container it shows up with an error as follows as the container is not running

[vamshi@node01 ~]$ docker top exited_container
Error response from daemon: Container 31b8ab84a175ce79b03ba1c9608e9d358db45f74914407212b0d2c7744324806 is not running

The docker also provides a stat command which prints out the current docker image stats
The syntax:
# sudo docker stats [CONTAINER ID | CONTAINER NAME]

This is demonstrated as follows:

CONTAINER ID        NAME                   CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
73caf780c813        do-nothing-container   0.00%               996KiB / 471.5MiB   0.21%               2.08kB / 0B         4.81MB / 0B         1

How to login and connect to docker container tty with a username

How to login to the docker through commandline?

The docker exec is similar to running a tty on a linux machine. the only difference is it can accept many more exec connections, Although you can enable ssh to the docker but it will only be possible while running an openssh server inside the running container.

You can exec to connect to a running container much like the ssh into the running machine, Here we use the special options -i or --interactive to provide interaction and -t or --tty which allocates a pseudo tty terminal of the type of shell that is followed
The syntax of docker exec is as below:

docker exec -it <container-id | container-name> bash
[vamshi@node01 ~]$ docker exec -it b511234ebe31 bash
jenkins@b511234ebe31:/$ id
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins)

We know that the docker exec command is used opens up a tty terminal to connect to the running docker container, but It will connect us with the Default USER that was activated during it docker build.

We can use the -u flag to connect to the container with the username that is enabled for that image, We see the below example

[vamshi@node01 ~]# docker exec -it  -u root b511234ebe31 bash
root@b511234ebe31:/# pwd
/
root@b511234ebe31:/# id
uid=0(root) gid=0(root) groups=0(root)