Redirect the Std error and std output 2>&1

The redirect operation > is used in conjunction with stdoutput 1 and stderr 2.

command > [/dev/null] 2>&1

2 Represents the stderror. The &1 here represents the first argument which is /dev/null

The character 2 represents the stderr which takes the entire errors printed to the screen and then appends them to the /dev/null which is the first argument represented by &1.

So the command demonstration will be the following:

$ du -sh /* > /dev/null 2>&1

This redirect command will dump the errors and the output to /dev/null.

Explanation: the default behaviour of redirection operator is to redirect the stdout and we are redirecting them to devnul and then we followup the command with 2>&1 which mentions the stderr 2 and then redirects is to /dev/null, which is denoted by &1 describing the &1 as the first argument which is /dev/null

 

 

Install java on Linux centos

In this tutorial we will quickly setup java on linux centos,

We will be using the yum command to download the openjdk 1.8 and install

[vamshi@node01 ~]$ sudo yum install java-1.8.0-openjdk.x86_64

We have installed the java openjdk 1.8 and we can check the version using java -version

[vamshi@node01 ~]$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

 

We make use of the alternatives command in centos which lists if we have any other version of java installed on the machine, and then enabling the default java version on the system wide.

[vamshi@node01 ~]$ alternatives --list | grep java
java auto /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/java
jre_openjdk auto /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
jre_1.8.0 auto /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
jre_1.7.0 auto /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre
[vamshi@node01 ~]$ sudo alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*  1           java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/java)
 + 2           java-1.7.0-openjdk.x86_64 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre/bin/java)

Enter to keep the current selection[+], or type selection number: 1

This enabled openjdk1.8 to be the default version of java.

Setting JAVA_HOME path
In order to set the system JAVA_HOME path on the system we need to export this variable, for the obvious reasons of other programs and users using the classpath such as while using maven or a servlet container.

Now there are two levels we can setup the visibility of JAVA_HOME environment variable.
1. Setup JAVA_HOME for single user profile
We need to update the changes to the ~/.bash_profile

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

PATH=$PATH:$JAVA_HOME

export PATH

Now we need enforce the changes with reloading the .bash_profile with a simple logout and then login into the system or we can source the file ~/.bash_profile as follows:

[vamshi@node01 ~]$ source .bash_profile

Verifying the changes:

[vamshi@node01 ~]$ echo $PATH
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/vamshi/.local/bin:/home/vamshi/bin:/home/vamshi/.local/bin:/home/vamshi/bin:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

2. Setup JAVA_HOME for the system wide profile and available to all the users.

[vamshi@node01 ~]$ sudo sh -c "echo -e 'export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/' > /etc/profile.d/java.sh"

This echo command writes the JAVA_HOME path to the system profile.d and creates a file java.sh which is read system wide level.

Ensure the changes are written to /etc/profile.d/java.sh

[vamshi@node01 ~]$ cat /etc/profile.d/java.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

Now source to apply the changes immediately to the file /etc/profile.d/java.sh as follows

[vamshi@node01 ~]$ sudo sh -c ' source /etc/profile.d/java.sh '

Or login to the root account and run the source command

Ensure to run the env command

[vamshi@node01 ~]$ env  | grep JAVA_HOME
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

How to allocate resources to docker images in Runtime?

We have seen the docker runtime environment takes up the overall available system resources on the system and tends to impact the base system.

To better utilize the containers, we can avail the resource cap and define the metric limits on specific containers while starting up the respective docker images..

The general syntax goes as follows:

# docker run --cpus ="x.x" --memory=x[M|G] docker-image

Here we see the demonstration

[root@node01 ~]# docker run --cpus="0.2" --memory="200M" jenkins:latest
/usr/bin/docker-current: Error response from daemon: Minimum memory limit allowed is 4MB.

It should be noticed that the minimum Memory limit allowed for the docker container to run is 4MB and the minimum CPU cores is at 0.01, any thing lower that this means the container runtime fails to allocate sufficient resources. These limits will be efficient when running on some test and debug scenarios.

How to login and connect to docker container tty with a username

How to login to the docker through commandline?

The docker exec is similar to running a tty on a linux machine. the only difference is it can accept many more exec connections, Although you can enable ssh to the docker but it will only be possible while running an openssh server inside the running container.

You can exec to connect to a running container much like the ssh into the running machine, Here we use the special options -i or --interactive to provide interaction and -t or --tty which allocates a pseudo tty terminal of the type of shell that is followed
The syntax of docker exec is as below:

docker exec -it <container-id | container-name> bash
[vamshi@node01 ~]$ docker exec -it b511234ebe31 bash
jenkins@b511234ebe31:/$ id
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins)

We know that the docker exec command is used opens up a tty terminal to connect to the running docker container, but It will connect us with the Default USER that was activated during it docker build.

We can use the [/code]-u[/code] flag to connect to the container with the username that is enabled for that image, We see the below example

[vamshi@node01 ~]# docker exec -it  -u root b511234ebe31 bash
root@b511234ebe31:/# pwd
/
root@b511234ebe31:/# id
uid=0(root) gid=0(root) groups=0(root)

 

 

Buildah – An alternate Docker build tool

What is Buildah? Is it a better Docker alternate?

Used to build container images as per OCI(Open Container Initiative) standard which provides API to build Images in docker format using golang.

The Buildah project provides a command line tool that be used to create an OCI or traditional Docker image format image and to then build a working container from the image. The container can be mounted and modified and then an image can be saved based on the updated container.

Podman – Used to manage the pulling, tagging the images similar to Docker based on OCI standard.
To be used for Production grade deployments. It also allows you to create, run, and maintain containers created from those images.

Buildah Concepts:

  • Buildah’s commands replicate all of the commands that are found in a Dockerfile.
  • Also has the feasibility to build docker images with Dockerfile while not requiring root privileges while providing lowerlevel coreutils interface to build images
  • Buildah uses simple fork-exec model and does not run as a daemon but it is based on a comprehensive API in golang.
  • Buildah uses runcrun commands when buildah run used, or when buildah build-using-dockerfile encounters a RUN instruction. And it is namespace dependent on the Host System
  • Buildah relies on the CNI library and plugins to set up interfaces and routing for network namespaces
  • Buildah on Red Hat Enterprise Linux or CentOS, version 7.4 or higher is required. On other Linux distributions Buildah requires a kernel version of 4.0 or higher in order to support the OverlayFS filesystem – https://github.com/containers/buildah/blob/master/install.md
  • The dependencies required as btree filesystem, containernetworking-cni, golang and bz2 and other common-container tools.

Buildah Configuration Files and its Directory Structure:

# /usr/share/containers/

# registries.conf
# mounts.conf
# seccomp.json
# policy.json

Git config setup on linux; Unable to pull or clone from git; fatal: unable to access git; Peer’s Certificate has expired

Facing an issue with pulling the repository while dealing with an expired SSL certificate.

[vamshi@workstation ~]$ git pull https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
fatal: unable to access 'https://gitlab.linuxcent.com/linuxcent/pipeline-101.git/': Peer's Certificate has expired.
[vamshi@workstation ~]$

SSL error while cloning git URL

If you have faced the error, then we can work around it by ignoring SSL certificate check and continue working with the git repo.

[vamshi@workstation ~]$ git clone https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
Cloning into 'pipeline-101'...
fatal: unable to access 'https://gitlab.linuxcent.com/linuxcent/pipeline-101.git/': Peer's Certificate has expired.

It doesn’t allow the clone or pull or push to the gitlab website as its certificate is not valid, and the certificate is unsigned by a Valid CA. In most cases, we will have the corporate gitlab repo in our internal network and not publicly exposed.
We therefore trust the gitlab server as we have a bunch of our code on it.. Why not, I say?
We have to ensure to disable the check for the SSL certificate verification

Set the Variable GIT_SSL_NO_VERIFY=1 or GIT_SSL_NO_VERIFY=false and try to execute your previous command.

[vamshi@workstation ~]$ GIT_SSL_NO_VERIFY=1 git clone https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
Cloning into 'pipeline-101'...
Username for 'https://gitlab.linuxcent.com': vamshi
Password for 'https://vamshi@gitlab.linuxcent.com': 
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.

Make a permanent entry to system wide user level profiles as below. The following change works at the system level

[vamshi@workstation ~]$ sudo bash -c "echo -e export GIT_SSL_NO_VERIFY=1 > /etc/profile.d/gitconfig.sh "
[vamshi@workstation ~]$ cat /etc/profile.d/gitconfig.sh
export GIT_SSL_NO_VERIFY=false

The practical use case of setting the environment variable can be made while building container images, using  the GIT_SSL_NO_VERIFY false as an environment variable in Dockerfile and building an image.

[vamshi@workstation ~]$ cat Dockerfile
FROM jetty:latest
-- CONTENT TRUNCATED --
env GIT_SSL_NO_VERIFY 1
-- CONTENT TRUNCATED --

We can also setup the container build agent with Jenkins Pipeline code with similar configuration to fetch a gitrepo in our next sessions.

Time and date setup in Linux systemd with timedatectl

The Linux command timedatectl is a systemd linux command that enables the Linux system admin to configure the date and time effectively.

The timedatectl is an important and a handy tool in linux to configure the time and date of the system, especially in a cluster setup where the synchronization is essential.

[vamshi@node01 .ssh]$ sudo timedatectl
Local time: Wed 2020-04-01 13:53:03 UTC
Universal time: Wed 2020-04-01 13:53:03 UTC
RTC time: Sun 2020-04-26 14:26:09
Time zone: UTC (UTC, +0000)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
List the timezones from commandline

[vamshi@node01 ~]$ sudo timedatectl list-timezones

How to sync the server to with ntp time sync using timedatectl ?

You have to ensure that the ntp in installed and running and then automatically the time sync will be enforced with the following command.

# sudo timedatectl set-ntp true

The manual time set can be performed using timedatectl as follows:

[vamshi@node01 ~]$ sudo timedatectl set-time 17:53:03
[vamshi@node01 ~]$ date
Wed Apr 1 17:53:04 UTC 2020

The best practice and practical approach is to set the timesync on with ntpd process enabled and in running state as show below:

# sudo systemctl enable ntpd --now

check linux version

The Linux OS has many Distributions and various versions with slight modifications in the kernel versions, a major and minor version identification is crucial to practical administration of a linux server. Linux being open source, the release cycles are continuous resulting in many changes with Longterm release and short term release cycles in some most known and popular linux Distributors like Centos/Debian.. A special mentioning of the generic Desktop PC/laptop based versions of fedora/Ubuntu/Arch/OpenSuse Leap-Tumbleweed Linux where fan following is heavy, the Release cycles are very aggressive with new versions releasing once in for every 2-3 weeks.

In this Section we will see how to check linux version, on most popular Distributions.
This will be handy during the kernel patch process to identify the current linux version.
Lets list our some practical linux commands and scenarios

Checking the Linux OS Version using /etc/os-release

[vamshi@node02 ]$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

Checking the Linux Version from the file /etc/issue

This file is being relinquished in latest Systemd variants of Linux, But tends to gives a less adequate information just lying around for aged linux user’s familiarity with systemd versions.

vamshi@node03:/$ cat /etc/issue
Debian GNU/Linux 10 \n \l

Check Linux Version using lsb_release:

The lsb_release prints the (Linux Standard Base) and Distribution information on the Linux host.

[vamshi@node02 cp-command]$ lsb_release -a
LSB Version:    :core-4.1-ia32:core-4.1-noarch
Distributor ID:    CentOS
Description:    CentOS Linux release 7.7.1908 (Core)
Release:    7.7.1908
Codename:    Core

How to check the running kernel version information ?

Check Linux Version information with uname command

[vamshi@node02 cp-command]$ uname -a
Linux node02.linuxcent.com 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Checking the Linux Version using hostnamectl

The command hostnamectl gives the complete information of the underlying architecture, kernel version and the Linux OS name

vamshi@node03:/$ hostnamectl 
Static hostname: node03
Icon name: computer-vm
Chassis: vm
Machine ID: b4adcdb84c724856b577524ebbfa0003
Boot ID: 1a2db0a0ae8b4c5ba86b390c68af7024
Virtualization: oracle
Operating System: Debian GNU/Linux 10 (buster)
Kernel: Linux 4.19.0-5-amd64
Architecture: x86-64

How to create Temporary filesystem on Linux

The greatest advantage with a tmpfs is that you can use the faster access Volatile memory to store files instead of Secondary storage system.

Lets head off to the Demonstration using tmpfs.

Create a Directory named test-docs in /mnt

[vamshi@SERVER02 mnt]$ sudo mkdir /mnt/test-docs

We will be using the dd command to dump the data and create a file with given Block size, as below and compare the write speeds on tmpfs (Linux temporary Filesystem) vs conventional storage filesystem.

[vamshi@SERVER02 mnt]$ sudo mount -t tmpfs -o size=1G tmpfs /mnt/test-docs

Output from the mount command:

[vamshi@SERVER02 mnt]$ df -hT /mnt/test-docs/
Filesystem     Type   Size Used Avail Use% Mounted on
tmpfs          tmpfs  1.0G     0  1.0G   0% /mnt/test-docs

Now add the entry to /etc/fstab to make the mount persistent across reboots(We should write How to create an fstab entry in another article)

tmpfs       /mnt/webdocs tmpfs   nodev,nosuid,noexec,nodiratime,size=1G   0 0

With these changes it is going to perform faster system read writes during the runtime.

Please advise caution as this uses the RAM.

Now to test the setting Temporary Filesystem vs Storage Filesystem write speeds

We will demonstrate see about the temporary(tmpfs) filesystem on linux in action and practical examples.

Writing a 500MB file on the same server to a Secondary storage vs Writing it to tmpfs.

Lets us write a single 500MB file and measure its write speed and time it’s operation

[vamshi@SERVER02 mnt]$ time sudo dd if=/dev/zero of=/mnt/test-docs/dump.txt bs=1k count=500000
500000+0 records in
500000+0 records out
512000000 bytes (512 MB) copied, 0.815591 s, 628 MB/s

real    0m0.871s
user    0m0.103s
sys    0m0.760s

 

This operation finished under 0.8 seconds time which is around 100 milliseconds lesser than 1 second and write operation completed with 628MB/s.

Lets check on the file size on the tmpfs mountpoint /mnt/test-docs/

[vamshi@SERVER02 mnt]$ df -hT /mnt/test-docs/
Filesystem     Type   Size Used Avail Use% Mounted on
tmpfs          tmpfs  1.0G 489M  536M 48% /mnt/test-docs

Now lets run the same write operation to the secondary storage filesystem

[vamshi@SERVER02 mnt]$ time sudo dd if=/dev/zero of=/mnt/dump.txt bs=1k count=500000
500000+0 records in
500000+0 records out
512000000 bytes (512 MB) copied, 1.38793 s, 369 MB/s

real    0m1.510s
user    0m0.028s
sys    0m1.227s

The write operation to the disk took twice the time at a write speed of 369MB/s

The space usage on the [/code]/mnt/test-docs/[/code] after the operation is 489MB out of 1024MB.

[vamshi@SERVER02 mnt]$ df -hT /mnt/test-docs/
Filesystem     Type   Size Used Avail Use% Mounted on
tmpfs          tmpfs  1.0G 489M  536M 48% /mnt/test-docs

CONCLUSION:

The write/read speeds to tmpfs were substantially faster compared to the secondary storage block device mount points, The fact of the matter is tmpfs although the clear winner is tmpfs, it cannot be used for persistent data storage as its just sits on top of the memory and in no way offers long term data storage persistence.

So what is a practical usage of tmpfs in linux ?

The practical usage can be storing of the static web content to serve the images,css and js to fasten the request speeds.
The secondary storage as we know for straight persistence and long term data storage capabilities.

Control Structures in Linux BASH

The Linux BASH provides many operators to check against test keyword
We are going to explain the Conditional comparison Operators for the Keyword parameter totest or [

For a detailed explanation of the Keyword test or [ Please refer to our Control Structure: Bash If then Else.

 

File Comparison Operators
Condition Explanation
-d “$file” Returns true if the file exists and is a directory
-e “$file” Returns true if the file exists.
-f “$file” Returns true if the file exists and is a regular file
-h “$file” Returns true if the file exists and is a symbolic link

 

String Comparators Condition  Explanation
“$str” = “$str2” True if string $str is equal to string $str2. Not best for integers.
“$str” != “$str2” True if the strings are not equal
-z “$str” True if length of string is zero
-n “$str” True if length of string is non-zero

 

Integer Comparators Condition Explanation
“$int1” -eq “$int2” True if the integers are equal
“$int1” -ne “$int2” True if the integers are not equals
“$int1” -gt “$int2” True if $int1 is greater than $int2
“$int1” -ge “$int2” True if $int1 is greater than or equal to $int2
“$int1” -lt “$int2” True if $int1 is less than $int2
“$int1” -le “$int2” True if $int1 is less than or equal to $int2

 

How to Remove Files and Directories in Linux using Command line

The Remove command in Unix/Linux ecosystem is very powerful and effective one time operation as it is unrecoverable.

To delete different kinds of content various options can be used with the rm command in linux.

In this section , we will demonstrate how to use the go about rm, unlink, and rmdir commands to remove files and directories in Linux

The Linux system treats the files and Directories by identifying them with Inodes.

Deleting a file is a simple operation but the user has to advise a lot of caution.

Sample Syntax

$ rm filename

Now lets Demonstrate deleting a file

Let’s use the -v flag to print the verbose information onto the screen.

[vamshi@linuxcent delete-dir]$ rm -v file1 file2 file3

removed ‘file1’

removed ‘file2’

removed ‘file3’

Using -i option asks the user for interaction and deletes the file upon accepting a “y” as user input.

[vamshi@linuxcent delete-dir]$ rm -i file3
rm: remove regular file ‘file3’? y

Deleting multiple files from the command line.

Using the wildcard * to delete similar extension filenames

[vamshi@linuxcent delete-dir]$ rm -v ./*.pdf

removed ‘./samplePDFfile1.pdf’

removed ‘./samplePDFfile2.pdf’

removed ‘./samplePDFfile3.pdf’

It’s a good practice to use (-v) verbose mode while running rm command.

Now let’s focus on the best practice

[vamshi@linuxcent ]$ alias rm="rm -iv"

Ensure to export the changes to your profile as demonstrated here(Exporting the changes to user login profile permanently)

[vamshi@linuxcent ]$ rm samplefilename.txt
rm: remove regular empty file ‘file10’? y
removed ‘samplefilename.txt’

But One has to always ensure to see the file permissions and if they are currently being used by any user or a program.(fuser and what it does)

You can long list the filename to see the permissions(long listing the files on Linux system)

How to Remove the Directory/s from Linux system?

A command-line utility rmdir exists on Linux to delete the directories. But, it deletes only the Empty directory(ies).

The command rm -d or --dir is also preferred to delete an empty directory

The rm command-line utility also lets you delete the directories and its contents recursively with -r option.

Sample Command Syntax:

[vamshi@linuxcent delete-dir]$ rmdir  sampleDIR1

We can use the -v option to print the verbose information:

[vamshi@linuxcent delete-dir]$ rmdir -v sampleDIR1

rmdir: removing directory, ‘sampleDIR1’

Deleting multiple subdirectories that are empty can be done with -p option, its demonstrated as follows:

[vamshi@linuxcent ]$ rmdir -v -p sampleDIR/subdir/
rmdir: removing directory, ‘sampleDIR/subdir/’
rmdir: removing directory, ‘sampleDIR’

Deleting multiple files from the command line.

[vamshi@linuxcent ]$ rmdir sampleDIR1 sampleDIR2 sampleDIR3

Now let’s shift our focus to rm command which offers the ability to delete the directories and its contents recursively.

[vamshi@linuxcent ]$ rm -v -r sampleDIR3/
removed directory: ‘sampleDIR3/subdir3’
removed ‘sampleDIR3/samplefile3.txt’
removed directory: ‘sampleDIR3/’
[vamshi@linuxcent delete-dir]$ rm -v -r sampleDIR6 sampleDIR7 sampleDIR8
removed directory: ‘sampleDIR6/subdir6’
removed directory: ‘sampleDIR6’
removed directory: ‘sampleDIR7/subdir7’
removed directory: ‘sampleDIR7’
removed directory: ‘sampleDIR8/subdir8’
removed directory: ‘sampleDIR8’

Please also see our post on How to make a file undeletable on Linux