What is Bash

What is Bash

Bash is a program (toolbox) used to interact between user and operating system.

The BASH acronym is for the ‘Bourne-Again SHell’. Bash is one of many shells, or command language interpreters, in Unix system.

I called bash a “toolbox”, because it contains large variety of shell tools, for example, tools for read files, calculate math operations, control devices…

The main difference between bash and any other program is that bash was not created to process certain task. Bash was developed to take commands from user using the text-based interface.

One of the advantages of bash is that users can put list of commands to a one (script) and execute these commands from file.

Most of bash scripts have ‘.sh’ extension, for example: script.sh. Linux is an extensionless system, but it is good practice to add .sh extension.

VI Basic Commands for UNIX in Nutshell

In this chapter, we will explore vi – the excellent Unix editor. There are many ways to edit text files in Unix; however, one of the best is using screen-oriented editors like vi who allow you to see context lines around a line that needs editing.

VIM, or Vi IMproved (commonly shortened to vim) is an improved version of the vi editor. It has rapidly grown in popularity because it can be used as both a command line and graphical interface text-editor with more features than standard vi that you might not find elsewhere like syntax highlighting and multitasking capabilities.

  • It’s usually available on all the flavors of Unix system.
  • Its implementations are very similar across the board.
  • It requires very few resources.
  • It is more user-friendly than other editors such as the ed or the ex.

VI Basic Commands for UNIX in Nutshell

Vi has 3 basic modes of operation: command (default), input, last line mode.

VI Editor Command mode

In command mode, you can run commands to search, copy, move, remove text.

VI Editor Input mode

In input mode, you can insert text into the file. Everything you type will be interpreted as text. many ways how to activate input mode (vi is case sensitive):

  • i – Inserts text before the cursor.
  • I – Inserts text at the beginning of the line.
  • o – Opens a new blank line below the cursor.
  • O – Opens a new blank line above the cursor.
  • a – Appends text after the cursor.
  • A – Appends text at the end of the line.

VI Editor Last line mode

To get into the last line mode type ‘:’ only from command mode. After type ‘:’ you will see a colon character appear at the beginning of the last line of your vi editor. It means vi is ready for type a “last line command”. To end vi type ‘q’ from last line mode.
You can return to command mode from input or last line mode pressing Esc.

Moving the Cursor

Key Cursor movement
w Forward one word.
b Back one word.
e To the end of the current word.
$ To the end of the line.
0(zero) To the beginning of the line.
^ To the first non-whitespace character on the line.
G Goes to the last line of the file.
IG Goes to the first line of the file.
Ctrl + F Pages forward one screen.
Ctrl + B Pages back one screen.
Ctrl + D Scrolls down one-half screen.
Ctrl + U Scrolls up one-half screen.
Ctrl + L Refreshes the screen.

Text-Deletion Commands

Command Function
R Overwrites or replaces characters on the line at and to the right of cursor. To terminate press Esc.
C Changes or overwrites characters from cursor to the end of the line.
s Substitutes a string for a character at the cursor.
x Deletes a character at the cursor.
dw Deletes a word or part of the word to the right of the cursor.
dd Deletes the line containing the cursor.
D Deletes the line from the cursor to the right end of the line.
:n, nd Deletes lines n-n. Example :2,80d deletes lines 2-80.

Text-Changing Commands

Command Function
cw Changes or overwrites characters at the cursor location to the end of that word.
r Replaces the character at the cursor with one other character.
J Join the current line and the line below.
xp Transposes the character at the cursor and the character to the right of the cursor.
~ Changes the case of the letter, either uppercase or lowercase, at the cursor.
u Undo the previous command.
. Repeats the previous command.

Text-Replacing Commands

Command Function
/string Searches forward for the string from the cursor.
?string Searches backward for the string.
n Searches for the next occurrence of the string. Use this command after searching for a string.
N Searches for the previous occurrence of the string. Use this command after searching for a string.
:%s/old/new/g Searches for the old string and replaces it iwth the new string globally.

Copy and Paste Commands

Command Function
yy Yanks a copy of the line
p Puts yanked or deleted text under the line containing the cursor.
P Put
:n,n co n Copies lines n-n and puts them after line n. Example: 1, 5 co 8 copies lines 1-5 and puts them after line 8.
:n,n m n Moves lines n-n to line n.
Example: 1,5 m 8 moves lines 1-5 to line 8.

File Save and Quit Commands

Command Function
:w Saves the file with changes by writing to the disk
:w new_file Writes the contents of the buffer to new_file.
wq Saves the changed file and quits editor vi.
😡 Saves the changed file and quits editor vi.
ZZ Saves the changed file and quits editor vi.
:q! Quits without saving changes.

Customizing vi Session

Command Function
:set nu Shows line numbers.
:set nonu Hides line numbers.
:set ic Instructs searches to ignore cases.
:set noic Instructs searches to be case-sensitive.
set list Display invisible characters.
:set showmode Display the current mode of operation.
:set noshowmode Turns off the mode of operation display.
:set Displays all the vi variables that are set.
:set all Display all vi variables and their values.

Customizing vi Session
To automatic customization for all vi sessions do the following steps:

  • Create a file in your home directory named ‘ . exrc’
  • Enter any of the set variables into the ‘ . exrc’ file.
  • Enter each ‘set variable’ command on one line.

Vi reads ‘exrc’ file every time before starting vi sessions.

Command Function
:set nu Shows line numbers.
:set nonu Hides line numbers.
:set ic Instructs searches to ignore case.
:set noic Instructs searches to be case-sensitive.
set list Display invisible characters.
:set showmode Display the current mode of operation.
:set noshowmode Turns off the mode of operation display.
:set Displays all the vi variables that are set.
:set all Display all vi variables and their values.

Redirect the Std error and std output 2>&1

The redirect operation > is used in conjunction with stdoutput 1 and stderr 2.

command > [/dev/null] 2>&1

2 Represents the stderror. The &1 here represents the first argument which is /dev/null

The character 2 represents the stderr which takes the entire errors printed to the screen and then appends them to the /dev/null which is the first argument represented by &1.

So the command demonstration will be the following:

$ du -sh /* > /dev/null 2>&1

This redirect command will dump the errors and the output to /dev/null.

Explanation: the default behaviour of redirection operator is to redirect the stdout and we are redirecting them to devnul and then we followup the command with 2>&1 which mentions the stderr 2 and then redirects is to /dev/null, which is denoted by &1 describing the &1 as the first argument which is /dev/null

 

 

Install java on Linux centos

In this tutorial we will quickly setup java on linux centos,

We will be using the yum command to download the openjdk 1.8 and install

[vamshi@node01 ~]$ sudo yum install java-1.8.0-openjdk.x86_64

We have installed the java openjdk 1.8 and we can check the version using java -version

[vamshi@node01 ~]$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

 

We make use of the alternatives command in centos which lists if we have any other version of java installed on the machine, and then enabling the default java version on the system wide.

[vamshi@node01 ~]$ alternatives --list | grep java
java auto /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/java
jre_openjdk auto /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
jre_1.8.0 auto /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
jre_1.7.0 auto /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre
[vamshi@node01 ~]$ sudo alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*  1           java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/java)
 + 2           java-1.7.0-openjdk.x86_64 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre/bin/java)

Enter to keep the current selection[+], or type selection number: 1

This enabled openjdk1.8 to be the default version of java.

Setting JAVA_HOME path
In order to set the system JAVA_HOME path on the system we need to export this variable, for the obvious reasons of other programs and users using the classpath such as while using maven or a servlet container.

Now there are two levels we can setup the visibility of JAVA_HOME environment variable.
1. Setup JAVA_HOME for single user profile
We need to update the changes to the ~/.bash_profile

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

PATH=$PATH:$JAVA_HOME

export PATH

Now we need enforce the changes with reloading the .bash_profile with a simple logout and then login into the system or we can source the file ~/.bash_profile as follows:

[vamshi@node01 ~]$ source .bash_profile

Verifying the changes:

[vamshi@node01 ~]$ echo $PATH
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/vamshi/.local/bin:/home/vamshi/bin:/home/vamshi/.local/bin:/home/vamshi/bin:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

2. Setup JAVA_HOME for the system wide profile and available to all the users.

[vamshi@node01 ~]$ sudo sh -c "echo -e 'export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/' > /etc/profile.d/java.sh"

This echo command writes the JAVA_HOME path to the system profile.d and creates a file java.sh which is read system wide level.

Ensure the changes are written to /etc/profile.d/java.sh

[vamshi@node01 ~]$ cat /etc/profile.d/java.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

Now source to apply the changes immediately to the file /etc/profile.d/java.sh as follows

[vamshi@node01 ~]$ sudo sh -c ' source /etc/profile.d/java.sh '

Or login to the root account and run the source command

Ensure to run the env command

[vamshi@node01 ~]$ env  | grep JAVA_HOME
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

How to allocate resources to docker images in Runtime?

We have seen the docker runtime environment takes up the overall available system resources on the system and tends to impact the base system.

To better utilize the containers, we can avail the resource cap and define the metric limits on specific containers while starting up the respective docker images..

The general syntax goes as follows:

# docker run --cpus ="x.x" --memory=x[M|G] docker-image

Here we see the demonstration

[root@node01 ~]# docker run --cpus="0.2" --memory="200M" jenkins:latest
/usr/bin/docker-current: Error response from daemon: Minimum memory limit allowed is 4MB.

It should be noticed that the minimum Memory limit allowed for the docker container to run is 4MB and the minimum CPU cores is at 0.01, any thing lower that this means the container runtime fails to allocate sufficient resources. These limits will be efficient when running on some test and debug scenarios.

How to login and connect to docker container tty with a username

How to login to the docker through commandline?

The docker exec is similar to running a tty on a linux machine. the only difference is it can accept many more exec connections, Although you can enable ssh to the docker but it will only be possible while running an openssh server inside the running container.

You can exec to connect to a running container much like the ssh into the running machine, Here we use the special options -i or --interactive to provide interaction and -t or --tty which allocates a pseudo tty terminal of the type of shell that is followed
The syntax of docker exec is as below:

docker exec -it <container-id | container-name> bash
[vamshi@node01 ~]$ docker exec -it b511234ebe31 bash
jenkins@b511234ebe31:/$ id
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins)

We know that the docker exec command is used opens up a tty terminal to connect to the running docker container, but It will connect us with the Default USER that was activated during it docker build.

We can use the [/code]-u[/code] flag to connect to the container with the username that is enabled for that image, We see the below example

[vamshi@node01 ~]# docker exec -it  -u root b511234ebe31 bash
root@b511234ebe31:/# pwd
/
root@b511234ebe31:/# id
uid=0(root) gid=0(root) groups=0(root)

 

 

Buildah – An alternate Docker build tool

What is Buildah? Is it a better Docker alternate?

Used to build container images as per OCI(Open Container Initiative) standard which provides API to build Images in docker format using golang.

The Buildah project provides a command line tool that be used to create an OCI or traditional Docker image format image and to then build a working container from the image. The container can be mounted and modified and then an image can be saved based on the updated container.

Podman – Used to manage the pulling, tagging the images similar to Docker based on OCI standard.
To be used for Production grade deployments. It also allows you to create, run, and maintain containers created from those images.

Buildah Concepts:

  • Buildah’s commands replicate all of the commands that are found in a Dockerfile.
  • Also has the feasibility to build docker images with Dockerfile while not requiring root privileges while providing lowerlevel coreutils interface to build images
  • Buildah uses simple fork-exec model and does not run as a daemon but it is based on a comprehensive API in golang.
  • Buildah uses runcrun commands when buildah run used, or when buildah build-using-dockerfile encounters a RUN instruction. And it is namespace dependent on the Host System
  • Buildah relies on the CNI library and plugins to set up interfaces and routing for network namespaces
  • Buildah on Red Hat Enterprise Linux or CentOS, version 7.4 or higher is required. On other Linux distributions Buildah requires a kernel version of 4.0 or higher in order to support the OverlayFS filesystem – https://github.com/containers/buildah/blob/master/install.md
  • The dependencies required as btree filesystem, containernetworking-cni, golang and bz2 and other common-container tools.

Buildah Configuration Files and its Directory Structure:

# /usr/share/containers/

# registries.conf
# mounts.conf
# seccomp.json
# policy.json

Git config setup on linux; Unable to pull or clone from git; fatal: unable to access git; Peer’s Certificate has expired

Facing an issue with pulling the repository while dealing with an expired SSL certificate.

[vamshi@workstation ~]$ git pull https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
fatal: unable to access 'https://gitlab.linuxcent.com/linuxcent/pipeline-101.git/': Peer's Certificate has expired.
[vamshi@workstation ~]$

SSL error while cloning git URL

If you have faced the error, then we can work around it by ignoring SSL certificate check and continue working with the git repo.

[vamshi@workstation ~]$ git clone https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
Cloning into 'pipeline-101'...
fatal: unable to access 'https://gitlab.linuxcent.com/linuxcent/pipeline-101.git/': Peer's Certificate has expired.

It doesn’t allow the clone or pull or push to the gitlab website as its certificate is not valid, and the certificate is unsigned by a Valid CA. In most cases, we will have the corporate gitlab repo in our internal network and not publicly exposed.
We therefore trust the gitlab server as we have a bunch of our code on it.. Why not, I say?
We have to ensure to disable the check for the SSL certificate verification

Set the Variable GIT_SSL_NO_VERIFY=1 or GIT_SSL_NO_VERIFY=false and try to execute your previous command.

[vamshi@workstation ~]$ GIT_SSL_NO_VERIFY=1 git clone https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
Cloning into 'pipeline-101'...
Username for 'https://gitlab.linuxcent.com': vamshi
Password for 'https://vamshi@gitlab.linuxcent.com': 
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.

Make a permanent entry to system wide user level profiles as below. The following change works at the system level

[vamshi@workstation ~]$ sudo bash -c "echo -e export GIT_SSL_NO_VERIFY=1 > /etc/profile.d/gitconfig.sh "
[vamshi@workstation ~]$ cat /etc/profile.d/gitconfig.sh
export GIT_SSL_NO_VERIFY=false

The practical use case of setting the environment variable can be made while building container images, using  the GIT_SSL_NO_VERIFY false as an environment variable in Dockerfile and building an image.

[vamshi@workstation ~]$ cat Dockerfile
FROM jetty:latest
-- CONTENT TRUNCATED --
env GIT_SSL_NO_VERIFY 1
-- CONTENT TRUNCATED --

We can also setup the container build agent with Jenkins Pipeline code with similar configuration to fetch a gitrepo in our next sessions.

Time and date setup in Linux systemd with timedatectl

The Linux command timedatectl is a systemd linux command that enables the Linux system admin to configure the date and time effectively.

The timedatectl is an important and a handy tool in linux to configure the time and date of the system, especially in a cluster setup where the synchronization is essential.

[vamshi@node01 .ssh]$ sudo timedatectl
Local time: Wed 2020-04-01 13:53:03 UTC
Universal time: Wed 2020-04-01 13:53:03 UTC
RTC time: Sun 2020-04-26 14:26:09
Time zone: UTC (UTC, +0000)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
List the timezones from commandline

[vamshi@node01 ~]$ sudo timedatectl list-timezones

How to sync the server to with ntp time sync using timedatectl ?

You have to ensure that the ntp in installed and running and then automatically the time sync will be enforced with the following command.

# sudo timedatectl set-ntp true

The manual time set can be performed using timedatectl as follows:

[vamshi@node01 ~]$ sudo timedatectl set-time 17:53:03
[vamshi@node01 ~]$ date
Wed Apr 1 17:53:04 UTC 2020

The best practice and practical approach is to set the timesync on with ntpd process enabled and in running state as show below:

# sudo systemctl enable ntpd --now

check linux version

The Linux OS has many Distributions and various versions with slight modifications in the kernel versions, a major and minor version identification is crucial to practical administration of a linux server. Linux being open source, the release cycles are continuous resulting in many changes with Longterm release and short term release cycles in some most known and popular linux Distributors like Centos/Debian.. A special mentioning of the generic Desktop PC/laptop based versions of fedora/Ubuntu/Arch/OpenSuse Leap-Tumbleweed Linux where fan following is heavy, the Release cycles are very aggressive with new versions releasing once in for every 2-3 weeks.

In this Section we will see how to check linux version, on most popular Distributions.
This will be handy during the kernel patch process to identify the current linux version.
Lets list our some practical linux commands and scenarios

Checking the Linux OS Version using /etc/os-release

[vamshi@node02 ]$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

Checking the Linux Version from the file /etc/issue

This file is being relinquished in latest Systemd variants of Linux, But tends to gives a less adequate information just lying around for aged linux user’s familiarity with systemd versions.

vamshi@node03:/$ cat /etc/issue
Debian GNU/Linux 10 \n \l

Check Linux Version using lsb_release:

The lsb_release prints the (Linux Standard Base) and Distribution information on the Linux host.

[vamshi@node02 cp-command]$ lsb_release -a
LSB Version:    :core-4.1-ia32:core-4.1-noarch
Distributor ID:    CentOS
Description:    CentOS Linux release 7.7.1908 (Core)
Release:    7.7.1908
Codename:    Core

How to check the running kernel version information ?

Check Linux Version information with uname command

[vamshi@node02 cp-command]$ uname -a
Linux node02.linuxcent.com 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Checking the Linux Version using hostnamectl

The command hostnamectl gives the complete information of the underlying architecture, kernel version and the Linux OS name

vamshi@node03:/$ hostnamectl 
Static hostname: node03
Icon name: computer-vm
Chassis: vm
Machine ID: b4adcdb84c724856b577524ebbfa0003
Boot ID: 1a2db0a0ae8b4c5ba86b390c68af7024
Virtualization: oracle
Operating System: Debian GNU/Linux 10 (buster)
Kernel: Linux 4.19.0-5-amd64
Architecture: x86-64

How to create Temporary filesystem on Linux

The greatest advantage with a tmpfs is that you can use the faster access Volatile memory to store files instead of Secondary storage system.

Lets head off to the Demonstration using tmpfs.

Create a Directory named test-docs in /mnt

[vamshi@SERVER02 mnt]$ sudo mkdir /mnt/test-docs

We will be using the dd command to dump the data and create a file with given Block size, as below and compare the write speeds on tmpfs (Linux temporary Filesystem) vs conventional storage filesystem.

[vamshi@SERVER02 mnt]$ sudo mount -t tmpfs -o size=1G tmpfs /mnt/test-docs

Output from the mount command:

[vamshi@SERVER02 mnt]$ df -hT /mnt/test-docs/
Filesystem     Type   Size Used Avail Use% Mounted on
tmpfs          tmpfs  1.0G     0  1.0G   0% /mnt/test-docs

Now add the entry to /etc/fstab to make the mount persistent across reboots(We should write How to create an fstab entry in another article)

tmpfs       /mnt/webdocs tmpfs   nodev,nosuid,noexec,nodiratime,size=1G   0 0

With these changes it is going to perform faster system read writes during the runtime.

Please advise caution as this uses the RAM.

Now to test the setting Temporary Filesystem vs Storage Filesystem write speeds

We will demonstrate see about the temporary(tmpfs) filesystem on linux in action and practical examples.

Writing a 500MB file on the same server to a Secondary storage vs Writing it to tmpfs.

Lets us write a single 500MB file and measure its write speed and time it’s operation

[vamshi@SERVER02 mnt]$ time sudo dd if=/dev/zero of=/mnt/test-docs/dump.txt bs=1k count=500000
500000+0 records in
500000+0 records out
512000000 bytes (512 MB) copied, 0.815591 s, 628 MB/s

real    0m0.871s
user    0m0.103s
sys    0m0.760s

 

This operation finished under 0.8 seconds time which is around 100 milliseconds lesser than 1 second and write operation completed with 628MB/s.

Lets check on the file size on the tmpfs mountpoint /mnt/test-docs/

[vamshi@SERVER02 mnt]$ df -hT /mnt/test-docs/
Filesystem     Type   Size Used Avail Use% Mounted on
tmpfs          tmpfs  1.0G 489M  536M 48% /mnt/test-docs

Now lets run the same write operation to the secondary storage filesystem

[vamshi@SERVER02 mnt]$ time sudo dd if=/dev/zero of=/mnt/dump.txt bs=1k count=500000
500000+0 records in
500000+0 records out
512000000 bytes (512 MB) copied, 1.38793 s, 369 MB/s

real    0m1.510s
user    0m0.028s
sys    0m1.227s

The write operation to the disk took twice the time at a write speed of 369MB/s

The space usage on the [/code]/mnt/test-docs/[/code] after the operation is 489MB out of 1024MB.

[vamshi@SERVER02 mnt]$ df -hT /mnt/test-docs/
Filesystem     Type   Size Used Avail Use% Mounted on
tmpfs          tmpfs  1.0G 489M  536M 48% /mnt/test-docs

CONCLUSION:

The write/read speeds to tmpfs were substantially faster compared to the secondary storage block device mount points, The fact of the matter is tmpfs although the clear winner is tmpfs, it cannot be used for persistent data storage as its just sits on top of the memory and in no way offers long term data storage persistence.

So what is a practical usage of tmpfs in linux ?

The practical usage can be storing of the static web content to serve the images,css and js to fasten the request speeds.
The secondary storage as we know for straight persistence and long term data storage capabilities.