Linux rsync command

Linux Command Utility [/code]rsync[/code] is a very robust, fast content copy command which can be used within the same linux host and over a connected network between 2 linux hosts. It is a special program which has intelligence in terms of not copying data repetitively if the destination has the same copy as source based on file checksum calculations.

We shall explore some of the practical rsync command features and demonstrate them

Syntax of command

$ rsync [OPTIONS] /source/path/ /dest/path/

Running rsync on the same host?

[vamshi@linuxcent ~]$ rsync -avx newfile.txt /tmp/
sending incremental file list
newfile.txt

sent 133 bytes  received 35 bytes  336.00 bytes/sec
total size is 21  speedup is 0.12

Running rsync between two hosts in a network

$ rsync [OPTIONS] host:/source/path/ /dest/path/

Run rsync in Dry-run mode by using [/code]-n[/code] option

$ rsync -avn Source_host:/source/path Destination_host:/dest/path

This generally runs over the SSH protocol and you are required to enter the login credentials appropriately.

How to invoke SSH remote shell in rsync?

In case you are using a SSH keys then you have to invoke the remote shell to authenticate to the remote server with your private keypair. This is Demonstrated as follows:

$ rsync -avxn --rsh="ssh -i ~/.ssh/vamshi_id_rsa" vamshi@<Your.Source.IP.DNS>:"/<Source_Path>" "/<Destination_Path>"

For more information about the SSH key setup, Please refer to our SSH keys section

How to persist Hard links on the system using rsync. Following is the Demonstration

Flag : -H. Using this option enables to preserve the HardLinks over the destination copy of the data.

$ rsync -avHx /path/to/source/ /path/to/destination/

The most practical example of working with rsync comes in replicating mission critical data or transferring Database dumps within the DB servers etc.,

How to exclude certain directories in rsync in linux ?

using --exclude filter option is demonstrated as follows:

$ rsync -avx /source/path/to/backup-v31/ /dest/databackups/backup-v31/ --exclude="DontTouchMyData/"

Using the delete option, enables us to delete the directories from the source upon completion of the operation.

Note: This operation has the same effects as the mv command on linux but performed over the network between source and destination hosts.

$ rsync -avx --delete /source/path/to/backup-v31/ /dest/databackups/backup-v31/ --exclude="data/" --exclude="data/board" --exclude="cache/apt" --exclude="opt"

Redirect the rsync output to a file by appending output redirection symbol to a file on current location

$rsync -avx --delete /source/path/to/backup-v31/ /dest/databackups/backup-v31/ --exclude="data/" --exclude="data/board" --exclude="cache/apt" --exclude="opt"  >>/tmp/rsync.log

What is rsync command Linux?

rsync or remote synchronization is a software utility for Unix-Like systems that efficiently sync files and directories between two hosts or machines. … Copying/syncing to/from another host over any remote shell like ssh, rsh.

How do I use rsync in Linux?

Syntax of rsync command:

  • -v, u2013verbose Verbose output.
  • -q, u2013quiet suppress message output.
  • -a, u2013archive archive files and directory while synchronizing ( -a equal to following options -rlptgoD)
  • -r, u2013recursive sync files and directories recursively.
  • -b, u2013backup take the backup during synchronization.

What is rsync in bash?

rsync is a fast and versatile command-line utility for synchronizing files and directories between two locations over a remote shell, or from/to a remote Rsync daemon. … Rsync can be used for mirroring data, incremental backups, copying files between systems, and as a replacement for scp , sftp , and cp commands.

How do I transfer files using rsync?

You can use SecureShell (SSH) or Remote Sync (Rsync) to transfer files to a remote server. Secure Copy (SCP) uses SSH to copy only the files or directories that you select. On first use, Rsync copies all files and directories and then it copies only the files and directories that you have changed.

What is rsync command do?

Rsync is typically used for synchronizing files and directories between two different systems. For example, if the command rsync local-file user@remote-host:remote-file is run, rsync will use SSH to connect as user to remote-host.

Why do we use rsync?

Syntax of rsync command:

  • -v, u2013verbose Verbose output.
  • -q, u2013quiet suppress message output.
  • -a, u2013archive archive files and directory while synchronizing ( -a equal to following options -rlptgoD)
  • -r, u2013recursive sync files and directories recursively.
  • -b, u2013backup take the backup during synchronization.

What is rsync in RHEL?

Rsync can be used to quickly move large amounts of data to both local and remote destinations. For this reason, rsync is often used to copy data, make backups, migrate hosts, and bridge the gap between site staging and production environments.

How does rsync work in Linux?

An rsync process operates by communicating with another rsync process, a sender and a receiver. At startup, an rsync client connects to a peer process. If the transfer is local (that is, between file systems mounted on the same host) the peer can be created with fork, after setting up suitable pipes for the connection.

How do I rsync a file in Linux?

Copy a single file locally If you want to copy a file from one location to another within your system, you can do so by typing rsync followed by the source file name and the destination directory. Note: Instead of u201c/home/tin/file1. txtu201d, we can also type u201cfile1u201d as we are currently working in the home directory.

How do I use rsync?

You can use secure shell (SSH) or Remote Sync (Rsync) to transfer files to a remote server. Secure Copy (SCP) uses SSH to copy only the files or directories that you select. On first use, Rsync copies all files and directories and then it copies only the files and directories that you have changed.

How to access Tar files in Linux/Unix

The Linux tar command abbreviation is “tar archive” released under POSIX standards initially and It now follows the GNU standards

Often when you download the files from the internet they are in the .tar, tar.gz. Or tar.bz2 compressed format using either bz2 or gz compression algorithm.

If you are familiar with the Opensource then there’s a good chance that you would have come across the extensions like Package-name.tar, tar.gz, tar.bz, tar.xz which are standard.

Well most open source software use tarballs to distribute programs/source codes in this format as it offers efficient compression and better organized way of grouping files.

It supports a vast range of compression programs such as gzip, bzip2, xz, lzip, lzma, lzop.

In the following tutorial we will show how to Compress/Extract the files into tar.gz or tgz.

How to specify the format POSIX Or GNU while using tar Command?

[vamshi@linuxcent ]$ tar --format=posix -cf posix.tar *
[vamshi@linuxcent ]$  tar --format=gnu -cf gnu.tar *
[vamshi@linuxcent ]$  file posix.tar gnu.tar 
posix.tar: POSIX tar archive
gnu.tar: POSIX tar archive (GNU)

GNU is based on an older POSIX format, So that’s why it says POSIX for both.

To print the contents information of an archived directory, use the directory name as a file name argument in conjunction with --list (-t). To find out file attributes, include the --verbose (-v) option.

Firstly start off by long listing ls in the present directory we have here:

vamshi@LinuxCent:~/Linux> ls
Debian-Distro/  OpenSuse-Distro/ README  Redhat-Distro/

We will now compress the present working directory using Linux GNU’s tar command line utility.. Creating a tar of the current directory mentioned by Asterisk *. The Options -c creates, -v: Verbose mode, -z: Uses GZIP algorithm
Demonstration shown below:
Either one of the following commands can be used

vamshi@LinuxCent:~/Linux> tar --gzip -cvf Linux-flavours.tar.gz *
vamshi@LinuxCent:~/Linux> tar -cvzf Linux-flavours.tar.gz *
Debian-Distro/
Debian-Distro/ubuntu.txt
Debian-Distro/debian.txt
Debian-Distro/README-Debian-Distro
Linux-flavours.tar.gz
OpenSuse-Distro/
OpenSuse-Distro/README-Opensuse-Distro
OpenSuse-Distro/opensuse.txt
README
Redhat-Distro/
Redhat-Distro/Fedora/
Redhat-Distro/Fedora/fedora.txt
Redhat-Distro/Centos/
Redhat-Distro/Centos/centos.txt
Redhat-Distro/Centos/CentOS-versions/
Redhat-Distro/Centos/CentOS-versions/centos7.txt
Redhat-Distro/Centos/CentOS-versions/centos5.5.txt
Redhat-Distro/Centos/CentOS-versions/centos6.9.txt
Redhat-Distro/Centos/CentOS-versions/centos5.8.txt
Redhat-Distro/Centos/CentOS-versions/centos6.1.txt
Redhat-Distro/Centos/README-CentOS
Redhat-Distro/README-Redhat-Distro
Redhat-Distro/RHEL-Versions/
Redhat-Distro/RHEL-Versions/redhat5.txt
Redhat-Distro/RHEL-Versions/redhat7.txt
Redhat-Distro/RHEL-Versions/redhat6.txt
Redhat-Distro/RHEL-Versions/redhat8.txt
Redhat-Distro/redhat.txt

How to List/view Archives

The option-t does a Dry-run of extract operation but only to print the contents of the mentioned compression.

vamshi@LinuxCent:~/Linux> tar -tvf Linux-flavours.tar.gz
Debian-Distro/
Debian-Distro/ubuntu.txt
Debian-Distro/debian.txt
Debian-Distro/README-Debian-Distro
…

How to Extracting tar.gz File?

Extracting the tar file using the options -x works out just fine as -x option chooses the tye of decompression based on the compression file type, and the content will be extracted to current working directory.

Here are various extraction options:

$ tar -xvf Linux-flavours.tar.gz
$ tar -zxvf Linux-flavours.tar.gz
$ tar --gzip -xvf Linux-flavours.tar.gz

The Filter / Options for compression types:
-z or --gzip : Used for Archival operation for .gzip type
-j -r --bzip2: Used for archival operation for .bzip2 type
[/code]-J[/code] or --xz: User for Archival for .xz type

How to Extract Only Specific Files from a compressed tar archive (tar) File

This doesn’t require any special option but you have to name the exact file and directory that you want to extract

[vamshi@linuxcent ~]$ tar -zxvf redmine-4.0.6.tar.gz redmine-4.0.6/bin/about
redmine-4.0.6/bin/about

Successful in extracting only the redmine-4.0.6/bin/about file from the redmine-4.0.6.tar.gz archive.

[vamshi@linuxcent ~]$ ls -l redmine-4.0.6/bin/about 
-rwxrwxr-x. 1 vamshi vamshi 167 Dec 20 11:46 redmine-4.0.6/bin/about

How to Extract specific files or Directories from the archive

vamshi@LinuxCent:/tmp/linux-test> tar -zxvf Linux-flavours.tar.gz Redhat-Distro/Centos/
Redhat-Distro/Centos/
Redhat-Distro/Centos/centos.txt
Redhat-Distro/Centos/CentOS-versions/
Redhat-Distro/Centos/CentOS-versions/centos7.txt
Redhat-Distro/Centos/CentOS-versions/centos5.5.txt
Redhat-Distro/Centos/CentOS-versions/centos6.9.txt
Redhat-Distro/Centos/CentOS-versions/centos5.8.txt
Redhat-Distro/Centos/CentOS-versions/centos6.1.txt
Redhat-Distro/Centos/README-CentOS

 

This way we are able to extract only the specific directories pertaining to Centos from Redhat-Distro.
Now we would do the extraction of the contents on Redhat-Distro and its sub directories.
Extracting Sub-directories inside the compressed tar.gz

How to Extract TAR file Contents to a New directory?

The tar program by default, extracts the archive contents to the present working directory.
By specifying the option --directory (-C) You can extract archive files in a Target directory.

vamshi.santhapuri@LinuxCent:/tmp/linux-test1> tar -xzvf ~/Linux/jdk-8u101-linux-x64.tar.gz  -C /usr/local/

The above operation extracts the java bundle to /usr/local/ directory.

How to extract only the specific directory from the Compression to a Target directory?

vamshi.santhapuri@LinuxCent:/tmp/linux-test1> tar -xzvf ~/Linux/Linux-flavours.tar.gz Redhat-Distro/RHEL-Versions/ -C /tmp/linux-test
Redhat-Distro/RHEL-Versions/
Redhat-Distro/RHEL-Versions/redhat5.txt
Redhat-Distro/RHEL-Versions/redhat7.txt
Redhat-Distro/RHEL-Versions/redhat6.txt
Redhat-Distro/RHEL-Versions/redhat8.txt

How to tar compress the specific Files and Directory and Sub-directories / Multiple Directories using tar command?

Below is the Demonstration of compression program using gz compression.

vamshi@LinuxCent:~/Linux> tar -cvzf Redhat-Distro/ Linux-flavours.tar.gz centos.txt fedora.txt opensuse.txt redhat.txt ubuntu.txt
centos.txt
fedora.txt
opensuse.txt
redhat.txt
Redhat-Distro/centos.txt
Redhat-Distro/fedora.txt
Redhat-Distro/redhat.txt
Redhat-Distro/Redhat-Versions/
Redhat-Distro/Redhat-Versions/redhat5.txt
Redhat-Distro/Redhat-Versions/redhat7.txt
Redhat-Distro/Redhat-Versions/redhat6.txt
Redhat-Distro/Redhat-Versions/redhat8.txt

You can also compress multiple directories and files into a single tar.gz file as demonstrated below

How to exclude particular directories and file from the compression using tar command?

We can make use of the --exclude="DontIncludethisPath" in Linux tar command, where in the base directory of mentioned pattern are excluded.. Lets run the tar command and see the results

vamshi@linuxCent:~/Linux/OSes> tar -czvf exclude-flavours.tar.gz --exclude="Redhat-Distro" .
./
./OpenSuse-Distro/
./OpenSuse-Distro/README-Opensuse-Distro
./OpenSuse-Distro/opensuse.txt
./Debian-Distro/
./Debian-Distro/ubuntu.txt
./Debian-Distro/debian.txt
./Debian-Distro/README-Debian-Distro
tar: .: file changed as we read it

The compression successfully completed and now lets list out the files

vamshi@linuxCent:~/Linux/OSes> ls -lthr
total 16K
drwxr-xr-x 2 vamshi users 4.0K Apr  8 14:09 OpenSuse-Distro
drwxr-xr-x 2 vamshi users 4.0K Apr  8 14:09 Debian-Distro
drwxr-xr-x 5 vamshi users 4.0K Apr  8 14:09 Redhat-Distro
-rw-r--r-- 1 vamshi users  718 Apr 8 19:18 exclude-flavours.tar.gz

From the dry-run extract output of the compression does not contain the Directory Redhat-Distro

vamshi@linuxCent:~/Linux/OSes> tar -tvf exclude-flavours.tar.gz
drwxr-xr-x vamshi/users 0 2020-04-08 19:18 ./
drwxr-xr-x vamshi/users 0 2020-04-08 14:09 ./OpenSuse-Distro/
-rw-r--r-- vamshi/users 0 2020-04-08 14:09 ./OpenSuse-Distro/README-Opensuse-Distro
-rw-r--r-- vamshi/users 9 2020-04-08 13:22 ./OpenSuse-Distro/opensuse.txt
drwxr-xr-x vamshi/users 0 2020-04-08 14:09 ./Debian-Distro/
-rw-r--r-- vamshi/users 7 2020-04-08 13:22 ./Debian-Distro/ubuntu.txt
-rw-r--r-- vamshi/users 7 2020-04-08 13:50 ./Debian-Distro/debian.txt
-rw-r--r-- vamshi/users 0 2020-04-08 14:09 ./Debian-Distro/README-Debian-Distro

How do I open a tar file in Linux?

tar Utility

  1. Let’s consider that we want to extract and open a doc file, and then we can use the below command to unzip the
  2. file on Linux:
  3. tar –xvzf doc.tar.gz. Remember that the tar. …
  4. tar –cvzf docs.tar.gz ~/Documents. …
  5. tar -cvf documents.tar ~/Documents. …
  6. tar –xvf docs.tar. …
  7. gzip xyz.txt. …
  8. gunzip test.txt. …
  9. gzip *.txt.

How do I view the contents of a tar file in Unix?

tar File Content. Use -t switch with tar command to list content of a archive. tar file without actually extracting.

How do I view a tar file?

How to open TAR files

  • Download and save the TAR file to your computer. …
  • Launch WinZip and open the compressed file by clicking File > Open. …
  • Select all of the files in the compressed folder or select only the files you want to extract by holding the CTRL key and left-clicking on them.

Where can I find tar files in Linux?

Combining find and tar commands so that we can find and tar files into a tarball

  1. -name “*. doc” : Find file as per given pattern/criteria. In this case find all *. doc files in $HOME.
  2. -exec tar … : Execute tar command on all files found by the find command.

How do I extract a tar file?

To extract (unzip) a tar. gz file simply right-click on the file you want to extract and select “Extract”. Windows users will need a tool named 7zip to extract tar.

How do I unzip a tar file in Terminal?

The most common uses of the tar command are to create and extract a tar archive. To extract an archive, use the tar -xf command followed by the archive name, and to create a new one use tar -czf followed by the archive name and the files and directories you want to add to the archive.

How do I extract a tar file from a directory in Linux?

Syntax For Tar Command To Extract Tar Files To a Different Directory

  1. x : Extract files.
  2. f : Tar archive name.
  3. –directory : Set directory name to extract files.
  4. -C : Set dir name to extract files.
  5. -z : Work on . tar. …
  6. -j : Work on . tar. …
  7. -J (capital J ) : Work on . tar. …
  8. -v : Verbose output i.e. show progress on screen.

Can 7 Zip open tar files?

7-Zip can also be used to unpack many other formats and to create tar files (amongst others). Download and install 7-Zip from 7-zip.org. … Move the tar file to the directory you wish to unpack into (usually the tar file will put everything into a directory inside this directory).

What is tar file in Linux?

The Linux ‘tar’ stands for tape archive, is used to create Archive and extract the Archive files. tar command in Linux is one of the important command which provides archiving functionality in Linux. We can use Linux tar command to create compressed or uncompressed Archive files and also maintain and modify them.

Install java on Linux centos

In this tutorial we will quickly setup java on linux centos,

We will be using the yum command to download the openjdk 1.8 and install

[vamshi@node01 ~]$ sudo yum install java-1.8.0-openjdk.x86_64

We have installed the java openjdk 1.8 and we can check the version using java -version

[vamshi@node01 ~]$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

 

We make use of the alternatives command in centos which lists if we have any other version of java installed on the machine, and then enabling the default java version on the system wide.

[vamshi@node01 ~]$ alternatives --list | grep java
java auto /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/java
jre_openjdk auto /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
jre_1.8.0 auto /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
jre_1.7.0 auto /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre
[vamshi@node01 ~]$ sudo alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*  1           java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/java)
 + 2           java-1.7.0-openjdk.x86_64 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre/bin/java)

Enter to keep the current selection[+], or type selection number: 1

This enabled openjdk1.8 to be the default version of java.

Setting JAVA_HOME path
In order to set the system JAVA_HOME path on the system we need to export this variable, for the obvious reasons of other programs and users using the classpath such as while using maven or a servlet container.

Now there are two levels we can setup the visibility of JAVA_HOME environment variable.
1. Setup JAVA_HOME for single user profile
We need to update the changes to the ~/.bash_profile

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

PATH=$PATH:$JAVA_HOME

export PATH

Now we need enforce the changes with reloading the .bash_profile with a simple logout and then login into the system or we can source the file ~/.bash_profile as follows:

[vamshi@node01 ~]$ source .bash_profile

Verifying the changes:

[vamshi@node01 ~]$ echo $PATH
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/vamshi/.local/bin:/home/vamshi/bin:/home/vamshi/.local/bin:/home/vamshi/bin:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

2. Setup JAVA_HOME for the system wide profile and available to all the users.

[vamshi@node01 ~]$ sudo sh -c "echo -e 'export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/' > /etc/profile.d/java.sh"

This echo command writes the JAVA_HOME path to the system profile.d and creates a file java.sh which is read system wide level.

Ensure the changes are written to /etc/profile.d/java.sh

[vamshi@node01 ~]$ cat /etc/profile.d/java.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

Now source to apply the changes immediately to the file /etc/profile.d/java.sh as follows

[vamshi@node01 ~]$ sudo sh -c ' source /etc/profile.d/java.sh '

Or login to the root account and run the source command

Ensure to run the env command

[vamshi@node01 ~]$ env  | grep JAVA_HOME
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/

How do I download and install Java on CentOS?

Install Java On CentOS

  1. Install OpenJDK 11. Update the package repository to ensure you download the latest software: sudo yum update. …
  2. Install OpenJRE 11. Java Runtime Environment 11 (Open JRE 11) is a subset of OpenJDK. …
  3. Install Oracle Java 11. …
  4. Install JDK 8. …
  5. Install JRE 8. …
  6. Install Oracle Java 12.

Is Java installed on CentOS?

OpenJDK, the open-source implementation of the Java Platform, is the default Java development and runtime in CentOS 7. The installation is simple and straightforward.

How do I install Java on Linux?

  • Java for Linux Platforms
  • Change to the directory in which you want to install. Type: cd directory_path_name. …
  • Move the . tar. gz archive binary to the current directory.
  • Unpack the tarball and install Java. tar zxvf jre-8u73-linux-i586.tar.gz. The Java files are installed in a directory called jre1. …
  • Delete the . tar.

How do I install latest version of Java on CentOS?

To install OpenJDK 8 JRE using yum, run this command: sudo yum install java-1.8. 0-openjdk.

Where is java path on CentOS?

They usually reside in /usr/lib/jvm . You can list them via ll /usr/lib/jvm . The value you need to enter in the field JAVA_HOME in jenkins is /usr/lib/jvm/java-1.8.

How do I know if java is installed on CentOS 7?

  • To check the Java version on Linux Ubuntu/Debian/CentOS:
  • Open a terminal window.
  • Run the following command: java -version.
  • The output should display the version of the Java package installed on your system. In the example below, OpenJDK version 11 is installed.

Where is java path set in Linux?

Steps

  • Change to your home directory. cd $HOME.
  • Open the . bashrc file.
  • Add the following line to the file. Replace the JDK directory with the name of your java installation directory. export PATH=/usr/java/<JDK Directory>/bin:$PATH.
  • Save the file and exit. Use the source command to force Linux to reload the .

How do I install java 14 on Linux?

Installing OpenJDK 14

  • Step 1: Update APT. …
  • Step 2: Download and Install JDK Kit. …
  • Step 3: Check Installed JDK Framework. …
  • Step 4: Update Path to JDK (Optional) …
  • Step 6: Set Up Environment Variable. …
  • Step 7: Open Environment File. …
  • Step 8: Save Your Changes.

How do I know where java is installed on Linux?

This depends a bit from your package system … if the java command works, you can type readlink -f $(which java) to find the location of the java command. On the OpenSUSE system I’m on now it returns /usr/lib64/jvm/java-1.6. 0-openjdk-1.6. 0/jre/bin/java (but this is not a system which uses apt-get ).

How do I install java 11 on Linux?

Installing the 64-Bit JDK 11 on Linux Platforms

  1. Download the required file: For Linux x64 systems: jdk-11. interim. …
  2. Change the directory to the location where you want to install the JDK, then move the . tar. …
  3. Unpack the tarball and install the downloaded JDK: $ tar zxvf jdk-11. …
  4. Delete the . tar.

Git config setup on linux; Unable to pull or clone from git; fatal: unable to access git; Peer’s Certificate has expired

Facing an issue with pulling the repository while dealing with an expired SSL certificate.

[vamshi@workstation ~]$ git pull https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
fatal: unable to access 'https://gitlab.linuxcent.com/linuxcent/pipeline-101.git/': Peer's Certificate has expired.
[vamshi@workstation ~]$

SSL error while cloning git URL

If you have faced the error, then we can work around it by ignoring SSL certificate check and continue working with the git repo.

[vamshi@workstation ~]$ git clone https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
Cloning into 'pipeline-101'...
fatal: unable to access 'https://gitlab.linuxcent.com/linuxcent/pipeline-101.git/': Peer's Certificate has expired.

It doesn’t allow the clone or pull or push to the gitlab website as its certificate is not valid, and the certificate is unsigned by a Valid CA. In most cases, we will have the corporate gitlab repo in our internal network and not publicly exposed.
We therefore trust the gitlab server as we have a bunch of our code on it.. Why not, I say?
We have to ensure to disable the check for the SSL certificate verification

Set the Variable GIT_SSL_NO_VERIFY=1 or GIT_SSL_NO_VERIFY=false and try to execute your previous command.

[vamshi@workstation ~]$ GIT_SSL_NO_VERIFY=1 git clone https://gitlab.linuxcent.com/linuxcent/pipeline-101.git
Cloning into 'pipeline-101'...
Username for 'https://gitlab.linuxcent.com': vamshi
Password for 'https://[email protected]': 
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.

Make a permanent entry to system wide user level profiles as below. The following change works at the system level

[vamshi@workstation ~]$ sudo bash -c "echo -e export GIT_SSL_NO_VERIFY=1 > /etc/profile.d/gitconfig.sh "
[vamshi@workstation ~]$ cat /etc/profile.d/gitconfig.sh
export GIT_SSL_NO_VERIFY=false

The practical use case of setting the environment variable can be made while building container images, using  the GIT_SSL_NO_VERIFY false as an environment variable in Dockerfile and building an image.

[vamshi@workstation ~]$ cat Dockerfile
FROM jetty:latest
-- CONTENT TRUNCATED --
env GIT_SSL_NO_VERIFY 1
-- CONTENT TRUNCATED --

We can also setup the container build agent with Jenkins Pipeline code with similar configuration to fetch a gitrepo in our next sessions.

Why is git clone not working?

If you have a problem cloning a repository, or using it once it has been created, check the following: Make sure that the path in the git clone call is correct. … If you have an authorization error, have an administrator check the ACLs in Administration > Repositories > <repoName> > Access.

How do I fix fatal unable to access?

How to resolve “git pull,fatal: unable to access ‘https://github.com… \’: Empty reply from server”

  1. If you have configured your proxy for a VPN, you need to login to your VPN to use the proxy.
  2. to use it outside the VPN use the unset command: git config –global –unset http.proxy.

How do I bypass SSL certificate in git?

Prepend GIT_SSL_NO_VERIFY=true before every git command run to skip SSL verification. This is particularly useful if you haven’t checked out the repository yet. Run git config http. sslVerify false to disable SSL verification if you’re working with a checked out repository already.

How do I open a cloned git repository?

Clone Your Github Repository

  • Open Git Bash. If Git is not already installed, it is super simple. …
  • Go to the current directory where you want the cloned directory to be added. …
  • Go to the page of the repository that you want to clone.
  • Click on “Clone or download” and copy the URL.

Can not clone from GitHub?

If you’re unable to clone a repository, check that:
You can connect using HTTPS. For more information, see “HTTPS cloning errors.”
You have permission to access the repository you want to clone. For more information, see “Error: Repository not found.”
The default branch you want to clone still exists.

Do I need git for GitLab?

To install GitLab on a Linux server, you first need Git software. We explain how to install Git on a server in our Git tutorial. Next, you should download the GitLab omnibus package from the official GitLab website.

How do I clone a project from GitHub?

Cloning a repository

  • In the File menu, click Clone Repository.
  • Click the tab that corresponds to the location of the repository you want to clone. …
  • Choose the repository you want to clone from the list.
  • Click Choose… and navigate to a local path where you want to clone the repository.
  • Click Clone.

How do I push code to GitHub?

Using Command line to PUSH to GitHub

  • Creating a new repository. …
  • Open your Git Bash. …
  • Create your local project in your desktop directed towards a current working directory. …
  • Initialize the git repository. …
  • Add the file to the new local repository. …
  • Commit the files staged in your local repository by writing a commit message.

What is Bash

What is Bash

Bash is a program (toolbox) used to interact between user and operating system.

The BASH acronym is for the ‘Bourne-Again SHell’. Bash is one of many shells, or command language interpreters, in Unix system.

I called bash a “toolbox”, because it contains large variety of shell tools, for example, tools for read files, calculate math operations, control devices…

The main difference between bash and any other program is that bash was not created to process certain task. Bash was developed to take commands from user using the text-based interface.

One of the advantages of bash is that users can put list of commands to a one (script) and execute these commands from file.

Most of bash scripts have ‘.sh’ extension, for example: script.sh. Linux is an extensionless system, but it is good practice to add .sh extension.

VI Basic Commands for UNIX in Nutshell

In this chapter, we will explore vi – the excellent Unix editor. There are many ways to edit text files in Unix; however, one of the best is using screen-oriented editors like vi who allow you to see context lines around a line that needs editing.

VIM, or Vi IMproved (commonly shortened to vim) is an improved version of the vi editor. It has rapidly grown in popularity because it can be used as both a command line and graphical interface text-editor with more features than standard vi that you might not find elsewhere like syntax highlighting and multitasking capabilities.

  • It’s usually available on all the flavors of Unix system.
  • Its implementations are very similar across the board.
  • It requires very few resources.
  • It is more user-friendly than other editors such as the ed or the ex.

VI Basic Commands for UNIX in Nutshell

Vi has 3 basic modes of operation: command (default), input, last line mode.

VI Editor Command mode

In command mode, you can run commands to search, copy, move, remove text.

VI Editor Input mode

In input mode, you can insert text into the file. Everything you type will be interpreted as text. many ways how to activate input mode (vi is case sensitive):

  • i – Inserts text before the cursor.
  • I – Inserts text at the beginning of the line.
  • o – Opens a new blank line below the cursor.
  • O – Opens a new blank line above the cursor.
  • a – Appends text after the cursor.
  • A – Appends text at the end of the line.

VI Editor Last line mode

To get into the last line mode type ‘:’ only from command mode. After type ‘:’ you will see a colon character appear at the beginning of the last line of your vi editor. It means vi is ready for type a “last line command”. To end vi type ‘q’ from last line mode.
You can return to command mode from input or last line mode pressing Esc.

Moving the Cursor

Key Cursor movement
w Forward one word.
b Back one word.
e To the end of the current word.
$ To the end of the line.
0(zero) To the beginning of the line.
^ To the first non-whitespace character on the line.
G Goes to the last line of the file.
IG Goes to the first line of the file.
Ctrl + F Pages forward one screen.
Ctrl + B Pages back one screen.
Ctrl + D Scrolls down one-half screen.
Ctrl + U Scrolls up one-half screen.
Ctrl + L Refreshes the screen.

Text-Deletion Commands

Command Function
R Overwrites or replaces characters on the line at and to the right of cursor. To terminate press Esc.
C Changes or overwrites characters from cursor to the end of the line.
s Substitutes a string for a character at the cursor.
x Deletes a character at the cursor.
dw Deletes a word or part of the word to the right of the cursor.
dd Deletes the line containing the cursor.
D Deletes the line from the cursor to the right end of the line.
:n, nd Deletes lines n-n. Example :2,80d deletes lines 2-80.

Text-Changing Commands

Command Function
cw Changes or overwrites characters at the cursor location to the end of that word.
r Replaces the character at the cursor with one other character.
J Join the current line and the line below.
xp Transposes the character at the cursor and the character to the right of the cursor.
~ Changes the case of the letter, either uppercase or lowercase, at the cursor.
u Undo the previous command.
. Repeats the previous command.

Text-Replacing Commands

Command Function
/string Searches forward for the string from the cursor.
?string Searches backward for the string.
n Searches for the next occurrence of the string. Use this command after searching for a string.
N Searches for the previous occurrence of the string. Use this command after searching for a string.
:%s/old/new/g Searches for the old string and replaces it iwth the new string globally.

Copy and Paste Commands

Command Function
yy Yanks a copy of the line
p Puts yanked or deleted text under the line containing the cursor.
P Put
:n,n co n Copies lines n-n and puts them after line n. Example: 1, 5 co 8 copies lines 1-5 and puts them after line 8.
:n,n m n Moves lines n-n to line n.
Example: 1,5 m 8 moves lines 1-5 to line 8.

File Save and Quit Commands

Command Function
:w Saves the file with changes by writing to the disk
:w new_file Writes the contents of the buffer to new_file.
wq Saves the changed file and quits editor vi.
😡 Saves the changed file and quits editor vi.
ZZ Saves the changed file and quits editor vi.
:q! Quits without saving changes.

Customizing vi Session

Command Function
:set nu Shows line numbers.
:set nonu Hides line numbers.
:set ic Instructs searches to ignore cases.
:set noic Instructs searches to be case-sensitive.
set list Display invisible characters.
:set showmode Display the current mode of operation.
:set noshowmode Turns off the mode of operation display.
:set Displays all the vi variables that are set.
:set all Display all vi variables and their values.

Customizing vi Session
To automatic customization for all vi sessions do the following steps:

  • Create a file in your home directory named ‘ . exrc’
  • Enter any of the set variables into the ‘ . exrc’ file.
  • Enter each ‘set variable’ command on one line.

Vi reads ‘exrc’ file every time before starting vi sessions.

Command Function
:set nu Shows line numbers.
:set nonu Hides line numbers.
:set ic Instructs searches to ignore case.
:set noic Instructs searches to be case-sensitive.
set list Display invisible characters.
:set showmode Display the current mode of operation.
:set noshowmode Turns off the mode of operation display.
:set Displays all the vi variables that are set.
:set all Display all vi variables and their values.

Puppet: Error: Could not parse for environment : Illegal class reference

The following errors appear due to missing of certain modules.

Error: Could not parse for environment production: Illegal class reference

To resolve this error you need to install the relevant puppet module from the module-repository.

For example :

Error: 'module' has no 'info' action.  See `puppet help module`.
[vamshi@node01 manifests]$ puppet module install puppetlabs-stdlib
Notice: Preparing to install into /home/vamshi/.puppetlabs/etc/code/modules ...
Notice: Created target directory /home/vamshi/.puppetlabs/etc/code/modules
Notice: Downloading from https://forgeapi.puppet.com ...
Notice: Installing -- do not interrupt ...
/home/vamshi/.puppetlabs/etc/code/modules
└── puppetlabs-stdlib (v6.3.0)

Redirect the Std error and std output 2>&1

The redirect operation > is used in conjunction with stdoutput 1 and stderr 2.

command > [/dev/null] 2>&1

2 Represents the stderror. The &1 here represents the first argument which is /dev/null

The character 2 represents the stderr which takes the entire errors printed to the screen and then appends them to the /dev/null which is the first argument represented by &1.

So the command demonstration will be the following:

$ du -sh /* > /dev/null 2>&1

This redirect command will dump the errors and the output to /dev/null.

Explanation: the default behaviour of redirection operator is to redirect the stdout and we are redirecting them to devnul and then we followup the command with 2>&1 which mentions the stderr 2 and then redirects is to /dev/null, which is denoted by &1 describing the &1 as the first argument which is /dev/null

 

 

How to allocate resources to docker images in Runtime?

We have seen the docker runtime environment takes up the overall available system resources on the system and tends to impact the base system.

To better utilize the containers, we can avail the resource cap and define the metric limits on specific containers while starting up the respective docker images..

The general syntax goes as follows:

# docker run --cpus ="x.x" --memory=x[M|G] docker-image

Here we see the demonstration

[root@node01 ~]# docker run --cpus="0.2" --memory="200M" jenkins:latest
/usr/bin/docker-current: Error response from daemon: Minimum memory limit allowed is 4MB.

It should be noticed that the minimum Memory limit allowed for the docker container to run is 4MB and the minimum CPU cores is at 0.01, any thing lower that this means the container runtime fails to allocate sufficient resources. These limits will be efficient when running on some test and debug scenarios.