Implementing ILM with Write Aliases (Logstash + Elasticsearch)

In this blog post, I demonstrate the creation of a new elasticsearch index with the ability to rollover using the aliases.

We will be implementing the ILM (Information lifecycle Management) in Elasticsearch with Logstash Using Write Aliases

Optimize Elasticsearch indexing with a clean, reliable setup: use Index Lifecycle Management (ILM) with a dedicated write alias, let Elasticsearch handle rollovers, and keep Logstash writing to the alias instead of hardcoded index names. This approach improves stability, reduces manual ops, and scales cleanly as log volume grows.

Implementing ILM with Write Aliases (Logstash + Elasticsearch)

Optimize Elasticsearch indexing with a clean, reliable setup: use Index Lifecycle Management (ILM) with a dedicated write alias, let Elasticsearch handle rollovers, and keep Logstash writing to the alias instead of hardcoded index names. This approach improves stability, reduces manual operations, and scales cleanly as log volume grows.

What you’ll set up

  • Write to a single write alias.
  • Apply ILM via an index template with a rollover alias.
  • Bootstrap the first index with the alias marked as is_write_index:true.
  • Point Logstash at ilm_rollover_alias (not a date-based index).

Prerequisites

  • Elasticsearch with ILM enabled.
  • Logstash connected to Elasticsearch.
  • An ILM policy (example: es_policy01).

1) Create index template with rollover alias

Define a template that applies the ILM policy and the alias all indices will use.

PUT _index_template/test-vks
{
  "index_patterns": ["vks-nginx-*"],
  "priority": 691,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "es_policy01",
          "rollover_alias": "vks-nginx-write-alias"
        },
        "number_of_shards": 1,
        "number_of_replicas": 0
      }
    },
    "mappings": {
      "dynamic": "runtime"
    }
  }
}

Notes:

  • Only set index.lifecycle.rollover_alias here; do not declare the alias body in the template.
  • Tune shards/replicas for your cluster and retention goals.

2) Bootstrap the first index

Create the first managed index and bind the write alias to it.

PUT /<vks-nginx-error-{now/d}-000001>
{
  "aliases": {
    "vks-nginx-write-alias": {
      "is_write_index": true
    }
  }
}

Notes:

  • The -000001 suffix is required for rollover sequencing.
  • is_write_index:true tells Elasticsearch where new writes should go.

3) Configure Logstash to use the write alias

Point Logstash to the rollover alias and avoid hardcoding an index name.

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    manage_template => false
    template_name   => "test-vks"
    # index => "vks-nginx-error-%{+YYYY.MM.dd}"   # keep commented when using ILM
    ilm_rollover_alias => "vks-nginx-write-alias"
  }
}

Notes:

  • manage_template => false prevents Logstash from overwriting your Elasticsearch template.
  • Restart Logstash after changes.

How rollover works

  • When ILM conditions are met, Elasticsearch creates the next index (...-000002), moves the write alias to it, and keeps previous indices searchable.
  • Reads via the alias cover all indices it targets; writes always land on the active write index.

Common issues and quick fixes

  • rollover_alias missing: Ensure index.lifecycle.rollover_alias is set in the template and matches the alias used in bootstrap and Logstash.
  • Docs landing in the wrong index: Remove index in Logstash; use only ilm_rollover_alias.
  • Alias conflicts on rollover: Don’t embed the alias body in the template—bind it during the bootstrap call only.
Complete Flow of Implementing ILM with Write Aliases (Logstash + Elasticsearch)
Implementing ILM with Write Aliases (Logstash + Elasticsearch)

Quick checklist

  • ILM policy exists (e.g., es_policy01).
  • Template includes index.lifecycle.name and index.lifecycle.rollover_alias.
  • First index created with -000001 and is_write_index:true.
  • Logstash writes to the alias (no concrete index).
  • Logstash restarted and ILM verified.

Verify your setup (optional)

Run these in Kibana Dev Tools or via curl:

GET _ilm/policy/es_policy01 GET _index_template/test-vks GET vks-nginx-write-alias/_alias POST /vks-nginx-write-alias/_rollover # non-prod/manual test 

Housekeeping on gitlab and practical examples using gitlab-ctl

In this section we will see some important housekeeping tasks while maintaining gitlab server.

We will list out some of the important configuration services present in gitlab and best housekeeping tips.
The important configuration file for gitlab is: /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
Once the changes are made to this file it should be followed up running the reconfigure to apply the changes.
The chef server reads the config files and then updates the system wide configuration resources.

Reconfiguring the gitlab

[vamshi@gitlab ~]$ sudo gitlab-ctl reconfigure

This command reloads the configuration from the configuration files and also migrated if any changes regiuired to the Postgresql database.

View all the current logs run the gitlab-ctl tail command.

[vamshi@node01 ~]$ sudo gitlab-ctl tail
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==

==> /var/log/gitlab/gitlab-rails/sidekiq.log <==
2020-04-o2_13:30:24.29475 2020-04-02T13:30:24.293Z 8872 TID-or0k2ve5w PagesDomainVerificationCronWorker JID-e46ed1d452563ca9b7d4adca INFO: done: 0.357 sec
-- OUTPUT TRUNCATED --

This in one of the first command you need when troubleshooting your gitlab server..
This prints the logs from various components present in gitlab and you as an administrator will be able to effectively identify the issues.
To start the interactive ruby console and debug mostly used for Database interaction operations, it takes -e as argument and takes you to the production environment. -s takes you to the sandbox environment.

gitlab-rails console <-e|-s> [environment]

For example we can get the complete gitlab information using the following gitlab:env:info:

[vamshi@gitlab ~]$ sudo gitlab-rails gitlab:env:info
[sudo] password for vamshi:

System information
System: 
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.5.3p105
Gem Version: 2.7.6
Bundler Version:1.17.3
Rake Version: 12.3.2
Redis Version: 3.2.12
Git Version: 2.18.1
GitLab information
Version: 11.10.4-ee
Revision: 88a3c791734

GitLab Shell
Version: 9.0.0

-- OUTPUT TRUNCATED --

Some more important commands for gitlab Houseeping:

[vamshi@gitlab ~]$ sudo gitlab-rails gitlab:check
[sudo] password for vamshi:
Checking GitLab subtasks ...

Checking GitLab Shell ...

-- OUTPUT TRUNCATED --

How to restart the gitlab serer from command line ?

[vamshi@gitlab ~]$ sudo gitlab-rails restart

How to clear the gitlab logs by truncating them to Zero bytes?

[vamshi@gitlab ~]$ sudo gitlab-rails log:clear

How To print the high level stats of gitlab server?

[vamshi@gitlab ~]$ sudo gitlab-rails stats

What is GitLab housekeeping?

GitLab supports and automates housekeeping tasks within your current repository such as: Compressing Git objects. Removing unreachable objects.

What is GitLab housekeeping?

How do I clean up GitLab repository?

  • To clean up a repository:
  • Go to the project for the repository.
  • Navigate to Settings > Repository.
  • Upload a list of objects. For example, a commit-map file created by git filter-repo which is located in the filter-
  • repo directory. …
  • Click Start cleanup.

What is Git repack?

DESCRIPTION. This command is used to combine all objects that do not currently reside in a “pack”, into a pack. It can also be used to re-organize existing packs into a single, more efficient pack.

What does Git prune do?

The git prune command is an internal housekeeping utility that cleans up unreachable or “orphaned” Git objects. Unreachable objects are those that are inaccessible by any refs. Any commit that cannot be accessed through a branch or tag is considered unreachable.

How do I know if GitLab is running?

When a user goes to your GitLab URL, they will be shown an arbitrary Deploy in progress page. To remove the page, you simply run sudo gitlab-ctl deploy-page down . You can also check the status of the deploy page with sudo gitlab-ctl deploy-page status.

How do I maintain a Git repository?

Manually reviewing large files in your repository

  • Download the script to your local system.
  • Put it in a well known location accessible to your Git repository.
  • Make the script an executable: …
  • Clone the repository to your local system.
  • Change directory to your repository root.
  • Run the Git garbage collector manually.

How do I purge Gitlab?

uninstall Gitlab:

  1. Stop the gitlab service. Command : sudo gitlab-ctl stop.
  2. Start commonde of service facility. command : sudo gitlab-ctl uninstall.
  3. Delete the package of gitlab. command : Locat gitlab.
  4. restart machine.

How do I empty a Gitlab project?

Delete a project

  1. On the top bar, select Menu > Projects and find your project.
  2. On the left sidebar, select Settings > General.
  3. Expand Advanced.
  4. In the “Delete project” section, select Delete project.
  5. Confirm the action when asked to.

Are git repos compressed?

Thirdly, Git stores its data as compressed objects, whereas SVN stores them as uncompressed copies. Go into any . svn/text-base directory, and you’ll find uncompressed copies of the (base) files.

kubernetes bash completion

Wouldn’t it be great when we are using kubernetes and just tab to get a list of possible subcommands?

We have the extended facility to automatically tab and get the kubectl command suggestions, This feature is inbuilt into the kubectl and kubeadmin, All we need to just extract the script and enable it to the .bash_profile and source it. Lets do it as show below

[vamshi@workstation ~]$ kubectl completion bash > ~/.kube/k8s_bash_completion.sh
[vamshi@workstation ~]$ echo -e "\n#kubectl shell completion\nsource '$HOME/.kube/k8s_bash_completion.sh'\n" >> $HOME/.bash_profile
[vamshi@workstation ~]$ source $HOME/.bash_profile

Once you have successfully sourced the .bash_profie file.. type kubectl commands and keep tabbing all the way to get the suitable suggestions.

What is kubectl bash completion?

The kubectl completion script for Bash can be generated with the command kubectl completion bash . … However, the completion script depends on bash-completion, which means that you have to install this software first (you can test if you have bash-completion already installed by running type _init_completion ).

How do I create a completion in bash?

How to add bash auto completion in Ubuntu Linux

  1. Open the terminal application.
  2. Refresh package database on Ubuntu by running: sudo apt update.
  3. Install bash-completion package on Ubuntu by running: sudo apt install bash-completion.
  4. Log out and log in to verify that bash auto completion in Ubuntu Linux working properly.

What is kubectl auto completion?

Shell command-line completion allows you to quickly build your command without having to type every character. … The Kubernetes documentation provides great instructions on how to set it up for your development environment.

How do you use kubectl autocomplete?

To use shell autocompletion with kubectl simply press tab while writing out a command. For example we can type g and then press tab to autocomplete to get .

What is difference between Docker and Kubernetes?

A fundamental difference between Kubernetes and Docker is that Kubernetes is meant to run across a cluster while Docker runs on a single node. … Kubernetes pods—scheduling units that can contain one or more containers in the Kubernetes ecosystem—are distributed among nodes to provide high availability.

Is zsh better than bash?

It has many features like Bash but some features of Zsh make it better and improved than Bash, such as spelling correction, cd automation, better theme, and plugin support, etc. Linux users don’t need to install the Bash shell because it is installed by default with Linux distribution.

How does bash completion work?

The programmable completion feature in Bash permits typing a partial command, then pressing the [Tab] key to auto-complete the command sequence. [1] If multiple completions are possible, then [Tab] lists them all. Let’s see how it works. Tab completion also works for variables and path names.

How do I know if bash completion is installed?

If the autocomplete results contain directories only (no files), then Bash Completion is installed. If the autocomplete results include files, then Bash Completion is not installed.

What is complete command?

complete is a bash command used to perform the auto-complete action when the user hit the TAB key in a terminal. Calling just complete will list all the functions registered for auto-completion of commands or services options.

What is the difference between Minikube and Kubernetes?

Kubernetes is an open source orchestration system for Docker containers. … On the other hand, minikube is detailed as “Local Kubernetes engine”. It implements a local Kubernetes cluster on macOS, Linux, and Windows.

How do I know if kubectl is installed?

Install the kubectl Command Line

  1. Check that kubectl is correctly installed and configured by running the kubectl cluster-info command: kubectl cluster-info. …
  2. You can also verify the cluster by checking the nodes. …
  3. To get complete information on each node, run the following: kubectl describe node.

Docker Networking basics and the types of networks

The docker networking comprises of a overlay network and enabled communication with the outside resources using it.

There are following main types of built in connectivity networking drivers namely the bridged, host, macvlan, overlay network and the null driver with no network.

The docker container networking Model CNM architecture manages the networking for Docker container.

IPAM which stands for the IP address management works in a single docker node, and aids in Enabling the network connectivity among the doccker containers. Its primary responsibility is to allocate the IP address space for the subnets, allocation of the IP addresses to the endpoints and the network etc,.

The networking in docker is essentially an isolated sandbox environment, The isolation of the networking resources is possible by the namespaces

The overlay network enables the communication enabled the network spanning across many docker nodes on an environment like the Docker swarm network, The same networking logic is evident in a bridge networking but it is ony limited to a single docker host unlike the overlay network.

Here’s the output snippet from the docker info command; Listing the available network drivers.

# docker info
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay

The container networking enables connectivity inbetween the docker containers and also the host machine to docker container and vice-versa.

Listing the default networks in docker:

[vamshi@node01 nginx]$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
68b2ffd36e8f        bridge              bridge              local
c1aca4c87a2b        host                host                local
d5e48683def8        none                null                local

When a container is created by default it connects to the bridge network unless an extra arguments are specified.

When you install the docker by default a docker0 virtual interface is created which behaves as a bridge between the docker containers and the host system.

[vamshi@node01 nginx]$ brctl show
bridge name	bridge id		STP enabled	interfaces
docker0		8000.0242654b42ef	no

For brctl command we need to Install the bridge-util package.

We now examine the docker networks with docker network inspect.

Inspecting the various docker networks:

Inspecting the bridge network:

The bridge networking enables the network connectivity over the dockers in a single docker server host.

[vamshi@node01 nginx]$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "68b2ffd36e8fcdc0c3b170dfdbdbc93bb58351d1b2c011abc80709928463f809",
        "Created": "2020-05-23T10:28:27.206979057Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

This bridge is shown with the ip addr command as follows:

# ip addr show docker0 
   docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:65:4b:42:ef brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:65ff:fe4b:42ef/64 scope link 
       valid_lft forever preferred_lft forever

Inspecting the host network.

[vamshi@node01 ~]$ docker network inspect host 
[
    {
        "Name": "host",
        "Id": "c1aca4c87a2b3e7db4661e0cdedc97245cd5dfdc8aa2c9e6fa4ff1d5ecf9f3c1",
        "Created": "2019-05-16T18:46:19.485377974Z",
        "Scope": "local",
        "Driver": "host",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]

Inspecting the null driver network

[vamshi@node01 ~]$ docker network inspect none
[
    {
        "Name": "none",
        "Id": "d5e48683def80b2e739b3be95e58fb11abc580ce29a33ba0df679a7a3972f532",
        "Created": "2019-05-16T18:46:19.477155061Z",
        "Scope": "local",
        "Driver": "null",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]

 

The following are special networking architectures to span across multihost docker servers enabling network connectivity among the docker containers.

1.Overlay network

2 macvlan network.

Let us inspect the multi host networking:

The core components of the docker interhost network consists of

Inspecting the overlay network:

[vamshi@docker-master ~]$ docker network inspect overlay-linuxcent 
[
    {
        "Name": "overlay-linuxcent",
        "Id": "qz5ucx9hthyva53cydei0y8yv",
        "Created": "2020-05-25T13:22:35.087032198Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.255.0.0/16",
                    "Gateway": "10.255.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "overlay-linuxcent-endpoint",
                "EndpointID": "165beb97b22c2857e3637119016ef88e462a05d3b3251c4f66aa0fc9176cfe67",
                "MacAddress": "02:42:0a:ff:00:03",
                "IPv4Address": "10.255.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "node01.linuxcent.com-95ad856b6f56",
                "IP": "10.100.0.20"
            }
        ]
    }
]

The endpoint is the Virtual IP addressing that routes the traffic to the respective containers running on individual docker nodes.

Inspecting the macvlan network:

vamshi@docker-master ~]$ docker network inspect macvlan-linuxcent 
[
    {
        "Name": "macvlan-linuxcent",
        "Id": "99c6a20bd4029ce5a37139c6e6792ec4f8a075c94b5f3e71efc32d92d41f3f89",
        "Created": "2020-05-25T14:20:00.655299409Z",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]

What is Docker networking?

Docker networking is primarily used to establish communication between Docker containers and the outside world via the host machine where the Docker daemon is running. … You can run hundreds of containers on a single-node Docker host, so it’s required that the host can support networking at this scale.

How does networking work with Docker?

Docker secures the network by managing rules that block connectivity between different Docker networks. Behind the scenes, the Docker Engine creates the necessary Linux bridges, internal interfaces, iptables rules, and host routes to make this connectivity possible.

How do I connect to a Docker network?

Connect a container to a network when it starts

You can also use the docker run –network= option to start a container and immediately connect it to a network.

Can a Docker container have multiple networks?

You can create multiple networks with Docker and add containers to one or more networks. Containers can communicate within networks but not across networks. A container with attachments to multiple networks can connect with all of the containers on all of those networks.

Why is docker network needed?

Some of the major benefits of using Docker Networking are: They share a single operating system and maintain containers in an isolated environment. It requires fewer OS instances to run the workload. It helps in the fast delivery of software.

What are the types of docker networks?

There are three common Docker network types – bridge networks, used within a single host, overlay networks, for multi-host communication, and macvlan networks which are used to connect Docker containers directly to host network interfaces.

How do I ping a Docker container?

Ping the IP address of the container from the shell prompt of your Docker host by running ping -c5 . Remember to use the IP of the container in your environment. The replies above show that the Docker host can ping the container over the bridge network.

What does Docker network create do?

When you install Docker Engine it creates a bridge network automatically. This network corresponds to the docker0 bridge that Engine has traditionally relied on. When you launch a new container with docker run it automatically connects to this bridge network.

Does Docker offer support for IPv6?

Before you can use IPv6 in Docker containers or swarm services, you need to enable IPv6 support in the Docker daemon. Afterward, you can choose to use either IPv4 or IPv6 (or both) with any container, service, or network. Note: IPv6 networking is only supported on Docker daemons running on Linux hosts.

How do I run a docker on a local network?

This article discusses four ways to make a Docker container appear on a local network.

  • Using NAT
  • It will create a veth interface pair.
  • Connect one end to the docker0 bridge.
  • Place the other inside the container namespace as eth0 .
  • Assign an ip address from the network used by the docker0 bridge.

Can a docker container have its own ip address?

The answer is: you can configure it. Create the container with –network host and it will use the host ip.

docker CMD vs ENTRYPOINT

The ENTRYPOINT or CMD usually determine the single and main startup process inside the running container, The ENTRYPOINT and the CMD are the final Docker DSL invocation statements in a Dockerfile.A Dockerfile can not be complete without either one of them.

Both the ENTRYPOINT and CMD will execute a /bin/sh -c , but have significant differences.

ENTRYPOINT usually is straight forward with the and is often the statement that enables the  docker image as a runnable container. The ENTRYPOINT statement doesnot let the run time arguments get overwritten during the container runtime, instead the commands are passed on as arguments to the ENTRYPOINT, for Example when the container is started as docker run my-image -d, then the resultant -d argument is passed on as an argument to the ENTRYPOINT.
It means that only last ENTRYPOINT statement is valid in the Dockerfile

The ENTRYPOINT takes CMD as its arguments.

ENTRYPOINT command arg1 arg2

OR

ENTRYPOINT command
CMD ["arg1", "arg2"]
ENTRYPOINT ["java","-jar","app.jar"]
CMD ["-Dserver.port=8080"]

CMD on the other hand is subversive to the ENTRYPOINT.
CMD start statement can be overwritten with container arguments during the container runtime. The CMD does not execute anything at the build time.

CMD ["command","arg1", "arg2"]
CMD ["java","-Djava.security.egd=file:/dev/urandom","-jar","-Dserver.port=8080","app.jar"]

The major practical use with CMD is that the you will be able to overwrite the docker runtime command line arguments but you cannot override with you have ENTRYPOINT.

The CMD will be very beneficial in cases which require sending arguments to containers in runtime which in development and QA stages and is great for debugging environments.

The [] are not mandatory for ENTRYPOINT and CMD instruction statements inside Dockerfile but can be given for better readability when you have lot of arguments

What is the difference between CMD and run in Dockerfile?

RUN is an image build step, the state of the container after a RUN command will be committed to the container image. A Dockerfile can have many RUN steps that layer on top of one another to build the image. CMD is the command the container executes by default when you launch the built image.

What is the use of CMD in docker?

The CMD command​ specifies the instruction that is to be executed when a Docker container starts. This CMD command is not really necessary for the container to work, as the echo command can be called in a RUN statement as well. The main purpose of the CMD command is to launch the software required in a container.

What is docker ENTRYPOINT script?

entrypoint scripts are plugins that can be written in any language. inherited images can add to existing entrypoint scripts by simply adding to the /docker-entrypoint.

What is the purpose of ENTRYPOINT in Dockerfile?

In Dockerfiles, an ENTRYPOINT instruction is used to set executables that will always run when the container is initiated. Unlike CMD commands, ENTRYPOINT commands cannot be ignored or overridden—even when the container runs with command line arguments stated.

Can Dockerfile have multiple ENTRYPOINT?

According to the documentation however, there must be only one ENTRYPOINT in a Dockerfile.

What is Docker ENTRYPOINT Initdb?

3. 3. /docker-entrypoint-initdb. d/init. sql is executed the moment your database container starts running, while your entrypoint.sh is executed the moment your web container starts running.

Does docker start run ENTRYPOINT?

So yes, the ‘ CMD ‘ commands are executed after a ‘ docker start ‘.

Can Dockerfile have multiple CMD?

You are right, the second Dockerfile will overwrite the CMD command of the first one. Docker will always run a single command, not more. So at the end of your Dockerfile, you can specify one command to run.

What is in docker Run command?

The docker run command creates a container from a given image and starts the container using a given command. It is one of the first commands you should become familiar with when starting to work with Docker.

What is difference between CMD and entrypoint?

In a nutshell: CMD sets default command and/or parameters, which can be overwritten from command line when docker container runs. ENTRYPOINT command and parameters will not be overwritten from command line. Instead, all command line arguments will be added after ENTRYPOINT parameters.

What is the default entrypoint for docker?

/bin/sh -c
Docker defaults the entrypoint to /bin/sh -c . This means you’ll end up in a shell session when you start the container.

Create a new User in Jenkins

Login to the Jenkins and navigate to the Manage Jenkins -> Manage Users. Now you will be able to see the current users on the Jenkins.

Create-new-user-in-jenkins-securityRealm
You should be able to land on the URL similar to this: http://jenkins.linuxcent.com:8080/securityRealm/

On the Left hand side menu Click on Create User and follow the process. .
Create-user-Jenkins-own-user-database
With this process you will be able to create a new user using the Jenkins’s own user database.

You can also ensure the Jenkins system user details from the jenkins server by navigating to the path $JENKINS_HOME and from the file users/users.xml

[root@node02 jenkins]# grep -w builduser * users/users.xml 2>/dev/null 
users/users.xml: <string>builduser</string>

How do I give user access to Jenkins?

Essentially you do this:

  • Go to Jenkins -> Manage Jenkins -> Configure Global Security.
  • Check “Enable security”.
  • Set “Jenkins own user database” as security realm.
  • Check “Allow users to sign up”
  • Choose “Matrix based security”
  • Check “Overall read” on Anonymous.
  • Add your admin account in the matrix, check every box.

How do I make the first admin in Jenkins?

Jenkins – Security

  • Step 1 − Click on Manage Jenkins and choose the ‘Configure Global Security’ option.
  • Step 2 − Click on Enable Security option. …
  • Step 3 − You will be prompted to add your first user. …
  • Step 4 − It’s now time to setup your users in the system.

How get manage users in Jenkins?

When you go to manage Jenkins and scroll down, you will see the ‘Manage Users’ option. Click on this option. Step 2: Just Like, you defined the admin user, and start creating other users for the system.

How do I change users in Jenkins?

To change the jenkins user, open the /etc/sysconfig/jenkins (in debian this file is created in /etc/default) and change the JENKINS_USER to whatever you want. Make sure that user exists in the system (you can check the user in the /etc/passwd file ).

How do I create multiple users in Jenkins?

in jenkins go to Manage Jenkins > Configure Global Security and select the “Enable Security” check box. select “Jenkins Own User Database” for the security realm and then select “Logged in Users can do anything” or a matrix based strategy (If you have multiple users with different permissions) for the Authorization.

What user runs Jenkins?

Similar to access control for users, builds in Jenkins run with an associated user authorization. By default, builds run as the internal SYSTEM user that has full permissions to run on any node, create or delete jobs, start and cancel other builds, etc.

Is it possible to create users and define roles for them in Jenkins?

By default, Jenkins comes with very basic user creation options. You can create multiple users but can only assign the same global roles and privileges to them. The Role Strategy Plugin enable you to assign different roles and privileges to different users.

How do I give Jenkins a user root permission?

Below are the steps to achieve this.

  • #1 open /etc/sudoers. type sudo vi /etc/sudoers . This will open your file in edit mode.
  • #2 Add/Modify jenkins user. Look for the entry for jenkins user. Modify as below if found or add a new line. …
  • #3 Save and Exit from edit mode. Press ESC and type :wq and hit Enter .

Where does Jenkins Store users?

Jenkins stores the configuration for each job within an eponymous directory in jobs/. The job configuration file is config. xml, the builds are stored in builds/, and the working directory is workspace/.

How do I create a user and group in Jenkins?

Follow the steps below:

  • Log in to Jenkins using the existing administrative user account credentials.
  • Navigate to the “Manage Jenkins -> Manage Users” page.
  • Select the “Create User” option.
  • Enter a username, password, name and email address to create a new user. …
  • Save the new account.

Initiating a docker swarm and getting the current docker swarm token

Creating a docker swarm cluster:

The docker swarm can be created by using the following command:

The syntax is defined as follows:

docker swarm init --advertise-addr [available interface IP adress]

The –advertise-addr is used to explicitly define the docker swarm advertise ip. If you have a single interface this option will not be needed but will be real handy if you have more than 1 active public accessible interfaces.
Let us initialize our docker swarm environment.

[vamshi@docker-swarm ~]$ docker swarm init --advertise-addr 10.100.0.20
Swarm initialized: current node (nodeidofmastercdq7nmmq3kcmb5l85k2e) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-verylongstringofcharactercontainingthedockerswarmjoinstring-70bouyqwhfgdcgtw6o0fw6wup \
    10.100.0.20:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

The docker swarm creation can be viewed from the docker info command as follows:

[vamshi@docker-swarm ~]$ docker info | grep -C 2 Swarm
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: nodeidofmastercdq7nmmq3kcmb5l85k2e
 Is Manager: true

The docker swarm explicitly uses the overlay and macvlan to enable the interhost network connectivity between the container over a swarm network

How to get the docker swarm join token:

This command can come in very handy when you forgot your docker swarm token and you need to join a new docker node to this docker swarm cluster.

[vamshi@docker-swarm ~]$ docker swarm join-token manager -q
SWMTKN-1-verylongstringofcharactercontainingthedockerswarmjoinstring-70bouyqwhfgdcgtw6o0fw6wup

How can I get token to join docker Swarm?

Run the command from the output on the worker to join the swarm: $ docker swarm join \ –token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \ 192.168. 99.100:2377 This node joined a swarm as a worker.

What is token in docker swarm join?

Join tokens are secrets that allow a node to join the swarm. There are two different join tokens available, one for the worker role and one for the manager role. You pass the token using the –token flag when you run swarm join. Nodes use the join token only when they join the swarm.

How do I join a docker swarm worker?

Once you’ve created a swarm with a manager node, you’re ready to add worker nodes.

  1. Open a terminal and ssh into the machine where you want to run a worker node. …
  2. Run the command produced by the docker swarm init output from the Create a swarm tutorial step to create a worker node joined to the existing swarm:

What is the docker command to set up a swarm?

When you run the command to create a swarm, the Docker Engine starts running in swarm mode. Run docker swarm init to create a single-node swarm on the current node. The Engine sets up the swarm as follows: switches the current node into swarm mode.

What is swarm built on?

Docker Swarm mode is built into the Docker Engine.

What is swarm built on CoinList?

Introducing Wrapped Filecoin (EFIL) Minting on CoinList

By design, Swarm is an economically self-sustaining system of peer-to-peer dragon coin nodes that create a decentralized storage and communication service.

What is swarm mode?

Swarm mode is a Docker feature that provides built in container orchestration capabilities, including native clustering of Docker hosts and scheduling of container workloads.

What is Swarm service?

A service is the definition of the tasks to execute on the manager or worker nodes. It is the central structure of the swarm system and the primary root of user interaction with the swarm. When you create a service, you specify which container image to use and which commands to execute inside running containers.

What is the difference between Docker swarm and Kubernetes?

Kubernetes focuses on open-source and modular orchestration, offering an efficient container orchestration solution for high-demand applications with complex configuration. Docker Swarm emphasizes ease of use, making it most suitable for simple applications that are quick to deploy and easy to manage.

What is Docker swarm command?

Run the following command to create a new swarm: $ docker swarm init –advertise-addr <MANAGER-IP> Note: If you are using Docker Desktop for Mac or Docker Desktop for Windows to test single-node swarm, simply run docker swarm init with no arguments. There is no need to specify –advertise-addr in this case.

How to push a docker image to a private repository

How to push a docker image to a private repository: In this tutorial we will cover the the stages dealing with tagging the image and pushing a docker image to the private repository.

For Docker login please refer Docker login to private registry

Now we need to see the two types of docker registry references when pushing the docker images, lets look at the standard docker registry references.

docker image registry reference types

Public registry format:

The public registry is a free to signup and start using immediately. Here is the docker tagging format.

[docker-hub-username]/[image-name]:[tag]

Private registry format:
The private registry is setup on-premises using a hosted docker registry software like docker-registry, nexus-docker registry are among are some of the popular tools to be named. The software is available over the connected network with the given server-name and the desired port to connect to the docker registry service. The format of it is as below

docker-registry-host:docker-registry-port/[user-name/]image-name[:tag]

 

STEP 1: Tagging a docker image

Tagging is a crucial part, As the docker image will be pre-fixed with the registry name which forms the complete unique docker image identity. [DOCKER-REGISTRY-NAME]/[DOCKER-IMAGE]:[TAG]

Eg of Public registry format:

# docker-repo/nginx:latest.

So that it registry name is be pushed to the prefixed Repository name, and Hence the docker image needs to be properly tagged.

The information about your current docker login repository can be found with docker info command.

The docker image has to be properly tagged with docker tag command and prefixed with the Docker-repository name as follows:

[root@docker03:~]#  sudo docker tag nginx:v1 nexusreg.linuxcent.com:5000/nginx:v1

the Image gets tagged with the relevant registry name with docker tag command:

[root@docker03:~]#  docker images
REPOSITORY                                       TAG                 IMAGE ID            CREATED             SIZE
docker.io/nginx                                  latest              602e111c06b6        2 weeks ago         127 MB
nexusreg.linuxcent.com:5000/nginx                v1                  0bc8c50014c7        20 minutes ago      193 MB
jfrog.linuxcent.com/debian-vamshi                v1                  0568d1ef9d81        1 day ago           253 MB

STEP 2: Push the image to a Private Docker registry

The docker push offers the user the capability to upload the docker image on disk to be shared onto a remote docker registry / repository with username authorization.

The process to docker push to private repository is a simple command and takes the following syntax:

[root@docker03:~]#  docker push [DOCKER-REGISTRY-NAME]/[DOCKER-IMAGE]:[TAG]

This will now enable docker image to be pushed to the destined repository.

The syntax command to push the docker image to the destination repository.
Example given:

[root@docker03:~]#  docker push nexusreg.linuxcent.com:5000/nginx:v1

How do I push to Docker repository?

To push an image to Docker Hub, you must first name your local image using your Docker Hub username and the repository name that you created through Docker Hub on the web. You can add multiple images to a repository by adding a specific :<tag> to them (for example docs/base:testing ).

How do I use private Docker repository?

Use a Private Docker Registry with the Docker Containerizer
Step 1: Compress Docker credentials. Log in to the private registry manually. Login creates a ~/. …
Step 2: Add URI path to app definition. Add the path to the archive file login credentials to the fetch parameter of your app definition.

Is Docker repositories can be public or private?

Public repositories can be used to host Docker images which can be used by everyone else. An example is the images which are available in Docker Hub. Most of the images such as Centos, Ubuntu, and Jenkins are all publicly available for all.

How do I pull an image from Docker Hub private repository?

In order to pull images from your private repository, you’ll need to login to Docker. If no registry URI is specified, Docker will assume you intend to use or log out from Docker Hub. Triton comes with several images built-in. You can view the available list with triton images

What is Docker push command?

Docker Push is a command that is used to push or share a local Docker image or a repository to a central repository; it might be a public registry like https://hub.docker.com or a private registry or a self-hosted registry.

How do I push a Docker image to Git?

In order to publish a container image on GitHub Container Registry using GitHub Actions, we have to do the following steps:

  • Activate improved container support.
  • Create a personal access token (PAT) and a repository secret.
  • Create GitHub Actions workflow and login to GitHub Container Registry using the PAT.

Does Docker build push to registry?

Once your application has been built into a Docker image, you’ll want to push it to a container registry for safe-keeping, ready for deployment. You’ll need to log into your container registry before pushing.

How do I create a .dockercfg file?

You can create this file by running docker login on your local system and entering your registry user name, password, and email address. You can also log in to a container instance and run the command there. Depending on your Docker version, this file is saved as either ~/. dockercfg or ~/.

What is the difference between Docker repo and Docker registry?

A Docker repository is where you can store 1 or more versions of a specific Docker image. An image can have 1 or more versions (tags). … It’s also worth pointing out that the Docker Hub and other third party repository hosting services are called “registries”. A registry stores a collection of repositories.

Jenkins API Copy Jenkins Job

Using the Jenkins API is perhaps the most effective way of managing your Devops Tasks, Be it getting the job information, Finding the users information, Copying the jobs,  Creating new jobs, triggering the build on multiple jobs, Clearing the Queues, Creating Credentials, Installing a Plugin, safe restart of Jenkins and many more through the API for ease of Jenkins Administration.

How to Copy a Jenkins job from Jenkins API?

A job can be referenced in order to be copied and replicated any number of times from the Jenkins UI dashboard.

First up we navigate to the Jenkins dashboard and then navigate to New Item

Enter an item name Copy jenkins job name
-> Go to the end of the page -> Enter the job name to be copied from.

The pattern name will be automatically matched from your typing and Jenkins auto suggests the source reference job name and click ok to proceed and then save in the next screen.

Using the Jenkins API:

We can use the jenkins API more effectively copy the job and create another job exact same preferences.
The syntax for the Jenkins API call will be as follows:

$ java -jar /tmp/jenkins-cli.jar -s http://jenkins.linuxcent.com:8080 -auth admin:admin copy-job SRC-JOB-NAME DST-JOB-NAME

You can also update the passwords to a text file and then pass the credentials to -auth with @<filename>
Know the Jenkins server version

$ java -jar /tmp/jenkins-cli.jar -s http://jenkins.linuxcent.com:8080 -auth @passfile -version
Version: 2.222.1

How to Download the Jenkins jarfile ?

If you are Unable to access jarfile jenkins-cli.jar on your jenkins server, Then you can download it from this URL path on your Jenkins Instance: http://YOUR-Jenkins-IP/jnlpJars/jenkins-cli.jar

$ wget http://YOUR-Jenkins-IP:8080/jnlpJars/jenkins-cli.jar -O /tmp/jenkins-cli.jar

 

Docker ADD vs COPY statement

The contents of the docker container image are created using personalized build artifacts and the configuration code which is copied from the build workspace during the docker build process, To achieve this purpose we use certain Docker DSL statements like ADD, COPY to dump the content to the docker image during the build process

It is good to throw some light into understanding the subtle differences between these statements.

It is important to send the up to date content to the Dockerfile and perform the build successfully, Lets see some practical case study of the Docker ADD vs COPY commands below:

Docker ADD statement

Docker COPY statement

Syntax:

ADD </host/Relative/path/to/source/> <Container/image/path/to/Destination/>
ADD [ "/host/path/source1/","/host/path/source2",.. "/container/path/target/" ]
Syntax:

COPY </host/Relative/path/to/source/> <container/image/path/to/Destination/>
COPY ["/host/path/source1/","/host/path/source2/",.. "/container/path/target/"  ]
ADD [source1,source2],,. /path/to/dest/

With multiple source files, the target container path must end with a /

COPY [source1,source2],,. /path/to/dest/

With multiple source files, the target container path must end with a /

The Destination path inside container image can be Absolute or Relative to WORKDIR The Destination path inside container image can be Absolute or Relative to WORKDIR
The Destination path will be automatically created on target container with 0755 permissions The Destination path will be automatically created on target container with 0755 permissions
ADD default.conf /tmp/default.conf

Creates the new files with the default 0644 permission

COPY default.conf /tmp/default.conf

Creates the new files with the default 0644 permission

The Destination content will be owned by the root user and the root group with uid & gid as 0 The Destination content will be owned by the root user and the root group with uid & gid as 0
ADD directory /app
COPY directory /app
ADD Directory Explanation: The whole directory will be copied from the source host to target container with directory permission as 0755 COPY Directory Explanation: The whole directory will be copied from the source host to target container with directory permission as 0755
ADD Portal.tar.gz /tmp/portal1/
COPY Portal.tar.gz /tmp/portal2/
Add Compressed file Explanation: The ADD command will extract the tar file and the extracted will be placed at the target container, with directory permissions as 0755 COPY Compressed file Explanation: The COPY command will NOT extract the tar files and places them as it is.. at the destination target container path as a single compressed tar file.
URL file as Source:

ADD https://ftp.gnu.org/gnu/wget/wget-1.6.tar.gz.sig /tmp/test-add/
URL file as Source – Not possible with COPY command

Source can't be a URL for COPY
ADD URL Explanation: With ADD the URL download and archive unpacking features cannot be used together Meaning it will be not extract the compressed tar.bz2 or tar.gz formats when it downloads from a URL inside the target container path. But it works just like RUN wget command and downloads the compressed package. Explanation:
COPY command will not be able treat source as a URL and Hence its not a valid command

 

Conclusion:
It is better to use ADD command if you have the source archive files like tar.gz, tar.bz2 and want to send them into your container image and extract them, ADD command does it automatically whilst the COPY command sends it as it is at source.

Both the ADD and COPY commands cannot access the source content which are outside of its current relative context path.
Eg:

ADD ../source-code /data
COPY ../source-code /data

You should always keep this in mind whilst writing Docker files.

Feel free to comment and share your experiences with the COPY / ADD commands.

Whats the difference between ADD and COPY Docker?

COPY only supports the basic copying of local files into the container, while ADD has some features (like local-only tar extraction and remote URL support) that are not immediately obvious. Consequently, the best use for ADD is local tar file auto-extraction into the image, as in ADD rootfs. tar.

What is the difference between COPY and add?

COPY copies a file/directory from your host to your image. ADD copies a file/directory from your host to your image, but can also fetch remote URLs, extract TAR files, etc

What is Docker add?

The ADD command is used to copy files/directories into a Docker image. It can copy data in three ways: Copy files from the local storage to a destination in the Docker image. Copy a tarball from the local storage and extract it automatically inside a destination in the Docker image.

What does COPY mean in Docker?

Dockerfiles can contain several different instructions, one of which is COPY. The COPY instruction lets us copy a file (or files) from the host system into the image. This means the files become a part of every container that is created from that image.

When should I use Docker COPY?

If you are copying local files to your Docker image, always use COPY because it’s more explicit. While functionality is similar, the ADD directive is more powerful in two ways: It can handle remote URLs. It can also auto-extract tar files.

Is Docker a recursive COPY?

Docker provides two commands for copying files from the host to the Docker image when building it: COPY and ADD . … COPY — copies local files recursively, given explicit source and destination files or directories.

Does Docker COPY overwrite?

It seems that docker build won’t overwrite a file it has previously copied. I have a dockerfile with several copy instructions, and files touched in earlier COPY directives don’t get overwritten by later ones.

How do I COPY a Docker container?

Solution

  1. To copy a file from the local file system to a container, run the command for Docker container or Kubernetes
  2. pod, respectively: docker cp <src-path> <container>:<dest-path> …
    To copy a file from the container to the local file system, use: docker cp <container>:<src-path> <local-dest-path>

Does Docker COPY create directory?

From the dockerfile reference about COPY and ADD , it says Note: The directory itself is not copied, just its contents. , so you have to specify a dest directory explicitly.

How do I copy a file in Dockerfile?

To do so follow the below steps:

  • Step 1: Create a Directory to Copy. …
  • Step 2: Edit the Dockerfile. …
  • Step 3: Build the Docker Image. …
  • Step 4: Verifying the Docker Image. …
  • Step 5: Running the Docker Container. …
  • Step 6: Verify the Copying of the Directory.

How does copy work in Dockerfile?

COPY and ADD are both Dockerfile instructions that serve similar purposes. They let you copy files from a specific location into a Docker image. COPY takes in a src and destination. It only lets you copy in a local file or directory from your host (the machine building the Docker image) into the Docker image itself.

What are Docker layers?

Basically, a layer, or image layer is a change on an image, or an intermediate image. Every command you specify ( FROM , RUN , COPY , etc.) in your Dockerfile causes the previous image to change, thus creating a new layer.

How do I copy a docker image?

To export your image to a tar file, run the docker save command, specifying a name for the . tar file, and the docker image name. This will save the docker image locally.

How to start Jenkins on a different port

The jenkins Build and Integration application based on java programming language, tends to have its roots stuck to the typical Tomcat web server
Out of the box the jenkins starts up on the default port 8080 and this can be customized on the system or a new proxy routing can be setup which might add a slight overhead on the jenkins inbound requests,

How to change the Jenkins default Port?

We have the Jenkins default home located at /var/lib/jenkins all the configuration files are present in this location.
The important configuration although is /etc/sysconfig/jenkins which is present outside $JENKINS_HOME
The Jenkins Default JENKINS_PORT parameter is set to 8080

[root@node02 jenkins]# grep JENKINS_PORT /etc/sysconfig/jenkins
JENKINS_PORT="8080"

If you are on Debian or Ubuntu linux, then the relevant file will be /etc/default/jenkins
We will be now changing the variable $JENKINS_HOME to listen to the port 9090 and restarting the jenkins service.

[root@node02 jenkins]# systemctl restart jenkins.service

You can check for the port information from the ps -ef | grep jenkins output as follows or checkout from the netstat command

[root@node02 jenkins]# ps -ef |grep jenkins
jenkins 5110 1 90 23:04 ? 00:00:02 /etc/alternatives/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=9090 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

How do I change the default Jenkins port in Linux?

  1. First, run this command to open jenkins configurations: sudo nano /etc/default/jenkins.
  2. The only part you need to change is: #port for HTTP connector (default 8080; disable with -1) Http_port = 8080. …
  3. Finally, Restart Jenkins service by running this command: sudo service jenkins restart.

How do I change my local Jenkins port?

  1. Go to the directory where you installed Jenkins (by default, it’s under Program Files/Jenkins)
  2. Open the Jenkins.xml configuration file.
  3. Search –httpPort=8080 and replace the 8080 with the new port number that you wish.
  4. Restart Jenkins for changes to take effect.

How do I change my Jenkins port from 8080 to 80?

  1. Go to /etc/default folder –> Open the file “jenkins”
  2. Modify the line HTTP_PORT=8080 as HTTP_PORT=80.
  3. Start jenkins as root by using the command: sudo /etc/init.d/jenkins start.
  4. Open a browser and browse as localhost:80.

What port is Jenkins running on?

The default Jenkins installation runs on ports 8080 and 8443. Typically, HTTP/HTTPS servers run on ports 80 and 443, respectively. But these ports are considered privileged on Unix/Linux systems, and the process using them must be owned by root.

How do I start Jenkins on a different port?

How to change the default port in Jenkins

  1. Go to C:\Program Files (x86)\Jenkins (I’m using Windows Server 2012 and assuming it’s installed to default location)
  2. Open Jenkins.xml.
  3. Edit the –httpPort argument (you may need to edit default permissions)
  4. Restart the Jenkins service.
  5. Now Jenkins will permanently use the new port.

How do I run a Jenkins war on another port?

All you need to do:

  1. Goto Jenkins folder present in C:\Program Files (x86)
  2. Open a notepad or text pad and run them as administrator and then try opening the jenkins. xml file present in the jenkins folder.
  3. Change the port number as below: <arguments>-Xrs -Xmx256m -Dhudson. lifecycle=hudson. lifecycle.

How do I change my Jenkins URL?

Fixing a root url

  1. Go to Jenkins > Manage Jenkins > Configure System, and locate the section titled “Jenkins Location”. You should see the warning here as well.
  2. Replace “localhost” with a valid hostname.
  3. Click Save.

How do I start Jenkins manually in Linux?

Go to the Jenkins installation, open the cmd and run:

  1. To stop: jenkins.exe stop.
  2. To start: jenkins.exe start.
  3. To restart: jenkins.exe restart.

What is Jenkins port 50000?

Jenkins runs on Tomcat, which uses port 8080 as the default. -p 5000:5000 required to attach slave servers; port 50000 is used to communicate between master and slaves.

How do I run Jenkins on port 443?

Basic solution (complete):

  1. You need a “Java keystore” of the SSL-certificate you want to use. …
  2. Copy the certificate, private key and (if present) intermediate CAs to your Jenkins host. …
  3. Convert the certificate-files to one single-filed PKCS12 container. …
  4. Make sure, that the Java “keystore”-command is present.