Docker for DevOps

What is a Docker?

It's a container management platform to develop, deploy and run your applications as a container. 

It enables you to separate your application from infrastructure. It runs directly on the host machine's kernel. It is quick to deploy and easy to use in the real world. It's cross-platform software. 

Two types of Versions:

Community Edition (CE) - Open Source

Enterprise Edition (EE) - Commercial

What is a container?

It's a runtime process of an image where our application process will be running. 

Container contains all application code and required dependencies (like Software, Environment variables) to run the applications. 

With the help of containers, we can carry the application from one machine to another without an additional configuration to create/deploy the application.

Each and every container has it's own file system.

What is required to host your applications?

We need Infrastructures like 

Physical Servers OR

Virtual Machines --- Hardware virtualization 

What is the Hypervisor software currently in use to virtualize the environment in AWS?

Citrix Xen

Virtual Machine vs Container

Virtual Machine:

i) It's a heavyweight 

ii) Configuration required on each VM to host the applications like installed Req Softwares, users, and setting up the environment to host the applications.

iii) Ecahc Virtual Machine has its own Operating System.

iv) It's a Hardware-level Virtualization.

Containers:

i) It's a lightweight 

ii) Container contains all application code and required dependencies (like Software, Environment variables) to run the applications.

iii) Container can also be created on top of Virtual Machines.

iv) Containers doesn't have Operation Systems since the container uses the Host Operating system's kernel and it just has required application software and code.

v) It's an Operating System-level Virtualization.

vi) It reduces/Eliminates platform compatibility issues.

Advantages of Container:

i) Containers are portable (you can run the application from any environment)

ii) We can quickly deploy the application using Container

iii) Easy is scale-up and down the application based on the load of the container

Docker Concept/Terminologies:

Docker File: It's a file that contains the instructions or it has steps to create an image and it's a source code for the image file.

Docker Image: It's a package that contains application code $ required dependencies (like Software, Environment variable, libraries, and config files )  

Docker Container: It's a runtime process of your image where our application process would be running.

Docker Registry/Repository: It maintains the Docker Images.

Docker Volumes: It retains the state of Docker Containers/applications 

Container Orchestration tool:

Docker Swarm

Kubernetes 

 

Docker Data Flow:

Docker File -> Create an Image -> Create Container

Docker Architecture:

Docker Client: It's a command-line interface (CLI) through which the user can send the commands to docker and it interacts with Docker Deamon.

Docker Deamon/Host/Engine: It's a program or software and it executes the commands sent by the user or CLI. The commands could install software, pulling an image from the registry. 

Docker Registry/Repository: It's a central repository maintained by Docker and called as Docker Hub. It contains different public images and it's a public registry.

Public Registry: Docker Hub (it has all the Public software as images)

Private Registry: 

ECR

Nexus

JFrog

DTR - Docker Trusted Registry and it's a part of EE of Docker.

UCP - Universal Control Pane and it's a part of EE of Docker.

What is Registry?

It's a place where we can have multiple Repositories.

Install Docker

$ sudo curl -fssL get.docker.com | /bin/bash

$ docker --version

By default, you can't able to access docker commands post the docker installation.

to enable docker access without sudo you need to add a user to the docker group and by default, docker installation creates a "docker" group.

$ sudo usermod -aG docker ubuntu

Note: post adding user to group exit the current shell and log on again.

Note: Docker is cross-platform, docker can be installed on Linux (Officially not supported), Mac, and Windows. Docker CE can be installed on any Linux flavor except Redhat. However, we could install it on RedHat but we don't get support from Docker.

Different methods to install Docker:

Method-1: Using the script available at get.docker.com written by Docker.

Method-2: using the curl command.

$ sudo curl -fssL get.docker.com | /bin/bash

Method-3: Package manager

$ sudo apt install docker.io

Create a Docker image:

Step-1: Copy the application folder on the local server and then run the below steps to update the git repository.

$ git add .

$ git commit -m "Dockerfile add"

$ git push

Step-2: Create a Docker file

$ cat Dockerfile

FROM tomcat:7-jdk8-corretto

COPY target/maven-web-application*.war /usr/local/tomcat/webapps/maven-web-application.war

The docker file instruction is to install java and copy the project.

Step-3: Create a Docker image

Pre-requisites:

i) Maven and Java to be installed (by default Maven will install Java)

$ sudo apt install maven 

ii) Create an application package

$ maven clean package

iii) Create an account in the hub.docker

Create a Docker image:

docker build -t <registry_name>/<repository_name>:<version/tag> .

Here the "." represent the docker context and t is the tag

Example: docker build -t feroz7861/maven-web-application:1.

Output:

$ docker build -t feroz7861/maven-web-application:1 .

Sending build context to Docker daemon  22.35MB

Step 1/2 : FROM tomcat:7-jdk8-corretto

 ---> c4a8b5f574fa

Step 2/2 : COPY target/maven-web-application*.war /usr/local/tomcat/webapps/maven-web-application.war

 ---> Using cache

 ---> 54543903ec62

Successfully built 54543903ec62

Successfully tagged feroz7861/maven-web-application:1

Note: Make sure you have Maven and java installed before create a Docker image.

list the Docker images:

$ docker images

REPOSITORY                            TAG               IMAGE ID       CREATED      SIZE

feroz7861/maven-web-application       1                 54543903ec62   2 days ago   380MB

tomcat                                7-jdk8-corretto   c4a8b5f574fa   2 days ago   375MB

Step-4: Push the image/images to the Docker Registry/Repository 

Initially, if you try to push the image to the repository it will fail because authentication is required.

Error: 

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/json: dial unix /var/run/docker.sock: connect: permission denied

$ docker login -u feroz781

password: xxxx

$ docker push feroz7861/maven-web-application:1

Output: 

$ docker push feroz7861/maven-web-application:1

The push refers to repository [docker.io/feroz7861/maven-web-application]

1bccf91405e8: Pushed

266914055bb4: Pushed

46ba1691b29a: Pushed

e4b6d59e3861: Pushed

7540e9affc96: Pushed

cee8d35c645b: Pushed

1: digest: sha256:f4df58463da7e3f42e4b4d10d64dd06bb0737d8baf46532645d17f4e24817d58 size: 1579

For now, all the images are in the Docker Repository. We now copy to the build server (locally).

Create Container:

Once all images are available in the Docker repository, connect to the Deployment server and create a container.

$ docker run -d --name mavenwebcontainer -p 8080:8080 feroz7861/maven-web-application:1

$ docker ps 

Output:

Container Creation details:

$ docker run -d --name mavenwebcontainer -p 8080:8080 feroz7861/maven-web-application:1

Unable to find image 'feroz7861/maven-web-application:1' locally

1: Pulling from feroz7861/maven-web-application

4232b9ee675b: Pull complete

638373d2fdad: Pull complete

51ceb0aff11f: Pull complete

d750517ee3c2: Pull complete

0bdcd24af4d1: Pull complete

f15725d1014f: Pull complete

Digest: sha256:f4df58463da7e3f42e4b4d10d64dd06bb0737d8baf46532645d17f4e24817d58

Status: Downloaded newer image for feroz7861/maven-web-application:1

5ef6c199515666d0342494a217402da116361166886bbd5e7de6da7d40660d57

Container running details: docker ps command to check the container status

$ docker ps

CONTAINER ID   IMAGE                               COMMAND             CREATED          STATUS          PORTS                    NAMES

5ef6c1995156   feroz7861/maven-web-application:1   "catalina.sh run"   29 seconds ago   Up 28 seconds   0.0.0.0:8080->8080/tcp   mavenwebcontainer

Docker Image Commands:

i) Create Docker image

docker build -t <registry>/<repo>:<version> .

Where "-t" represents the tag and "." represents the current directory or docker build context.

Example: 

$docker build -t feroz7861/maven-web-application:1

Note: If we don't mention the version, by default it takes "latest" as a version. 

ii)  List the Docker images

$ docker images

$ docker image ls

$ docker inspect <id/name>:version

$ docker image inspect <id/name>:version 

iii) List the layers of an image

$docker history <image_name/id>

iv) Push the image to the resitry

$docker push <imageName> 

Where <imageName> is <registry>/<repo>:<version> 

v) Authenticate to push the image to the repository 

$ docker login -u <username> -p <password> <registry>   ---- Public Repsository

$ docker login -u <username> -p <password> <URL>    ------  Private Repository

vi) Pull the image 

$docker pull <image_name>

vii) Display only the image ids

$docker images -q

vii) Remove image from the local server

$docker rmi <image_id/name> 

$docker rmi <image_id/name> ,   <image_id/name>  ---- multiple images deletion 

$ docker rmi $ (docker images -q)   ------------- delete all the images

viii) Remove dangling images

$ docker image prune     ------ removes all dangling images

$ docker system prune  --- removes all stopped containers, all network not used at least one container,  

                                           all dangling images, all dangling build cache

ix) Tar the Docker image

$docker save <image_name/id> -o <file_name>.tar

x) load the image from the tar file

$docker load -i <file_name>.tar

Configure the Private Repository :

NExus/JFrag

DTR --> Docker Trusted Registry 

ECR - > Elastic Container Registry (AWS)

Steps to configure Private Repository:

Step-1: Create an ECR repository in AWS

ECR -> Repositories -> Visibility Setting -> Private -> Reposiroty name -> Create Repository 

Step-2: To maintain the images in ECR Repository, you need to tag the images with the ECR URL.

$ docker build -t 193080714279.dkr.ecr.ap-south-1.amazonaws.com/maven-web-application:3 .

ECR URL: 193080714279.dkr.ecr.ap-south-1.amazonaws.com

Step-3: Authenticate the docker login with ECR since it's a private Repository

i) AWS CLI should be installed before you run push the image to the private repository.

If AWS CLI not installed run the below steps to install it

$sudo apt install awscli

ii) Get the password from ECR repository by selecting the repository name and View push command 

ECR -> Select ECR repository -> View push command -> copy the lines from "Retrieve an authentication token and authenticate your Docker client to your registry."

While pushing the image to the ECR repository got the below auth credential error.

Error:

$ docker push 193080714279.dkr.ecr.ap-south-1.amazonaws.com/maven-web-application:3

The push refers to repository [193080714279.dkr.ecr.ap-south-1.amazonaws.com/maven-web-application]

8abadb54bb72: Preparing

266914055bb4: Preparing

46ba1691b29a: Preparing

e4b6d59e3861: Preparing

7540e9affc96: Preparing

cee8d35c645b: Preparing

no basic auth credentials

Follow the below steps to fix the authentciaton issues with ECR repository:

i) create a role

IAM-> Role-> Create Role -> Choose use cases as "EC2" -> Click on Permissions -> AmazonEC2ContainerRegistyFullAccess -< Next -> Define Role_name-> Create role

ii) Attach a role to the EC2 instance

Select EC2 Server-> Actions -> Security -> Modify IAM role -> Select role to attach -> Save 

iii) get the credentials /token  using AWSCLI command line (it generate the token)

$ aws ecr get-login-password --region ap-south-1

iv) authenticate the login 

$docker login -u AWS -p <token> OR

aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin 193080714279.dkr.ecr.ap-south-1.amazonaws.com

$ aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin 193080714279.dkr.ecr.ap-south-1.amazonaws.com

WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.

Configure a credential helper to remove this warning. See

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Step-3: Push the image

$ docker push 193080714279.dkr.ecr.ap-south-1.amazonaws.com/maven-web-application:3

 

$ docker push 193080714279.dkr.ecr.ap-south-1.amazonaws.com/maven-web-application:3

The push refers to repository [193080714279.dkr.ecr.ap-south-1.amazonaws.com/maven-web-application]

8abadb54bb72: Pushed

266914055bb4: Pushed

46ba1691b29a: Pushed

e4b6d59e3861: Pushed

7540e9affc96: Pushed

cee8d35c645b: Pushed

3: digest: sha256:7f3eb760002ac4ca768ba8bf34318a4cd30c0018b0d24fcb4e8866eee5232d5d size: 1579

Container Commands

i) Container create and run

$docker create --name <ContainerName> -p <HostPort>:<ContainerPort> <ImageName>

$docker run --name <ContainerName> -p <HostPort>:<ContainerPort> <ImageName>

$docker run -d  --name <ContainerName> -p <HostPort>:<ContainerPort> <ImageName> 

Setting limits on resources while creating containers:

$ docker run --memory="2048M" --cpus="4"  

$ docker create --memory="2048M" --cpus="4"  

docker create: It just creates the container but it will not start up the container or process.

docker run: It creates the container and startup the container as well.

-p -> publish port OR port mapping

ii) list all runing containers

$docker ps

$docker container ls 

iii) list of available containers and status includes both Running and Stopped 

$docker ps -a 

$docke container ls -a 

iv) Startup the Container

$docker start <ContainerName/Id>

v) To list more details for Containers

$docker inspect <ContainerName/Id>

$docker container inspect <ContainerName/Id>

vi) Remove containers

$docker rm <ContainerName/Id>

$docker rm -f <ContainerName/Id>

$docker rm -f $(docker ps -qa)   ------ to remove all containers

vii) Check what process is running as part of Container

$docker top <Container_name/ID>

viii) validate the details on a running container

docker exec <Container_name/ID>

To see the directory structure:

docker exec <Container_name/ID> ls

To see the working directory:

docker exec <Container_name/ID> pwd

To see the application files part of the container

docker exec <Container_name/ID> ls webapps

ix) Check the status of the Container

docker stats <ContainerName/Id>

x) Execute the commands at the Container level

docker exec <<ContainerName/Id> <command>

a)To see the directory structure:

$docker exec <Container_name/ID> ls

b)To see the working directory:

$docker exec <Container_name/ID> pwd

c)To see the application files part of the container

$docker exec <Container_name/ID> ls webapps 

d) run the bash command at container level

$docker exec -it <ContainerName/ID> /bin/bash

$docker exec -it <ContainerName/ID> /bin/sh

where "-it" is the interactive 

xi) Monitor the ongoing or current activity on Container or application or process level

$docker attach <ContainerName/ID>

xii) Rename the Container

$docker rename <Old_ContainerName> <New_ContainerName>

xiii) Stopping the Container

$docker stop <ContainerName/ID>

xiv) Kill the  Container

$docker kill <ContainerName/ID>

Difference between kill and stop container:

docker stop: Safely Stopping

SIGSTOP

SIGTERMINATION

docker kill: like abort

xv) Copy files or folders from one docker host (Server) to Container or Container to docker host (Server)

$ docker cp <source> <destination>

$docker cp <file/Folder_location> <ContainerId/Name>: /<LocationofDesitnaion>

$docker cp <ContainerId/Name>: /<LocationofDesitnaion> <file/Folder_location>

Additional commands for Docker Container:

docker start <Container_name/ID>

docker stop <Container_name/ID>

docker restart <Container_name/ID>

docker pause <Container_name/ID>

docker unpause <Container_name/ID>

docker commit <Container_name/ID> <ImageName>

docker image prune

docker container prune

docker network prune

docker system prune

Steps to troubleshoot/Debug the application that is running as a container: 

i) Check Container is up and running

docker ps

ii) Check your expected processes are up and running

docker top <ContainerName/ID>

iii) Check the container logs

docker logs <ContainerName/ID>

iv) Check more details inside the container

a)To see the directory structure:

docker exec <Container_name/ID> ls

b)To see the working directory:

docker exec <Container_name/ID> pwd

c)To see the application files part of the container

docker exec <Container_name/ID> ls webapps 

d) Monitor the ongoing or current activity on Container or application or process level

$docker attach <ContainerName/ID>

Docker File Instructions

It's a file that has instructions to create an image. We write a docker file using docker domain-specific keywords.

FROM

MAINTAINER

COPY

ADD

RUN

CMD

ENTRYPOINT

WORKDIR

ENV

ARG

LABEL

EXPOSE

VOLUME

USER

FROM: 

It indicates the base image on top of it we can our own images.

FROM <registry/repository>:<version>

MAINTAINER:

It's like an Author of the docker file.

COPY

It's an instruction in a Docker file, which copy local files/folder to the image while creating an image.

COPY <Source> <Destination> 

The Source file should be part of the Docker build context.

The Destination is basically the location of the image where you want to copy.

 

ADD

Copy local files and can also download the files from HTTP(s) locations. It works like "wget". If we add a tar file using ADD instruction having an extension ".tar" it will add the file and also extract the file.

ADD <URL> <Destination_Location>

ADD <SourceLocation> <Destination_Location>

  

RUN

Run Instruction command can be executed while creating an image. It's executed on top of your base image like if you have wget installed then you could RUN wget available.

We use RUN instruction to configure/install the required software while creating an image.

RUN <cmd>

Example: 

RUN mkdir test

RUN wget <URL>

CMD

CMD instruction will execute while creating (Starting) a container and it's used to start up the process inside the container. CMD instruction can be overridden while creating a container.

CMD will be used as an argument for ENTRYPOINT.

ENTRYPOINT

ENTRYPROINT instruction will be executed while creating a container. We can set ENTRYPOINT for the container. ENTRYPOINT can't be overridden.

ENV

We can set the environment variable using ENV instruction.

EN <KEY> <VALUE>

WORKDIR

We can see a working directory for a container or an image using WORKDIR instruction in the Docker File.

WOKDIR <Location>

USER

We can define the owner for the container to start up and running using USER instruction.

USER <UserName>

EXPOSE

The EXPOSE indicates on what port number the container is running. It's a kind of documentation about port usage.

LABEL

Labels are like a tag where we can attach them to the container.

ARG

It's like a Variable can be referred to in the Doker File and it's only used inside the Docker File Whereas ENV.

Best Practices while creating a Docker File

i) Use only official base images.

ii) Reduce the number of layers when and where needed (using &&). Because, if you have more layers the performance of the application/container will impact.

iii) Install only needed software while creating containers. Avoid installing unnecessary software/packages to avoid CPU resource utilization. 

iv) Try to use alpine images wherever it's possible.

Questions and Answers

1. Can we delete the image if the container is up and running and the image is being used?

No, it won't delete the image with the force option it just removes the reference but not the image.

$docker rmi -f <image_name> 

2. What is a dangling image?

The image doesn't have a repository reference. 

3. How to tag a dangling image to any repository?

$docker tag <Image_id/name>  <registry>/<repo>:<version>

$docker tag <Image_id/name> <ECR>/<repository>:1

4. What is the difference between Docker Repository and Docker Registry?

The registry is a collection of repositories.

Ther Repository is a collection of images with their different versions. 

5. What is the location where docker images store or What is the location of the Docker Working directory?

$ /var/lib/docker

$ ls /var/lib/docker/

buildkit  containers  image  network  overlay2  plugins  runtimes  swarm  tmp  trust  volumes

6. How to see the layer of an image?

$docker history <image_name/id> 

7. What does "docker system prune" does in the background?

It removes:

all stopped containers, 

all network not used at least one container,  

all dangling images, and 

all dangling build cache

8. How to move the image from One Server to Another Server without Docker Registry/Repository to run the container out of that image?

zip/tar the docker image using docker save command and scp the image file.

$docker save <image_name/id> -o <file_name>.tar

9. What is the "docker save" command?

it creates the tar for the docker images which contains all the layers of the image.

10. What is the docker load command?

It loads the images from the tar file

11. What is Port mapping or Port Publish?

If you want to access the application which is running as a container and want to access the application outside the container server then you need to do the port Mapping or Publish.

Example: -p <HostPort>:<ContainerPort>

12. What is the difference between Docker cp and Copy?

COPY -> It's an instruction in a Docker file, which copy local files/folders to the image while creating an image.

cp -> Copy the files to the container

13. What is a "docker commit" do?

You can retain all your container changes as an image. 

14. How to delete only stopped container?

$docker container prune

$docker rm -f $(docker ps -aq -f "status=exited")

15. What is the difference between COPY and ADD?

16. Can we have more than one instruction in the Docker File?

Yes, We can any number of instructions. The Docker will process the instructions from the top to bottom approach.  

17. What is the difference between RUN and CMD?

18. Can we have more than one CMD instruction in the Docker File?

Yes, we can have but only one CMD will be executed as per the order/sequence or the last one.

19. What is the difference between CMD and ENTRYPOINT?

CMD

CMD instruction will execute while creating (Starting) a container and it's used to start up the process inside the container. CMD instruction can be overridden while creating a container.

ENTRYPOINT

ENTRYPROINT instruction will be executed while creating a container. We can set ENTRYPOINT for the the container. ENTRYPOINT can't be overridden.

20. How to avoid using cache while creating the container?

$docker build --no-cache -t <ContainerName>

21. How to use your own Docker file instead the default named Docker file whicle creating a Container?

$ docker build -t <ContainerName> -f <YourOwnDockerFileName> .

22. Can we have both CMD and ENTRYPOINT in the Docke File?

Yes, but the CMD can be taken as an argument for the ENTRYPOINT.

Sample Docker File:

FROM centOS

CMD ["file1"]

ENTRYPOINT["ls","/"]

The output would be:

ls /

ls file1

23. In what scenario we use both CMD and ENTRYPOINT in the Docke File?

24. What is the difference between RUN and ENTRYPOINT?

25. What is the difference between Docker run and RUN?

26. What is the difference between ENV and LABEL?

27. How can we pass an argument while creating an image?

with $ symbol

28. What is the difference between ENV and ARG?