Excerpt

hidden	true

CLI-suite for container

Terminology

Container: a docker container
Container type: a term describing a set of containers that have the same role, e.g. api, core, booking-engine. It is defined by a corresponding Task Definition.
ECS Service: manages the containers of a certain container type.
Image: a docker image
Docker Cluster: A logical entity consisting of a number of docker hosts
Task Definition: an extended, ECS-specific docker compose file (JSON)
ECS Task: Instance of a Task Definition, usually represents a single container
Desired Tasks: A number of containers based on the same image (horizontal scale)
Revision: Version of an ECS Task, that has a set of properties, such as an image-URL pointing to an image in a registry
Registry: a system for storing and managing images
Repository: a logical space seperator within the registry to separate images e.g. by type or purpose

The deployment process

Docker Container Deployment describes a process of updating a fleet of running docker containers that are operated via Amazon AWS EC2 Container Service (ECS).

Deployment strategies

Per default a rolling update is used, starting new containers based on a new docker image. The method is explained below in the context of a web application. Knowledge of typical ECS terms are useful. Besides rolling updates it is also possible to have replacing updates. This means running containers of a container type are terminated and then new ones are launched based on the submitted image.

The main difference between replacing and rolling updates is that deployments run with 0 downtime only in the later case. However, additional capacity has to be provided permanently. Thus increasing cost of permanently running docker hosts in the cluster.

Process example

Initiation of deployment from the bastion host via CLI for a Container Type providing a new image-URL
Creation of a new ECS Task Definition with the new image-URL, which has its own version number (revision)
Update of the respective ECS Service managing the Containers of the Container Type of consideration
Creation of new Containers based on the new image-URL in the Docker Cluster
All new Containers are taken into Service from the Load Balancer
Connections to all old containers are drained by the Load Balancer, meaning established ones are allowed to finish, bu no new ones are established to them
Shutdown of all old containers
Finish

Safeguards

Several mechanism are in place to recognize success and handle failure and rollback:

Container state verification
- A deployment is successful if all new containers are recognized as running and the Load Balancer has taken them into service successfully
Image-URL verification
- Checking to given Image-URL to be a valid String
- If the docker registry images should be pulled from is part of your environment (within the same AWS account) we try to locate the given image
AWS Deployment Circuit Breaker

Native docker commands

The following commands can be called via "sudo" by default, meaning in every environment:

docker ps, docker ps -a
docker images, docker images -a
docker logs [OPTION] CONTAINERID
docker exec -it CONTAINERID bash, docker exec -it CONTAINERID sh
docker stats, docker stats CONTAINERID
docker inspect CONTAINERID

For non-productive environments root360 can also enable access to the following commands e.g. to support debugging and analysis:

docker run
docker pull
docker start
docker create

You can find further details about the individual commands in docker cli reference.

Example usage scenarios

A great example is the docker start command. If you deployed a new container image to your test environment and it wouldn't come up you are in a dilemma.

On the one hand you want to look into the failing container, but on the other hand AWS ECS keeps trying to launch and terminate containers. Thus it will constantly kill the container you are actually want to keep a little bit longer for analysis.

By using docker start to run a container image AWS ECS has created on a docker host before, you can do so without AWS ECS interfering anymore.

CLI-suite container commands

All commands operating on docker hosts and containers and the tasks to manipulate them are available in the r3 container subcommand. A general description how the r3 CLI Suite works can be found in CLI-Suite (root360)

In the following actual usage examples are presented.

What are the general command line options?

r3 container overview

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container --help
usage: r3 container [-h] [-d] [-q] {deploy,list,templates,instances,status,image,events,logs,show} ...

options:
  -h, --help            show this help message and exit
  -d, --debug           Run in debug mode. (default: 0)
  -q, --quiet           Run in quiet mode (only errors are shown). (default: False)

command overview:
  {deploy,list,templates,instances,status,image,events,logs,show}
    deploy              Deploy containers to hosts.
    list                List running containers.
    templates           List available container templates.
    instances           List container instances.
    status              Show container status.
    image               Show container image.
    events              Show container event log.
    logs                Show container logs.
    show                Show detailed information about containers.

Show which container is running on which host including resources overview as table

r3 container list

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container list
 
+---------------------------------------+----------------------+--------------+-----------------+---------+-----------------+------------+----------+---------------------+
|             Container Name            |        Image         | Reserved CPU | Reserved Memory |  Status | Running/Desired | LaunchType |   Host   |      LaunchTime     |
+---------------------------------------+----------------------+--------------+-----------------+---------+-----------------+------------+----------+---------------------+
|   company-project-setting-hello-ec2   |    hello-ec2:1.0     |     500      |       1000      | running |      2 / 2      |    EC2     | 10.x.y.z | 2021-05-04_00:59:30 |
+---------------------------------------+----------------------+--------------+-----------------+---------+-----------------+------------+----------+---------------------+
|   company-project-setting-hello-ec2   |    hello-ec2:1.0     |     500      |       1000      | running |      2 / 2      |    EC2     | 10.x.y.z | 2021-05-04_00:59:41 |
+---------------------------------------+----------------------+--------------+-----------------+---------+-----------------+------------+----------+---------------------+
| company-project-setting-hello-fargate | hello-fargate:latest |     512      |       1024      | running |      1 / 1      |  FARGATE   | 10.x.y.z | 2021-05-04_00:21:30 |
+---------------------------------------+----------------------+--------------+-----------------+---------+-----------------+------------+----------+---------------------+
|  company-project-setting-hello-multi  | hello-fargate:latest |     512      |       1024      | running |      1 / 1      |  FARGATE   | 10.x.y.z | 2021-05-04_00:21:30 |
|                                       | hello-ec2:1.0        |              |                 |         |                 |            |          |                     |
+---------------------------------------+----------------------+--------------+-----------------+---------+-----------------+------------+----------+---------------------+

Show which container is running on which host including resources overview as JSON

Using --format json you get JSON output which can be parsed using e.g. jq command line tool (Manual).

r3 container list --format json

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container list --format json | jq
{
  "instances": [
    {
      "instance_ip": "10.x.y.z",
      "mem_registered": 7624,
      "mem_remaining": 4712,
      "cpu_registered": 4096,
      "cpu_remaining": 2096,
      "tasks": [
        {
          "name": "company-project-setting-hello-ec2",
          "images": [
            "hello-ec2:1.0"
          ]
          "cpu": "500",
          "memory": "1000",
          "status": "running"
        },
        {
          "name": "company-project-setting-hello-ec2",
          "images": [
            "hello-ec2:1.0"
          ]
          "cpu": "500",
          "memory": "1000",
          "status": "running"
        },
        {
          "name": "company-project-setting-multi",
          "images": [
            "hello-fargate:latest",
            "hello-ec2:1.0",
          ]
          "cpu": "500",
          "memory": "1000",
          "status": "running"
        }
      ]
    }
  ]
}

Show all containers running on FARGATE using JSON and JQ

Code Block

language	bash

USER@BASTIONHOST:~$ containers="$(r3 container list --format json)"
USER@BASTIONHOST:~$ jq -r '.services[] | select(.launchtype=="FARGATE") | .name' <<<$containers
company-project-setting-hello-fargate

Show all containers including EC2 host ip address using JSON and JQ formated as CSV

Code Block

language	bash

USER@BASTIONHOST:~$ containers="$(r3 container list --format json)"
USER@BASTIONHOST:~$ jq -r '.services[] | select(.launchtype=="EC2") | [.name, .host] | @csv' <<<$containers
"company-project-setting-hello-ec2","10.x.y.z"
"company-project-setting-hello-ec2","10.x.y.z"

Show all broken (not running) containers using JSON and JQ

Code Block

language	bash

USER@BASTIONHOST:~$ containers="$(r3 container list --format json)"
USER@BASTIONHOST:~$ jq -r '.services[] | select(.status!="running")' <<<$containers

Show all container instances and their services as table

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container list --instances
+----------+----------+---------+-------------+------------+------------------------------------------------+
|   Host   | CPU free | CPU max | Memory free | Memory max |          Tasks (Name - CPU - Memory)           |
+----------+----------+---------+-------------+------------+------------------------------------------------+
| 10.x.y.z |   2096   |   4096  |     4712    |    7624    | company-project-setting-hello-ec2 - 500 - 1000 |
|          |          |         |             |            | company-project-setting-hello-ec2 - 500 - 1000 |
+----------+----------+---------+-------------+------------+------------------------------------------------+

Show all container instances and their services as JSON (more details)

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container list --instances --format json
{
  "instances": [
    {
      "instance_ip": "10.x.y.z",
      "mem_registered": 7624,
      "mem_remaining": 4712,
      "cpu_registered": 4096,
      "cpu_remaining": 2096,
      "tasks": [
        {
          "name": "company-project-setting-hello-ec2",
          "image": "hello-ec2:1.0",
          "cpu": "500",
          "memory": "1000",
          "status": "running"
        },
        {
          "name": "company-project-setting-hello-ec2",
          "image": "hello-ec2:1.0",
          "cpu": "500",
          "memory": "1000",
          "status": "running"
        }
      ]
    }
  ]
}

Find instance where specific service is running using JSON and JQ

Code Block

language	bash

USER@BASTIONHOST:~$ containers="$(r3 container list --instances --format json)"
USER@BASTIONHOST:~$ jq -r '.instances[] | select(.tasks[].name == "company-project-setting-hello-ec2") | .instance_ip' <<<$instances

Show container logs for all containers

r3 container logs

Show container logs for specific container

r3 container logs --service ServiceName

Which container services can be deployed?

r3 container list --templates

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container list --container-types
 
# Command Response:
['container-type-a', 'container-type-b', ...]

How to start a new deployment for container-type x?

Info
You can start deployments in background using `--detach`. To keep track of a detached deployment, you can use the `r3 deployments --service ServiceName --deployment <id> --watch`

Single container tasks

r3 container deploy --service Service-Name --image-url Registry/Repository:Tag

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container deploy --service [company]-[project]-[setting]-[name] --image-url amazon/amazon-ecs-sample
 
# Command Response:
Initiating Deployment for [company]-[project]-[setting]-[name]: amazon/amazon-ecs-sample
Registered task definition arn:aws:ecs:eu-central-1:385026310112:task-definition/company-project-setting-name:3 with image amazon/amazon-ecs-sample
Initiated deployment for [company]-[project]-[setting]-[name] with arn:aws:ecs:eu-central-1:385026310112:task-definition/company-project-setting-name:3. Waiting for deployment to finish ...
(service [company]-[project]-[setting]-[name]) has started 1 tasks: (task <new-task-id>).
(service [company]-[project]-[setting]-[name]) has stopped 1 tasks: (task <old-task-id>).
(service [company]-[project]-[setting]-[name]) has reached a steady state.

Multi container tasks

r3 container deploy --service Service-Name --image-url container1=Registry/Repository:Tag container2=Registry/Repository:Tag

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container deploy --service [company]-[project]-[setting]-[name] --image-url foo=amazon/amazon-ecs-sample bar=docker.io/caddy:alpine
 
# Command Response:
Initiating Deployment for [company]-[project]-[setting]-[name]: foo=amazon/amazon-ecs-sample,bar=docker.io/caddy:alpine
Registered task definition arn:aws:ecs:eu-central-1:385026310112:task-definition/company-project-setting-name:3 with image amazon/amazon-ecs-sample
Initiated deployment for [company]-[project]-[setting]-[name] with arn:aws:ecs:eu-central-1:385026310112:task-definition/company-project-setting-name:3. Waiting for deployment to finish ...
(service [company]-[project]-[setting]-[name]) has started 1 tasks: (task <new-task-id>).
(service [company]-[project]-[setting]-[name]) has stopped 1 tasks: (task <old-task-id>).
(service [company]-[project]-[setting]-[name]) has reached a steady state.

How to list deployments?

r3 deployments --service ServiceName

Code Block

USER@BASTIONHOST:~$ r3 container deployments --service ServiceName
+---------------------+-------------------------------+--------------+---------+----------------------------------+-------------+--------------+--------------+
|          Id         |           createdAt           | rolloutState |  status |          taskDefinition          | failedTasks | runningCount | desiredCount |
+---------------------+-------------------------------+--------------+---------+----------------------------------+-------------+--------------+--------------+
| 7308241944486273163 | Thu, 04 May 2023 08:58:30 GMT |  COMPLETED   | PRIMARY |         arn:aws:ecs:eu-          |      0      |      2       |      2       |
|                     |                               |              |         |   central-1:000000000000:task-   |             |              |              |
|                     |                               |              |         |   definition/company-project-    |             |              |              |
|                     |                               |              |         |          setting-name:67         |             |              |              |
+---------------------+-------------------------------+--------------+---------+----------------------------------+-------------+--------------+--------------+

On which image do the currently running containers of type X base?

r3 container image --service Service-Name

Code Block
USER@BASTIONHOST:~$ r3 container image --service ServiceName +--------------------------+ \| Images \| +--------------------------+ \| amazon/amazon-ecs-sample \| +--------------------------+

Show the latest events for a given container type

r3 container show events --service ServiceName

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container events --service ServiceName
 
 
# Command Response:
[
  {
    "events": [
      {
        "message": "(service CONTAINER-TYPE-Name) has reached a steady state.",
        "id": "4c23824b-ae16-4c33-aeb1-f4f9fd0b713b",
        "createdAt": 1448643717.709
      },
      {
        "message": "(service CONTAINER-TYPE-Name) stopped 1 running tasks.",
        "id": "55bd871a-ab68-4f1b-87fe-77f8f0135e59",
        "createdAt": 1448643675.905
      },
      {
        "message": "(service CONTAINER-TYPE-Name) has begun draining connections on 1 tasks.",
        "id": "5608da62-6697-44f0-891e-9a5911b36bdc",
        "createdAt": 1448643665.745
      },
      {
        "message": "(service CONTAINER-TYPE-Name) deregistered 1 instances in (elb port-r3WebELB-TOKW867P8Q3N)",
        "id": "ff828587-a1d8-4a64-bed2-b935db28f2ac",
        "createdAt": 1448643665.744
      },
      {
        "message": "(service CONTAINER-TYPE-Name) registered 1 instances in (elb port-r3WebELB-TOKW867P8Q3N)",
        "id": "04226066-b501-42a8-a9cf-118cbc076a62",
        "createdAt": 1448643643.593
      },
      {
        "message": "(service CONTAINER-TYPE-Name) deregistered 1 instances in (elb port-r3WebELB-TOKW867P8Q3N)",
        "id": "341d0d15-1d9e-4a19-bb8f-5be2ddea5ed7",
        "createdAt": 1448643465.881
      },
      {
        "message": "(service CONTAINER-TYPE-Name) has started 1 tasks: (task 5d1c1443-0dff-4378-bce3-ed116db70b96).",
        "id": "6a85e881-2514-43c3-b039-7ba9daa20cab",
        "createdAt": 1448643465.881
      },
...
      {
        "message": "(service CONTAINER-TYPE-Name) has reached a steady state.",
        "id": "428eaa0f-2c0b-484b-883d-782fa5af65a7",
        "createdAt": 1448534247.066
      }
    ]
  }
]

Show the deployment status for a container-type

Code Block

language	bash

USER@BASTIONHOST:~$ r3 container list
+--------------------------------+-----------------------------------------+------------------------+--------------+-----------------+---------+-----------------+-----------+------------+------------+---------------------+
|            Cluster             |              Container Name             |         Image          | Reserved CPU | Reserved Memory |  Status | Running/Desired | Deploying | LaunchType |    Host    |      LaunchTime     |
+--------------------------------+-----------------------------------------+------------------------+--------------+-----------------+---------+-----------------+-----------+------------+------------+---------------------+
|    company-project-setting     |      company-project-setting-service-a  |  image-name:image-tag  |     512      |       1024      | running |      1 / 1      |   False   |  FARGATE   |  10.x.y.28 | 2021-08-03_14:13:05 |
+--------------------------------+-----------------------------------------+------------------------+--------------+-----------------+---------+-----------------+-----------+------------+------------+---------------------+
|    company-project-setting     |     company-project-setting-service-b   |  image-name:image-tag  |     1024     |       2048      | running |      2 / 2      |   False   |  FARGATE   |  10.x.y.99 | 2021-08-19_14:34:39 |
+--------------------------------+-----------------------------------------+------------------------+--------------+-----------------+---------+-----------------+-----------+------------+------------+---------------------+
|    company-project-setting     |     company-project-setting-service-b   |  image-name:image-tag  |     1024     |       2048      | running |      2 / 2      |   False   |  FARGATE   | 10.x.y.149 | 2021-08-19_14:34:42 |
+--------------------------------+-----------------------------------------+------------------------+--------------+-----------------+---------+-----------------+-----------+------------+------------+---------------------+

Show detailed information about all containers of container-type X (see following example for explanation)

r3 container status --service Service-Name

Code Block

USER@BASTIONHOST:~$ r3 container status --service Service-Name
 
# Command Response:
[
  [
    {
      "taskArn": "arn:aws:ecs:eu-west-1:XXXX:task/29dce5e3-16d8-45fe-acc2-0da74d93f069",
      "group": "service:company-project-prod-monitor",
      "containers": [
        {
          "containerArn": "arn:aws:ecs:eu-west-1:XXXX:container/0844bc9c-ff24-459e-b230-715cc72df95b",
          "taskArn": "arn:aws:ecs:eu-west-1:XXXX:task/29dce5e3-16d8-45fe-acc2-0da74d93f069",
          "lastStatus": "RUNNING",
          "name": "company-project-prod-monitor",
          "networkBindings": [
            {
              "bindIP": "0.0.0.0",
              "protocol": "tcp",
              "containerPort": 9100,
              "hostPort": 9100
            },
            {
              "bindIP": "0.0.0.0",
              "protocol": "tcp",
              "containerPort": 9126,
              "hostPort": 9126
            },
            {
              "bindIP": "0.0.0.0",
              "protocol": "tcp",
              "containerPort": 8080,
              "hostPort": 8085
            }
          ]
        }
      ],
      "overrides": {
        "containerOverrides": [
          {
            "name": "company-project-prod-monitor"
          }
        ]
      },
      "lastStatus": "RUNNING",
      "containerInstanceArn": "arn:aws:ecs:eu-west-1:XXXX:container-instance/f5f5d9ac-ded3-4780-9453-a07ea3108350",
      "version": 3,
      "clusterArn": "arn:aws:ecs:eu-west-1:XXXX:cluster/company-project-prod",
      "desiredStatus": "RUNNING",
      "startedAt": 1508248278.273,
      "taskDefinitionArn": "arn:aws:ecs:eu-west-1:XXXX:task-definition/company-project-prod-monitor:24",
      "startedBy": "ecs-svc/9223370528606525657",
      "createdAt": 1508248275.161
    }
  ],
  [
    {
      "taskArn": "arn:aws:ecs:eu-west-1:XXXX:task/2c12b253-3f2f-4277-8b9a-4b8d26a5a75e",
      "group": "service:company-project-prod-monitor",
      ...
    }
  ],
  [
    {
      "taskArn": "arn:aws:ecs:eu-west-1:XXXX:task/83ba896d-582e-487c-bc94-19b3e02b8742",
      "group": "service:company-project-prod-monitor",
    ...   
    }
  ]
]

The important points here are:

"lastStatus" describes the current status
"desiredStatus" describes the target state
"stoppedReason": is only to be found with stopped containers and represents in detail the reason for the state.

Related components

Filter by label (Content by label)

showLabels	false
max	10
sort	title
showSpace	false
cql	label in ( "container" , "cli-suite" ) and ancestor = "2014350220" and space = currentSpace ( )

Status

colour	Yellow
title	intermediate

Table of Contents

exclude	(Related * \| Recommended * \|Table of contents).*

Filter by label (Content by label)

showLabels	false
max	10
sort	title
showSpace	false
title	Related questions
cql	label in ( "container" , "cli-suite" ) and ancestor = "2014351598" and space = currentSpace ( )

Versions Compared

Old Version 18

New Version Current

Key

Terminology

The deployment process

Deployment strategies

Process example

Safeguards

Native docker commands

Example usage scenarios

CLI-suite container commands

What are the general command line options?

Show which container is running on which host including resources overview as table

Show which container is running on which host including resources overview as JSON

Show all containers running on FARGATE using JSON and JQ

Show all containers including EC2 host ip address using JSON and JQ formated as CSV

Show all broken (not running) containers using JSON and JQ

Show all container instances and their services as table

Show all container instances and their services as JSON (more details)

Find instance where specific service is running using JSON and JQ

Show container logs for all containers

Show container logs for specific container

Which container services can be deployed?

How to start a new deployment for container-type x?

Single container tasks

Multi container tasks

How to list deployments?

On which image do the currently running containers of type X base?

Show the latest events for a given container type

Show the deployment status for a container-type

Show detailed information about all containers of container-type X (see following example for explanation)

Related tutorials

Related components

Page Comparison

Versions Compared

Old Version 18

New Version Current

Key

Terminology

The deployment process

Deployment strategies

Process example

Safeguards

Native docker commands

Example usage scenarios

CLI-suite container commands

What are the general command line options?

Show which container is running on which host including resources overview as table

Show which container is running on which host including resources overview as JSON

Show all containers running on FARGATE using JSON and JQ

Show all containers including EC2 host ip address using JSON and JQ formated as CSV

Show all broken (not running) containers using JSON and JQ

Show all container instances and their services as table

Show all container instances and their services as JSON (more details)

Find instance where specific service is running using JSON and JQ

Show container logs for all containers

Show container logs for specific container

Which container services can be deployed?

How to start a new deployment for container-type x?

Single container tasks

Multi container tasks

How to list deployments?

On which image do the currently running containers of type X base?

Show the latest events for a given container type

Show the deployment status for a container-type

Show detailed information about all containers of container-type X (see following example for explanation)

Related tutorials

Related components