Why you should not trust Docker image tags


Marcin Budny
Head of R&D
Marcin Budny
Architect and developer, over 10 years in IT. He works with intelligent transport systems, researches new technology and explores ways to apply it. Passionate about building software, always looking to learn something new. Focused on .NET, yet curious about the rest of the dev world.

When deploying an application to Docker, one of the things I expect, is the predictability and repeatability. This is why I try to always use specific image tags for the components my application uses. However, this doesn’t always ensure the application runs exactly the same on all the environments.

The problem with Docker image tags

Let’s take a look at mongo Docker Hub repository:

Docker image tags

If I wanted to use MongoDB version 3.4.3, I could use one of following tags: `latest`, `3`, `3.4`, `3.4.3`. If I don’t specify a version tag, `latest` will be used as default. Once Docker downloads an image to a local cache on my machine, all containers created with the selected tag will always use said image. However when I decide to deploy my application to another environment, it may turn out that the tag I used on my machine now points to another version of MongoDB. The `latest` tag may now point to version 3.5 (and this will also be true for the `3` tag). My application will therefore work with a version of the database engine it was not tested against. This is why it is recommended to use the most specific version tag, in this case `3.4.3`.

One could expect, that being this specific about the mongo version will result in predictable behavior on all environments, as you should always get exactly the same image. However this may not be true. Please note, that the version tag specifies the version of the MongoDB embedded in the image. It does not uniquely identify the image. A docker image can be built in many different ways, having different base image, preinstalled components, entrypoint script, etc.

So if the way the image is being built changes, effectively you get a different image. If it gets tagged with one of already existing tags, you may be under impression that it’s still the same image you used before. However this is not true. When deploying the application to a new environment (one, that does not have the specified tag in the cache), you will get a version of the image that differs from what your application was tested against.

In fact, this is exactly what happened in my case. I used the `mongo:3.4.3` image on the development environment. Few days later I wanted to promote the new application version to test environment. In the meantime, the underlying image changed. This change introduced a bug, that completely prevented me from running the application.

The solution

It seems Docker does not have a perfect way of handling this problem. However you have at least two options.

First option is to push the image to a private registry. This way you have a full control over the image tags, so there is no possibility that the image will change unexpectedly.

Second option is to use image digest in the place of the tag. The digest is a SHA256 calculated from the image and identifies it uniquely. Docker displays it when you pull or push the image:

Docker image tags

Or you can use following command for images in your local cache (verified in Docker version 17.04, may not work in previous versions):

docker images --digests --no-trunc

Once you have the digest, you can use it to pull the image in following way:

docker pull docker pull mongo@sha256:2b1c1183829ad5c509b07a617cc53ef24ca77fd60276882608c3635fcd407257


You have to be careful when using the docker image version tags. Try to be as specific as possible. If you suspect the underlying image could change for a tag you are using, use a private repository or pull the image by digest.