Private dockerhub registry: Failed to pull image

Troubleshooting
1

Hi there:

I set up an external registry named “docker” with endpoint “docker.com” to let the replicated access my containers in dockerhub. In the K8s YAML I’m referencing my images as “COMPANYNAME/IMAGE”. I always get “Failed to pull image” errors.

What might be the problem?

2

index.docker.io should be the correct endpoint for DockerHub registry

3

Thanks, I changed it, but I still have the same problem. Do I need to prefix the images with the docker.io?

4

You’ll also need to make sure you’ve filled out the images section as described in Docker Registries and added the replicatedregistrykey image pull secret.

5

After doing the changes to the images section, I can see the dashboard downloading the images, but the pods still get Error: ErrImagePull on the pods.

6

You’ll need to also ensure that the pods have the registry key set as a secret (also in the doc). An example is something like

spec:
  containers:
  - name: worker
     image: registry.replicated.com/myapp/worker:1.0.1
  imagePullSecrets:
  - name: replicatedregistrykey
7

I still cannot get it to work. I try to provide more context:

My external repository is set up as docker with index.docker.io as endpoint.

After previous help I have changed the replicated section to include:

images:
- name: smaccio/my-image
  source: docker
  tag: 0.5.0

I also made sure to reference the endpoint and secret in the pod section:

    spec:
      containers:
      - name: my-image
        image: "index.docker.io/smaccio/my-image:0.5.0"
      imagePullSecrets:
        - name: replicatedregistrykey

When updating the release, I can see the dasbhoard download the images, but the pods will stay in the state ImagePullBackOff. When I look at the journalctl --unit docker, I get something similar like this:

"Handler for GET /v1.24/images/index.docker.io/smaccio/my-image:0.5.0/json returned error: No such image: index.docker.io/smaccio/my-image:0.5.0"

Any more ideas?

8

Can you try removing index.docker.io from your spec?

9

Nope. Also doesn’t work. Same error. Here is what I found out after doing a sudo docker images:

The images are downloaded but they have a strange and non-unique prefix, in the pattern of:

registry.replicated.com/PRODUCTNAME/xxxxxx.smaccio-my-image
10.32.0.11:9874/smaccio-my-image

The first type of prefix is non unique and If I replace that in my manifest.yaml it actually works after doing a manual kubectl apply -f

10

Its a bit of an implementation detail, but in order to deliver your proxied private images, Replicated will rewrite

"index.docker.io/smaccio/my-image:0.5.0"

to

registry.replicated.com/..../TOKEN.smaccio-my-image:0.5.0

you found in your deployments and other workloads. It will also be re-tagged to Replicated’s on-prem registry.

It seems like for some reason replicated is not re-writing the images section in your deployment. You should be able to use the index.docker.io/smaccio/my-image:0.5.0 without needing to write the registry.replicated.com version yourself.

12

TIL another reason this could fail is if the apiVersion for Deployment is not apps/v1. Without going too deep into the implementations details, Replicated only rewrites images to the local registry if apps/v1 is used.

In the example below when I used apps/v1beta2 it did not rewrite the image but after I used apps/v1 it rewrote the image fine.

Does not work

---
# kind: scheduler-kubernetes
apiVersion: apps/v1beta2
kind: Deployment

Works

---
# kind: scheduler-kubernetes
apiVersion: apps/v1
kind: Deployment