Unable to pull or push image deal to the insecure registry configuration.

[verils]

Description

Problem

As mentioned in #2096 and #5667 , it seems to be a coding issue in util/resolver/resolver.go from line 47 to 65 (in v0.20.1):

        if c.Insecure != nil && *c.Insecure {
		h2 := h

		var transport http.RoundTripper = httpsTransport
		if isHTTP {
			// TODO: Replace this with [docker.NewHTTPFallback] once
			// backported to vendored version of containerd
			transport = &httpFallback{super: transport}
		}
		h2.Client = &http.Client{
			Transport: tracing.NewTransport(transport),
		}
		tc.InsecureSkipVerify = true
		return &h2, nil
	} else if isHTTP {
		h2 := h
		h2.Scheme = "http"
		return &h2, nil
	}

It first handles the insecure situation then handles the plain HTTP situation.

This causes some issues when users are handling a plain HTTP registry, and it's hard to figure out what's happening, because lack of documentation about this.

Let's see what happened in this situation:

I have a private registry, which didn't enable HTTPS in the first deployment. To push images built by buildkit onto that registry, I have 2 options:

1. Configure buildkitd

I can edit /etc/buildkit/buildkitd.toml file (for rootful mode) or ~/.config/buildkit/buildkitd.toml file (for rootless mode) as following:

[registry."my-registry.local"]
  http = true
  insecure = true

It looks fine, but in fact, I can only use http = true without insecure = true. Once I put the insecure = true in the toml file, the resolver will force to use HTTPS schema to request the registry, and there is no way to fallback to HTTP.

Unfortunately, there are no documentations mention about this.

2. Use registry.insecure option in the build command

The second method we thought to handle the insecure registry is to use --output type=image,registry.insecure=true in the docker buildx build command. It's wrong, it totally can't help anything for a plain HTTP registry. Either use this option or not, the buildkit always request in HTTPS.

How to solve

Users can only fix the plain HTTP registry issue by editing the buildkitd.toml file, use http = true option without insecure = true option.

I whish this answer can lead everyone to go to the right way.

Sugguestion

1. Fix the documentations, tell the differences and importances of http and insecure option in the buildkitd.toml file, tell users we can not use insecure option for plain HTTP registry.
2. Make the export option registry.insecure more precise, and document it. Many people have a misunderstanding to it, thought it can fix the plain HTTP connection issue.

I have no idea what else I can do. Just welcom to discuss it.