protobuf: avoid registering generic protobuf files globally

[aead]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Building and running a go binary depending on the go.etcd.io/etcd/api/[email protected] module fails with:

panic: proto: file "rpc.proto" is already registered
        See https://protobuf.dev/reference/go/faq#namespace-conflict


goroutine 1 [running]:
google.golang.org/protobuf/reflect/protoregistry.init.func1({0xe055320?, 0x87e2a20?}, {0xa8732c0, 0xc001668760})
        /home/ramon/go/pkg/mod/google.golang.org/[email protected]/reflect/protoregistry/registry.go:56 +0x1ec
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc000010120, {0xa91b950, 0xc000ede1e0})
        /home/ramon/go/pkg/mod/google.golang.org/[email protected]/reflect/protoregistry/registry.go:130 +0xb63
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x0, 0x0}, {0xc000f86000, 0x3cef, 0x4000}, 0x8, 0x5f, 0x0, 0x6, {0xa89c278, ...}, ...})
        /home/ramon/go/pkg/mod/google.golang.org/[email protected]/internal/filedesc/build.go:112 +0x1d6
github.com/golang/protobuf/proto.RegisterFile({0x88b51c8, 0x9}, {0xe19ea60, 0x100e, 0x100e})
        /home/ramon/go/pkg/mod/github.com/golang/[email protected]/proto/registry.go:48 +0x148
go.etcd.io/etcd/api/v3/etcdserverpb.init.5()
        /home/ramon/go/pkg/mod/go.etcd.io/etcd/api/[email protected]/etcdserverpb/rpc.pb.go:6154 +0x34

What did you expect to happen?

No panic at runtime due to conflicting files.

How can we reproduce it (as minimally and precisely as possible)?

1. Create a rpc.proto or record.proto file containing some arbitrary protobuf definitions and generate the corresponding Go code using protoc.
2. Create a minimal go program importing go.etcd.io/etcd/api/v3
3. Run the go program.

Anything else we need to know?

This issue is caused by the generated *.pb.go files - specially:

• etcd/server/storage/wal/walpb/record.pb.go

  Line 119 in c983744

  func init() { proto.RegisterFile("record.proto", fileDescriptor_bf94fd919e302a1d) }

• etcd/api/etcdserverpb/rpc.pb.go

  Line 6154 in ac31c34

  func init() { proto.RegisterFile("rpc.proto", fileDescriptor_77a6da22d6a3feb1) }

I'm not sure why the protobuf files contain the init function registering their file names globally without any qualifying package prefix. However, I can confirm that:

$ protoc --version
libprotoc 29.3

$ ll internal/protobuf/
total 264
-rw-r--r--@ 1 andreas  staff   5.0K Mar  6 12:08 cluster.pb.go
-rw-r--r--@ 1 andreas  staff   390B Jan  9 16:20 cluster.proto
-rw-r--r--@ 1 andreas  staff    47K Mar  6 12:08 db.pb.go
-rw-r--r--@ 1 andreas  staff   3.0K Mar  6 12:07 db.proto
-rw-r--r--@ 1 andreas  staff   1.6K Jan 20 17:21 export.go
-rw-r--r--@ 1 andreas  staff   3.0K Jan 20 17:21 proto.go
-rw-r--r--@ 1 andreas  staff    50K Mar  6 12:08 rpc.pb.go
-rw-r--r--@ 1 andreas  staff   5.1K Feb 17 10:06 rpc.proto

$ grep -ni 'RegisterFile' internal/protobuf/*

The grep does not find any such call to RegisterFile in my protobuf package containing the generated go code. Maybe, older versions or protoc generated such a call but not sure.

In general it seems problematic to "manually" register the filename like this since any other project depending on etcd could have a name conflict causing the builds to fail. Another issue related to this is: #18341

A potential fix could be to delete the two linked auto-generated pb.go files and re-generate them.

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

[ivanvc]

The grep does not find any such call to RegisterFile in my protobuf package containing the generated go code. Maybe, older versions or protoc generated such a call but not sure.

It seems you're looking for matches in the main branch, but your issue refers to v3.5, so you should look at the release-3.5 branch.

Note that the dependency versions from older release branches do not necessarily match the ones from main. So, even deleting and recrating the file wouldn't address the issue (we have checks in our CI to avoid having changes in the protobuffs that don't update the generated code). It sounds like a dependency (likely google.golang.org/protobuf) could be the culprit.

[aead]

It seems you're looking for matches in the main branch, but your issue refers to v3.5, so you should look at the release-3.5 branch.

The specific line is also in the release-3.5 branch. Refer to:

etcd/api/etcdserverpb/rpc.pb.go

Line 6154 in a17edfd

func init() { proto.RegisterFile("rpc.proto", fileDescriptor_77a6da22d6a3feb1) }

Note that the dependency versions from older release branches do not necessarily match the ones from main. So, even deleting and recrating the file wouldn't address the issue (we have checks in our CI to avoid having changes in the protobuffs that don't update the generated code). It sounds like a dependency (likely google.golang.org/protobuf) could be the culprit.

Yes, that is what I'm referring to. As mentioned above, protoc does not generate such RegisterFile calls in any *.pb.go file generated from the .proto definitions within my project. Hence, protoc may have generated these calls at some point in time but now has stopped doing so. Therefore, the suggestion to re-generate the entire *.pb.go files with a reasonably up-to-date version of protoc.

[ivanvc]

Thanks for the reply, @aead. I missed some of your initial input. Sorry for the confusion.

I've tried regenerating the Go files from the .proto inputs using different protoc versions. But unless I'm doing something very wrong, they don't have any changes. Did you by any chance try recreating them with our scripts/genproto.sh script? Or did you run it stand alone? I'm guessing it could be either another dependency (i.e., gogo/protobuf, golang/protobuf or genproto) or protoc option we're passing as an argument.