Add sideCars to the MinIO tenant Helm chart

[ThoSap]

Description

This PR adds the ability to configure the tenant Helm chart sideCars.

---

With the replacement of the MinIO Console with the simple Object Browser, more and more administrative tasks like the creation of service accounts have to be done with the MinIO Client mc command, which is a good thing, as IaC is more in the focus instead of conveniently using a web UI.
This configuration can be done, for example, using a K8s sidecar using the quay.io/minio/mc container image.
The Operator Tenant CRD already supports sideCars, but the tenant Helm chart currently does not support them.

https://github.com/minio/operator/blob/v7.1.1/docs/tenant_crd.adoc#sidecars

https://github.com/minio/operator/blob/v7.1.1/resources/base/crds/minio.min.io_tenants.yaml#L3805
https://github.com/minio/operator/blob/v7.1.1/helm/operator/templates/minio.min.io_tenants.yaml#L3805

Related Issue

fixes: #2465

Type of Change

• Bug fix 🐛
• New feature 🚀
• Breaking change 🚨
• Documentation update 📖
• Refactor 🔨
• Other (please describe) ⬇️

Checklist

• I have tested these changes
• I have updated relevant documentation (if applicable)
• I have added necessary unit tests (if applicable)

Test Steps

1. Use the sideCars configuration example in helm/tenant/values.yaml, same as it was done with the initContainers example.

[ThoSap]

@pjuarezd @cniackz I think the failing checks are unrelated.

[ramondeklein]

Although I think it's fine to add sidecars to the Helm chart, this is not the correct use. Sidecars are typically used to add functionality to a pod (i.e. logging, monitoring, networking, ...). It's not to be used for a single task, like setting up a service account for the following reasons:

1. When running a 4 node cluster, all 4 nodes will attempt to create the service account.
2. When mc finishes, the pod terminates and will be restarted. Keeping pods alive forever (while doing nothing) isn't the right way to deal with this.

This is better fixed by deploing a Kubernetes batch job (docs) instead that will perform the required setup tasks.

[ThoSap]

@ramondeklein our use case is exactly that: start a MinIO configuration reconciler sidecar container that mounts a K8s secret as a volume.
With a single job, this won't work (update: a cronjob will).

As the Tenant CRD supports sideCars, I don't see why the Helm chart should limit this.

Whenever Terraform updates this secret, it will also update the secret within the container. Our reconciler script, which runs every 60 seconds, will then remove or add resources such as console users or service accounts.
Also, my example in the PR does this: run and then sleep for 300 seconds.

We don't want to expose the MinIO API using an ingress (which would allow us to interact with MinIO again using Terraform).
We also don't want to use kubectl port-forward for it, as TF should exclusively handle the configuration part and maintain the state.

Therefore, a reconciler sidecar is the approach we've chosen.

[ThoSap]

A CronJob would also work, I see your point:

1. When running a 4 node cluster, all 4 nodes will attempt to create the service account.

Regarding point 2, we run the reconciler script every 60 seconds (and we catch any potential errors), therefore the container will never exit, and the pod keeps running.
Still, a CronJob would be the more elegant solution for our reconciler use case.

2. When mc finishes, the pod terminates and will be restarted. Keeping pods alive forever (while doing nothing) isn't the right way to deal with this.

[ramondeklein]

A CronJob meant to run at a specific interval and a Job is used to run a single time. Sidecars are definitely not to be used for this, so we can't accept this PR with the example value. We don't want to suggest using sidecars for this.

[ThoSap]

You are right, I'll remove the example value.