DT-436 adding tags to aws resources

[christopher-comet]

https://comet-ml.atlassian.net/browse/DT-436

This also includes edits to make the tfvars ready to go with defaults and updates the redis version.

[darenjacobs]

LGTM

[christopher-comet]

Ready for review. @darenjacobs @liyaka

Old

[christopher-comet]

There are a couple of modules wherein the resource explicitly sets tags.
eg. in terraform-aws-comet/modules/comet_s3/main.tf

resource "aws_s3_bucket" "comet_airflow_bucket" {
  count = var.enable_mpm_infra ? 1 : 0

  bucket = "comet-airflow-${local.suffix}"

  force_destroy = var.s3_force_destroy

  tags = {
    Name = "comet-airflow-${local.suffix}"
  }
}

In this situation the S3 bucket will only get the Name tag, and it will NOT inherit common_tags automatically because the tags block inside aws_s3_bucket is explicitly defined, and default_tags from providers.tf does not override manually set tags.

Therefore for these cases we need to merge the common_tags.

Stale review

[christopher-comet]

Summary of Terraform Changes in This PR
Provider Updates

Bumped versions for the following providers:

    aws → 5.97.0

    helm → 2.17.0

    kubernetes → 2.36.0

    Others: tls, time, cloudinit, and added null

Why: Modernization, bug fixes, and compatibility with new module versions.

EKS Module & Provider Refactor

Migrated to terraform-aws-modules/eks v20.33.1

Set authentication_mode = "API_AND_CONFIG_MAP" for streamlined auth

Replaced deprecated aws_eks_cluster_auth with output-based kubeconfig logic

Moved helm.eks and kubernetes.eks provider definitions into eks_providers.tf

Added explicit providers = { ... } block for comet_eks

Enables one-stage apply without Helm/Kubernetes provider race conditions.
Unified Tagging Across Modules

Replaced hardcoded tags = { Terraform = "true", Environment = var.environment }

Now using common_tags passed into all modules

Centralized in local.all_tags and merged with per-resource tags

Improves tag consistency across all infrastructure components.
Module Improvements

Updated modules for:

    comet_eks (with conditional merge for node groups)

    comet_ec2, comet_vpc, comet_elasticache, comet_rds, comet_s3

Added missing common_tags variables and passed them to resources

Output Cleanup

Removed obsolete comet_eks_token output (no longer available via token-based auth)

Updated comet_eks_cert and comet_eks_endpoint to match new module structure

IRSA + Addons

Added IRSA role creation for EBS CSI driver

Integrated aws-ia/eks-blueprints-addons module to install:

    CoreDNS, kube-proxy, vpc-cni, aws-ebs-csi-driver

    AWS Load Balancer Controller (with Helm release)