feat(deployment): add sequencer readiness check stage to system test workflow

[nadin-Starkware]

No description provided.

[reviewable-StarkWare]

This change is 

[nadin-Starkware]

• feat(deployment): add run simulator stage to system test workflow #6669
• feat(deployment): add copy storage files stage to system test workflow #6668
• feat(deployment): add sequencer liveness check stage to system test workflow #6645
• feat(deployment): add sequencer readiness check stage to system test workflow #6641 👈 (View in Graphite)
• feat(deployment): add sequencer deployment stage to system test workflow #6638
• feat(deployment): add generate k8s manifests stage to system test workflow #6633
• feat(deployment): add config secrets injector stage to the system test #6627
• feat(deployment): add base consolidated system test workflow #6623
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Itay-Tsabary-Starkware]

Reviewable status: 0 of 2 files reviewed, 1 unresolved discussion

---

scripts/system_tests/readiness_check.py line 43 at r2 (raw file):

def wait_for_services_ready(deployment_config_path: str, namespace: str):
    config.load_kube_config()  # Or load_incluster_config() if inside a pod

?

Code quote:

# Or load_incluster_config() if inside a pod

[Itay-Tsabary-Starkware]

Reviewable status: 0 of 2 files reviewed, 2 unresolved discussions (waiting on @nadin-Starkware)

---

scripts/system_tests/readiness_check.py line 56 at r2 (raw file):

        controller = service["controller"]
        service_name_lower = service_name.lower()
        controller_lower = controller.lower()

This chunk is repeated, maybe create a fn?

Code quote:

        service_name = service["name"]
        controller = service["controller"]
        service_name_lower = service_name.lower()
        controller_lower = controller.lower()

[Itay-Tsabary-Starkware]

Reviewable status: 0 of 2 files reviewed, 3 unresolved discussions (waiting on @nadin-Starkware)

---

scripts/system_tests/readiness_check.py line 72 at r2 (raw file):

            else:
                print(f"❌ Unknown controller: {controller}. Skipping...")
                continue

Is this not a reason to fail the test?

Code quote:

                print(f"❌ Unknown controller: {controller}. Skipping...")
                continue

[Itay-Tsabary-Starkware]

Reviewable status: 0 of 2 files reviewed, 4 unresolved discussions (waiting on @nadin-Starkware)

---

scripts/system_tests/readiness_check.py line 78 at r2 (raw file):

                continue
            else:
                raise

What does this do?

Code quote:

        except ApiException as e:
            if e.status == 404:
                print(f"❌ {controller} {resource_name} not found. Skipping...")
                continue
            else:
                raise

[Itay-Tsabary-Starkware]

Reviewable status: 0 of 2 files reviewed, 5 unresolved discussions (waiting on @nadin-Starkware)

---

scripts/system_tests/readiness_check.py line 104 at r2 (raw file):

                    ).status
                    ready = status.ready_replicas or 0
                    desired = status.replicas or 0

Please use "else if", and fail on the 3rd default branch

Code quote:

                else:  # deployment
                    status = apps_v1.read_namespaced_deployment_status(
                        resource_name, namespace
                    ).status
                    ready = status.ready_replicas or 0
                    desired = status.replicas or 0

[Itay-Tsabary-Starkware]

Reviewable status: 0 of 2 files reviewed, 6 unresolved discussions (waiting on @nadin-Starkware)

---

scripts/system_tests/readiness_check.py line 131 at r2 (raw file):

        "--namespace",
        help="Kubernetes namespace",
    )

Mark as required

Code quote:

    parser.add_argument(
        "--namespace",
        help="Kubernetes namespace",
    )

[Itay-Tsabary-Starkware]

Reviewable status: 0 of 2 files reviewed, 7 unresolved discussions (waiting on @nadin-Starkware)

---

scripts/system_tests/readiness_check.py line 116 at r2 (raw file):

        else:
            print(
                f"⚠️ Timeout waiting for {controller} {resource_name} to become ready."

This is an error, right?

Code quote:

⚠️