feat(KONFLUX-6960): Introduce hourly jobs #1534

jhutar · 2025-03-14T12:40:57Z

Description

This introduces new hourly job as discussed with Gal to measure build speed on each main cluster and the changes needed along the way.

Issue ticket number and link

https://issues.redhat.com/browse/KONFLUX-6960

Type of change

Breaking change (fix or feature that would cause existing functionality to not work as expected) - but hopefully not :-)

How Has This Been Tested?

It is running already for few weeks in my fork.

openshift-ci · 2025-03-14T12:41:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tisutisu for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jhutar · 2025-03-18T06:58:04Z

/retest

jhutar · 2025-03-18T12:58:19Z

/retest

jhutar · 2025-03-28T14:26:46Z

e2e test failed:

Summarizing 2 Failures:
  [FAIL] [integration-service-suite Status Reporting of Integration tests] with status reporting of Integration tests in CheckRuns when Integration PipelineRuns are created [It] should eventually complete successfully [integration-service, github-status-reporting]
  /tmp/tmp.MRoPsrol3V/tests/integration-service/status-reporting-to-pullrequest.go:162
  [FAIL] [build-service-suite Build service E2E tests] test pac with multiple components using same repository when components are created in same namespace [It] merging the PR should be successful [build-service, pac-build, multi-component]
  /tmp/tmp.MRoPsrol3V/tests/build/build.go:775

Ran 292 of 367 Specs in 1960.709 seconds
FAIL! -- 290 Passed | 2 Failed | 26 Pending | 49 Skipped

…gressive

…ts forks

…Gitlab

…"task"] not in self.data_pipelineruns[pod["pipelinerun"]]["taskRuns"]: KeyError: 'taskRuns''

… absolute

…n use for loadtest Created based on https://konflux.pages.redhat.com/docs/users/getting-started/getting-access.html#_logging_into_your_workspace_via_service_account This was needed to setup a load-test probe test on private clusters.

konflux-ci-qe-bot · 2025-03-28T15:56:59Z

@jhutar: The following test has Failed, say /retest to rerun failed tests.

PipelineRun Name	Status	Rerun command	Build Log	Test Log
`konflux-e2e-vmzl6`	Failed	`/retest`	View Pipeline Log	View Test Logs

Inspecting Test Artifacts

To inspect your test artifacts, follow these steps:

Install ORAS (see the ORAS installation guide).
Download artifacts with the following commands:

mkdir -p oras-artifacts
cd oras-artifacts
oras pull quay.io/konflux-test-storage/konflux-team/e2e-tests:konflux-e2e-vmzl6

Test results analysis

🚨 Error occurred while running the E2E tests, list of failed Spec(s):

➡️ [failed] [It] [integration-service-suite Integration Service E2E tests] with happy path for general flow of Integration service when a new Component is created triggers a build PipelineRun [integration-service]

Click to view logs

Unexpected error:
    <*errors.StatusError | 0xc000610140>: 
    Internal error occurred: failed calling webhook "vintegrationtestscenario.kb.io": failed to call webhook: Post "https://integration-service-webhook-service.integration-service.svc:443/validate-appstudio-redhat-com-v1beta2-integrationtestscenario?timeout=10s": context deadline exceeded
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"vintegrationtestscenario.kb.io\": failed to call webhook: Post \"https://integration-service-webhook-service.integration-service.svc:443/validate-appstudio-redhat-com-v1beta2-integrationtestscenario?timeout=10s\": context deadline exceeded",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"vintegrationtestscenario.kb.io\": failed to call webhook: Post \"https://integration-service-webhook-service.integration-service.svc:443/validate-appstudio-redhat-com-v1beta2-integrationtestscenario?timeout=10s\": context deadline exceeded",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
occurred

➡️ [failed] [It] [release-service-suite Release service tenant pipeline] Post-release verification verifies that a Release CR should have been created in the dev namespace [release-service, tenant]

Click to view logs

Unexpected error:
    <*errors.StatusError | 0xc000df74a0>: 
    Internal error occurred: failed calling webhook "mauthor.kb.io": failed to call webhook: Post "https://release-service-webhook-service.release-service.svc:443/mutate-appstudio-redhat-com-v1alpha1-author?timeout=10s": context deadline exceeded
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"mauthor.kb.io\": failed to call webhook: Post \"https://release-service-webhook-service.release-service.svc:443/mutate-appstudio-redhat-com-v1alpha1-author?timeout=10s\": context deadline exceeded",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"mauthor.kb.io\": failed to call webhook: Post \"https://release-service-webhook-service.release-service.svc:443/mutate-appstudio-redhat-com-v1alpha1-author?timeout=10s\": context deadline exceeded",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
occurred

➡️ [failed] [It] [release-service-suite [HACBS-2469]test-releaseplan-owner-ref-added] ReleasePlan verification verifies that the ReleasePlan has an owner reference for the application [release-service, releaseplan-ownerref]

Click to view logs

Unexpected error:
    <*errors.StatusError | 0xc000acd7c0>: 
    Internal error occurred: failed calling webhook "vreleaseplan.kb.io": failed to call webhook: Post "https://release-service-webhook-service.release-service.svc:443/validate-appstudio-redhat-com-v1alpha1-releaseplan?timeout=10s": context deadline exceeded
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"vreleaseplan.kb.io\": failed to call webhook: Post \"https://release-service-webhook-service.release-service.svc:443/validate-appstudio-redhat-com-v1alpha1-releaseplan?timeout=10s\": context deadline exceeded",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"vreleaseplan.kb.io\": failed to call webhook: Post \"https://release-service-webhook-service.release-service.svc:443/validate-appstudio-redhat-com-v1alpha1-releaseplan?timeout=10s\": context deadline exceeded",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
occurred

➡️ [failed] [It] [release-service-suite Release service happy path] Post-release verification verifies that a Release CR should have been created in the dev namespace [release-service, happy-path]

Click to view logs

Unexpected error:
    <*errors.StatusError | 0xc001276dc0>: 
    Internal error occurred: failed calling webhook "vreleaseplanadmission.kb.io": failed to call webhook: Post "https://release-service-webhook-service.release-service.svc:443/validate-appstudio-redhat-com-v1alpha1-releaseplanadmission?timeout=10s": context deadline exceeded
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"vreleaseplanadmission.kb.io\": failed to call webhook: Post \"https://release-service-webhook-service.release-service.svc:443/validate-appstudio-redhat-com-v1alpha1-releaseplanadmission?timeout=10s\": context deadline exceeded",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"vreleaseplanadmission.kb.io\": failed to call webhook: Post \"https://release-service-webhook-service.release-service.svc:443/validate-appstudio-redhat-com-v1alpha1-releaseplanadmission?timeout=10s\": context deadline exceeded",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
occurred

➡️ [failed] [It] [release-service-suite [HACBS-2360] Release CR fails when missing ReleasePlan and ReleasePlanAdmission.] post-release verification. missing ReleasePlan makes a Release CR set as failed in both IsReleased and IsValid with a proper message to user. [release-service, release-neg, negMissingReleasePlan]

Click to view logs

Unexpected error:
    <*errors.StatusError | 0xc000f02460>: 
    Internal error occurred: failed calling webhook "mrelease.kb.io": failed to call webhook: Post "https://release-service-webhook-service.release-service.svc:443/mutate-appstudio-redhat-com-v1alpha1-release?timeout=10s": context deadline exceeded
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "Internal error occurred: failed calling webhook \"mrelease.kb.io\": failed to call webhook: Post \"https://release-service-webhook-service.release-service.svc:443/mutate-appstudio-redhat-com-v1alpha1-release?timeout=10s\": context deadline exceeded",
            Reason: "InternalError",
            Details: {
                Name: "",
                Group: "",
                Kind: "",
                UID: "",
                Causes: [
                    {
                        Type: "",
                        Message: "failed calling webhook \"mrelease.kb.io\": failed to call webhook: Post \"https://release-service-webhook-service.release-service.svc:443/mutate-appstudio-redhat-com-v1alpha1-release?timeout=10s\": context deadline exceeded",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 500,
        },
    }
occurred

➡️ [failed] [It] [integration-service-suite Integration Service E2E tests] with happy path for general flow of Integration service with ephemeral environment when a new Component is created verifies if the build PipelineRun contains the finalizer [integration-service, integration-env]

Click to view logs

Timed out after 60.001s.
timeout when waiting for finalizer to be added
Expected success, but got an error:
    <*errors.errorString | 0xc000ddb8e0>: 
    build pipelineRun integration-env-qiej-tenant/test-component-pac-ehodus-on-pull-request-lp958 doesn't contain the finalizer: test.appstudio.openshift.io/pipelinerun yet
    {
        s: "build pipelineRun integration-env-qiej-tenant/test-component-pac-ehodus-on-pull-request-lp958 doesn't contain the finalizer: test.appstudio.openshift.io/pipelinerun yet",
    }

➡️ [failed] [It] [konflux-demo-suite] Maven project - Default build when Build PipelineRun completes successfully should find the related Snapshot CR [konflux, upstream-konflux]

Click to view logs

Timed out after 240.000s.
timed out when trying to check if the Snapshot exists for PipelineRun konflux-heqw-tenant/konflux-demo-component-wdos-on-push-pmrkk
Expected success, but got an error:
    <*errors.errorString | 0xc0008121a0>: 
    no snapshot found for component '', pipelineRun 'konflux-demo-component-wdos-on-push-pmrkk' in 'konflux-heqw-tenant' namespace
    {
        s: "no snapshot found for component '', pipelineRun 'konflux-demo-component-wdos-on-push-pmrkk' in 'konflux-heqw-tenant' namespace",
    }

➡️ [failed] [It] [integration-service-suite Status Reporting of Integration tests] with status reporting of Integration tests in CheckRuns when the PaC build pipelineRun run succeeded checks if the Snapshot is created [integration-service, github-status-reporting]

Click to view logs

Unexpected error:
    : 
    context deadline exceeded
    {}
occurred

➡️ [failed] [It] [integration-service-suite Gitlab Status Reporting of Integration tests] Gitlab with status reporting of Integration tests in the assosiated merge request when the PaC build pipelineRun run succeeded checks if the Snapshot is created [integration-service, gitlab-status-reporting]

Click to view logs

Unexpected error:
    : 
    context deadline exceeded
    {}
occurred

➡️ [failed] [It] [integration-service-suite Creation of group snapshots for monorepo and multiple repos] with status reporting of Integration tests in CheckRuns when the Build PLRA is finished successfully checks if the Snapshot is created [integration-service, group-snapshot-creation]

Click to view logs

Unexpected error:
    : 
    context deadline exceeded
    {}
occurred

openshift-ci bot requested review from Dannyb48 and kasemAlem March 14, 2025 12:41

github-actions bot added the > 60 min label Mar 14, 2025

jhutar force-pushed the fix43-probes branch from b6ed96e to 1218b15 Compare March 21, 2025 10:29

jhutar added 19 commits March 28, 2025 15:27

feat: Add GitHub action workflow to run in Prod

4f2987f

feat: Remove unneeded loop as reason for it was removed in 766cb58

a83352a

fix: Use non-hidden artifacts dir and explicitly use $PWD

9187ae4

refactor: Rename this variable as it is not about stage only now

0f75e41

feat: Collect API endpoint we are talking to

2324511

feat: Use matrix job to run on 4 different clusters

5dee504

feat: Compute duration of PRs, TRs and steps

09eb5b4

feat: Now also compute idle duration time for PRs and TRs

1334a38

feat: Retry when repo renaming fails after fork and do not be that ag…

07bc88d

…gressive

feat: Make sure this action does not get scheduled by cron on e2e-tes…

23e6ca1

…ts forks

refactor: Make getting asAdmin bit more readable

38446f8

feat: Introduce authentication via service account token

0bd7d4b

feat: Add hack that allows us to run on private cluster with repo in …

e4d202e

…Gitlab

fix: Make sure 'taskRuns' dict is there for all PLRs. Solves 'if pod[…

8d24b0c

…"task"] not in self.data_pipelineruns[pod["pipelinerun"]]["taskRuns"]: KeyError: 'taskRuns''

feat: Ignore errors this is printing: warning: safe.directory '*' not…

06b265c

… absolute

feat: Add Jenkins specific variables

f62a2a5

feat: Only run jobs from main repo

d1e6336

feat: Count all 4 build-images tasks separately for MPC build

b7530fe

jhutar force-pushed the fix43-probes branch from 1218b15 to b7530fe Compare March 28, 2025 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(KONFLUX-6960): Introduce hourly jobs #1534

feat(KONFLUX-6960): Introduce hourly jobs #1534

jhutar commented Mar 14, 2025

openshift-ci bot commented Mar 14, 2025

jhutar commented Mar 18, 2025

jhutar commented Mar 18, 2025

jhutar commented Mar 28, 2025

konflux-ci-qe-bot commented Mar 28, 2025

feat(KONFLUX-6960): Introduce hourly jobs #1534

Are you sure you want to change the base?

feat(KONFLUX-6960): Introduce hourly jobs #1534

Conversation

jhutar commented Mar 14, 2025

Description

Issue ticket number and link

Type of change

How Has This Been Tested?

openshift-ci bot commented Mar 14, 2025

jhutar commented Mar 18, 2025

jhutar commented Mar 18, 2025

jhutar commented Mar 28, 2025

konflux-ci-qe-bot commented Mar 28, 2025

Inspecting Test Artifacts

Test results analysis