Skip to content

Improve PR pipeline for run examples test #856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jgchn opened this issue Mar 31, 2025 · 0 comments · Fixed by #908
Closed

Improve PR pipeline for run examples test #856

jgchn opened this issue Mar 31, 2025 · 0 comments · Fixed by #908
Labels

Comments

@jgchn
Copy link
Collaborator

jgchn commented Mar 31, 2025

Is your feature request related to a problem? Please describe.
Currently, the Run Example test is executed nightly, meaning newly added or modified .pdl files in a pull request are not automatically validated before merging. As a result, developers can introduce changes to .pdl programs and merge their PRs without verifying that Run Example still functions correctly. This leads to frequent failures and inconsistencies, which negatively impact the reliability of Run Example and reflect poorly on the main page.

To improve stability, we should ensure that Run Example runs as part of the PR checks, preventing unverified changes from being merged.

Describe the solution you'd like
A "Run Changed PDL Examples" test should be executed each time a PR is opened. This pipeline should detect modifications to .pdl programs in the PR and execute a Pytest run for each of those files, ensuring that 0.result exists and is validated for that file.

Here's more criterias to consider:

  • Certain files should be skipped. In that case, 0.result doesn't exist
  • Certain files could result in runtime and parse errors. We need to ensure that this is reflected in test_example_runs.py
  • Certain files require inputs. Again, we need to ensure that this is captured in test_example_run.py
  • The check shouldn't take long for each PR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant