Introducing Run Examples check in PRs #908
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Cleaning up run examples pipeline Signed-off-by: Jing Chen <[email protected]> Rm run examples composite Signed-off-by: Jing Chen <[email protected]> Halt early if no file patch Signed-off-by: Jing Chen <[email protected]> Better msg Signed-off-by: Jing Chen <[email protected]> Modify some PDL files Signed-off-by: Jing Chen <[email protected]> Revert revisions Signed-off-by: Jing Chen <[email protected]> Modify some PDL programs Signed-off-by: Jing Chen <[email protected]> Fix if Signed-off-by: Jing Chen <[email protected]> Composite run examples Signed-off-by: Jing Chen <[email protected]> Checkout before using composite Test if Test if again Rm bash test Fix unfound path Update if check Delete a folder and see if test runs Bring back deleted example. Expects test to run Test if examples config is patched Mv yaml patch to composite action Try another patch strategy Change some files Checkout again in run examples Modify files Typo in composite action Use json in check parse Try to escape json Use changes instead in yaml action Bring back Ollama set up Gather new results from GH env Clean up workflows Fix lint Modify chatbot example Fix run examples config file * Add possible results for react examples * Preserve trailing white space * See if a commit is pushed on my behalf * Rename config file * Diff pdl file * Check if commit happens despite pytet failures * Allow pytest failure and continue run * github-actions[bot]: Updated results file when running examples on Mon Apr 7 23:24:42 UTC 2025 Signed-off-by: github-actions[bot] <[email protected]> * Set max parallel to 1 to prevent race conditions * See if yaml action is triggered * Change master branch * Fix pytest * Set update results to true * Test yaml actio again * Update result to true * Reverting update_results to False. If you wish to update again, change locally and push with change. * WRite permissions * Reverting update_results to False. If you wish to update again, change locally and push with change. * Don't commit using yaml action * Set update results to true * github-actions[bot]: Updated results file when running examples on Tue Apr 8 20:11:04 UTC 2025 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Reverting update_results to False. Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Try pytest failure * Test pytest behavior again * Test set +e * Test again * Set update to true to see if changed-files can capture results * PR test * github-actions[bot]: Updated results file when running examples Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * github-actions[bot]: Reverting update_result to false Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Test only with PR, not push * github-actions[bot]: Updated results file when running examples Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * github-actions[bot]: Reverting update_result to false Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * No pdl file change * Clean up and fix run examples nightly Signed-off-by: Jing Chen <[email protected]> * Fix empty parse * Fix * Bring back more models on ollama set up * Contrib readme updates --------- Signed-off-by: github-actions[bot] <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Jing Chen <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <75508867+github-actions[bot][email protected]> Get new results for failing progs github-actions[bot]: Run examples: updated result files on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> github-actions[bot]: Run examples: reverting update_result to false on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> See if ollama is cached Try caching ollama Set up Ollama outside composite action Revert changes Update runexamples config Signed-off-by: Jing Chen <[email protected]>
Signed-off-by: Jing Chen <[email protected]>
Signed-off-by: Jing Chen <[email protected]>
github-actions[bot]: Run examples: reverting update_result to false on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> github-actions[bot]: Run examples: updated result files on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Revert patches Signed-off-by: Jing Chen <[email protected]>
Signed-off-by: Jing Chen <[email protected]>
…n your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…n your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #856
This PR refactored
test_run_examples.pyby making it more modular. Specifically, it now reads a config file, namelytest_run_examples.yaml, which outlines what files to check, skip, requires input, and expects errors during execution. This separation not only makes the code more readable, but also allows testing in different environments.This PR also improved the logic of Run Examples. The nightly test remains. In addition, whenever a PR is opened, three new jobs are initiated. Each will execute Run Examples on PDL files that were modified and added to the repo. Below is a high-level overview of the implementation to achieve this behavior.
update-results: 'false'(link)update_result: truein the PR. (link)This PR finally updates
contrib.mdwith more details on the behavior of Run Examples in local dev and GH environments (see link).