Soft reboot prep #4133
Merged
Soft reboot prep #4133
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Go `exec.Command` when used naively like this captures stderr and then it gets lost. It turned out with soft reboot it was this `mkfifo` that was failing but all we got is "Child process exited with code 1" but there are like 5 different processes on two different hosts that could be talking about, and we *really* thought it was talking about the code-under-test, not the framework setting it up.
Amazingly we had *two* places that dropped stderr, this was the second. The default `c.SSH` captures logs from test failures but this was an *infra* failure so we need to drop to the raw ssh tool. (This could use a big cleanup but that's...a bigger project)
This definitely adds some chatter at debug level but I think it's really worth it.
|
/retest-required |
jlebon
approved these changes
Jun 6, 2025
|
Prow failing on coreos/rhel-coreos-config#20. /override ci/prow/rhcos |
|
@jlebon: Overrode contexts on behalf of jlebon: ci/prow/rhcos In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prep for #4119
kolet: Capture error messages from mkfifo
The Go
exec.Commandwhen used naively like this capturesstderr and then it gets lost.
It turned out with soft reboot it was this
mkfifothat wasfailing but all we got is "Child process exited with code 1"
but there are like 5 different processes on two different hosts
that could be talking about, and we really thought it
was talking about the code-under-test, not the framework setting
it up.
kola: Don't drop stderr on the floor from unit starting function
Amazingly we had two places that dropped stderr, this was
the second. The default
c.SSHcaptures logs from test failuresbut this was an infra failure so we need to drop to the
raw ssh tool.
(This could use a big cleanup but that's...a bigger project)
mantle: Log commands executed over ssh and return code
This definitely adds some chatter at debug level but I think
it's really worth it.