[SR-8223] Running swift on a runtime test file would crash

[swift-ci]

Previous ID	SR-8223
Radar	None
Original Reporter	hongm (JIRA User)
Type	Bug

Additional Detail from JIRA

Votes	0
Component/s	Swift for TensorFlow
Labels	Bug
Assignee	None
Priority	Medium

md5: 26db0e07501f2a24eac26715804f400f

relates to:

• TF-121 Loop test failing on macOS

Issue Description:

This two-step workflow works:

$ ../build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swiftc test/TensorFlowRuntime/control_flow.swift -L../build/bazel-bin/tensorflow -ltensorflow -ltensorflow_framework

./control_flow

But this one-step workflow does not:

$ ../build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift -O /usr/local/google/home/hongm/ssd_part/git/swift-base/swift/test/TensorFlowRuntime/control_flow.swift -L../build/bazel-bin/tensorflow -ltensorflow -ltensorflow_framework
/usr/local/google/home/hongm/ssd_part/git/swift-base/swift/test/TensorFlowRuntime/control_flow.swift:14:14: warning: value implicitly copied to the accelerator, use .toAccelerator() to make transfer explicit
public func testSwitchEnum(_ a: Tensor<Float>?,
~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/google/home/hongm/ssd_part/git/swift-base/swift/test/TensorFlowRuntime/control_flow.swift:18:7: note: value used here
b += a
~~^~~~
[ RUN ] ControlFlow.testSwitchEnum_CPU_OR_GPU
Permission denied
#&#8203;0 0x0000000003d1ef84 PrintStackTraceSignalHandler(void*) (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x3d1ef84)
#&#8203;1 0x0000000003d1d202 llvm::sys::RunSignalHandlers() (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x3d1d202)
#&#8203;2 0x0000000003d1f132 SignalHandler(int) (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x3d1f132)
#&#8203;3 0x00007f3c39f820c0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x110c0)
#&#8203;4 0x00007f3bfea36c2f $S22SwiftPrivateLibcExtras10spawnChildys5Int32V3pid_AD7stdinFDAD06stdoutJ0AD06stderrJ0tSaySSGF (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftSwiftPrivateLibcExtras.so+0x3c2f)
#&#8203;5 0x00007f3bfcc4e07e $S14StdlibUnittest14_ParentProcessV11_spawnChildyyF (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftStdlibUnittest.so+0x4007e)
#&#8203;6 0x00007f3bfcc4e3bc $S14StdlibUnittest14_ParentProcessV15_runTestInChild__9parameterSb15anyExpectFailed_Sb04seenK5Crash22SwiftPrivateLibcExtras0D17TerminationStatusOSg6statusSays9SubstringVG11crashStdoutAO0W6StderrtAA0F5SuiteC_SSSiSgtF (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftStdlibUnittest.so+0x403bc)
#&#8203;7 0x00007f3bfcc508d5 $S14StdlibUnittest14_ParentProcessV10runOneTest04fullG4Name9testSuite0J00J9ParameterAC01_G6StatusOSS_AA0gK0CAL01_G0VSiSgtF (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftStdlibUnittest.so+0x428d5)
#&#8203;8 0x00007f3bfcc5225d $S14StdlibUnittest14_ParentProcessV3runyyF (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftStdlibUnittest.so+0x4425d)
#&#8203;9 0x00007f3bfcc5462b $S14StdlibUnittest11runAllTestsyyF (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftStdlibUnittest.so+0x4662b)
#&#8203;10 0x00007f3c3a38a0c0
#&#8203;11 0x0000000001066645 llvm::MCJIT::runFunction(llvm::Function*, llvm::ArrayRef<llvm::GenericValue>) (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x1066645)
#&#8203;12 0x000000000106a4ee llvm::ExecutionEngine::runFunctionAsMain(llvm::Function*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, char const* const*) (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x106a4ee)
#&#8203;13 0x00000000004ecba8 swift::RunImmediately(swift::CompilerInstance&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, swift::IRGenOptions&, swift::SILOptions const&) (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x4ecba8)
#&#8203;14 0x00000000004d330c performCompile(swift::CompilerInstance&, swift::CompilerInvocation&, llvm::ArrayRef<char const*>, int&, swift::FrontendObserver*, swift::UnifiedStatsReporter*) (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x4d330c)
#&#8203;15 0x00000000004cee49 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x4cee49)
#&#8203;16 0x0000000000489cc5 main (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x489cc5)
#&#8203;17 0x00007f3c261fc2b1 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b1)
#&#8203;18 0x000000000048776a _start (/usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift+0x48776a)
Stack dump:
0.  Program arguments: /usr/local/google/home/hongm/ssd_part/git/swift-base/build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift -frontend -interpret /usr/local/google/home/hongm/ssd_part/git/swift-base/swift/test/TensorFlowRuntime/control_flow.swift -disable-objc-interop -O -module-name control_flow -- -L../build/bazel-bin/tensorflow -ltensorflow -ltensorflow_framework 
Illegal instruction

I've only tested this on Linux, and on our runtime unit tests.

Feel free to take this if anyone is interested. Otherwise I can triage this later.

[swift-ci]

Comment by Ding Ye (JIRA)

Hi Mingsheng, I think it has nothing to do with TenserFlow. It is a StdlibUnittest issue/feature. I tested with vanilla swift with a TensorFlow-free case, and got the same crash as well. Maybe StdlibUnittest is not designed to support such one-step workflow.

[swift-ci]

Comment by Mingsheng Hong (JIRA)

Thanks Ding. If it's a unit-test only issue, at least it does not affect end users on Linux, which is good. Do you know if there is a known bug filed for the master branch already (or maybe we can convert this one for that purpose)?

[swift-ci]

Comment by Ding Ye (JIRA)

I cannot find any existing bug related to this one, and it even seems not to be a bug but incorrect usage due to missing env.

After diving a bit deeper into the stdlib source code, it tries to check `SWIFT_INTERPRETER` env here, If such env is not correctly set, errors may occur in interpret mode (i.e., the one-step process).

Please try the following command:

$ env SWIFT_INTERPRETER="../build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift" ../build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift -O /usr/local/google/home/hongm/ssd_part/git/swift-base/swift/test/TensorFlowRuntime/control_flow.swift -L../build/bazel-bin/tensorflow -ltensorflow -ltensorflow_framework

[swift-ci]

Comment by Mingsheng Hong (JIRA)

Sorry about the delay. I tried this and am getting a different error now (we need the flag -tf-strict-deabstraction now).

hongm@hongm:~/ssd_part/git/swift-base/swift$ env SWIFT_INTERPRETER="../build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift" ../build/Ninja-ReleaseAssert+stdlib-Release/swift-linux-x86_64/bin/swift -Xllvm -tf-strict-deabstraction -O -L../build/bazel-bin/tensorflow -ltensorflow -ltensorflow_framework test/TensorFlowRuntime/control_flow.swift
test/TensorFlowRuntime/control_flow.swift:92:13: warning: value implicitly copied to the accelerator, use .toAccelerator() to make transfer explicit
public func testEnumWithPayload(_ x: EnumWithPayload, _ expectedVal: Float) \{
 ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test/TensorFlowRuntime/control_flow.swift:101:26: note: value used here
 let tx = Tensor<Float>(x).toAccelerator(shape: [])
 ~~~~~~~~~~~~^~~~
test/TensorFlowRuntime/control_flow.swift:101:26: warning: value implicitly copied to the host, use .toHost() to make transfer explicit
 let tx = Tensor<Float>(x).toAccelerator(shape: [])
 ~~~~~~~~~~~~^~~~
test/TensorFlowRuntime/control_flow.swift:101:31: note: value used here
 let tx = Tensor<Float>(x).toAccelerator(shape: [])
 ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
test/TensorFlowRuntime/control_flow.swift:125:30: warning: 'a' implicitly copied to the accelerator, use .toAccelerator() to make transfer explicit
public func testCondBranch(_ a: Bool,
 ~~^~~~~~~
test/TensorFlowRuntime/control_flow.swift:128:6: note: value used here
 if a \{
 ^
[ RUN ] ControlFlow.weighPet_CPU_OR_GPU
stderr>>> test/TensorFlowRuntime/control_flow.swift:36:14: error: attribute '__shapes' requires a constant argument
stderr>>> weight = _scalarTensorWithShape(weight)
stderr>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

(This is not super high priority. But i'm curious what's going on here.)

[swift-ci]

Comment by Ding Ye (JIRA)

Hi Mingsheng, thanks for your feedback!

I tried with this commit, and the result here is different from yours (I got flooded with a large amount of stderr>>> messages). Anyway, in either case, something is not right. I will investigate this later, since it is not prioritised.

[swift-ci]

Comment by Ding Ye (JIRA)

Hi Mingsheng, here is a follow up with this issue. Overall, the problem is due to mishandled command line option. Let me briefly introduce some knowledge of the related two aspects.

[1] On the one hand, for Swift immediate mode (i.e. interpreter mode), if we have the following command for example:

"swift -opt1 -opt2 target.swift -arg1 -arg2"

, the frontend considers -opt1 and -opt2 as the compiler options while -arg1 and -arg2 as the arguments to be passed to target.swift. So when target.swift is being interpreted, the command line becomes "target.swift -arg1 -arg2", while the other three (i.e. swift, -opt1 and -opt2) are not propagated.

[2] On the other hand, StdlibUnittest is designed to work in a parent-child multi-process manner. When spawning a child process, an executable plus a list of arguments should be provided. When target.swift is being interpreted, everything works well in the parent process until it tries to spawn a child. What we'd like the child to execute is the command:

"swift -opt1 -opt2 target.swift --stdlib-unittest-run-child -arg1 -arg2"

However, the first three (i.e. swift, -opt1 and -opt2) are not directly available. With the help of env `SWIFT_INTERPRETER`, we can get the executable swift. So, the best we can do at the moment is to trigger the following command:

"swift target.swift --stdlib-unittest-run-child -arg1 -arg2"

, where -opt1 -opt2 are missing. In the particular case of SR-8223, the important -O is missing by the child, and thus causes the problem observed.

To resolve this issue, we should let parent process be aware of the opt list. This can be achieved by either introducing a new env in addition to `SWIFT_INTERPRETER` to hold such information, or just putting executable plus options altogether into `SWIFT_INTERPRETER` with its string parser enhanced. What do you think?

[swift-ci]

Comment by Mingsheng Hong (JIRA)

Thank you Ding for this awesome investigation and write-up! I learned quite a few things. 🙂

I don't have a strong opinion on which way we go, and imagine people who develop/use SWIFT_INTERPRETER might have a stronger opinion. 🙂 Given this is not an S4TF specific issue, please feel free to take the discussion to swift community forum and continue there.

Again, thanks for the in-depth research and insightful analysis!

[swift-ci]

Comment by Ding Ye (JIRA)

Well, I don't have a strong willingness to continue, Mingsheng. StdlibUnittest already works well with the two-step compilation workflow, which is the way to go for swift test suite. It seems to be little demand there to use it in interpret mode. Anyway, as long as you are happy with our investigation so far, I'd rather spend time with other issues with higher priorities 🙂

[swift-ci]

Comment by Mingsheng Hong (JIRA)

SGTM. Thanks again for your insights, Ding!

[rxwei]

I just observed something similar in https://bugs.swift.org/browse/SR-8541. A test case fails on macOS but would run correctly if the code was compiled as a separate source file.