Test failure on i386 and armhf platforms

[avalentino]

============================= test session starts ==============================
platform linux -- Python 3.8.1, pytest-4.6.8, py-1.8.0, pluggy-0.13.0
rootdir: /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d
collected 265 items / 1 deselected / 264 selected

compyle/tests/test_array.py ..s..s..s..s..s..s..s..s..sF.s..s..s..s..s.. [ 16%]
s                                                                        [ 17%]
compyle/tests/test_ast_utils.py .......                                  [ 19%]
compyle/tests/test_capture_stream.py .....                               [ 21%]
compyle/tests/test_config.py ............                                [ 26%]
compyle/tests/test_cython_generator.py ...............                   [ 31%]
compyle/tests/test_ext_module.py ......                                  [ 34%]
compyle/tests/test_gpu_struct.py s                                       [ 34%]
compyle/tests/test_jit.py .......................                        [ 43%]
compyle/tests/test_low_level.py s.s...                                   [ 45%]
compyle/tests/test_parallel.py s.s...s..s..s..s..s..s...s...s...s...s.s. [ 60%]
..s..s..s..s..s..s...s...s...s...                                        [ 73%]
compyle/tests/test_template.py .....                                     [ 75%]
compyle/tests/test_translator.py ....................................... [ 90%]
.............                                                            [ 95%]
compyle/tests/test_transpiler.py ..                                      [ 95%]
compyle/tests/test_types.py .......                                      [ 98%]
compyle/tests/test_utils.py ....                                         [100%]

=================================== FAILURES ===================================
_________________________ test_align_multiple[cython] __________________________

backend = 'cython'

    @test_all_backends
    def test_align_multiple(backend):
        check_import(backend)
    
        # Given
        dev_array_a = Array(np.uint32, backend=backend)
        dev_array_b = Array(np.float32, backend=backend)
        orig_array_a = array.arange(0, 1024, 1, dtype=np.uint32, backend=backend)
        orig_array_b = array.arange(
            1024, 2048, 1, dtype=np.float32, backend=backend)
        dev_array_a.set_data(orig_array_a)
        dev_array_b.set_data(orig_array_b)
    
        indices = array.arange(1023, -1, -1, dtype=np.int64, backend=backend)
    
        # When
>       dev_array_a, dev_array_b = array.align([dev_array_a, dev_array_b],
                                               indices)

compyle/tests/test_array.py:191: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
compyle/array.py:643: in align
    align_multiple_elwise(*args_list)
compyle/parallel.py:525: in __call__
    self.elementwise(*args, **kwargs)
compyle/jit.py:316: in __call__
    c_func(*c_args, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   ValueError: Buffer dtype mismatch, expected 'long' but got 'long long'

m_2a00dba787c0caa83a32476b5532225f.pyx:22: ValueError
----------------------------- Captured stdout call -----------------------------
running build_ext
cythoning /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.pyx to /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.cpp
building 'm_2a00dba787c0caa83a32476b5532225f' extension
i686-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -O2 -ffile-prefix-map=/build/1st/compyle-0.6~dev0~20190922.gitaa5a50d=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/lib/python3/dist-packages/numpy/core/include -I/usr/local/include -I/usr/include/python3.8 -c /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.cpp -o /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/build/temp.linux-i386-3.8/build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.o
i686-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-z,relro -g -O2 -ffile-prefix-map=/build/1st/compyle-0.6~dev0~20190922.gitaa5a50d=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/build/temp.linux-i386-3.8/build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.o -o /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/build/lib.linux-i386-3.8/m_2a00dba787c0caa83a32476b5532225f.cpython-38-i386-linux-gnu.so

[avalentino]

Is there any progress on this?
I have just tested the new release 0.7 and the problem seems to be still there.

[prabhuramachandran]

Sorry about that, is this both on i386 and armhf? I suspect this is just a poor mapping between the types that we have -- we have perhaps assumed that the world is now 64 bit. Is there a simple way for us to test this ourselves? I can try to spend some time on this later this week but cannot test it, would you be willing to help test this if I push to a public branch for instance?

[avalentino]

Thanks @prabhuramachandran for the quick reply.
Sure I can help to test and I can also try to help with the fix if you provide some guidance.
I can test quite easily on i386, Unfortunately testing on armhf is more complex for me, I have to wait for the standard debian CI.

[prabhuramachandran]

@avalentino -- sorry my response times can be quite terrible sometimes. Thank you for your kind offer! I will try to push a fix to a branch and let you know this week.

[prabhuramachandran]

@avalentino -- can you please try the fix-32bit branch I have just pushed and see if that fixes the problem? Many thanks! Also there a way to easily test this on travis-ci to prevent issues going forward?

[avalentino]

Hi @prabhuramachandran unfortunately the test still fails on i386:

https://salsa.debian.org/science-team/compyle/-/jobs/1047959

[prabhuramachandran]

Thanks @avalentino, I have pushed another small change. Would it be possible to test the latest version of the branch? Thanks!

[avalentino]

Dear @prabhuramachandran the test still fails: https://salsa.debian.org/science-team/compyle/-/jobs/1049211
This time I have also enabled verbose test output and do not caprure output streams.
Please note that the problem seems t affect only the cython backend:

compyle/tests/test_array.py::test_align_multiple[cython] FAILED
compyle/tests/test_array.py::test_align_multiple[opencl] 1 warning generated.
2 warnings generated.
PASSED
compyle/tests/test_array.py::test_align_multiple[cuda] SKIPPED

[prabhuramachandran]

@avalentino -- Thanks for testing. I've made another fix which removes some unnecessary code. Let's hope this works.

[avalentino]

Hi, @prabhuramachandran it seems you are on the right way.
The test still fails unfortunately [1].
The good news is that now the cython backend works, while the pyopencl is broken (in the previous run it was the contrary)

compyle/tests/test_array.py::test_align_multiple[cython] PASSED
compyle/tests/test_array.py::test_align_multiple[opencl] FAILED
compyle/tests/test_array.py::test_align_multiple[cuda] SKIPPED

Is there a way to change the dtype_to_ctype behavior depending on the calling backend?

[1] https://salsa.debian.org/science-team/compyle/-/jobs/1054489

[prabhuramachandran]

@avalentino -- I suspected as much, that cython will pass and opencl would not. At least I know where the issue is potentially and can work on a fix that works for both. Thanks for your patience.

[avalentino]

Hi @prabhuramachandran, just a reminder.
Now that we have found the origin of the issue it would be nice to have this fixed for Debian 11.

[prabhuramachandran]

@avalentino -- yes sorry, I am chasing down another urgent bug and should get to the issue later this week.

[prabhuramachandran]

@avalentino -- I have attempted a more robust fix and force pushed to the fix-32bit branch. Please let me know if this works. Thanks!

[avalentino]

@prabhuramachandran I can confirm that the new fix works correctly. thanks a lot.

[prabhuramachandran]

@avalentino -- thank you very much for testing!

[prabhuramachandran]

@avalentino -- this is now merged in master. Thanks for your patience!