PyOpenCL May 2014

pyopencl@tiker.net

10 participants
18 discussions

Invalid command Queue when using big data sets on nVidia

by Justin Heinermann

Dear all, we are trying to implement a K nearest neighbor search on GPUs with PyOpenCL. The goal of the algorithm is: For a given target point, find the nearest points from a given set (training data). The distance between two points is computed by the squared euclidean distance. One of our implementations is a brute force approach, which aims at processing big data sets in parallel, e.g. 1 million training data and some millions of targets (test data). For every target point one kernel instance is created which finds the k nearest points out of the training points. Our problem is the following. Everything works fine for small data sets and the results are as expected on both GPU (GeForce GTX 650 with nVidia Driver 313.09.) and CPU(Intel Core i5-3450 with AMD APP SDK) running Ubuntu 12.10, PyOpenCL 2013.1-py2.7-linux-x86_64. But if we increase the size of the data sets, the GPU version crashes with the following error: > File "brutegpu.py", line 65, in query > cl.enqueue_copy(self.queue, d_min, self.d_min_buf).wait() > File "/usr/local/lib/python2.7/dist-packages/ > pyopencl-2013.1-py2.7-linux-x86_64.egg/pyopencl/__init__.py", > line 935, in enqueue_copy > return _cl._enqueue_read_buffer(queue, src, dest, **kwargs) > pyopencl.LogicError: clEnqueueReadBuffer failed: invalid command queue The CPU-Version still works fine with 1 million training points and 1 million of test points. Attached you can find the corresponding source code as working minimal example, which consists of on Host-Python-File and one OpenCL-Kernel-File. We would highly apprecriate any help - maybe we made a mistake which is already known to you. So the big question for us is: Why is it working on CPU and why isn't it working on the GPU? Are there nVidia-specific pitfalls for such big data sets? The compiler says: > ptxas info : Compiling entry function 'find_knn' for 'sm_30' > ptxas info : Function properties for find_knn > 0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads > ptxas info : Used 17 registers, 336 bytes cmem[0], 4 bytes cmem[3] Or are there any rules for using a kernel for big data sets such as setting the work group sizes or maximum memory usage? The error message "invalid command queue" is confusing and I wasn't able to find any helpful information (except that oftentimes "invalid command queue" means segfault, but i could not find any wrong array adress yet.) Maybe one of you could have a look at our code and finds some stupid mistake. We would be very grateful for every hint. Best regards, Justin Heinermann, University Oldenburg

6 years, 11 months

Re: [PyOpenCL] PyOpenCL, PyPy, and future directions

by Andreas Kloeckner

Bogdan Opanchuk <mantihor(a)gmail.com> writes: > Hi Andreas, > > On Wed, May 28, 2014 at 4:04 PM, Andreas Kloeckner > <lists(a)informa.tiker.net> wrote: >> - PyOpenCL's array and algorithms functionality will be spun off into >> separate packages ("clarray" and "clalgorithms"?). > > Isn't `compyte` supposed to be the array package (and I remember it > containing some algorithms as well)? Will its status change somehow? Compyte has a role across PyCUDA and PyOpenCL. The things I'm describing here are CL-only. Andreas

7 years, 1 month

PyOpenCL in Debian - change of section

by Tomasz Rybak

Hello. PyOpenCL packages in Debian has been moved from contrib to main. To those less familiar with Debian taxonomy it means that PyOpenCL is in "True Debian(TM)" in fully supported section. It also means that it is visible in default install - user does not need to enable additional sections to have it available. Debian currently contains 4 OpenCL implementations: 1. AMD 2. Beignet 3. Mesa 4. NVIDIA so there is place to play. More details of this change is in post: http://tomaszrybak.wordpress.com/2014/05/27/pyopencl-in-main/ Best regards. -- Tomasz Rybak GPG/PGP key ID: 2AD5 9860 Fingerprint A481 824E 7DD3 9C0E C40A 488E C654 FB33 2AD5 9860 http://member.acm.org/~tomaszrybak

7 years, 2 months

Non-empty compiler output warning from Intel OpenCL

by Freddie Witherden

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, Recently I have been playing around with OpenCL on the Intel Xeon Phi. A quirk of the OpenCL stack (which also applies to the Intel CPU stack) is that the compiler is extremely chatty. Running a simple axpy kernel gives: /usr/local/lib/python2.7/site-packages/pyopencl/__init__.py:57: Build on <pyopencl.Device 'Intel(R) Many Integrated Core Acceleration Card' on 'Intel(R) OpenCL' at 0x1a207d8> succeeded, but said: Compilation started Compilation done Linking started Linking done Build started Kernel <sum> was successfully vectorized Done. warn(text, CompilerWarning) /usr/local/lib/python2.7/site-packages/pyopencl/__init__.py:57: CompilerWarning: From-binary build succeeded, but resulted in non-empty logs: Build on <pyopencl.Device 'Intel(R) Many Integrated Core Acceleration Card' on 'Intel(R) OpenCL' at 0x1a207d8> succeeded, but said: Linking started Linking done Build started Kernel <sum> was successfully vectorized Done. warn(text, CompilerWarning) I am not sure the best way to handle this; PyOpenCL does not appear to expose an API for programmatic control over if these 'warnings' are generated. It is of course possible to sink the output but this does not help us to decide if a genuine warning has been generated or not (which are absolutely of interest and should be displayed). How have other people handled this? Regards, Freddie. -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.20 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlOHbSEACgkQ/J9EM/uoqVfbxQCdFv/SKh3JAGGvg2jVsRSSScJs Dn8An0E4hWkjXBT2+K9gbJzePiIRX6YZ =4cPY -----END PGP SIGNATURE-----

7 years, 2 months

Re: [PyOpenCL] import pyopencl fails on py3k

by Andreas Kloeckner

Hi Guilherme, Guilherme Gonçalves Ferrari <gg.ferrari(a)gmail.com> writes: > So, I think the problem is with the pytools' wheel. > > using "pip install --no-use-wheel pytools" everything works just fine. > > however, here is what happened when I used just "pip install pytools": Thanks for investigating! I've deleted the wheels from my 2to3-using packages for the time being--it'd be great if you could verify that this addresses the issue. It does for me. Thanks again, Andreas

7 years, 2 months

Re: [PyOpenCL] import pyopencl fails on py3k

by Guilherme Gonçalves Ferrari

Hi Andreas, So, I think the problem is with the pytools' wheel. using "pip install --no-use-wheel pytools" everything works just fine. however, here is what happened when I used just "pip install pytools": ------------------------------------------- Downloading/unpacking pytools Downloading pytools-2014.2.1-py2.py3-none-any.whl (51kB): 51kB downloaded Requirement already satisfied (use --upgrade to upgrade): decorator>=3.2.0 in ./Modelos/python/conda/miniconda/envs/py33/lib/python3.3/site-packages (from pytools) Installing collected packages: pytools *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/__init__.py'... File "/tmp/pip_build_guilherme/pytools/pytools/__init__.py", line 1313 print value, bin_nr, bin_starts ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/arithmetic_container.py'... File "/tmp/pip_build_guilherme/pytools/pytools/arithmetic_container.py", line 271 raise ValueError, "matrix width does not match ArithmeticList" ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/debug.py'... File "/tmp/pip_build_guilherme/pytools/pytools/debug.py", line 18 os.O_CREAT | os.O_WRONLY | os.O_EXCL, 0444), "w") ^ SyntaxError: invalid token *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/diskdict.py'... File "/tmp/pip_build_guilherme/pytools/pytools/diskdict.py", line 23 except OSError, e: ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/lex.py'... File "/tmp/pip_build_guilherme/pytools/pytools/lex.py", line 51 print "Trying", rule, "on", s[start:] ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/log.py'... File "/tmp/pip_build_guilherme/pytools/pytools/log.py", line 557 print "while adding datapoint for '%s':" % name ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/persistent_dict.py'... File "/tmp/pip_build_guilherme/pytools/pytools/persistent_dict.py", line 258 except OSError, e: ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/prefork.py'... File "/tmp/pip_build_guilherme/pytools/pytools/prefork.py", line 25 except OSError, e: ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/spatial_btree.py'... File "/tmp/pip_build_guilherme/pytools/pytools/spatial_btree.py", line 4 def do_boxes_intersect((bl1,tr1), (bl2,tr2)): ^ SyntaxError: invalid syntax *** Error compiling '/tmp/pip_build_guilherme/pytools/pytools/stopwatch.py'... File "/tmp/pip_build_guilherme/pytools/pytools/stopwatch.py", line 36 print "%s..." % name ^ SyntaxError: invalid syntax Successfully installed pytools Cleaning up... -------------------------------------------

7 years, 2 months

Re: [PyOpenCL] import pyopencl fails on py3k

by Andreas Kloeckner

Guilherme Gonçalves Ferrari <gg.ferrari(a)gmail.com> writes: > import pyopencl fails because pytools seems to be broken on py3k. I'm using > the latest version of both packages. > > ------------------------------- > import pyopencl as cl > > File > "/home/travis/miniconda/envs/testenv3.4/lib/python3.4/site-packages/pyopencl/__init__.py", > line 716, in <module> > > _add_functionality() > > File > "/home/travis/miniconda/envs/testenv3.4/lib/python3.4/site-packages/pyopencl/__init__.py", > line 333, in _add_functionality > > from pytools import memoize_method > > File > "/home/travis/miniconda/envs/testenv3.4/lib/python3.4/site-packages/pytools/__init__.py", > line 1313 > > print value, bin_nr, bin_starts > > ^ > > SyntaxError: invalid syntax > ------------------------------- How did you install pytools? It uses 2to3, so that print statement would have gotten translated if 2to3 had run as intended. Apparently it has not, and I'm trying to figure out why. Andreas

7 years, 2 months

PyOpenCL, PyPy, and future directions

by Andreas Kloeckner

Hi all, thanks to the tireless efforts of Marko Bencun and Yichao Yu, PyOpenCL now has a reasonably functional branch [1] based on cffi instead of Boost.Python. [1] https://github.com/pyopencl/pyopencl/tree/cffi What this means is that there's real hope that at some point in the not-too-distant future there will be a fully functional PyOpenCL that runs on PyPy, which I personally find very exciting. This would amount to a frighteningly nice environment within which to do high-performance work. Now, we're not quite there yet. Let me quickly summarize the state of things: - I can go grab PyPy 2.3 with apt-get (Debian has it, for example), make a virtualenv, install numpy following the instructions [2], install PyOpenCL as usual, run the tests, and passes 85% of them. (79/93 on my machine) I'll admit that I had never really played much with PyPy. The extent to which all of this 'just works' with it is really impressive. - That said, important bits are still missing. It's not ready for production use. If you'd like to help out, see this file [3]. We're tantalizingly close. [2] https://bitbucket.org/pypy/numpy [3] https://github.com/pyopencl/pyopencl/blob/cffi/TODOs This has important implications for the future of PyOpenCL. Here's where I imagine we might be headed: - For a while, there will be two equivalent implementations of PyOpenCL, pyopencl_bpl and pyopencl_cffi. Eventually, pyopencl_bpl may be deprecated. Both will install as "pyopencl". The "pyopencl" package index entry will be an alias of one of the two--likely "pyopencl_bpl" initially. - PyOpenCL's array and algorithms functionality will be spun off into separate packages ("clarray" and "clalgorithms"?). To ease the transition, both pyopencl versions will depend on these packages, but using them through their old names ("pyopencl.array" etc) will be deprecated. Naturally, this will come with a long transition period to give dependencies time to adapt. Interfaces will stay the same, so that all that's needed in downstream software is a change of the import name. This step not just eases maintenance of both base wrappers, it is also fairer to alternative CL algorithms packages like Bogdan's reikna and potential alternative array libraries, which now don't have to face the question of why they're replacing something that already comes bundled with the wrapper. In addition, I've found that much of the algorithms functionality often gets overlooked, because the name 'PyOpenCL' doesn't indicate that there's quite a bit more in the package. - To make all of this easier, I will make one more release (2014.1) that supports Python 2.4 and later, and for all releases after that, Python 2.6 will be required. Comments? Andreas

7 years, 2 months

import pyopencl fails on py3k

by Guilherme Gonçalves Ferrari

import pyopencl fails because pytools seems to be broken on py3k. I'm using the latest version of both packages. ------------------------------- import pyopencl as cl File "/home/travis/miniconda/envs/testenv3.4/lib/python3.4/site-packages/pyopencl/__init__.py", line 716, in <module> _add_functionality() File "/home/travis/miniconda/envs/testenv3.4/lib/python3.4/site-packages/pyopencl/__init__.py", line 333, in _add_functionality from pytools import memoize_method File "/home/travis/miniconda/envs/testenv3.4/lib/python3.4/site-packages/pytools/__init__.py", line 1313 print value, bin_nr, bin_starts ^ SyntaxError: invalid syntax -------------------------------

7 years, 2 months

Basic GL manipulation issue

by Keith Brafford

I am having (what I hope is) a basic OpenGL user-interface issue with a sample program I am working on. This glut program has a visualized OpenCL data structure whose points are calculated via an OpenCL kernel. The scene is then combined with a standard Utah Teapot and dodecahedron straight out of the glut library. The teapot and polyhedron are rendered using shaders--so if anyone is looking for an example using OpenCL with shader visualizations this example might be useful. The problem I have is that if you manipulate the scene with the mouse you can tell that the OpenCL particle fountain is always moving according to the mouse input, but the teapot and polyhedron sometimes seem to move in the opposite direction. I know this isn't an OpenCL question per se, but it's almost impossible to find OpenGL people who are set up to also run and evaluate OpenCL code, but people who understand and use OpenCL are often times also able to run OpenGL samples. Can someone help me figure out what I'm doing wrong? https://gist.github.com/kbrafford/18665bdafa2b0a822f4f Thanks! --Keith

7 years, 2 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

PyOpenCL May 2014