we are trying to implement a K nearest neighbor search on GPUs with PyOpenCL. The goal of the algorithm is: For a given target point, find the nearest points from a given set (training data). The distance between two points is computed by the squared euclidean distance. One of our implementations is a brute force approach, which aims at processing big data sets in parallel, e.g. 1 million training data and some millions of targets (test data). For every target point one kernel instance is created which finds the k nearest points out of the training points. Our problem is the following. Everything works fine for small data sets and the results are as expected on both GPU (GeForce GTX 650 with nVidia Driver 313.09.) and CPU(Intel Core i5-3450 with AMD APP SDK) running Ubuntu 12.10, PyOpenCL 2013.1-py2.7-linux-x86_64. But if we increase the size of the data sets, the GPU version crashes with the following error: > File "brutegpu.py", line 65, in query > cl.enqueue_copy(self.queue, d_min, self.d_min_buf).wait() > File "/usr/local/lib/python2.7/dist-packages/ > pyopencl-2013.1-py2.7-linux-x86_64.egg/pyopencl/__init__.py", > line 935, in enqueue_copy > return _cl._enqueue_read_buffer(queue, src, dest, **kwargs) > pyopencl.LogicError: clEnqueueReadBuffer failed: invalid command queue The CPU-Version still works fine with 1 million training points and 1 million of test points. Attached you can find the corresponding source code as working minimal example, which consists of on Host-Python-File and one OpenCL-Kernel-File. We would highly apprecriate any help - maybe we made a mistake which is already known to you. So the big question for us is: Why is it working on CPU and why isn't it working on the GPU? Are there nVidia-specific pitfalls for such big data sets? The compiler says: > ptxas info : Compiling entry function 'find_knn' for 'sm_30' > ptxas info : Function properties for find_knn > 0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads > ptxas info : Used 17 registers, 336 bytes cmem[0], 4 bytes cmem[3] Or are there any rules for using a kernel for big data sets such as setting the work group sizes or maximum memory usage? The error message "invalid command queue" is confusing and I wasn't able to find any helpful information (except that oftentimes "invalid command queue" means segfault, but i could not find any wrong array adress yet.)

thank you very much for your reply, Andreas. > As you mention, this is very likely a bug in your code. I unfortunately > don't have time to go in and debug it for you, but I've got a few hints > that might help: > > - Try and use valgrind on your CPU code. Unfortunately, that spews a lot > of unrelated nonsense for both the Intel and AMD OpenCL runtimes, but > it has helped me a few times. Ok I tried that but was not able to find any useful information yet. > - Have you tried with both Intel's and AMD's CPU runtime? Yes, and on both Intel's and AMD's runtime the kernel runs as expected. > - AMD's CPU runtime allows you to use gdb (in 32 bit). See their > documentation for how to do that. I have a 64 bit OS and gdb seems not work with the PyOpenCL stuff. Because of that, we tested the whole kernel code in C and debugged it. Everything seems right (see attachment) - for very large data sets, too. So the code, when executed instance by instance, works fine, but executed in parallel on nVidia GPU it crashes. Is there a chance that the code could work with other work group sizes? I could think of a problem of resource allocation when many kernel instances are created.

Hi, thanks again for the suggestions. Altough your assumption was right, the problem is a different (see below) Am 12.03.2013 20:47, schrieb Andreas Kloeckner: > > See if the output of 'dmesg' contains lines with 'NVRM Xid ... 13'. If > so, then that's the GPU equivalent of a segmentation fault. If not, then > something else may be up. In any case, the fact that the code executes > as desired on the CPU does not prove that it is correct... When our application crashes with "invalid command queue" error, there also appears the mentioned line in dmesg, so it really looks like a segmentation fault. We also tried a version which holds the temporary lists only in private memory and the input buffers are READ_ONLY and the output buffers are WRITE_ONLY, but the problem remains. We found out, that the problem had s.th. to do with a big number of loop-iterations, which lead to a long run time. On nVidia Platform (both Windows and Linux, don't know about Mac) the processes on the GPU are killed after a time like 5-10 seconds or so. I think the intention is to keep the GUI responsive. The problem can be found here, too: http://stackoverflow.com/questions/497685/how-do-you-get-around-the-maxim... The funny thing is, that this behaviour only occurs when the X11-Server is running. If I kill X11 Server and run our application on console, it runs as expected. Ok, maybe a better solution is to only use kernels which are fast enough. We can divide our algorithm into smaller steps, which was planned anyway. So the problem is fixed, but if anyone has other ideas to solve the runtime problem for nVidia devices, let us know. This might also be useful for other OpenCL programmers. Best regards, Justin Heinermann _______________________________________________ PyOpenCL mailing list PyOpenCL(a)tiker.net http://lists.tiker.net/listinfo/pyopencl