Justin Heinermann <Justin.Heinermann(a)Informatik.Uni-Oldenburg.DE>
we are trying to implement a K nearest neighbor search
on GPUs with
PyOpenCL. The goal of the algorithm is: For a given target point,
find the nearest points from a given set (training data). The distance
between two points is computed by the squared euclidean distance.
One of our implementations is a brute force approach, which aims
at processing big data sets in parallel, e.g. 1 million training data and
some millions of targets (test data). For every target point one kernel
instance is created which finds the k nearest points out of the
Our problem is the following. Everything works fine for small data sets
and the results are as expected on both GPU (GeForce GTX 650 with
nVidia Driver 313.09.) and CPU(Intel Core i5-3450 with AMD APP SDK)
running Ubuntu 12.10, PyOpenCL 2013.1-py2.7-linux-x86_64.
But if we increase the size of the data sets, the GPU version crashes
with the following error:
File "brutegpu.py", line 65, in query
cl.enqueue_copy(self.queue, d_min, self.d_min_buf).wait()
line 935, in enqueue_copy
return _cl._enqueue_read_buffer(queue, src, dest, **kwargs)
pyopencl.LogicError: clEnqueueReadBuffer failed: invalid command queue
CPU-Version still works fine with 1 million training points
and 1 million of test points. Attached you can find the corresponding
source code as working minimal example, which consists of on
and one OpenCL-Kernel-File.
We would highly apprecriate any help - maybe we made a
mistake which is already known to you.
So the big question for us is: Why is it working on CPU and why isn't it
working on the GPU?
Are there nVidia-specific pitfalls for such big data sets?
The compiler says:
ptxas info : Compiling entry function
'find_knn' for 'sm_30'
ptxas info : Function properties for find_knn
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 17 registers, 336 bytes cmem, 4 bytes cmem
Or are there any rules for using a kernel for big data sets such as setting
the work group sizes or maximum memory usage?
The error message "invalid command queue" is confusing and I wasn't able
to find any helpful information (except that oftentimes "invalid command
queue" means segfault, but i could not find any wrong array adress yet.)
As you mention, this is very likely a bug in your code. I unfortunately
don't have time to go in and debug it for you, but I've got a few hints
that might help:
- Try and use valgrind on your CPU code. Unfortunately, that spews a lot
of unrelated nonsense for both the Intel and AMD OpenCL runtimes, but
it has helped me a few times.
- Have you tried with both Intel's and AMD's CPU runtime?
- AMD's CPU runtime allows you to use gdb (in 32 bit). See their
documentation for how to do that.