11 Mar
2013
11 Mar
'13
3:39 p.m.
Dear all,
we are trying to implement a K nearest neighbor search on GPUs with
PyOpenCL. The goal of the algorithm is: For a given target point,
find the nearest points from a given set (training data). The distance
between two points is computed by the squared euclidean distance.
One of our implementations is a brute force approach, which aims
at processing big data sets in parallel, e.g. 1 million training data and
some millions of targets (test data). For every target point one kernel
instance is created which finds the k nearest points out of the
training points.
Our problem is the following. Everything works fine for small data sets
and the results are as expected on both GPU (GeForce GTX 650 with
nVidia Driver 313.09.) and CPU(Intel Core i5-3450 with AMD APP SDK)
running Ubuntu 12.10, PyOpenCL 2013.1-py2.7-linux-x86_64.
But if we increase the size of the data sets, the GPU version crashes
with the following error:
File "brutegpu.py", line 65, in query
cl.enqueue_copy(self.queue, d_min, self.d_min_buf).wait()
File "/usr/local/lib/python2.7/dist-packages/
pyopencl-2013.1-py2.7-linux-x86_64.egg/pyopencl/__init__.py",
line 935, in enqueue_copy
return _cl._enqueue_read_buffer(queue, src, dest, **kwargs)
pyopencl.LogicError: clEnqueueReadBuffer failed: invalid command queue
The
CPU-Version still works fine with 1 million training points
and 1 million of test points. Attached you can find the corresponding
source code as working minimal example, which consists of on
Host-Python-File
and one OpenCL-Kernel-File.
We would highly apprecriate any help - maybe we made a
mistake which is already known to you.
So the big question for us is: Why is it working on CPU and why isn't it
working on the GPU?
Are there nVidia-specific pitfalls for such big data sets?
The compiler says:
ptxas info : Compiling entry function
'find_knn' for 'sm_30'
ptxas info : Function properties for find_knn
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 17 registers, 336 bytes cmem[0], 4 bytes cmem[3]
Or are there any rules for using a kernel for big data sets such as setting
the work group sizes or maximum memory usage?
The error message "invalid command queue" is confusing and I wasn't able
to find any helpful information (except that oftentimes "invalid command
queue" means segfault, but i could not find any wrong array adress yet.)
Maybe one of you could have a look at our code and finds some stupid
mistake.
We would be very grateful for every hint.
Best regards,
Justin Heinermann,
University Oldenburg
11 Mar
11 Mar
4:08 p.m.
New subject: Invalid command Queue when using big data sets on nVidia
Hi Justin,
Justin Heinermann <Justin.Heinermann(a)Informatik.Uni-Oldenburg.DE>
writes:
we are trying to implement a K nearest neighbor search
on GPUs with
PyOpenCL. The goal of the algorithm is: For a given target point,
find the nearest points from a given set (training data). The distance
between two points is computed by the squared euclidean distance.
One of our implementations is a brute force approach, which aims
at processing big data sets in parallel, e.g. 1 million training data and
some millions of targets (test data). For every target point one kernel
instance is created which finds the k nearest points out of the
training points.
Our problem is the following. Everything works fine for small data sets
and the results are as expected on both GPU (GeForce GTX 650 with
nVidia Driver 313.09.) and CPU(Intel Core i5-3450 with AMD APP SDK)
running Ubuntu 12.10, PyOpenCL 2013.1-py2.7-linux-x86_64.
But if we increase the size of the data sets, the GPU version crashes
with the following error:
As you mention, this is very likely a bug in your code. I unfortunately
don't have time to go in and debug it for you, but I've got a few hints
that might help:
- Try and use valgrind on your CPU code. Unfortunately, that spews a lot
of unrelated nonsense for both the Intel and AMD OpenCL runtimes, but
it has helped me a few times.
- Have you tried with both Intel's and AMD's CPU runtime?
- AMD's CPU runtime allows you to use gdb (in 32 bit). See their
documentation for how to do that.
HTH,
Andreas
File "brutegpu.py", line 65, in query
cl.enqueue_copy(self.queue, d_min, self.d_min_buf).wait()
File "/usr/local/lib/python2.7/dist-packages/
pyopencl-2013.1-py2.7-linux-x86_64.egg/pyopencl/__init__.py",
line 935, in enqueue_copy
return _cl._enqueue_read_buffer(queue, src, dest, **kwargs)
pyopencl.LogicError: clEnqueueReadBuffer failed: invalid command queue
The
CPU-Version still works fine with 1 million training points
and 1 million of test points. Attached you can find the corresponding
source code as working minimal example, which consists of on
Host-Python-File
and one OpenCL-Kernel-File.
We would highly apprecriate any help - maybe we made a
mistake which is already known to you.
So the big question for us is: Why is it working on CPU and why isn't it
working on the GPU?
Are there nVidia-specific pitfalls for such big data sets?
The compiler says:
ptxas info : Compiling entry function
'find_knn' for 'sm_30'
ptxas info : Function properties for find_knn
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 17 registers, 336 bytes cmem[0], 4 bytes cmem[3]
Or are there any rules for using a kernel for big data sets such as setting
the work group sizes or maximum memory usage?
The error message "invalid command queue" is confusing and I wasn't able
to find any helpful information (except that oftentimes "invalid command
queue" means segfault, but i could not find any wrong array adress yet.)
12 Mar
12 Mar
5:41 p.m.
Hi,
thank you very much for your reply, Andreas.
As you mention, this is very likely a bug in your
code. I unfortunately
don't have time to go in and debug it for you, but I've got a few hints
that might help:
- Try and use valgrind on your CPU code. Unfortunately, that spews a lot
of unrelated nonsense for both the Intel and AMD OpenCL runtimes, but
it has helped me a few times.
Ok I tried that but was not able to find any
useful information yet.
- Have you tried with both Intel's and AMD's
CPU runtime?
Yes, and on both Intel's and AMD's runtime the kernel runs as
expected.
- AMD's CPU runtime allows you to use gdb (in 32
bit). See their
documentation for how to do that.
I have a 64 bit OS and gdb seems not work with
the PyOpenCL stuff.
Because of that, we tested the whole kernel code in C and debugged it.
Everything seems right (see attachment) - for very large data sets, too.
So the code, when executed instance by instance, works fine,
but executed in parallel on nVidia GPU it crashes.
Is there a chance that the code could work with other work group sizes?
I could think of a problem of resource allocation when many kernel
instances are created.
Best regards,
Justin
7:47 p.m.
New subject: Invalid command Queue when using big data sets on nVidia
Hi Justin,
Justin Heinermann <Justin.Heinermann(a)Informatik.Uni-Oldenburg.DE>
writes:
thank you very much for your reply, Andreas.
See if the output of 'dmesg' contains lines with 'NVRM Xid ... 13'. If
so, then that's the GPU equivalent of a segmentation fault. If not, then
something else may be up. In any case, the fact that the code executes
as desired on the CPU does not prove that it is correct...
Hope that helps,
Andreas
As you mention, this is very likely a bug in your
code. I unfortunately
don't have time to go in and debug it for you, but I've got a few hints
that might help:
- Try and use valgrind on your CPU code. Unfortunately, that spews a lot
of unrelated nonsense for both the Intel and AMD OpenCL runtimes, but
it has helped me a few times.
Ok I tried that but was not able to find any
useful information yet.
- Have you tried with both Intel's and
AMD's CPU runtime?
Yes, and on both Intel's and AMD's runtime the
kernel runs as expected.
- AMD's CPU runtime allows you to use gdb (in
32 bit). See their
documentation for how to do that.
I have a 64 bit OS and gdb seems not work
with the PyOpenCL stuff.
Because of that, we tested the whole kernel code in C and debugged it.
Everything seems right (see attachment) - for very large data sets, too.
So the code, when executed instance by instance, works fine,
but executed in parallel on nVidia GPU it crashes.
Is there a chance that the code could work with other work group sizes?
I could think of a problem of resource allocation when many kernel
instances are created.
13 Mar
13 Mar
12:16 p.m.
New subject: Invalid command Queue when using big data sets on nVidia (FIXED)
Hi,
thanks again for the suggestions. Altough your assumption was right,
the problem is a different (see below)
Am 12.03.2013 20:47, schrieb Andreas Kloeckner:
See if the output of 'dmesg' contains lines
with 'NVRM Xid ... 13'. If
so, then that's the GPU equivalent of a segmentation fault. If not, then
something else may be up. In any case, the fact that the code executes
as desired on the CPU does not prove that it is correct...
When our application
crashes with "invalid command queue" error,
there also appears the mentioned line in dmesg, so it really looks like a
segmentation fault.
We also tried a version which holds the temporary lists only in private
memory and
the input buffers are READ_ONLY and the output buffers are WRITE_ONLY,
but the problem remains.
We found out, that the problem had s.th. to do with a big number of
loop-iterations,
which lead to a long run time. On nVidia Platform (both Windows and
Linux, don't know about Mac)
the processes on the GPU are killed after a time like 5-10 seconds or
so. I think the intention is to keep the GUI responsive.
The problem can be found here, too:
http://stackoverflow.com/questions/497685/how-do-you-get-around-the-maximum…
The funny thing is, that this behaviour only occurs when the X11-Server
is running.
If I kill X11 Server and run our application on console, it runs as
expected.
Ok, maybe a better solution is to only use kernels which are fast enough.
We can divide our algorithm into smaller steps, which was planned anyway.
So the problem is fixed, but if anyone has other ideas to solve the
runtime problem for nVidia devices,
let us know. This might also be useful for other OpenCL programmers.
Best regards,
Justin Heinermann
12:36 p.m.
New subject: Invalid command Queue when using big data sets on nVidia (FIXED)
Oh, that's probably why I wasn't seeing it. I have a slow video card
for GUI, and separate faster cards to run stuff on (if you take into
account the work required to modify your programs, this may be
actually the cheapest solution).
On Wed, Mar 13, 2013 at 11:16 PM, Justin Heinermann
<Justin.Heinermann(a)informatik.uni-oldenburg.de> wrote:
Hi,
thanks again for the suggestions. Altough your assumption was right,
the problem is a different (see below)
Am 12.03.2013 20:47, schrieb Andreas Kloeckner:
See if the output of 'dmesg' contains lines with 'NVRM Xid ... 13'. If
so, then that's the GPU equivalent of a segmentation fault. If not, then
something else may be up. In any case, the fact that the code executes
as desired on the CPU does not prove that it is correct...
When our application crashes with "invalid command queue" error,
there also appears the mentioned line in dmesg, so it really looks like a
segmentation fault.
We also tried a version which holds the temporary lists only in private
memory and
the input buffers are READ_ONLY and the output buffers are WRITE_ONLY,
but the problem remains.
We found out, that the problem had s.th. to do with a big number of
loop-iterations,
which lead to a long run time. On nVidia Platform (both Windows and Linux,
don't know about Mac)
the processes on the GPU are killed after a time like 5-10 seconds or so. I
think the intention is to keep the GUI responsive.
The problem can be found here, too:
http://stackoverflow.com/questions/497685/how-do-you-get-around-the-maximum…
The funny thing is, that this behaviour only occurs when the X11-Server is
running.
If I kill X11 Server and run our application on console, it runs as
expected.
Ok, maybe a better solution is to only use kernels which are fast enough.
We can divide our algorithm into smaller steps, which was planned anyway.
So the problem is fixed, but if anyone has other ideas to solve the runtime
problem for nVidia devices,
let us know. This might also be useful for other OpenCL programmers.
Best regards,
Justin Heinermann
_______________________________________________
PyOpenCL mailing list
PyOpenCL(a)tiker.net
http://lists.tiker.net/listinfo/pyopencl 
19 Aug
19 Aug
12:04 p.m.
New subject: Invalid command Queue when using big data sets on nVidia (FIXED)
Justin Heinermann <Justin.Heinermann@...> writes:
Hi,
thanks again for the suggestions. Altough your assumption was right,
the problem is a different (see below)
Am 12.03.2013 20:47, schrieb Andreas Kloeckner:
maximum-cuda-run-time
See if the output of 'dmesg' contains
lines with 'NVRM Xid ... 13'. If
so, then that's the GPU equivalent of a segmentation fault. If not, then
something else may be up. In any case, the fact that the code executes
as desired on the CPU does not prove that it is correct...
When our application
crashes with "invalid command queue" error,
there also appears the mentioned line in dmesg, so it really looks like a
segmentation fault.
We also tried a version which holds the temporary lists only in private
memory and
the input buffers are READ_ONLY and the output buffers are WRITE_ONLY,
but the problem remains.
We found out, that the problem had s.th. to do with a big number of
loop-iterations,
which lead to a long run time. On nVidia Platform (both Windows and
Linux, don't know about Mac)
the processes on the GPU are killed after a time like 5-10 seconds or
so. I think the intention is to keep the GUI responsive.
The problem can be found here, too:
http://stackoverflow.com/questions/497685/how-do-you-get-around-the-
The funny thing is, that this behaviour only occurs when the X11-Server
is running.
If I kill X11 Server and run our application on console, it runs as
expected.
Ok, maybe a better solution is to only use kernels which are fast enough.
We can divide our algorithm into smaller steps, which was planned anyway.
So the problem is fixed, but if anyone has other ideas to solve the
runtime problem for nVidia devices,
let us know. This might also be useful for other OpenCL programmers.
Best regards,
Justin Heinermann
I wanted to thank you for this point as I have run into this same problem a
few times during the last few months, when writing experimental not-
necessarily-optimal code. A very annoying feature and one everyone working
with nVidia GPUs, OpenCL and some OS with X11 server should be aware of.
The problem itself can be very painful to identify if the timeout point does
not pop into one's mind, as removing a line there and another here might
just squeeze the execution of the kernel to the right side of the timeout
and one will faultly think that e.g. "hmm, ok, so I can't perform
normalize() inside three for loops? That's a tad strange..."
Thanks!
Best regards,
Kalle Karhu
2392
days inactive
2918
days old
6 comments
4 participants
participants (4)
-
Andreas Kloeckner -
Bogdan Opanchuk -
Justin Heinermann -
Kalle Karhu