29 May
2012
29 May
'12
8:17 p.m.
Dear PyCuda community,
First of all I would like to introduce myself: I am a scientific
developer and I am pretty new to PyCuda (even if I followed a CUDA
course). I would like to port part of a very big application to GPU,
switching from FFTw to scikit.cuda (cu_fft part). This was straight
forward, thanks to the very good abstraction done in PyCuda. I got
already speed-up of 5x with exactly the same result compared to fftw.
My problems starts when integrating the code into python-threads;
indeed the large application will make all PyCuda calls from different
threads and ends with memory leaks on the GPU crashing after a couple
of minutes. So I need to enforce all python threads to use the same
context on the GPU.
I have another question: why is the data1_gpu.ptr changing whereas data2_gpu.ptr and plan
fixed (as expected in my code ?
Thanks for your help.
Cheers
--
Jérôme Kieffer
Data analysis unit - ESRF
29 May
29 May
11:02 p.m.
On Tue, 29 May 2012 22:17:52 +0200, Jerome Kieffer <Jerome.Kieffer(a)esrf.fr> wrote:
Dear PyCuda community,
First of all I would like to introduce myself: I am a scientific
developer and I am pretty new to PyCuda (even if I followed a CUDA
course). I would like to port part of a very big application to GPU,
switching from FFTw to scikit.cuda (cu_fft part). This was straight
forward, thanks to the very good abstraction done in PyCuda. I got
already speed-up of 5x with exactly the same result compared to fftw.
My problems starts when integrating the code into python-threads;
indeed the large application will make all PyCuda calls from different
threads and ends with memory leaks on the GPU crashing after a couple
of minutes. So I need to enforce all python threads to use the same
context on the GPU.
Right. IOW, it's a mess. Multiple threads fighting over one GPU context
is not a good idea, and if you have *any* other way to design your
program, use that.
If you *need* to use this design, there's a way to prevent the leaks:
Also manage all your memory manually (see, it's getting prettier by the
minute). The problem is that it's not guaranteed that PyCUDA can
activate an object's home context at garbage collection time.
Alternatively, PyOpenCL (and PyFFT) make threading, even for a single
context across multiple host threads, completely painless. Nearly all of
the CL API is thread-safe by definition. End of story. (CL 1.2 standard,
Section A.2)
I have another question: why is the data1_gpu.ptr
changing whereas
data2_gpu.ptr and plan fixed (as expected in my code ?
No idea. When I wrote GPUArray, I thought of .ptr as a read-only
attribute. Seems scikits.cuda has a different opinion.
Andreas
30 May
30 May
3:37 p.m.
On Tue, 29 May 2012 19:02:38 -0400
Andreas Kloeckner <lists(a)buster.tiker.net> wrote:
Right. IOW, it's a mess. Multiple threads fighting
over one GPU context
is not a good idea, and if you have *any* other way to design your
program, use that.
Thanks Anreas,
The core of the program has been developed for 5 year and the application for >2 years
now.
The PyCuda part is very promising, especially because I only spent 6 hours on it. But
moving to a pure ctypes interface or a Cython interface is still an option for now. I
liked pretty much the scikit.cuda interface to cuda-fft.
If you *need* to use this design, there's a way to
prevent the leaks:
Also manage all your memory manually (see, it's getting prettier by the
minute). The problem is that it's not guaranteed that PyCUDA can
activate an object's home context at garbage collection time.
I understand the problem.
Alternatively, PyOpenCL (and PyFFT) make threading,
even for a single
context across multiple host threads, completely painless.
I would have preferred PyOpenCL and PyFFT but the later is limited to power of 2 array
size and we have have a PCO-edge camera witch does not fulfil this requirement.
Nearly all of
the CL API is thread-safe by definition. End of story. (CL 1.2 standard,
Section A.2)
great to know.
No idea. When I wrote GPUArray, I thought of .ptr as a
read-only
attribute. Seems scikits.cuda has a different opinion.
OK it is then in scikits.cuda. Sorry for the mistake.
Thanks a lot for all precision.
Cheers,
--
Jérôme Kieffer
Data analysis unit - ESRF
9:30 p.m.
On Wed, 30 May 2012 17:37:09 +0200, Jerome Kieffer <Jerome.Kieffer(a)esrf.fr> wrote:
On Tue, 29 May 2012 19:02:38 -0400
Andreas Kloeckner <lists(a)buster.tiker.net> wrote:
Keith Brafford and Gregor Thalhammer are working on wrapping AMD's
OpenCL FFT library for use with PyOpenCL--they even have some working
code, I believe. Might be worth checking out.
Andreas
Right. IOW, it's a mess. Multiple threads
fighting over one GPU context
is not a good idea, and if you have *any* other way to design your
program, use that.
Thanks Anreas,
The core of the program has been developed for 5 year and the application for >2 years
now.
The PyCuda part is very promising, especially because I only spent 6 hours on it. But
moving to a pure ctypes interface or a Cython interface is still an option for now. I
liked pretty much the scikit.cuda interface to cuda-fft.
If you *need* to use this design, there's a
way to prevent the leaks:
Also manage all your memory manually (see, it's getting prettier by the
minute). The problem is that it's not guaranteed that PyCUDA can
activate an object's home context at garbage collection time.
I understand the problem.
Alternatively, PyOpenCL (and PyFFT) make
threading, even for a single
context across multiple host threads, completely painless.
I would have preferred PyOpenCL and PyFFT but the later is limited to
power of 2 array size and we have have a PCO-edge camera witch does
not fulfil this requirement. 
6:02 p.m.
Received from Jerome Kieffer on Tue, May 29, 2012 at 04:17:52PM EDT:
Dear PyCuda community,
First of all I would like to introduce myself: I am a scientific
developer and I am pretty new to PyCuda (even if I followed a CUDA
course). I would like to port part of a very big application to GPU,
switching from FFTw to scikit.cuda (cu_fft part). This was straight
forward, thanks to the very good abstraction done in PyCuda. I got
already speed-up of 5x with exactly the same result compared to fftw.
My problems starts when integrating the code into python-threads;
indeed the large application will make all PyCuda calls from different
threads and ends with memory leaks on the GPU crashing after a couple
of minutes. So I need to enforce all python threads to use the same
context on the GPU.
I have another question: why is the data1_gpu.ptr changing whereas
data2_gpu.ptr and plan fixed (as expected in my code ?
The address changes because the fft() and ifft() functions effectively
allocate a new array when scaling of the result is requested. I'll
modify the functions to perform the scaling operation in-place.
L.G.
31 May
31 May
6:11 p.m.
On Wed, 30 May 2012 14:02:12 -0400
Lev Givon <lev(a)columbia.edu> wrote:
The address changes because the fft() and ifft()
functions effectively
allocate a new array when scaling of the result is requested. I'll
modify the functions to perform the scaling operation in-place.
This raises a question:
Why is scikits.cuda not merged into PyCUDA ?
I don't want to hurt anybody but it looks like a natural extension as cufft, cula or
cublas are natural extension to cuda.
Cheers,
--
Jérôme Kieffer
Data analysis unit - ESRF
7:37 p.m.
Received from Jerome Kieffer on Thu, May 31, 2012 at 02:11:14PM EDT:
On Wed, 30 May 2012 14:02:12 -0400
Lev Givon <lev(a)columbia.edu> wrote:
I'm not opposed to eventually merging scikits.cuda with pycuda (the
licenses, after all, are essentially compatible), but since they are
currently developed by different developers and many users of the
latter don't need the functionality provided by the former, I suspect
that maintaining them as separately developed projects that can each
be developed at a different pace is preferable for the time being.
L.G.
The address changes because the fft() and ifft()
functions effectively
allocate a new array when scaling of the result is requested. I'll
modify the functions to perform the scaling operation in-place.
This raises a question:
Why is scikits.cuda not merged into PyCUDA ?
I don't want to hurt anybody but it looks like a natural extension
as cufft, cula or cublas are natural extension to cuda.
1 Jun
1 Jun
3:24 p.m.
Lev Givon <lev(a)columbia.edu> writes:
Received from Jerome Kieffer on Thu, May 31, 2012 at
02:11:14PM EDT:
Completely agree with Lev. I'm not opposed to merging scikits.cuda, if
there's a community consensus that that's the right thing to do. It's
mainly maintenance logistics that make the current arrangement
convenient. Anyway, if you feel strongly about this, now would be a good
time to speak up.
Andreas
On Wed, 30 May 2012 14:02:12 -0400
Lev Givon <lev(a)columbia.edu> wrote:
I'm not opposed to eventually merging scikits.cuda with pycuda (the
licenses, after all, are essentially compatible), but since they are
currently developed by different developers and many users of the
latter don't need the functionality provided by the former, I suspect
that maintaining them as separately developed projects that can each
be developed at a different pace is preferable for the time being. The address changes because the fft() and ifft()
functions effectively
allocate a new array when scaling of the result is requested. I'll
modify the functions to perform the scaling operation in-place.
This raises a question:
Why is scikits.cuda not merged into PyCUDA ?
I don't want to hurt anybody but it looks like a natural extension
as cufft, cula or cublas are natural extension to cuda.
7:58 p.m.
On Fri, 01 Jun 2012 11:24:03 -0400
Andreas Kloeckner <lists(a)buster.tiker.net> wrote:
Completely agree with Lev. I'm not opposed to
merging scikits.cuda, if
there's a community consensus that that's the right thing to do. It's
mainly maintenance logistics that make the current arrangement
convenient. Anyway, if you feel strongly about this, now would be a good
time to speak up.
I cannot claim I am representing the community, far from that.
I have just checked the dependency tree of libcufft & libcublas and it is clear that
many PyCUDA users will not have them installed. Creating extra dependencies in PyCUDA is
not needed. You convinced me for the status-quo.
Cheers,
--
Jérôme Kieffer
Data analysis unit - ESRF
3157
days inactive
3160
days old
8 comments
3 participants
participants (3)
-
Andreas Kloeckner -
Jerome Kieffer -
Lev Givon