PyCUDA January 2015

pycuda@tiker.net

6 participants
10 discussions

Compute several FFT with GPU using Python multiprocessing and pyfft: how to avoid GPU memory leak?

by Marco Tazzari

I am trying to implement in Python the following pattern for **multi-CPU and single-GPU** computation using **pycuda** and **pyfft** packages. I would like to have **several processes** (e.g. launched with multiprocessing.Pool()), with **each of them** able to perform **FFTs using the GPU (using NVIDIA CUDA)**. However, I have the following problem: If I run too many processes or too many FFTs per process, **the overall script remains on hold without terminating** (and without computing all the FFTs that are due). From further investigations I suppose this is due to the **memory limit** on the GPU (currently 2048MB on NVIDIA GeForce GT 750M). It seems that the multiprocessing pool is not able to acquire the control back. Is there any way to avoid this? Since each process requires less than 2048 MB, I would like to have something like a **queue** where each process can *book* the usage of the GPU and, when a process releases the context, the next process in the queue starts using it. Is this doable? Alternatively, is it possible to force that only one process uses the GPU at a given time? I have tried separately these solutions but they do not work (or probably I have not implemented them correctly): 1. synchronize the stream, with proc_stream.synchronize() 2. clear context cache, with pycuda.tools.clear_context_caches() 3. change the compute mode, with cuda.compute_mode = cuda.compute_mode.EXCLUSIVE **Note:** The solution 2. seems to free some memory, but it makes the computation way slower, and does not solve the problem: e.g. increasing the number of ffts to be computed, the script shows the same behaviour. Here the code. To start from a simple task, here each process computes 1 FFT (then one can use batch option in execute() to do more FFTs in a row). import multiprocessing import pycuda.driver as cuda import pycuda.gpuarray as gpuarray from pycuda.tools import make_default_context from pyfft.cuda import Plan def main(): # generates simple matrix, (e.g. image with a signal at the center) size = 4096 center = size/2 in_matrix = np.zeros((size, size), dtype='complex64') in_matrix[center:center+2, center:center+2] = 10. pool_size = 4 # integer up to multiprocessing.cpu_count() pool = multiprocessing.Pool(processes=pool_size) func = FuncWrapper(in_matrix, size) nffts = 16 # total number of ffts to be computed par = np.arange(nffts) results = pool.map(func, par) pool.close() pool.join() print results And here the function wrapper: class FuncWrapper(object): def __init__(self, matrix, size): self.in_matrix = matrix self.size = size print("Func initialized with matrix size=%i" % size) def __call__(self, par): proc_id = multiprocessing.current_process().name # take control over the GPU cuda.init() context = make_default_context() device = context.get_device() proc_stream = cuda.Stream() # move data to GPU # multiplication self.in_matrix*par is just to have each process computing # different matrices in_map_gpu = gpuarray.to_gpu(self.in_matrix*par) # create Plan, execute FFT and get back the result from GPU plan = Plan((self.size, self.size), dtype=np.complex64, fast_math=False, normalize=False, wait_for_finish=True, stream=proc_stream) plan.execute(in_map_gpu, wait_for_finish=True) result = in_map_gpu.get() # free memory on GPU del in_map_gpu mem = np.array(cuda.mem_get_info())/1.e6 print("%s free=%f\ttot=%f" % (proc_id, mem[0], mem[1])) # release context context.pop() return par Now, with nffts=16 and pool_size=4 the script terminates correctly and gives this output: Func initialized with matrix size=4096 PoolWorker-1 free=1481.019392 tot=2147.024896 PoolWorker-2 free=1331.011584 tot=2147.024896 PoolWorker-3 free=1181.003776 tot=2147.024896 PoolWorker-4 free=1030.631424 tot=2147.024896 PoolWorker-1 free=881.074176 tot=2147.024896 PoolWorker-2 free=731.746304 tot=2147.024896 PoolWorker-3 free=582.418432 tot=2147.024896 PoolWorker-4 free=433.090560 tot=2147.024896 PoolWorker-1 free=582.754304 tot=2147.024896 PoolWorker-2 free=718.946304 tot=2147.024896 PoolWorker-3 free=881.254400 tot=2147.024896 PoolWorker-4 free=1030.684672 tot=2147.024896 PoolWorker-1 free=868.028416 tot=2147.024896 PoolWorker-2 free=731.713536 tot=2147.024896 PoolWorker-3 free=582.402048 tot=2147.024896 PoolWorker-4 free=433.090560 tot=2147.024896 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] But with nffts=18 and pool_size=4 the script does not terminate and gives this output, remaining stuck at the last line: Func initialized with matrix size=4096 PoolWorker-1 free=1416.392704 tot=2147.024896 PoolWorker-2 free=982.544384 tot=2147.024896 PoolWorker-1 free=1101.037568 tot=2147.024896 PoolWorker-2 free=682.991616 tot=2147.024896 PoolWorker-3 free=815.747072 tot=2147.024896 PoolWorker-4 free=396.918784 tot=2147.024896 PoolWorker-3 free=503.046144 tot=2147.024896 PoolWorker-4 free=397.144064 tot=2147.024896 PoolWorker-1 free=531.361792 tot=2147.024896 PoolWorker-1 free=397.246464 tot=2147.024896 PoolWorker-2 free=518.610944 tot=2147.024896 PoolWorker-2 free=397.021184 tot=2147.024896 PoolWorker-3 free=517.193728 tot=2147.024896 PoolWorker-4 free=397.021184 tot=2147.024896 PoolWorker-3 free=504.336384 tot=2147.024896 PoolWorker-4 free=149.123072 tot=2147.024896 PoolWorker-1 free=283.340800 tot=2147.024896 ...on hold... Many thanks for your help! -- View this message in context: http://pycuda.2962900.n2.nabble.com/Compute-several-FFT-with-GPU-using-Pyth… Sent from the PyCuda mailing list archive at Nabble.com.

6 years, 6 months

Compute several FFT with GPU using Python multiprocessing and pyfft: how to avoid GPU memory leak?

by Marco Tazzari

I am trying to implement in Python the following pattern for *multi-CPU and single-GPU* computation using *pycuda* and *pyfft* packages.I would like to have *several processes* (e.g. launched with multiprocessing.Pool()), with *each of them* able to perform *FFTs using the GPU (using NVIDIA CUDA)*.However, I have the following problem:If I run too many processes or too many FFTs per process, *the overall script remains on hold without terminating* (and without computing all the FFTs that are due). From further investigations I suppose this is due to the *memory limit* on the GPU (currently 2048MB on NVIDIA GeForce GT 750M). It seems that the multiprocessing pool is not able to acquire the control back.Is there any way to avoid this?Since each process requires less than 2048 MB, I would like to have something like a *queue* where each process can /book/ the usage of the GPU and, when a process releases the context, the next process in the queue starts using it.Is this doable?Alternatively, is it possible to force that only one process uses the GPU at a given time? I have tried separately these solutions but they do not work (or probably I have not implemented them correctly): 1. synchronize the stream, with proc_stream.synchronize() 2. clear context cache, with pycuda.tools.clear_context_caches() 3. change the compute mode, with cuda.compute_mode = cuda.compute_mode.EXCLUSIVE*Note:* The solution 2. seems to free some memory, but it makes the computation way slower, and does not solve the problem: e.g. increasing the number of ffts to be computed, the script shows the same behaviour.Here the code. To start from a simple task, here each process computes 1 FFT (then one can use batch option in execute() to do more FFTs in a row). import multiprocessing import pycuda.driver as cuda import pycuda.gpuarray as gpuarray from pycuda.tools import make_default_context from pyfft.cuda import Plan def main(): # generates simple matrix, (e.g. image with a signal at the center) size = 4096 center = size/2 in_matrix = np.zeros((size, size), dtype='complex64') in_matrix[center:center+2, center:center+2] = 10. pool_size = 4 # integer up to multiprocessing.cpu_count() pool = multiprocessing.Pool(processes=pool_size) func = FuncWrapper(in_matrix, size) nffts = 16 # total number of ffts to be computed par = np.arange(nffts) results = pool.map(func, par) pool.close() pool.join() print resultsAnd here the function wrapper: class FuncWrapper(object): def __init__(self, matrix, size): self.in_matrix = matrix self.size = size print("Func initialized with matrix size=%i" % size) def __call__(self, par): proc_id = multiprocessing.current_process().name # take control over the GPU cuda.init() context = make_default_context() device = context.get_device() proc_stream = cuda.Stream() # move data to GPU # multiplication self.in_matrix*par is just to have each process computing # different matrices in_map_gpu = gpuarray.to_gpu(self.in_matrix*par) # create Plan, execute FFT and get back the result from GPU plan = Plan((self.size, self.size), dtype=np.complex64, fast_math=False, normalize=False, wait_for_finish=True, stream=proc_stream) plan.execute(in_map_gpu, wait_for_finish=True) result = in_map_gpu.get() # free memory on GPU del in_map_gpu mem = np.array(cuda.mem_get_info())/1.e6 print("%s free=%f\ttot=%f" % (proc_id, mem[0], mem[1])) # release context context.pop() return parNow, with nffts=16 and pool_size=4 the script terminates correctly and gives this output: Func initialized with matrix size=4096 PoolWorker-1 free=1481.019392 tot=2147.024896 PoolWorker-2 free=1331.011584 tot=2147.024896 PoolWorker-3 free=1181.003776 tot=2147.024896 PoolWorker-4 free=1030.631424 tot=2147.024896 PoolWorker-1 free=881.074176 tot=2147.024896 PoolWorker-2 free=731.746304 tot=2147.024896 PoolWorker-3 free=582.418432 tot=2147.024896 PoolWorker-4 free=433.090560 tot=2147.024896 PoolWorker-1 free=582.754304 tot=2147.024896 PoolWorker-2 free=718.946304 tot=2147.024896 PoolWorker-3 free=881.254400 tot=2147.024896 PoolWorker-4 free=1030.684672 tot=2147.024896 PoolWorker-1 free=868.028416 tot=2147.024896 PoolWorker-2 free=731.713536 tot=2147.024896 PoolWorker-3 free=582.402048 tot=2147.024896 PoolWorker-4 free=433.090560 tot=2147.024896 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] But with nffts=18 and pool_size=4 the script does not terminate and gives this output, remaining stuck at the last line: Func initialized with matrix size=4096 PoolWorker-1 free=1416.392704 tot=2147.024896 PoolWorker-2 free=982.544384 tot=2147.024896 PoolWorker-1 free=1101.037568 tot=2147.024896 PoolWorker-2 free=682.991616 tot=2147.024896 PoolWorker-3 free=815.747072 tot=2147.024896 PoolWorker-4 free=396.918784 tot=2147.024896 PoolWorker-3 free=503.046144 tot=2147.024896 PoolWorker-4 free=397.144064 tot=2147.024896 PoolWorker-1 free=531.361792 tot=2147.024896 PoolWorker-1 free=397.246464 tot=2147.024896 PoolWorker-2 free=518.610944 tot=2147.024896 PoolWorker-2 free=397.021184 tot=2147.024896 PoolWorker-3 free=517.193728 tot=2147.024896 PoolWorker-4 free=397.021184 tot=2147.024896 PoolWorker-3 free=504.336384 tot=2147.024896 PoolWorker-4 free=149.123072 tot=2147.024896 PoolWorker-1 free=283.340800 tot=2147.024896Many thanks for your help! -- View this message in context: http://pycuda.2962900.n2.nabble.com/Compute-several-FFT-with-GPU-using-Pyth… Sent from the PyCuda mailing list archive at Nabble.com.

6 years, 6 months

Re: [PyCUDA] (no subject)

by Andreas Kloeckner

"David A. Markowitz" <david.a.markowitz(a)gmail.com> writes: > Many thanks Andreas, I've solved the problem now. While digging through the > compiler.py code, I noticed a check for the PYCUDA_DEFAULT_NVCC_FLAGS > environment variable, which is then passed to nvcc. Ultimately I was able > to solve my problem by putting a file in /etc/profile.d/ with the contents: > > export PYCUDA_DEFAULT_NVCC_FLAGS="--dont-use-profile > -ldir=/usr/local/cuda/nvvm/libdevice" > > This offers a simple but effective way for PyCUDA to point nvcc in the > right direction. > > I'm still not sure why I wasn't able to fix the "libdevice library not > found" error by modifying nvcc.profile directly, but the above is probably > a better solution regardless, because it doesn't require modifications to a > file that affects nvcc's operation in all use cases (i.e. even when it's > not called by PyCUDA). Since I've only encountered this problem using > PyCUDA, it makes sense that the solution should only kick in when nvcc is > called by PyCUDA. > > I'm really looking forward to using this software package. Thank you for > all of your hard work putting it together! Glad to hear you got things to work! Andreas

6 years, 6 months

Re: [PyCUDA] (no subject)

by Andreas Kloeckner

"David A. Markowitz" <david.a.markowitz(a)gmail.com> writes: > Thanks again, Andreas. I'm really looking forward to getting started with > PyCUDA. > > Unfortunately, I've already tried your suggested approach (updating > nvcc.profile with NVVMIR_LIBRARY_DIR = /usr/local/cuda-6.5/nvvm/libdevice, > which contains libdevice.compute_35.10.bc in my installation), but it did > not solve the problem. When I asked NVIDIA about this directly (hoping not > to bother you further), they told me I should never need to modify > nvcc.profile under any circumstances, which wasn't very helpful. > > Is there some other line in nvcc.profile that I can modify so that nvcc > will be able to find the appropriate libdevice library when called by > PyCUDA's compiler script? e.g. perhaps I could append something to the > INCLUDES or LIBRARIES variables? > > Is there an easy way for me to see which environment variables are > available when PyCUDA's compiler.py code calls nvcc? This might help > diagnose the problem. The call happens here: https://github.com/inducer/pycuda/blob/master/pycuda/compiler.py#L114 using this: https://github.com/inducer/pytools/blob/master/pytools/prefork.py#L34 https://docs.python.org/2.7/library/subprocess.html says that processes should inherit the parent's environment unless otherwise specified (and it isn't), so it's not clear to me what would override your variable... You can also use the 'keep=True' flag and try and run nvcc yourself in the temp directory that PyCUDA creates. That's perhaps the best way of figuring out what's up. Andreas

6 years, 6 months

Re: [PyCUDA] (no subject)

by Andreas Kloeckner

"David A. Markowitz" <david.a.markowitz(a)gmail.com> writes: > Hi, thanks for the quick reply (and good advice!). I wiped my cuda 6.5 > installation and reinstalled from scratch. nvcc now works when called from > the command line on simple CUDA samples. It compiles for my GPU's > architecture (3.5) by default, so PATH and LD_LIBRARY_PATH are definitely > configured correctly. > > I also wiped PyCUDA and reinstalled from scratch, per the instructions for > Ubuntu 14.04 64 bit on the PyCUDA Installation page. No errors during this > process, and I can successfully import pycuda.autoinit. However, now when I > try to run any of the PyCUDA examples, I get the following error: > > nvcc fatal : Path to libdevice library not specified > > Since I do not encounter this error when compiling CUDA samples or my own > CUDA code with nvcc, my guess is that my environment variables aren't being > seen by PyCUDA. I googled this error and found a few threads on the > subject, but no effective solutions. > > I was wondering if I could trouble this list for a pointer or two? > Hopefully there's a quick fix. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=725649 suggests that you may be able to tweak /etc/nvcc.profile. HTH, Andreas

6 years, 6 months

(no subject)

by David A. Markowitz

Hi, thanks for the quick reply (and good advice!). I wiped my cuda 6.5 installation and reinstalled from scratch. nvcc now works when called from the command line on simple CUDA samples. It compiles for my GPU's architecture (3.5) by default, so PATH and LD_LIBRARY_PATH are definitely configured correctly. I also wiped PyCUDA and reinstalled from scratch, per the instructions for Ubuntu 14.04 64 bit on the PyCUDA Installation page. No errors during this process, and I can successfully import pycuda.autoinit. However, now when I try to run any of the PyCUDA examples, I get the following error: nvcc fatal : Path to libdevice library not specified Since I do not encounter this error when compiling CUDA samples or my own CUDA code with nvcc, my guess is that my environment variables aren't being seen by PyCUDA. I googled this error and found a few threads on the subject, but no effective solutions. I was wondering if I could trouble this list for a pointer or two? Hopefully there's a quick fix. Many thanks, -David On Sat, Jan 17, 2015 at 2:41 PM, Andreas Kloeckner <lists(a)informa.tiker.net> wrote: > "David A. Markowitz" <david.a.markowitz(a)gmail.com> writes: > > > Hi, I just installed PyCUDA, but test_driver.py crashes with the > > following error: > > > > CompileError: nvcc compilation of /tmp/tmpNht4bp/kernel.cu failed > > [command: nvcc --cubin -arch sm_35 > > > -I/usr/local/lib/python2.7/dist-packages/pycuda-2014.1-py2.7-linux-x86_64.egg/pycuda/cuda > > kernel.cu] > > [stderr: error in open: > > /usr/bin/../nvvm/libdevice/libdevice.compute_35.10.bc No such > > file or directory ] > > > This looks like nvcc is unable to find its own parts. Does nvcc work > when called from the command line on a simple CUDA sample? (My guess is > no.) > > Andreas >

6 years, 6 months

Re: [PyCUDA] Error running PyCUDA Tests

by Andreas Kloeckner

"David A. Markowitz" <david.a.markowitz(a)gmail.com> writes: > Hi, I just installed PyCUDA, but test_driver.py crashes with the > following error: > > CompileError: nvcc compilation of /tmp/tmpNht4bp/kernel.cu failed > [command: nvcc --cubin -arch sm_35 > -I/usr/local/lib/python2.7/dist-packages/pycuda-2014.1-py2.7-linux-x86_64.egg/pycuda/cuda > kernel.cu] > [stderr: error in open: > /usr/bin/../nvvm/libdevice/libdevice.compute_35.10.bc No such > file or directory ] This looks like nvcc is unable to find its own parts. Does nvcc work when called from the command line on a simple CUDA sample? (My guess is no.) Andreas

6 years, 6 months

Re: [PyCUDA] NPP library with PyCUDA

by Lev Givon

Received from Shivang Ghetia on Fri, Jan 16, 2015 at 01:23:18PM EST: > Hi PyCUDA community, > > I want to know whether PyCUDA support NPP library as I want to use some > functions from NPP library. I went through documentation of CULA and > scikits.cuda > provides way to accss libraries like cuFFT and cuBLAS. I am not able to > find any way to access NPP libray. > > Regards, > Shivang. scikits.cuda (which is developed independently of PyCUDA) doesn't provide wrappers for the NPP functions yet, although I would be happy to integrate support for them if someone writes wrappers for the functions. -- Lev Givon Bionet Group | Neurokernel Project http://www.columbia.edu/~lev/ http://lebedov.github.io/ http://neurokernel.github.io/

6 years, 6 months

NPP library with PyCUDA

by Shivang Ghetia

Hi PyCUDA community, I want to know whether PyCUDA support NPP library as I want to use some functions from NPP library. I went through documentation of CULA and scikits.cuda provides way to accss libraries like cuFFT and cuBLAS. I am not able to find any way to access NPP libray. Regards, Shivang.

6 years, 6 months

Error running PyCUDA Tests

by David A. Markowitz

Hi, I just installed PyCUDA, but test_driver.py crashes with the following error: CompileError: nvcc compilation of /tmp/tmpNht4bp/kernel.cu failed [command: nvcc --cubin -arch sm_35 -I/usr/local/lib/python2.7/dist-packages/pycuda-2014.1-py2.7-linux-x86_64.egg/pycuda/cuda kernel.cu] [stderr: error in open: /usr/bin/../nvvm/libdevice/libdevice.compute_35.10.bc No such file or directory ] Note the last line- it seems to think my CUDA_ROOT is "/usr/bin/..", which is clearly incorrect! All of my CUDA environment variables are set properly (e.g. I'm able to compile+run the CUDA samples and my own CUDA code without problems), so my guess is that PyCUDA ignored one or more of these variables during installation. I've tried forcing PyCUDA to use the correct CUDA environment variables during installation, as follows: sudo ./configure.py --cuda-root=/usr/local/cuda --cudadrv-lib-dir=/usr/local/cuda/lib64 However, I still encounter the same error as above. I would be grateful for any tips for how to address this problem. Happy to provide more information if that would help. Thanks, -David

6 years, 6 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

PyCUDA January 2015