python - Thread error: can't start new thread -

here's mwe of larger code i'm using. performs monte carlo integration on kde (kernel density estimate) values located below threshold (the integration method suggested on @ question: integrate 2d kernel density estimate) iteratively number of points in list , returns list made of these results.

import numpy np scipy import stats multiprocessing import pool import threading  # define kde integration function. def kde_integration(m_list):      # put of values m_list 2 new lists.     m1, m2 = [], []     item in m_list:         # x data.         m1.append(item[0])         # y data.         m2.append(item[1])      # define limits.     xmin, xmax = min(m1), max(m1)     ymin, ymax = min(m2), max(m2)      # perform kernel density estimate on data:     x, y = np.mgrid[xmin:xmax:100j, ymin:ymax:100j]     values = np.vstack([m1, m2])     kernel = stats.gaussian_kde(values)      # list returned @ end of function.     out_list = []      # iterate through points in list , calculate each integral     # of kde domain of points located below value of point     # in kde.     point in m_list:          # compute point below integrate.         iso = kernel((point[0], point[1]))          # sample kde distribution         sample = kernel.resample(size=1000)          #choose number of cores , split input array.         cores = 4         torun = np.array_split(sample, cores, axis=1)          # print number of active threads.         print threading.active_count()          #calculate         pool = pool(processes=cores)         results = pool.map(kernel, torun)          #reintegrate , calculate results         insample_mp = np.concatenate(results) < iso          # integrate values below iso.         integral = insample_mp.sum() / float(insample_mp.shape[0])          # append integral value point list return.         out_list.append(integral)      return out_list   # generate random two-dimensional data: def measure(n):     "measurement model, return 2 coupled measurements."     m1 = np.random.normal(size=n)     m2 = np.random.normal(scale=0.5, size=n)     return m1+m2, m1-m2  # create list pass kde integral function. m_list = [] in range(100):     m1, m2 = measure(5)     m_list.append(m1.tolist())     m_list.append(m2.tolist())  # call kde integration function. print 'integral result: ', kde_integration(m_list) 

the multiprocessing in code suggested on @ question speed sampling of kernel estimate speed code (which ~3.4x).

the code works ok until try pass kde function list of more ~62-63 elements (ie: set value on 63 in line for in range(100)) if following error:

traceback (most recent call last):   file "~/gauss_kde_temp.py", line 78, in <module>     print 'integral result: ', kde_integration(m_list)   file "~/gauss_kde_temp.py", line 48, in kde_integration     pool = pool(processes=cores)   file "/usr/lib/python2.7/multiprocessing/__init__.py", line 232, in pool     return pool(processes, initializer, initargs, maxtasksperchild)   file "/usr/lib/python2.7/multiprocessing/pool.py", line 144, in __init__     self._worker_handler.start()   file "/usr/lib/python2.7/threading.py", line 494, in start     _start_new_thread(self.__bootstrap, ()) thread.error: can't start new thread 

usually (9 out of 10 times) around active thread 374. i'm way out of league in terms of python coding here , have no clue how fix issue. appreciated.

---

add

i tried adding while loop prevent code using many threads. did replacing print threading.active_count() line bit of code:

    # print number of active threads.     exit_loop = true     while exit_loop:         if threading.active_count() < 300:             exit_loop = false         else:             # pause 10 seconds.             time.sleep(10.)             print 'waiting: ', threading.active_count() 

the code halted (ie: got stuck inside loop) when reached 302 active threads. waited more 10 minutes , code never exited loop , number of active threads never dropped 302. shouldn't number of active threads diminish after while?