Analysis of Custom Process Pool Instance in Python [Producer and Consumer Model Issues]

This article analyzes the Python custom process pool. Shared for everyone's reference, as follows:

Code speaks for itself:

#encoding=utf-8
#author: walker
#date: 2014-05-21
# function: Custom process pool to traverse files under directory
from multiprocessing import Process, Queue, Lock
import time, os
# Consumer
class Consumer(Process):
  def __init__(self, queue, ioLock):
    super(Consumer, self).__init__()
    self.queue = queue
    self.ioLock = ioLock
  def run(self):
    while True:
      task = self.queue.get()  # Will block the process if there is no task in the queue
      if isinstance(task, str) and task == 'quit':
        break;
      time.sleep(1)  # Assuming task processing requires1seconds
      self.ioLock.acquire()
      print(str(os.getpid()) + ' + task)
      self.ioLock.release()
    self.ioLock.acquire()
    print 'Bye-bye'
    self.ioLock.release()
# Producer
def Producer():
  queue = Queue()  # This queue is for processes/thread-safe
  ioLock = Lock()
  subNum = 4  #子进程数量
  workers = build_worker_pool(queue, ioLock, subNum)
  start_time = time.time()
  for parent, dirnames, filenames in os.walk(r'D:\test'):
    for filename in filenames:
      queue.put(filename)
      ioLock.acquire()
      print('qsize:') + str(queue.qsize())
      ioLock.release()
      while queue.qsize() > subNum * 10: #Control the number of tasks in the queue
        time.sleep(1)
  for worker in workers:
    queue.put('quit')
  for worker in workers:
    worker.join()
  ioLock.acquire()
  print('Done! Time taken: {}'.format(time.time() - start_time))
  ioLock.release()
#Create process pool
def build_worker_pool(queue, ioLock, size):
  workers = []
  for _ in range(size):
    worker = Consumer(queue, ioLock)
    worker.start()
    workers.append(worker)
  return workers
if __name__ == '__main__':
  Producer()

ps:

self.ioLock.acquire()
...
self.ioLock.release()

Available:

with self.ioLock:
  ...

replacement.

Here is another fun example:

#encoding=utf-8
#author: walker
#date: 2016-01-06
#function: An example of a multi-process fun
import os, sys, time
from multiprocessing import Pool
cur_dir_fullpath = os.path.dirname(os.path.abspath(__file__))
g_List = ['a']
#修改全局变量g_List
def ModifyDict_1():
  global g_List
  g_List.append('b')
#修改全局变量g_List
def ModifyDict_2():
  global g_List
  g_List.append('c')
#处理一个
def ProcOne(num):
  print('ProcOne ' + str(num) + ', g_List:' + repr(g_List))
#处理所有
def ProcAll():
  pool = Pool(processes = 4)
  for i in range(1, 20):
    #ProcOne(i)
    #pool.apply(ProcOne, (i,))
    pool.apply_async(ProcOne, (i,))
  pool.close()
  pool.join()
ModifyDict_1() # Modify global variable g_List
if __name__ == '__main__':
  ModifyDict_2() # Modify global variable g_List
  print('In main g_List :' + repr(g_List))
  ProcAll()

Windows7 The result of running below:

λ python3 demo.py
In main g_List :['a', 'b', 'c']
ProcOne 1, g_List:['a', 'b']
ProcOne 2, g_List:['a', 'b']
ProcOne 3, g_List:['a', 'b']
ProcOne 4, g_List:['a', 'b']
ProcOne 5, g_List:['a', 'b']
ProcOne 6, g_List:['a', 'b']
ProcOne 7, g_List:['a', 'b']
ProcOne 8, g_List:['a', 'b']
ProcOne 9, g_List:['a', 'b']
ProcOne 10, g_List:['a', 'b']
ProcOne 11, g_List:['a', 'b']
ProcOne 12, g_List:['a', 'b']
ProcOne 13, g_List:['a', 'b']
ProcOne 14, g_List:['a', 'b']
ProcOne 15, g_List:['a', 'b']
ProcOne 16, g_List:['a', 'b']
ProcOne 17, g_List:['a', 'b']
ProcOne 18, g_List:['a', 'b']
ProcOne 19, g_List:['a', 'b']

Ubuntu 14.04The result of running below:

In main g_List :['a', 'b', 'c']
ProcOne 1, g_List:['a', 'b', 'c']
ProcOne 2, g_List:['a', 'b', 'c']
ProcOne 3, g_List:['a', 'b', 'c']
ProcOne 5, g_List:['a', 'b', 'c']
ProcOne 4, g_List:['a', 'b', 'c']
ProcOne 8, g_List:['a', 'b', 'c']
ProcOne 9, g_List:['a', 'b', 'c']
ProcOne 7, g_List:['a', 'b', 'c']
ProcOne 11, g_List:['a', 'b', 'c']
ProcOne 6, g_List:['a', 'b', 'c']
ProcOne 12, g_List:['a', 'b', 'c']
ProcOne 13, g_List:['a', 'b', 'c']
ProcOne 10, g_List:['a', 'b', 'c']
ProcOne 14, g_List:['a', 'b', 'c']
ProcOne 15, g_List:['a', 'b', 'c']
ProcOne 16, g_List:['a', 'b', 'c']
ProcOne 17, g_List:['a', 'b', 'c']
ProcOne 18, g_List:['a', 'b', 'c']
ProcOne 19, g_List:['a', 'b', 'c']

You can see Windows7The second modification did not succeed, but it succeeded under Ubuntu. According to the author limodou of uliweb, the reason is that under Windows, it is implemented by a child process after reboot; under Linux, it is implemented by fork.

Readers who are interested in more about Python-related content can check the special topics on this site: 'Summary of Python URL Operation Skills', 'Summary of Python Image Operation Skills', 'Python Data Structures and Algorithms Tutorial', 'Summary of Python Socket Programming Skills', 'Summary of Python Function Usage Skills', 'Summary of Python String Operation Skills', 'Classic Tutorial of Python入门与进阶', and 'Summary of Python File and Directory Operation Skills'.

I hope the content described in this article will be helpful to everyone in Python program design.

Statement: The content of this article is from the Internet, and the copyright belongs to the original author. The content is contributed and uploaded by Internet users spontaneously. This website does not own the copyright, has not been manually edited, and does not assume relevant legal liability. If you find any content suspected of copyright infringement, please send an email to: notice#oldtoolbag.com (Please replace # with @ when sending an email for reporting. Provide relevant evidence, and once verified, this site will immediately delete the content suspected of infringement.)