From 6e1b3773c5e1dda6cc71797caa533a6d39ed97bf Mon Sep 17 00:00:00 2001 From: Jamesgo1 Date: Fri, 29 Sep 2023 15:59:49 +0100 Subject: [PATCH] Add explanation of chunksize for map method in multiprocessing.rst --- Doc/library/multiprocessing.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 38d24a86072..6aee9d2fc71 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -2287,6 +2287,16 @@ with the :class:`Pool` class. the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting *chunksize* to a positive integer. + It is important to distinguish between *maxtasksperchild* in the + :class:`Pool` class and *chunksize*. Whereas *maxtasksperchild* denotes + how many tasks can be run at once before those resources are freed, + *chunksize* defines how many separate chunks in which to split all of the + iterations of the function. In order for each iteration of the function to + be executed separately, *chunksize* needs to be set to 1. + + Whereas :meth:`imap` defaults to a *chunksize* of 1, :meth:`map` defaults + to ``None`` and needs to be declared explicitly. + Note that it may cause high memory usage for very long iterables. Consider using :meth:`imap` or :meth:`imap_unordered` with explicit *chunksize* option for better efficiency.