a ƹDf87@sdZdaddZddZGdddeZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZGdddeZdS)a This module contains functions for profiling in other threads and processes. Functions for identifying a thread/process: process_id - get the identifier (process id) for the current process thread_id - get the identifier for the current thread Functions for controlling profiling: enable_profiling - initialize a profiler in the current thread/process start_profiling - begin profiling everything in the current thread/process stop_profiling - stop profiling everything in the current thread/process disable_profiling - remove the profiler from the current thread/process Functions that control profile statstics (pstats) output clear_stats - clear stored pstats from the current thread/process get_stats - get stored pstats for the current thread/process print_stats - print stored pstats for the current thread/process dump_stats - dump stored pstats for the current thread/process Functions that add/remove profiling: profiled - decorator to add profiling to a function not_profiled - decorator to remove profiling from a function profile - decorator for profiling a function (will enable_profiling) Usage ===== Typical calls to pathos profiling will roughly follow this example:: >>> import time >>> import random >>> import pathos.profile as pr >>> >>> # build a worker function >>> def _work(i): ... x = random.random() ... time.sleep(x) ... return (i,x) >>> >>> # generate a 'profiled' work function >>> config = dict(gen=pr.process_id) >>> work = pr.profiled(**config)(_work) >>> >>> # enable profiling >>> pr.enable_profiling() >>> >>> # profile the work (not the map internals) in the main process >>> for i in map(work, range(-10,0)): ... print(i) ... >>> # profile the map in the main process, and work in the other process >>> from pathos.helpers import mp >>> pool = mp.Pool(10) >>> _uimap = pr.profiled(**config)(pool.imap_unordered) >>> for i in _uimap(work, range(-10,0)): ... print(i) ... >>> # deactivate all profiling >>> pr.disable_profiling() # in the main process >>> tuple(_uimap(pr.disable_profiling, range(10))) # in the workers >>> for i in _uimap(work, range(-20,-10)): ... print(i) ... >>> # re-activate profiling >>> pr.enable_profiling() >>> >>> # print stats for profile of 'import math' in another process >>> def test_import(module): ... __import__(module) ... >>> import pathos.pools as pp >>> pool = pp.ProcessPool(1) >>> pr.profile('cumulative', pipe=pool.pipe)(test_import, 'pox') 10 function calls in 0.003 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.003 0.003 :1(test_import) 1 0.002 0.002 0.003 0.003 {__import__} 1 0.001 0.001 0.001 0.001 __init__.py:8() 1 0.000 0.000 0.000 0.000 shutils.py:11() 1 0.000 0.000 0.000 0.000 _disk.py:15() 1 0.000 0.000 0.000 0.000 {eval} 1 0.000 0.000 0.000 0.000 utils.py:11() 1 0.000 0.000 0.000 0.000 :1() 1 0.000 0.000 0.000 0.000 info.py:2() 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} >>> pool.close() >>> pool.join() >>> pool.clear() Notes ===== This module leverages the python's cProfile module, and is primarily a high-level interface to that module that strives to make profiling in a different thread or process easier. The use of pathos.pools are suggested, however are not required (as seen in the example above). In many cases, profiling in another thread is not necessary, and either of the following can be sufficient/better for timing and profiling:: $ python -c "import time; s=time.time(); import pathos; print (time.time()-s)" $ python -c "import cProfile; p=cProfile.Profile(); p.enable(); import pathos; p.print_stats('cumulative')" This module was inspired by: http://stackoverflow.com/a/32522579/4646678. NcCsddlm}|jS)z7get the identifier (process id) for the current processmp)Zpathos.helpersrcurrent_processpidrr[/nfs/NAS7/SABIOD/METHODE/ermites/ermites_venv/lib/python3.9/site-packages/pathos/profile.py process_idzs rcCsddl}|jS)z)get the identifier for the current threadrN) threadingcurrent_threadident)thrrr thread_idsr c@s"eZdZdZd ddZddZdS) profiledzCdecorator for profiling a function (does not call enable profiling)Nid-.profcCsV||_||_t|ttfvr:t|tur,dn||_t|_nd|_|durLtn||_dS)ay=gen(), with y an indentifier (e.g. current_process().pid) Important class members: prefix - string prefix for pstats filename [default: 'id-'] suffix - string suffix for pstats filename [default: '.prof'] pid - function for obtaining id of current process/thread sort - integer index of column in pstats output for sorting Example: >>> import time >>> import random >>> import pathos.profile as pr >>> >>> config = dict(gen=pr.process_id) >>> @pr.profiled(**config) ... def work(i): ... x = random.random() ... time.sleep(x) ... return (i,x) ... >>> pr.enable_profiling() >>> # profile the work (not the map internals); write to file for pstats >>> for i in map(work, range(-10,0)): ... print(i) ... NOTE: If gen is a bool or string, then sort=gen and pid is not used. Otherwise, pid=gen and sort is not used. Output can be ordered by setting gen to one of the following: 'calls' - call count 'cumulative' - cumulative time 'cumtime' - cumulative time 'file' - file name 'filename' - file name 'module' - file name 'ncalls' - call count 'pcalls' - primitive call count 'line' - line number 'name' - function name 'nfl' - name/file/line 'stdname' - standard name 'time' - internal time 'tottime' - internal time N)prefixsuffixtypeboolstrsortrr)selfgenrrrrr__init__s-zprofiled.__init__csfdd}|_|S)Ncsztd}Wn*ty&d}Ynty:d}Yn0|i|}|rtjturntjnt dj j f|S)NTF%s%s%s) profilerenableAttributeError NameErrordisablerr print_statsr dump_statsrr)argskwdsZdoitresfrrr proactivesz$profiled.__call__..proactive) __wrapped__)rr'r(rr&r__call__s zprofiled.__call__)Nrr__name__ __module__ __qualname____doc__rr*rrrrrs 6rcs2t|dddkrt|d|n|fdd}|S)zAdecorator to remove profiling (due to 'profiled') from a functionr,Nr(r)cs|i|SNr)r#r$_frrwrappersznot_profiled..wrapper)getattr)r'r3rr1r not_profileds  r5cGsddl}|adS)z>> import time >>> import random >>> import pathos.profile as pr >>> ... def work(): ... x = random.random() ... time.sleep(x) ... return x ... >>> # profile the work; print pstats info >>> pr.profile()(work) 4 function calls in 0.136 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.136 0.136 :1(work) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 {method 'random' of '_random.Random' objects} 1 0.136 0.136 0.136 0.136 {time.sleep} 0.1350568110491419 >>> NOTE: pipe provided should come from pool built with nodes=1. Other configuration keywords (config) are passed to 'pr.profiled'. Output can be ordered by setting sort to one of the following: 'calls' - call count 'cumulative' - cumulative time 'cumtime' - cumulative time 'file' - file name 'filename' - file name 'module' - file name 'ncalls' - call count 'pcalls' - primitive call count 'line' - line number 'name' - function name 'nfl' - name/file/line 'stdname' - standard name 'time' - internal time 'tottime' - internal time pipeN)rFr) SerialPool__self__z-pipe must draw from a pool with only one node) r<rrrCrBrDZ pathos.poolsrH_poolrGr4nodes ValueError)rrrDrGrHrrrrs1    zprofile.__init__cOsD|td|jtfi|j|g|Ri|}|td|Sr0)rGr7rrDr=)rfunctionr#r$resultrrrr*]s ( zprofile.__call__)Nr+rrrrrFs ?rF)r/rrr objectrr5r7r9r:r=r?r@r!r"rFrrrrspH I