a ‘·Df¢‚ã@sVdZddlZddlZddlZddlZddlZddlZddlZddlZeZ z$ddl Z ddl m Z ddl mZWn*ey–ddl Z dd„Z dd„ZYn0d d lmZd d lmZd d lmZejZejZZd ZdZddlZdZGdd„deƒZGdd„deƒZ Gdd„dej!ƒZ"Gdd„deƒZ#Gdd„deƒZ$Gdd„deƒZ%Gdd„de&ƒZ'dS)z, ppft: the parallel python execution server éN)Ú importable)ÚgetnamecCs(t |¡d}|d ¡|d<d |¡S)NrÚ)ÚinspectÚgetsourcelinesÚlstripÚjoin)ÚfuncÚ sourcelines©r úU/nfs/NAS7/SABIOD/METHODE/ermites/ermites_venv/lib/python3.9/site-packages/ppft/_pp.pyr/srcCs|jS)N)Ú__name__)Úobjr r r r5sré)Ú transport)Úauto)ÚcommonéFTc@s<eZdZdZddd„Zdd„Zdd d „Zd d „Zdd„ZdS)Ú_Taskz'Class describing single task (job) Nr ÚdefaultcCsBt ¡|_|j ¡||_||_||_||_||_d|_ d|_ dS)zInitializes the taskFN) Ú threadingÚLockÚlockÚacquireÚtidÚserverÚcallbackÚ callbackargsÚgroupÚfinishedÚ unpickled)Úselfrrrrrr r r Ú__init__Ms  z_Task.__init__cCs(||_|jr| ¡|j ¡d|_dS)z5Finalizes the task. For internal use onlyTN)ÚsresultrÚ_Task__unpicklerÚreleaser)r!r#r r r ÚfinalizeZs  z_Task.finalizeFcCsD|js|jjrtdƒ‚| ¡|js0|s0| ¡|r:|jS|jSdS)zRetrieves result of the taskz.Server was destroyed before the job completionN) rrÚ_exitingÚDestroyedServerErrorÚwaitr r$r#Úresult)r!Z raw_resultr r r Ú__call__ds z_Task.__call__cCs|js|j ¡|j ¡dS)zWaits for the taskN)rrrr%©r!r r r r)rs z _Task.waitcCsXt t |j¡¡\|_}d|_t|ƒdkr6t|dd|j rT|j |jf}|j |ŽdS)z Unpickles the result of the taskTrú )ÚendN) ÚpickleÚloadsÚppcZb_r#r*r ÚlenÚprintrr)r!ZsoutÚargsr r r Z __unpicklexs  z_Task.__unpickle)Nr r)F) r Ú __module__Ú __qualname__Ú__doc__r"r&r+r)r$r r r r rIsÿ  rc@sPeZdZdZejdddgZe d¡dd„Zdd „Z d d „Z d d „Z dd„Z dS)Ú_WorkerzLocal worker class z-uz-mZppftz 2>/dev/nullcCs||_||_| ¡dS)zInitializes local workerN)Úrestart_on_freeÚ pickle_protoÚstart)r!r9r:r r r r"Šsz_Worker.__init__cCsXtj|jtjtjtjd}t |j|j¡|_t |j  ¡ƒ|_ |j  t |jƒ¡d|_dS)zStarts local worker)ÚstdinÚstdoutÚstderrTN)Ú subprocessÚPopenÚcommandÚPIPEÚ pptransportZCPipeTransportr=r<ÚtÚintÚreceiveÚpidÚsendÚstrr:Úis_free)r!Úprocr r r r;s ÿz _Worker.startcCs d|_|j d¡|j ¡dS)zStops local workerFZEXITN)rJrDrHÚcloser,r r r ÚstopŸs z _Worker.stopcCs| ¡| ¡dS)zRestarts local workerN)rMr;r,r r r Úrestart¦sz_Worker.restartcCs|jr| ¡nd|_dS)zFrees local workerTN)r9rNrJr,r r r Úfree«s z _Worker.freeN) r r5r6r7ÚsysÚ executablerAÚappendr"r;rMrNrOr r r r r8ƒs r8c@s*eZdZdZdd„Zdd„Zd dd„ZdS) Ú_RWorkerzRemote worker class cCsf||_||_||_||_||_||f|_|dt|ƒ|_|jj  d|j|f¡||_ |  |¡dS)zInitializes remote workerú:z$Creating Rworker id=%s persistent=%sN) rÚ persistentÚhostÚportÚsecretÚaddressrIÚidÚloggerÚdebugÚsocket_timeoutÚconnect)r!rVrWrXrÚmessagerUr]r r r r"·s  ÿz_RWorker.__init__cCs | ¡dS)z$Closes connection with remote serverN)rLr,r r r Ú__del__Åsz_RWorker.__del__Nc Csâ|jjsÞzhtj |d|j¡| |j|j¡|  |j ¡sX|jj   d|j|jf¡WdS|rf|  |¡d|_WdStrŽ|jj jddd|js°|jj  d|jf¡YdS|jj  d|j|jtf¡t t¡Yq0qdS) zConnects to a remote serverTNz*Authentication failed for host=%s, port=%sFz/Exception in connect method (possibly expected)©Úexc_infozDeleting from queue Rworker %szCFailed to reconnect with (host=%s, port=%i), will try again in %i s)rr'rCZSocketTransportr"r]Ú_connectrVrWZ authenticaterXr[ÚerrorrHrJÚSHOW_EXPECTED_EXCEPTIONSr\rUrZÚinfoÚRECONNECT_WAIT_TIMEÚtimeÚsleep)r!r_r r r r^És6   ÿ  ÿ ÿ  þz_RWorker.connect)N)r r5r6r7r"r`r^r r r r rS³srSc@seZdZdZddd„ZdS)Ú _Statisticsz9Class to hold execution statisitcs for a single node NcCs||_d|_d|_||_dS)z!Initializes statistics for a nodeçrN)ÚncpusrhÚnjobsÚrworker)r!rlrnr r r r"ísz_Statistics.__init__)N)r r5r6r7r"r r r r rjésrjc@s"eZdZdZd dd„Zdd„ZdS) ÚTemplatezTemplate class r Nrc Cs4||_||_||_||_||_||_||_||_dS)aùCreates Template instance jobs_server - pp server for submitting jobs func - function to be executed depfuncs - tuple with functions which might be called from 'func' modules - tuple with module names to import callback - function which will be called with argument list equal to callbackargs+(result,) as soon as calculation is done callbackargs - additional arguments for callback function group - job group, is used when wait(group) is called to wait for jobs in a given group to finish globals - dictionary from which all modules, functions and classes will be imported, for instance: globals=globals() N)Ú job_serverr ÚdepfuncsÚmodulesrrrÚglobals) r!rpr rqrrrrrrsr r r r"ùszTemplate.__init__c Gs(|j |j||j|j|j|j|j|j¡S)zDSubmits function with *arg arguments to the execution queue ) rpÚsubmitr rqrrrrrrs)r!r4r r r rts þzTemplate.submit)r r Nr rN)r r5r6r7r"rtr r r r roõs ÿ roc@sìeZdZdZdZdZd>d d „Zd?d d„Zd@dd„Zdd„Z dAdd„Z dd„Z dd„Z dd„Z dBdd„ZdCdd„Zd d!„Zd"d#„Zd$d%„Zd&d'„Zd(d)„Zd*d+„Zd,d-„Zd.d/„Zd0d1„Zd2d3„Zd4d5„Zd6d7„Zd8d9„Zd:d;„Zd| dkrÚd} |  6dd¡} |  6dd¡} |j. 5| | f| | ff¡qpt  ¡|_7|durBt|t8ƒs6tdƒ‚t8|ƒ|_9n>t:t;dƒrxt|_9| ?¡t@ @¡|_A|j d|jBf¡dS)a£Creates Server instance ncpus - the number of worker processes to start on the local computer, if parameter is omitted it will be set to the number of processors in the system ppservers - list of active parallel python execution servers to connect with secret - passphrase for network connections, if omitted a default passphrase will be used. It's highly recommended to use a custom passphrase for all network connections. restart - restart the worker process after each task completion proto - protocol number for pickle module socket_timeout - socket timeout in seconds, which is the maximum time a remote job could be executed. Increase this value if you have long running jobs or decrease if connectivity to remote ppservers is often lost. With ncpus = 1 all tasks are executed consequently. For the best performance either use the default "autodetect" value or set ncpus to the total number of processors in the system. z"ppservers argument must be a tupleÚppzCreating server instance (pp-ú)zRunning on Python %s %sr-rFTÚ PYTHONPATHcs"g|]}|ˆvrˆ |¡p|‘qSr )Úadd©Ú.0Úx©Zdirsetr r Ú Zóz#Server.__init__..ÚlocalrTrÚ*éÿÿÿÿz*.*.*.*Ú0Z255Nzsecret must be of a string typeZ pp_secretz'pp local server started with %d workers)CÚ isinstanceÚtupleÚ TypeErrorÚloggingÚ getLoggerr[rfÚversionrPÚsplitÚplatformÚ _Server__tidÚ_Server__active_tasksrrÚ_Server__active_tasks_lockÚ_Server__queueÚ_Server__queue_lockÚ_Server__workersÚ_Server__rworkersÚ_Server__rworkers_reservedÚ_Server__sourcesHMÚ_Server__sfuncHMÚ_Server__waittasksÚ_Server__waittasks_lockr'Ú_Server__accurate_statsÚ autopp_listÚ"_Server__active_rworkers_list_lockÚ_Server__restart_on_freeÚ_Server__pickle_protoÚ_Server__connect_locksÚosÚgetcwdÚpathÚenvironÚpathsepÚsetrÚatexitÚregisterÚdestroyrjÚ_Server__statsÚ set_ncpusÚ ppserversÚauto_ppserversr]r2rEr1Z randomportÚfindrRÚreplaceÚ_Server__stats_lockrIrXÚhasattrÚuserÚgetattrruÚdefault_secretÚ_Server__connectrhÚ_Server__creation_timeÚ_Server__ncpus) r!rlr¬rXrNÚprotor]Z pythondirsÚppserverrVrWZ interfaceÚ broadcastr r€r r"!sˆ  ÿ            ÿ          ÿzServer.__init__rc CsN|jrtdƒ‚t|tƒs tdƒ‚t|tƒs2tdƒ‚t|tƒsDtdƒ‚t|tƒsVtdƒ‚|durpt|tƒsptdƒ‚|D]} t| tƒsttdƒ‚qt| ¡} tj } |r|t|  d |¡ƒ7}tt |ƒƒ}|j   d | t|ƒf¡| ¡D]$} t| tj ƒsút| | ƒrà|| f7}qàt|| |||ƒ} |j ¡|j | ¡|j ¡t|tjƒr^|j}|dur^|f|}|D]\}tt|ƒƒdd …d krbtt |¡d dƒdvršn"t ||¡rb|tt |j¡ƒ7}qb|D]}t|tj ƒrÄ||f7}qÄ| |f||¡}t  !||j"¡}|j# ¡|j$ | ||f¡|j# ¡|j   d| t%|ƒf¡| &¡| S)aSubmits function to the execution queue func - function to be executed args - tuple with arguments of the 'func' depfuncs - tuple with functions which might be called from 'func' modules - tuple with module names to import callback - function which will be called with argument list equal to callbackargs+(result,) as soon as calculation is done callbackargs - additional arguments for callback function group - job group, is used when wait(group) is called to wait for jobs in a given group to finish globals - dictionary from which all modules, functions and classes will be imported, for instance: globals=globals() z6Cannot submit jobs: server instance has been destroyedzargs argument must be a tuplez!depfuncs argument must be a tuplez modules argument must be a tuplez%callbackargs argument must be a tupleNz%globals argument must be a dictionaryz*modules argument must be a list of stringsrz(Task %i will autoimport next modules: %séz 0csg|]}tˆjˆjƒ‘qSr )r8ržrŸr}r,r r rsÿÿz$Server.set_ncpus..rƒN) Ú_Server__detect_ncpusr‡rEr‰Ú ValueErrorr2r”ÚextendÚrangerªrlr·©r!rlr r,r r«øs þ zServer.set_ncpuscCsBi}|j ¡D].\}}|dks2||jvr|j|r|j||<q|S)zKReturns active nodes as a dictionary [keys - nodes, values - ncpus]rƒ)rªÚitemsrœrl)r!Z active_nodesÚnodeÚstatr r r Úget_active_nodes sÿ zServer.get_active_nodescCsZ|j ¡D]H\}}|jr z |j d¡t|j ¡ƒ|_Wq d|_d|_Yq 0q |jS)z0Returns job execution statistics as a dictionaryZTIMEFrk)rªrÖrnrHÚfloatrFrhr›)r!r×rØr r r Ú get_statss zServer.get_statscCsÆtdƒt ¡|j}t| ¡ ¡ƒ}d}|D]\}}||j7}q.tdƒ|D]:\}}|jrNtd|jd|j||j|j|j|fƒqNtd|ƒtd|j| ¡fƒ|j sºtdƒtd ƒd S) zOPrints job execution statistics. Useful for benchmarking on clusterszJob execution statistics:rkzE job count | % of all jobs | job time sum | time per job | job serverz1 %6i | %6.2f | %8.4f | %11.6f | %sgY@z%Time elapsed since server creation %sz%s active tasks, %s coreszJWARNING: statistics provided above is not accurate due to job reschedulingrN) r3rhr¶ÚlistrÛrÖrmrrÐr›)r!ZwalltimeÚ statisticsZ totaljobsr¹rØr r r Ú print_stats s*    ÿÿ ÿzServer.print_statscCs\|s| ¡}t||ƒ}|j ¡|j |||f¡|j ¡|j d|j f¡|  ¡|S)zqInserts function into the execution queue. It's intended for internal use only (in ppserver). zTask %i inserted) r¾rr“rr’rRr%r[r\rrÊ)r!rÎrÏrÌrr r r Úinsert<s   z Server.insertTc Cs\|dt|ƒ}|j |t ¡¡}| ¡z zð||jvrJWW| ¡dSt|||j |d||j ƒ}t |  ¡ƒ}t ||ƒ|j|<t|ƒD]6}t|||j |d||j ƒ}| |jd¡|j |¡q†t|ƒD]6}t|||j |d||j ƒ}| |jd¡|j |¡qÆ|j d|||f¡| ¡Wn"tr<|jjddd Yn0W| ¡n | ¡0dS) z7Conects to a remote ppserver specified by host and portrTNZSTATÚEXECrzLConnected to ppserver (host=%s, port=%i) with %i workersz0Exception in connect1 method (possibly expected)Tra)rIr Ú setdefaultrrrrœr%rSrXr]rErFrjrªrÔÚ_Server__update_active_rworkersrZr•rRr–r[r\rÊre) r!rVrWrUZhostidrrnrlrr r r Úconnect1Ks6 å   ÿ zServer.connect1cCsL|jD]}t d|j|¡qt |d¡|_|jD]}t d|jj|¡q0dS)z Connects to all remote ppserversrãTz discover.runN) r¬r1Ú start_threadrãÚppautoZDiscoverZdiscoverr­Úrun)r!r¹r r r Z __connectos   zServer.__connectcCstttdƒrLdtjvr6t d¡}t|tƒrL|dkrL|Sntt d¡d ¡ƒSdtjvrpttjdƒ}|dkrp|SdS)z2Detects the number of effective CPUs in the systemÚsysconfÚSC_NPROCESSORS_ONLNrzsysctl -n hw.ncpurZNUMBER_OF_PROCESSORS) r±r¡Ú sysconf_namesrçr‡rEZpopen2Úreadr¤rÕr r r Z__detect_ncpusxs    zServer.__detect_ncpuscsTt||ƒ}|ˆjvrJ‡fdd„|Dƒ}t t|dƒ||fˆj¡ˆj|<ˆj|S)z Serializes functions and modulescsg|]}ˆ |¡‘qSr )Ú_Server__get_source)r~r r,r r r‘r‚z&Server.__dumpsfunc..r)Úhashr˜r/rÉrrŸ)r!ÚfuncsrrZhashsÚsourcesr r,r Z __dumpsfuncs  þ zServer.__dumpsfunccCsjg}| ¡D]X\}}t|tjƒr |dvr |j||ks>|dkr | |j¡| | |jd|j¡¡q |S)z)recursively finds all the modules in dict)Ú __builtins__ryrÚ.) rÖr‡r¿Ú ModuleTyper rRrÓrÁÚ__dict__)r!Úprefixr½rrÚnameÚobjectr r r Z__find_modules˜s ÿ  ÿzServer.__find_modulescCs||j ¡|jrn|j|jkr¢|j d¡}|jD]}|jr0d|_qZq0|j  d¡t dƒ‚|  d¡z.|j dj d7_ t d|j||f¡Wn Yn0q |jD]L}|jr¨d|_|j d¡}|j |jj d7_ t d|j||f¡q q¨t|jƒ|jkrn|jD]R}|jrd|_|j d¡}|j |jj d7_ t d|j||f¡qlqqnq qnq |j ¡d S) zSchedules jobs for executionrFzThere are no free workers leftzError: No free workersrrƒZ run_localZ run_remoteN)r“rr’rr·Úpopr”rJr[rdÚ RuntimeErrorÚ_Server__add_to_active_tasksrªrmr1räÚ _run_localr•rZÚ _run_remoter2r–r%)r!rÌÚworkerrnr r r Z __scheduler¤sD           zServer.__schedulercCs*t|ƒ}||jvr t|ƒ|j|<|j|S)zFetches source of the function)rìr—r)r!r Zhashfr r r Z __get_sourceÐs zServer.__get_sourcecCsî|jr dS|j d|j¡t ¡}z0|j |¡|j |¡|j ¡}|  |¡Wn,|jrfYdSt rz|jj dddYn0|j r¦|j  ¡|j  |¡|j  ¡| ¡| d¡|jsÒ| dt ¡|¡|j  d|j¡| ¡dS) zRuns a job locallyNzTask %i startedz+Exception in _run_local (possibly expected)Trar…rƒz Task %i ended)r'r[rfrrhrDÚcsendrHrFr&rer\r™ršrÚremover%rOrøÚ_Server__stat_add_timerÊ)r!ÚjobrÎrÏrûÚ start_timer#r r r rù×s0       zServer._run_localcCsæ|j d|j¡z0| |¡| |¡| ¡}d|_| |¡Wnb|j d|j¡| |||¡|  ¡|  |j d¡|  d¡rœ|  |j d¡|  ¡YdS0|j rÊ|j ¡|j  |¡|j ¡|j d|j¡|  ¡dS) zRuns a job remotellyzTask (remote) %i startedTz>Task %i failed due to broken network connection - reschedulingr…ràrNzTask (remote) %i ended)r[r\rrürHrFrJr&rßrÊrârZr^r™ršrrýr%)r!rÿrÎrÏrnr#r r r rúþs0  ÿ    zServer._run_remotecCs&|j ¡|j|7_|j ¡dS)z"Updates the number of active tasksN)r‘rrr%)r!Únumr r r Z__add_to_active_tasks"s zServer.__add_to_active_taskscCs,|j ¡|j|j|7_|j ¡dS)z!Updates total runtime on the nodeN)r°rrªrhr%)r!r×Ztime_addr r r Z__stat_add_time(s zServer.__stat_add_timecCs,|j ¡|j|jd7_|j ¡dS)z Increments job count on the noderN)r°rrªrmr%)r!r×r r r Z__stat_add_job.s zServer.__stat_add_jobcCs>|j ¡||jvrd|j|<|j||7<|j ¡dS)zUpdates list of active rworkersrN)rrrœr%)r!rZÚcountr r r Z__update_active_rworkers4s    zServer.__update_active_rworkerscCs|jd7_|jdS)z Generates a unique job ID numberr)rr,r r r Z__gentid>szServer.__gentidcCs d|_dS)NT)r'r,r r r r`CszServer.__del__cCsŽd|_|j ¡g|_|j ¡|jD]b}zP|j ¡tj   d¡r\t   dt |jƒd¡nt  |jd¡t  |jd¡Wq&Yq&0q&dS)z%Kills ppworkers and closes open filesTÚwinzTASKKILL /PID z /F 2>NULé rN)r'r“rr’r%r”rDrLrPrŽÚ startswithr¡ÚpopenrIrGÚkillÚwaitpid)r!rûr r r r©Fs     zServer.destroy)rvr NFrwrx)r r r Nr rN)N)rv)N)T)r r5r6r7Ú default_portr´r"rtr)rÐr«rÙrÛrÞrßrãrµrÑrÈrÁrÊrërùrúrørþZ_Server__stat_add_jobrâr¾r`r©r r r r rus@ÿ dÿ `      $   ,'$ ruc@s eZdZdS)r(N)r r5r6r r r r r(Ysr()(r7r¡rrŠrrPr¿rhr§r²Zdillr/Z dill.sourcerrÚ ImportErrorrrrCrrårr1Ú copyrightÚ __version__rŒrgrer?Z_USE_SUBPROCESSrõrr8ZCSocketTransportrSrjrorur÷r(r r r r ÚsJ      :06 %C