a vDfV@sddlmZddlZddlZddlmZddlZddlm Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd l m!Z!dd l"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0m1Z1m2Z2m3Z3m4Z4e5de6fiZ7ddZ8ddZ9e%e.ddddddZ:ddZ;ddZGd$d%d%Z Gd&d'd'e Z?dS)() annotationsN)Integral)is_datetime64_any_dtype)Rolling) normalize_arg)tokenize)BlockwiseDepDict)methods)check_axis_keyword_deprecation)Scalar_Frame_get_divisions_map_partitions_get_meta_map_partitions_maybe_from_pandasapply_and_enforce new_dd_objectpartitionwise_graph) from_pandas)_maybe_align_partitions)insert_meta_param_descriptionis_dask_collectionis_dataframe_likeis_series_like)unpack_collections)HighLevelGraph) no_default)Mapply derived_fromfuncname has_keywordCombinedOutputcCsd}|dur,t|tr,|jd|kr,t||durTt|trT|jd|krTt|dd|||fD}t|}t||durt|nd|durt|ndfS)NzqPartition size is less than overlapping window size. Try using ``df.repartition`` to increase the partition size.rcSsg|]}|dur|qSN).0pr#r#c/nfs/NAS7/SABIOD/METHODE/ermites/ermites_venv/lib/python3.9/site-packages/dask/dataframe/rolling.py 9z#_combined_parts..) isinstancershapeNotImplementedErrorr concatr!len)Z prev_partZ current_partZ next_partbeforeaftermsgpartscombinedr#r#r&_combined_parts*s  r3c Osdd|D}|d\}}}dd|D}||i|} |durDd}t|tjrT|}d} |jddkrz| jd|jd} |r| r|| 9}|dur| j|dSt|tjr|}|r| r|| 9}| j|| S)NcSsg|]}t|tr|qSr#r)r!r$dfr#r#r&r'Fr(z!overlap_chunk..rcSs"g|]}t|tr|dn|qSrr4r$argr#r#r&r'Ir()r)datetime timedeltar*iloc) funcr.r/argskwargsdfsr2Zprev_part_lengthZnext_part_lengthoutZ expansionr#r#r& overlap_chunkEs(  rBT)metaenforce_metadatatransform_divisionsalign_dataframesc st|st|r"t|s"t|dn|}|f|}ttrDtttrXtttj spttj rt |j j j stdn,ttrdkrttrdkstd| dd} | dd} t|sJ| durtg|Ri| } n(dt|} t|g|Ri| } | d | } |rt|}z t|}Wn6ty} zt| d | WYd} ~ n d} ~ 00d d |D}t|||| | td d|Dr| dft|tdd |Df| fi}tj| ||d}t|| Sg}g}t|||||| fdd}|D]l}t|trV||}| || |q&t!|}t"|\}}|r| ||#|n | |q&i}d}| $D]:\}}t!|}t"|\}}|#||||<|rd}qt%|dr&ddt&ddD}|'dt(||fdd}|rTt)t*| |g|R|t+d|}n4|r^| n|}t)t+| |g|Ri|d|i}tj| ||d}t,|| S)a] Apply a function to each partition, sharing rows with adjacent partitions. Parameters ---------- func : function The function applied to each partition. If this function accepts the special ``partition_info`` keyword argument, it will receive information on the partition's relative location within the dataframe. df: dd.DataFrame, dd.Series args, kwargs : Positional and keyword arguments to pass to the function. Positional arguments are computed on a per-partition basis, while keyword arguments are shared across all partitions. The partition itself will be the first positional argument, with all other arguments passed *after*. Arguments can be ``Scalar``, ``Delayed``, or regular Python objects. DataFrame-like args (both dask and pandas) will be repartitioned to align (if necessary) before applying the function; see ``align_dataframes`` to control this behavior. enforce_metadata : bool, default True Whether to enforce at runtime that the structure of the DataFrame produced by ``func`` actually matches the structure of ``meta``. This will rename and reorder columns for each partition, and will raise an error if this doesn't work, but it won't raise if dtypes don't match. before : int, timedelta or string timedelta The rows to prepend to partition ``i`` from the end of partition ``i - 1``. after : int, timedelta or string timedelta The rows to append to partition ``i`` from the beginning of partition ``i + 1``. transform_divisions : bool, default True Whether to apply the function onto the divisions and apply those transformed divisions to the output. align_dataframes : bool, default True Whether to repartition DataFrame- or Series-like args (both dask and pandas) so their divisions align before applying the function. This requires all inputs to have known divisions. Single-partition inputs will be split into multiple partitions. If False, all inputs must have either the same number of partitions or a single partition. Single-partition inputs will be broadcast to every partition of multi-partition inputs. $META See Also -------- dd.DataFrame.map_overlap zMMust have a `DatetimeIndex` when using string offset for `before` and `after`rz*before and after must be positive integerstokenN parent_metazoverlap--zx. If you don't want the partitions to be aligned, and are calling `map_overlap` directly, pass `align_dataframes=False`.cSsg|]}t|tr|qSr#)r)r r5r#r#r&r'r(zmap_overlap..css|]}t|tVqdSr")r)r r8r#r#r& r(zmap_overlap..cSsg|]}|jdfqSr7)_namer8r#r#r&r'r( dependenciesc si}t|\}}||t|\}}||dt|}tt|||D]*\}\}} } ||f} t|| | f|| <qTtj |||gd} t | |S)Nzoverlap-concat-rM) _get_previous_partitionsupdate_get_nexts_partitionsr enumeratezipZ __dask_keys__r3rfrom_collectionsr) r9dskZprevs_parts_dskprevsZnexts_parts_dsknextsname_aiprevcurrentnextkeygraph)r/r. divisionsrCr#r&_handle_frame_arguments   z+map_overlap.._handle_frame_argumentTFpartition_infocSsi|]\}}|f||dqS))numberdivisionr#)r$rYrcr#r#r& szmap_overlap..cs|i|d|iS)Nrar#)rar>r?) orig_funcr#r&r=szmap_overlap..func)rNZ_func_metarN)-rrrrr)strpdZ to_timedeltar:r;rindex_meta_nonemptyZ inferred_type TypeErrorr ValueErrorpopcallablerrrrrallrtuplerrTr r r appendrrextenditemsr rRinsertrrrrBr)r=r6r.r/rCrDrErFr>r?namerIrHer@Zlayerr^Zargs2rNr`r9Zarg2 collectionsZkwargs3simplekvrarUZkwargs4r#)r/r.r_rCrfr& map_overlap`sB                          r|c Csi}|j}d}dt||}|rrt|trrg}td|jD]*}||f}tj||f|f||<||q:|dnt|t j rt |j jdd} || krt|g}td|jD]2}||f}t||df||f|f||<||q|dn dg|j}||fS)zE Helper to get the nexts partitions required for the overlap ziPartition size is less than specified window. Try using ``df.repartition`` to increase the partition sizezoverlap-append-rGNrer)rLrr)rrange npartitionsrheadrrr:r;riSeriesr_diffr<anyrm_head_timedelta) r6r/rUdf_nameZtimedelta_partition_messageZname_brWrYr]deltasr#r#r&rQ=s0       rQcsi}|jdt||}|rjt|trjdg}t|jdD]*}||f}tj|f|f||<||q:nLt|t j rt |j }|jdd}||kr\|d} dg}t|jdD]}||d} t| || } |||} } | | kr| dkr| || } | d} q||f}tfddt| |dD|df|f||<||qnLdg}t|jdD]6}||f}t|fg|df|f||<||qpn dg|j}||fS)zH Helper to get the previous partitions required for the overlap zoverlap-prepend-NrGrercsg|] }|fqSr#r#)r$rzrr#r&r'r(z,_get_previous_partitions..)rLrr)rr}r~rtailrrr:r;rirr_rr<rmax_tail_timedelta)r6r.rUrXrVrYr]ZdivsrZpt_zZpt_iZlbfirstjr#rr&rObsP       rOcCs||j|j|kS)zReturn rows of ``next_`` whose index is before the last observation in ``current`` + ``after``. Parameters ---------- current : DataFrame next_ : DataFrame after : timedelta Returns ------- overlapped : DataFrame )rjr)r[Znext_r/r#r#r&rsrcstfdd|D}|S)a4Return the concatenated rows of each dataframe in ``prevs`` whose index is after the first observation in ``current`` - ``before``. Parameters ---------- current : DataFrame prevs : list of DataFrame objects before : timedelta Returns ------- overlapped : DataFrame cs$g|]}||jjkqSr#)rjmin)r$rZr.r[r#r&r'r(z#_tail_timedelta..)r r,)rVr[r.selectedr#rr&rsrc@s>eZdZdZddddefddZddZedd Ze d d Z d d Z e e ddZe e ddZe e ddZe e ddZe e ddZe e ddZe e ddZe e d.ddZe e d/dd Ze e d!d"Ze e d#d$Ze e d%d&Ze e d0d(d)Ze e d*d+ZeZd,d-ZdS)1rz%Provides rolling window calculations.NFcCsj||_||_||_||_||_||_|jjfi|t |jt rJdnd|_ |jdvrft dtdS)Nfreq)rjrGrowszTUsing axis=1 in Rolling has been deprecated and will be removed in a future version.)objwindow min_periodscenteraxiswin_typergrolling_rolling_kwargsr)int _win_typewarningswarn FutureWarning)selfrrrrrrr#r#r&__init__s  zRolling.__init__cCs.|j|j|j|jd}|jtur*|j|d<|S)N)rrrrr)rrrrrrrr?r#r#r&rs  zRolling._rolling_kwargscCs,|jdvp*t|jtr |jdkp*|jjdkS)zm Indicator for whether the object has a single partition (True) or multiple (False). )rGcolumnsrG)rr)rrrr~rr#r#r&_has_single_partitions   zRolling._has_single_partitioncOsJt |jfi|}Wdn1s,0Yt|||i|Sr")r rgetattr)r6rolling_kwargsrvr>r?rr#r#r&pandas_rolling_methods.zRolling.pandas_rolling_methodcOs|}|j|jj||g|Ri|}|jrT|jj|j||g|R||d|S|jrt|jd}|j|d}n*|jdkrt |j}d}n|jd}d}t |j|j||||g|R||d|S)N)rHrCrGrr) rrrrkrZmap_partitionsrrrriZ Timedeltar|)r method_namer>r?rrCr.r/r#r#r& _call_methodsX       zRolling._call_methodcCs |dS)Ncountrrr#r#r&r/sz Rolling.countcCs |dS)Ncovrrr#r#r&r3sz Rolling.covcCs |dS)Nsumrrr#r#r&r7sz Rolling.sumcCs |dS)Nmeanrrr#r#r&r;sz Rolling.meancCs |dS)Nmedianrrr#r#r&r?szRolling.mediancCs |dS)Nrrrr#r#r&rCsz Rolling.mincCs |dS)Nrrrr#r#r&rGsz Rolling.maxrGcCs|jdddS)NstdrGddofrrrr#r#r&rKsz Rolling.stdcCs|jdddS)NvarrGrrrr#r#r&rOsz Rolling.varcCs |dS)Nskewrrr#r#r&rSsz Rolling.skewcCs |dS)Nkurtrrr#r#r&rWsz Rolling.kurtcCs |d|S)Nquantiler)rrr#r#r&r[szRolling.quantilecythonc Cs(|pi}|pd}|jd||||||dS)Nr#r)rawengine engine_kwargsr>r?r)rr=rrrr>r?r#r#r&r_s z Rolling.applycOs|jd|g|Ri|S)Naggr)rr=r>r?r#r#r& aggregateuszRolling.aggregatec CsJdd}|}|j|d<|j|d<ddddt||d DS) NcSs |\}}dddddd}||S)NrrGrrrrrrr#)itemrzr{_orderr#r#r&order|szRolling.__repr__..orderrrz Rolling [{}],css(|] \}}|dur|d|VqdS)N=r#)r$rzr{r#r#r&rKsz#Rolling.__repr__..)r])rrrformatjoinsortedrt)rrrr#r#r&__repr__{s    zRolling.__repr__)rG)rG)FrNNN)__name__ __module__ __qualname____doc__rrrpropertyr staticmethodrrr pd_Rollingrrrrrrrrrrrrrrrrr#r#r#r&rs`    (              rcsJeZdZd fdd ZfddZedddd d Zfd d ZZS)RollingGroupbyNFrc s|j|_|j|_|j}|jdurnt|jtr6|jg}n t|j}t|jtrZ||jn | |j||}t j ||||||ddS)Nr) _groupby_kwargsZ_slice_groupby_slicerr)rhlistZbyrrrssuperr) rgroupbyrrrrrrZ sliced_plus __class__r#r&rs&       zRollingGroupby.__init__cs(t}|dddvr$|d|S)Nr)rrj)rrgetrnrrr#r&rs  zRollingGroupby._rolling_kwargsgroupby_kwargs groupby_slicec OsH|jfi|}|r||}|jfi|}t|||i|jddS)Nre)level)rrrZ sort_index) r6rrvrrr>r?rrr#r#r&rs z$RollingGroupby.pandas_rolling_methodcs$tj|g|R|j|jd|S)Nr)rrrr)rrr>r?rr#r&rszRollingGroupby._call_method)NNFNr) rrrrrrrr __classcell__r#r#rr&rs! r)@ __future__rr:rZnumbersrZpandasriZpandas.api.typesrZpandas.core.windowrrZdask.array.corerZ dask.baserZdask.blockwiserZdask.dataframer Zdask.dataframe._compatr Zdask.dataframe.corer r r rrrrrZdask.dataframe.iorZdask.dataframe.multirZdask.dataframe.utilsrrrrZ dask.delayedrZdask.highlevelgraphrZ dask.typingrZ dask.utilsrrrrr typerqr!r3rBr|rQrOrrrr#r#r#r&sH         (     ]%AL