a tDf@sxddlmZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl ZddlmZddlmZdd lmZmZmZmZdd lmZdd lmZdd lmZmZmZm Z Gd ddZ!GdddeZ"Gddde"Z#Gddde"Z$GdddeZ%d'ddZ&ddZ'ddZ(GdddeZ)Gdd d e)Z*Gd!d"d"eZ+Gd#d$d$eZ,Gd%d&d&eZ-dS)( annotationsN) defaultdict)Callable)product)Any)map)tokenize) Blockwise BlockwiseDepBlockwiseDepDictblockwise_token)flatten)Layer)apply cached_cumsumconcreteinsertc@s eZdZdZddZddZdS)CallableLazyImportzFunction Wrapper for Lazy Importing. This Class should only be used when materializing a graph on a distributed scheduler. cCs ||_dSN) function_path)selfrrX/nfs/NAS7/SABIOD/METHODE/ermites/ermites_venv/lib/python3.9/site-packages/dask/layers.py__init__"szCallableLazyImport.__init__cOs ddlm}||j|i|S)Nr) import_term)Zdistributed.utilsrr)rargskwargsrrrr__call__%s zCallableLazyImport.__call__N)__name__ __module__ __qualname____doc__rrrrrrrsrc@sJeZdZUdZded<ded<dZded<dd d d Zdd d dZdS)ArrayBlockwiseDepzg Blockwise dep for array-likes, which only needs chunking information to compute its data. tuple[tuple[int, ...], ...]chunkstuple[int, ...] numblocksFboolproduces_tasksr%cCs$||_tdd|D|_d|_dS)Ncss|]}t|VqdSr)len).0chunkrrr >z-ArrayBlockwiseDep.__init__..F)r%tupler'r)rr%rrrr<szArrayBlockwiseDep.__init__idxcCs tddS)Nz%Subclasses must implement __getitem__)NotImplementedErrorrr3rrr __getitem__AszArrayBlockwiseDep.__getitem__N)rr r!r"__annotations__r)rr6rrrrr#2s  r#c@seZdZdZddddZdS)ArrayChunkShapeDepz(Produce chunk shapes given a chunk indexr&r2cCstddt||jDS)Ncss|]\}}||VqdSrr)r,ir-rrrr.Ir/z1ArrayChunkShapeDep.__getitem__..)r0zipr%r5rrrr6HszArrayChunkShapeDep.__getitem__N)rr r!r"r6rrrrr8Esr8cs>eZdZUdZded<ddfdd Zddd d ZZS) ArraySliceDepz>Produce slice(s) into the full-sized array given a chunk indexr$startsr*cs$t|tdd|D|_dS)Ncss|]}t|ddVqdS)T)Z initial_zeroN)r)r,crrrr.Sr/z)ArraySliceDep.__init__..)superrr0r<r1 __class__rrrQs zArraySliceDep.__init__r0r2cCs,tddt||jD}tdd|DS)Ncss&|]\}}||||dfVqdSNr)r,r9startrrrr.Vr/z,ArraySliceDep.__getitem__..css |]}tg|dRVqdSrslicer,srrrr.Wr/)r0r:r<)rr3locrrrr6UszArraySliceDep.__getitem__)rr r!r"r7rr6 __classcell__rrr?rr;Ls r;csneZdZdZfddZddZeddZdd Zd d Z d d Z ddZ ddZ ddZ dddZZS)ArrayOverlapLayera\Simple HighLevelGraph array overlap layer. Lazily computed High-level graph layer for a array overlap operations. Parameters ---------- name : str Name of new output overlap array. array : Dask array axes: Mapping Axes dictionary indicating overlap in each dimension, e.g. ``{'0': 1, '1': 1}`` cs2t||_||_||_||_||_d|_dSr)r>rnameaxesr%r'token _cached_keys)rrKrLr%r'rMr?rrris zArrayOverlapLayer.__init__cCsd|jdS)NzArrayOverlapLayerr/z>ArrayOverlapLayer._dask_keys..keys..csg|]}|fqSrrrh)rrfrrrir/)r+range)rindresultr%rfrKr')rrrfsz*ArrayOverlapLayer._dask_keys..keys)rNrKr%r')rrlrrmr _dask_keyss   zArrayOverlapLayer._dask_keysFcCs |j}|j}|j}|}d|j}d|j}|r\}}d}|dkr<|d7}||dkrT|d7}||q fddt|ddD}|dur|gg|}tt|} fddt|D} t| | } | S) aGet all neighboring keys around center Parameters ---------- k: Key The key around which to generate new keys dims: Sequence[int] The number of chunks in each dimension name: Option[str] The name to include in the output keys, or none to include no name axes: Dict[int, int] The axes active in the expansion. We don't expand on non-active axes Examples -------- >>> _expand_keys_around_center(('x', 2, 3), dims=[5, 5], name='y', axes={0: 1, 1: 1}) # noqa: E501 # doctest: +NORMALIZE_WHITESPACE [[('y', 1.1, 2.1), ('y', 1.1, 3), ('y', 1.1, 3.9)], [('y', 2, 2.1), ('y', 2, 3), ('y', 2, 3.9)], [('y', 2.9, 2.1), ('y', 2.9, 3), ('y', 2.9, 3.9)]] >>> _expand_keys_around_center(('x', 0, 4), dims=[5, 5], name='y', axes={0: 1, 1: 1}) # noqa: E501 # doctest: +NORMALIZE_WHITESPACE [[('y', 0, 3.1), ('y', 0, 4)], [('y', 0.9, 3.1), ('y', 0.9, 4)]] csNg}|ddkr||d|||d|dkrJ||d|S)Ng?rrB)append)r9rkrv)rprrindss  z(_expand_keys_around_center..indsrBNrcs2g|]*\}}t|dfr(||n|gqS)ranyget)r,r9rk)rLrrrrisz._expand_keys_around_center..cs*g|]"\}}t|dfr"|ndqS)rrBr)r,r9d)rLrrrir/) enumerater}rqr reshapelist) r{rprKrLshaper9rknumrseqZshape2rlr)rLrprrrts$      rtcsFtdkrt|Stt|d}fddt||DSdS)zgReshape iterator to nested shape >>> reshapelist((2, 3), range(6)) [[0, 1, 2], [3, 4, 5]] rBrcsg|]}tdd|qSrA)rr,partrrrrir/zreshapelist..N)r+rqintru partition)rrnrrrr s rc Cs|dftdd|ddD}g}tt|dd|ddD]\}\}}||d}t|trz|d}|d} n|}|} ||kr|tdddqF||kr| r|td| qF||kr|r|t| dqF|tddqFt|}tdd|Dr|Stj ||fSdS)a >>> fractional_slice(('x', 5.1), {0: 2}) (, ('x', 5), (slice(-2, None, None),)) >>> fractional_slice(('x', 3, 5.1), {0: 2, 1: 3}) (, ('x', 3, 5), (slice(None, None, None), slice(-3, None, None))) >>> fractional_slice(('x', 2.9, 5.1), {0: 2, 1: 3}) (, ('x', 3, 5), (slice(0, 2, None), slice(-3, None, None))) rcss|]}tt|VqdSr)rroundrhrrrr.$r/z#fractional_slice..rBNcss|]}|tdddkVqdSrrD)r,rkrrrr.:r/) r0rr:r isinstancer}rEalloperatorgetitem) ZtaskrLZroundedindexr9trdepthZ left_depthZ right_depthrrrrxs( $*     rxcseZdZdZd fdd ZddZddZd d Zd d Ze d dZ ddZ ddZ ddZ ddZd!ddZddZddZd"ddZZS)#SimpleShuffleLayeraSimple HighLevelGraph Shuffle layer High-level graph layer for a simple shuffle operation in which each output partition depends on all input partitions. Parameters ---------- name : str Name of new shuffled output collection. column : str or list of str Column(s) to be used to map rows to output partitions (by hashing). npartitions : int Number of output partitions. npartitions_input : int Number of partitions in the original (un-shuffled) DataFrame. ignore_index: bool, default False Ignore index during shuffle. If ``True``, performance may improve, but index values will not be preserved. name_input : str Name of input collection. meta_input : pd.DataFrame-like object Empty metadata of input collection. parts_out : list of int (optional) List of required output-partition indices. annotations : dict (optional) Layer annotations Nc sv||_||_||_||_||_||_||_|p4t||_d|j|_ | pJi} d|_ d| vrd|j | d<t j | ddS)Nsplit-priorityr)rKcolumn npartitionsnpartitions_input ignore_index name_input meta_inputrj parts_out split_nameZ _split_keys _key_priorityr>r) rrKrrrrrrrrr?rrrds   zSimpleShuffleLayer.__init__cCs(t|tsJ|d|jkr dSdSdS)NrrB)rr0rr\rrrrsz SimpleShuffleLayer._key_prioritycsfddjDS)Ncsh|]}j|fqSrrPrrQrr r/z5SimpleShuffleLayer.get_output_keys..rrQrrQrrgsz"SimpleShuffleLayer.get_output_keyscCsd|j|jS)Nz-SimpleShuffleLayer)formatrKrrQrrrrRszSimpleShuffleLayer.__repr__cCs t|dSrcrdrQrrrresz"SimpleShuffleLayer.is_materializedcCs$t|dr|jS|}||_|jSrSrUrXrrrrZs  zSimpleShuffleLayer._dictcCs |j|Srr[r\rrrr6szSimpleShuffleLayer.__getitem__cCs t|jSrr^rQrrrr`szSimpleShuffleLayer.__iter__cCs t|jSrrarQrrrrbszSimpleShuffleLayer.__len__c CsNt}|D]>}z |\}}Wnty0Yq Yn0||jkr>q ||q |Sz4Simple utility to convert keys to partition indices.set ValueErrorrKaddrrfpartsr]_name_partrrr_keys_to_partss     z!SimpleShuffleLayer._keys_to_partscsNtt}|p|}|D].}|j|ffddtjDO<q|S)zDetermine the necessary dependencies to produce `keys`. For a simple shuffle, output partitions always depend on all input partitions. This method does not require graph materialization. csh|]}j|fqSrrrhrQrrrsz8SimpleShuffleLayer._cull_dependencies..)rrrrKrjr)rrfrdepsrrrQr_cull_dependenciess z%SimpleShuffleLayer._cull_dependenciesc Cs&t|j|j|j|j|j|j|j|dSNr)rrKrrrrrrrrrrr_cullszSimpleShuffleLayer._cullcCsD||}|j||d}|t|jkr8||}||fS||fSdS)aCull a SimpleShuffleLayer HighLevelGraph layer. The underlying graph will only include the necessary tasks to produce the keys (indices) included in `parts_out`. Therefore, "culling" the layer only requires us to reset this parameter. rNrrrrrrrfall_keysrZ culled_deps culled_layerrrrculls   zSimpleShuffleLayer.cullFc sdj}|r td}td}nddlm}ddlm}i}jD]fddtjD}||j f|jf<|D]^\}}} t j || f|f|j || f<|| f|vrz|j | fjdjjj jf||| f<qzqB|S) z/Construct graph for a simple shuffle operation.group-dask.dataframe.core._concat$dask.dataframe.shuffle.shuffle_groupr_concat shuffle_groupcsg|]}j|fqSr)r)r,Zpart_inZpart_outrrrrisz7SimpleShuffleLayer._construct_graph..)rKrdask.dataframe.corerdask.dataframe.shufflerrrjrrrrrrrr) rrzshuffle_group_name concat_funcshuffle_group_funcrY _concat_list_Z _part_outZ_part_inrrrrWsB       z#SimpleShuffleLayer._construct_graph)NN)N)F)rr r!r"rrrgrRrer|rZr6r`rbrrrrrWrIrrr?rrGs$%*   rcsFeZdZdZdfdd ZddZdddZd d Zdd d ZZ S) ShuffleLayera"Shuffle-stage HighLevelGraph layer High-level graph layer corresponding to a single stage of a multi-stage inter-partition shuffle operation. Stage: (shuffle-group) -> (shuffle-split) -> (shuffle-join) Parameters ---------- name : str Name of new (partially) shuffled collection. column : str or list of str Column(s) to be used to map rows to output partitions (by hashing). inputs : list of tuples Each tuple dictates the data movement for a specific partition. stage : int Index of the current shuffle stage. npartitions : int Number of output partitions for the full (multi-stage) shuffle. npartitions_input : int Number of partitions in the original (un-shuffled) DataFrame. k : int A partition is split into this many groups during each stage. ignore_index: bool, default False Ignore index during shuffle. If ``True``, performance may improve, but index values will not be preserved. name_input : str Name of input collection. meta_input : pd.DataFrame-like object Empty metadata of input collection. parts_out : list of int (optional) List of required output-partition indices. annotations : dict (optional) Layer annotations Nc s@||_||_||_tj|||||| | | p4tt|| d dS)N)rr)inputsstagensplitsr>rrjr+) rrKrrrrrrrrrrrr?rrrAszShuffleLayer.__init__cCsd|j|j|j|jS)Nz=ShuffleLayer)rrKrrrrQrrrrR_szShuffleLayer.__repr__c Cstt}|p||}ddt|jD}|D]}|j|}t|jD]j}t||j|}||} |jdkr| |j kr||j |f d|j |dfqF||j |f |j | fqFq.|S)zqDetermine the necessary dependencies to produce `keys`. Does not require graph materialization. cSsi|]\}}||qSrrr,r9inprrr kr/z3ShuffleLayer._cull_dependencies..rrempty) rrrrrrjrrrrrKrr) rrfrr inp_part_maproutr{_inprrrrrds "zShuffleLayer._cull_dependenciesc Cs2t|j|j|j|j|j|j|j|j|j |j |d Sr) rrKrrrrrrrrrrrrrrwszShuffleLayer._cullFc Csld|j}|r td}td}nddlm}ddlm}i}ddt|jD}|jD]}|j|}g} t |j D].} t ||j | } ||j } | |j| | fqt|| |jf||j|f<| D]\} } } tj|| f| f||j| | f<|| f|vr|| }|j dkr2||jkr|j|f}n|| d f}|j||<n |j|f}|||j|j |j |j|j|jf||| f<qqV|S) z2Construct graph for a "rearrange-by-column" stage.rrrrrrcSsi|]\}}||qSrrrrrrrr/z1ShuffleLayer._construct_graph..r)rKrrrrrrrrrjrrrr}rrrrrrrrr)rrzrrrrYrrrrr9rZ_idxrrZ input_keyrrrrWsX              zShuffleLayer._construct_graph)NN)N)F) rr r!r"rrRrrrWrIrrr?rrs0 rcseZdZdZd fdd ZddZddZd d Zed d Z d dZ ddZ ddZ ddZ eddZd!ddZddZddZd"ddZZS)#BroadcastJoinLayera;Broadcast-based Join Layer High-level graph layer for a join operation requiring the smaller collection to be broadcasted to every partition of the larger collection. Parameters ---------- name : str Name of new (joined) output collection. lhs_name: string "Left" DataFrame collection to join. lhs_npartitions: int Number of partitions in "left" DataFrame collection. rhs_name: string "Right" DataFrame collection to join. rhs_npartitions: int Number of partitions in "right" DataFrame collection. parts_out : list of int (optional) List of required output-partition indices. annotations : dict (optional) Layer annotations. **merge_kwargs : **dict Keyword arguments to be passed to chunkwise merge func. Nc  stj|d||_||_||_||_||_||_|pBtt |j|_ t | t rXt | n| |_t | t rpt | n| |_| |_|jd|_|j|jd<|j|jd<dS)Nrhowleft_onright_on)r>rrKrlhs_namelhs_npartitionsrhs_namerhs_npartitionsrrjrrrqr0rr merge_kwargsrr) rrKrrrrrrrrrrr?rrrs zBroadcastJoinLayer.__init__csfddjDS)Ncsh|]}j|fqSrrPrrQrrrr/z5BroadcastJoinLayer.get_output_keys..rrQrrQrrgsz"BroadcastJoinLayer.get_output_keyscCsd|j|j|j|jS)Nz5BroadcastJoinLayer)rrKrrrrQrrrrR szBroadcastJoinLayer.__repr__cCs t|dSrcrdrQrrrresz"BroadcastJoinLayer.is_materializedcCs$t|dr|jS|}||_|jSrSrUrXrrrrZs  zBroadcastJoinLayer._dictcCs |j|Srr[r\rrrr6szBroadcastJoinLayer.__getitem__cCs t|jSrr^rQrrrr`szBroadcastJoinLayer.__iter__cCs t|jSrrarQrrrrb!szBroadcastJoinLayer.__len__c CsNt}|D]>}z |\}}Wnty0Yq Yn0||jkr>q ||q |Srrrrrrr$s     z!BroadcastJoinLayer._keys_to_partscCs8|j|jkr |j|j|j|jfS|j|j|j|jfSdSr)rrrrrrrQrrr_broadcast_plan1s  z"BroadcastJoinLayer._broadcast_plancs||jdd\}}tt}|p(||}|D]H}||j|ffddt|DO<||j|f||fhO<q.|S)zDetermine the necessary dependencies to produce `keys`. For a broadcast join, output partitions always depend on all partitions of the broadcasted collection, but only one partition of the "other" collection. Ncsh|] }|fqSrrrh bcast_namerrrXr/z8BroadcastJoinLayer._cull_dependencies..)rrrrrKrj)rrfr bcast_size other_namerrrrrrKs( z%BroadcastJoinLayer._cull_dependenciescCs0t|j|j|j|j|j|jf|j|d|jS)N)rr) rrKrrrrrrrrrrrr^s zBroadcastJoinLayer._cullcCsD||}|j||d}|t|jkr8||}||fS||fSdS)aCull a BroadcastJoinLayer HighLevelGraph layer. The underlying graph will only include the necessary tasks to produce the keys (indices) included in `parts_out`. Therefore, "culling" the layer only requires us to reset this parameter. rNrrrrrrks   zBroadcastJoinLayer.cullFcCs2d|j}d|j}|r2td}td}td}n$ddlm}ddlm}dd lm}|j\}}} } |j|jkrtd nd } i} |j D]} |j d kr|| | f| |f| || f<g}t |D]d}|j d krt j || f|fn| | f||fg}| d kr||| |f}t|||jf| |<||q||f| |j| f<q| S) z/Construct graph for a broadcast join operation.zinter-rz%dask.dataframe.multi._split_partitionz$dask.dataframe.multi._concat_wrapperz)dask.dataframe.multi._merge_chunk_wrapperr)_concat_wrapper)_merge_chunk_wrapper)_split_partitionleftrightinner)rKrZdask.dataframe.multirrrrrrrrrjrrreverserrr})rrzZ inter_namerZsplit_partition_funcrZmerge_chunk_funcrrrZother_onZ bcast_siderYr9rjZ _merge_argsZ inter_keyrrrrW{sZ            z#BroadcastJoinLayer._construct_graph)NNNN)N)F)rr r!r"rrgrRrer|rZr6r`rbrrrrrrWrIrrr?rrs*"     rcs>eZdZdZd fdd ZeddZdd Zd d ZZ S) DataFrameIOLayera DataFrame-based Blockwise Layer with IO Parameters ---------- name : str Name to use for the constructed layer. columns : str, list or None Field name(s) to read in as columns in the output. inputs : list or BlockwiseDep List of arguments to be passed to ``io_func`` so that the materialized task to produce partition ``i`` will be: ``(, inputs[i])``. Note that each element of ``inputs`` is typically a tuple of arguments. io_func : callable A callable function that takes in a single tuple of arguments, and outputs a DataFrame partition. Column projection will be supported for functions that satisfy the ``DataFrameIOFunction`` protocol. label : str (optional) String to use as a prefix in the place-holder collection name. If nothing is specified (default), "subset-" will be used. produces_tasks : bool (optional) Whether one or more elements of `inputs` is expected to contain a nested task. This argument in only used for serialization purposes, and will be deprecated in the future. Default is False. creation_info: dict (optional) Dictionary containing the callable function ('func'), positional arguments ('args'), and key-word arguments ('kwargs') used to produce the dask collection with this underlying ``DataFrameIOLayer``. annotations: dict (optional) Layer annotations to pass through to Blockwise. NFc s||_||_||_||_||_||_||_||_t|t sZt ddt |jD|jd} n|} |j|t dfi} t j|jd| | dfgi|ddS)NcSsi|]\}}|f|qSrrrrrrrr/z-DataFrameIOLayer.__init__..)r)rr9)outputZoutput_indicesrYindicesr'r)rK_columnsrio_funclabelr)r creation_inforr r rr r>r) rrKcolumnsrrrr)rrZ io_arg_maprYr?rrrs.  zDataFrameIOLayer.__init__cCs|jS)z(Current column projection for this layer)rrQrrrrszDataFrameIOLayer.columnsc Csddlm}t|}|jdus.t|j|rt|j|rH|j|}n|j}t |j pXddt |j |||j ||j |j|jd}|S|SdS)zProduce a column projection for this IO layer. Given a list of required output columns, this method returns the projected layer. r)DataFrameIOFunctionNZsubset-)rr)r)Zdask.dataframe.io.utilsrrqrr issupersetrrproject_columnsrrr rKrr)r)rrrrZlayerrrrrs"   z DataFrameIOLayer.project_columnscCsd|jt|j|jS)Nz3DataFrameIOLayer)rrKr+rrrQrrrrR<szDataFrameIOLayer.__repr__)NFNN) rr r!r"rr|rrrRrIrrr?rrs*( rc seZdZUdZded<ded<ded<ded<ded <d ed <ded <ded <ded<ded<ded<ded<d;dddddd dddddd fdd ZddddZd=1, the input tasks must contain data that can be indexed by a ``getitem`` operation with a key in the range ``[0, split_out)``. output_partitions : list, optional List of required output partitions. This parameter is used internally by Dask for high-level culling. tree_node_name : str, optional Name to use for intermediate tree-node tasks. strrKrrrrrtree_node_funczCallable | None finalize_func split_every split_outz list[int]output_partitionstree_node_namewidthsheightN z int | Nonezlist[int] | Nonez str | Nonezdict[str, Any] | None) rKrrrrrrrrrrc stj| d||_||_||_||_||_||_||_||_ | durXt t |j pRdn| |_ | pjd|j|_ |j} | g|_| dkrt| |j} |jt| q|t|j|_dS)NrrBz tree_node-)r>rrKrrrrrrrrqrjrrrmathceilr}rr+r) rrKrrrrrrrrrrrr?rrrus(zDataFrameTreeReduction.__init__rsplitcGs|jr||fS|Sr)r)rrZ name_partsrrr _make_keysz DataFrameTreeReduction._make_keyFcCs(|r|jr|j}n|j}tj||j|fSr)rrrurvr)r input_keys final_taskZ outer_funcrrr _define_tasks z#DataFrameTreeReduction._define_taskc si}js|Sjjr`d7jD]6tjD]&}tjj|ff|j|d<q6q(jdkrpjD]tdjD]tj D]}j d}j |}t |j |}dkrfddt||D}nfddt||D}jdkrF|dks,Jd |d j |d d |j f<qj |d d |jj|d<qqqrn8jD]0jddg}j |d d |j f<qv|S)z%Construct graph for a tree reduction.z-splitrrBcsg|]}j|dqS)r)rr,p)name_input_userGrrrrisz;DataFrameTreeReduction._construct_graph..cs$g|]}jj|ddqS)rBr)rrr )rrGrrrrisrzgroup = z%, not 0 for final tree reduction taskT)r F)rrrrjrrrrrrrminr rKr)rrYrgroupZp_maxZlstartZlstopr r)rrrGrrrWsX         z'DataFrameTreeReduction._construct_graphcCsd|j|j|jS)Nz>DataFrameTreeReduction)rrKrrrQrrrrRs zDataFrameTreeReduction.__repr__csfddjDS)Ncsh|]}j|fqSrrPrFrQrrrr/z6DataFrameTreeReduction._output_keys..)rrQrrQr _output_keyssz#DataFrameTreeReduction._output_keyscCs$t|dr|jS|}||_|jS)N_cached_output_keys)rVrr)rZ output_keysrrrrgs  z&DataFrameTreeReduction.get_output_keyscCs t|dSrcrdrQrrrresz&DataFrameTreeReduction.is_materializedcCs$t|dr|jS|}||_|jSrSrUrXrrrrZs  zDataFrameTreeReduction._dictcCs |j|Srr[r\rrrr6sz"DataFrameTreeReduction.__getitem__cCs t|jSrr^rQrrrr`szDataFrameTreeReduction.__iter__cCs>t|jddpd|jpd}|jr:||jt|jS|S)NrB)sumrrrr+r)rZ tree_sizerrrrbs zDataFrameTreeReduction.__len__c CsNt}|D]>}z |\}}Wnty0Yq Yn0||jkr>q ||q |S)z;Simple utility to convert keys to output partition indices.r)rrfsplitsr]r_splitrrr_keys_to_output_partitionss     z1DataFrameTreeReduction._keys_to_output_partitionsc Cs2t|j|j|j|j|j|j|j|j||j |j d S)N)rrrrrr) rrKrrrrrrrrr)rrrrrr,szDataFrameTreeReduction._cullcsXjdffddtjDi}|}|tjkrL|}||fS|fSdS)z2Cull a DataFrameTreeReduction HighLevelGraph layerrcsh|]}j|fqSrrrhrQrrr>sz.DataFrameTreeReduction.cull..N)rKrjrrrrr)rrfrrrrrrQrr;s  zDataFrameTreeReduction.cull)NrNNNN)F)rr r!r"r7rrr rWrRrrgrer|rZr6r`rbrrrrIrrr?rrBsF % (' J  r)NN). __future__rrrrr collectionsrcollections.abcr itertoolsrtypingrZtlzruZ tlz.curriedrZ dask.baser Zdask.blockwiser r r r Z dask.corerZdask.highlevelgraphrZ dask.utilsrrrrrr#r8r;rJrtrrxrrrrrrrrrs<          z 7 /V3~w