a tDf*@sddlmZddlZddlZddlmZddlmZddlm Z ddl m Z ddl Z ddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZmZdZddZddZ ddZ!dddZ"dS)) annotationsN)Counter)reduce)product)mul)config)Arraynormalize_chunks)meta_from_array)tokenize)flatten)HighLevelGraph)M parse_bytesa Dask's reshape only supports operations that merge or split existing dimensions evenly. For example: >>> x = da.ones((6, 5, 4), chunks=(3, 2, 2)) >>> x.reshape((3, 2, 5, 4)) # supported, splits 6 into 3 & 2 >>> x.reshape((30, 4)) # supported, merges 6 & 5 into 30 >>> x.reshape((4, 5, 6)) # unsupported, existing dimensions split unevenly To work around this you may call reshape in multiple passes, or (if your data is small enough) call ``compute`` first and handle reshaping in ``numpy`` directly. c sTtddDsJtd}t|d}ddttD}ddtt|D}|dksl|dkrD|||kr|||<|||<|d8}|d8}qZ|}||}|dkrd||<|d8}qZ|dkrd||<|d8}qZ||krZ|d} | dkr.tt| |d|kr.| d8} qtt| |d|krRtttfd dt|Drt|dD]} | || <q||tt t| |||<nt| d|dD]} | f|| <qttt t| d|d} t | | || <tt| d|dt fd d|| D||<|d8}| d}qZ||krZ|d} | dkrtt|| |d|kr| d8} qjtt|| |d|krtttt|| d|dt |||<t| d|dD]} || f|| <qt fd d||D|| <| d}|d8}qZt |t |fS) Ncss|]}t|tVqdSN) isinstancetuple.0cr_/nfs/NAS7/SABIOD/METHODE/ermites/ermites_venv/lib/python3.9/site-packages/dask/array/reshape.py $z"reshape_rechunk..cSsg|]}dqSrrrirrr 'rz#reshape_rechunk..cSsg|]}dqSrrrrrrr(rrrc3s"|]}t||kVqdSr)lenr)inchunksinshaperrrDrc3s|]}|VqdSrrr)prodrrrRsc3s|]}|VqdSrrr)csrrrfr) allrrangerrNotImplementedError_not_implemented_messagemathr"map expand_tuplercontract_tuple) r!Zoutshaper iiZoiZresult_inchunksZresult_outchunksdindoutZileftrZchunk_reductionZoleftr)r#r r!r"rreshape_rechunk#sv              &  r/cCs~|dkr |Sg}|D]L}|}t||d}|d|krR|t||t|8}q*|r||qt|t|ksvJt|S)z >>> expand_tuple((2, 4), 2) (1, 1, 2, 2) >>> expand_tuple((2, 4), 3) (1, 1, 1, 1, 2) >>> expand_tuple((3, 4), 2) (1, 2, 2, 2) >>> expand_tuple((7, 4), 3) (2, 2, 3, 1, 1, 2) r)maxappendintsumr)chunksfactoroutrxpartrrrr*ns  r*cCs\t||dksJg}d}|D]2}||7}||}||}||}|r ||q t|S)zReturn simple chunks tuple such that factor divides all elements Examples -------- >>> contract_tuple((2, 2, 8, 4), 4) (4, 8, 4) r)r4r2r)r5r6r7ZresidualchunkdivZgoodrrrr+s  r+TcsBddlm}ddlm}tt||}dd|D}t|t|krt|t|dkr`tdt|dkrz|jdkrz|S||j t t |dtfdd |D}t t|jrtd t|jt t |d|j krtd |j|kr|St|t|}d t||}|jdkrztt|} |fd t|tj| |fi} tdd |D} tj|| |gd} t| || |dSt|j} t|}|s| |kr|ddt| |D}t|j||j\}}t t dd|D|j j!}|durt"t#$d}t#$dd}n t"|}d}||kr|durDd}t%j&||ddnn|rt'|t'|@}g}|D]:}||dkr|(|||d8<n |(dqbt)||||j |d}||}t*t+|j,ggdd|DR}t*t+|ggdd|DR}t*t+|}ddt-|||D} tj|| |gd} t| |||dS) abReshape array to new shape Parameters ---------- shape : int or tuple of ints The new shape should be compatible with the original shape. If an integer, then the result will be a 1-D array of that length. One shape dimension can be -1. In this case, the value is inferred from the length of the array and remaining dimensions. merge_chunks : bool, default True Whether to merge chunks using the logic in :meth:`dask.array.rechunk` when communication is necessary given the input array chunking and the output shape. With ``merge_chunks==False``, the input array will be rechunked to a chunksize of 1, which can create very many tasks. limit: int (optional) The maximum block size to target in bytes. If no limit is provided, it defaults to using the ``array.chunk-size`` Dask config value. Notes ----- This is a parallelized version of the ``np.reshape`` function with the following limitations: 1. It assumes that the array is stored in `row-major order`_ 2. It only allows for reshapings that collapse or merge dimensions like ``(1, 2, 3, 4) -> (1, 6, 4)`` or ``(64,) -> (4, 4, 4)`` .. _`row-major order`: https://en.wikipedia.org/wiki/Row-_and_column-major_order When communication is necessary this algorithm depends on the logic within rechunk. It endeavors to keep chunk sizes roughly the same when possible. See :ref:`array-chunks.reshaping` for a discussion the tradeoffs of ``merge_chunks``. See Also -------- dask.array.rechunk numpy.reshape r)PerformanceWarning)sanitize_indexcSsg|]}|dkr|qS)rrsrrrrrzreshape..rz&can only specify one unknown dimensionc3s|]}|dkrn|VqdS)r>Nrr?Z missing_sizerrrrzreshape..z_Array chunk size or shape is unknown. shape: %s Possible solution with x.compute_chunk_sizes()z)total size of new array must be unchangedzreshape-)rcss|] }|fVqdSrr)rdrrrrr)Z dependencies)metacSsi|] }|dqSrrrrrr rzreshape..cSsg|] }t|qSr)r1rrrrrrNzarray.chunk-sizez array.slicing.split-large-chunksTaReshaping is producing a large chunk. To accept the large chunk and silence this warning, set the option >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}): ... array.reshape(shape) To avoid creating the large chunks, set the option >>> with dask.config.set(**{'array.slicing.split_large_chunks': True}): ... array.reshape(shape)Explicitly passing ``limit`` to ``reshape`` will also silence this warning >>> array.reshape(shape, limit='128 MiB')) stacklevelauto)shapelimitdtypeZprevious_chunkscSsg|]}tt|qSrr%rrrrrr(rcSsg|]}tt|qSrrKrrrrr)rcSs i|]\}}}|tj||fqSr)rreshape)rabrHrrrrD+r).dask.array.corer<Zdask.array.slicingr=rr)r ValueErrorndimsizerrnpisnanr4rHstrr r Z npartitionsnextr Z __dask_keys__rrLr Zfrom_collectionsrZrechunkr%r/r5rJitemsizerrgetwarningswarnrr2r listrnamezip)r8rHZ merge_chunksrIr<r=Z known_sizesrCr\keyZdskr5graphr-r.r Z outchunksZmax_chunksize_in_bytessplitmsgZmatching_chunksZ chunk_planr7Zx2Zin_keysZout_keysZshapesrrArrLs*           "  rL)TN)# __future__rr(rY collectionsr functoolsr itertoolsroperatorrnumpyrSZdaskrrOrr Zdask.array.utilsr Z dask.baser Z dask.corer Zdask.highlevelgraphr Z dask.utilsrrr'r/r*r+rLrrrrs&          K