a iDf2H@sUddlmZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl ZddlmZddlmZddlmZmZmZdd lmZdd lmZe rdd lmZdd lmZdd l mZ!dd l"mZdd l#m$Z$ddl%m&Z&e'e(Z)dZ*ddZ+dddddZ,ddZ-ddZ.dddddd Z/d8d"d#Z0d$d%Z1e2d&d'fd(d)Z3Gd*d+d+eej4Z5Gd,d-d-Z6Gd.d/d/Z7Gd0d1d1e6Z8Gd2d3d3e8Z9Gd4d5d5Z:iZ;d6e>> from pathlib import Path >>> directory = Path(xr.backends.common.__file__).parent >>> paths_path = Path(directory).joinpath("comm*n.py") >>> paths_str = xr.backends.common._normalize_path(paths_path) >>> print([type(p) for p in (paths_str,)]) [] ) isinstanceosPathLikefspathstrr pathabspath expanduser)rrc/nfs/NAS7/SABIOD/METHODE/ermites/ermites_venv/lib/python3.9/site-packages/xarray/backends/common.py_normalize_path$s   rz5str | os.PathLike | NestedSequence[str | os.PathLike]z list[str])pathsreturnc st|trt|r|dddkrzddlm}Wn.ty`}ztd|WYd}~n d}~00||d|did id d \}}|}fd d |D}n*t|rt d|dnt tt |}n(t|t j rt |g}ndd |D}|S)a Find absolute paths from the pattern. Parameters ---------- paths : Path(s) to file(s). Can include wildcards like * . **kwargs : Extra kwargs. Mainly for fsspec. Examples -------- >>> from pathlib import Path >>> directory = Path(xr.backends.common.__file__).parent >>> paths = str(Path(directory).joinpath("comm*n.py")) # Find common with wildcard >>> paths = xr.backends.common._find_absolute_paths(paths) >>> [Path(p).name for p in paths] ['common.py'] ZengineNZzarrr)get_fs_token_pathszCThe use of remote URLs for opening zarr requires the package fsspecrbZbackend_kwargsstorage_optionsF)moder#expandcsg|]}|qSr)Z get_mapper).0rfsrr iz(_find_absolute_paths..zjcannot do wild-card matching for paths that are remote URLs unless engine='zarr' is specified. Got paths: z7. Instead, supply paths as an explicit list of strings.cSs&g|]}t|tjrt|n|qSr)rrrr)r&prrrr)ur*)rrr getZ fsspec.corer! ImportErrorrZ_strip_protocol ValueErrorsortedrrrr)rkwargsr!e_Z tmp_pathsrr'r_find_absolute_paths@s>    r3cCs|dur t}|SN NONE_VAR_NAMEnamerrr_encode_variable_namezsr9cCs|tkr d}|Sr4r5r7rrr_decode_variable_namesr:zncDataset | ncDatasetLegacyH5;str | os.PathLike[Any] | BufferedIOBase | AbstractDataStorer) ncDatasetfilename_or_objr c Ksddlm}ddlm}ddlm}||fi|}|d|i}||ddZ}t|D]@} ||fd| i|} || j} || | d } |j | | d d d qXWdn1s0Y|S) Nr) open_datasetrNodePath/r)r$group)r8dataFT)Zallow_overwriteZnew_nodes_along_path) Zxarray.backends.apir>xarray.core.datatreerxarray.core.treenoder@ from_dict_iter_nc_groupsr8Z _set_item) r<r=r0r>rr@dsZ tree_rootZncdsrZ subgroup_dsZ node_namenew_noderrr_open_datatree_netcdfs"      &rKrAccsPddlm}||}|jD],\}}||}t|Vt||dEdHqdS)Nrr?)parent)rFr@groupsitemsrrH)rootrLr@rrCZgpathrrrrHs   rHcCsBd}|jdur,|jddf|}|j}qdd|}||fS)z;Find the root and group name of a netCDF4/h5netcdf dataset.rNrA)rLr8splitjoin)rIZ hierarchyrCrrrfind_root_and_groups  rSic Cs|dks Jt|dD]}z||WS|y||krB|d|}|tj|}d|d||dt}t|t d|Yq0qdS) z Robustly index an array, using retry logic with exponential backoff if any of the errors ``catch`` are raised. The initial_delay is measured in ms. With the default settings, the maximum delay will be in the range of 32-64 seconds. rzgetitem failed, waiting z ms before trying again (z# tries remaining). Full traceback: gMbP?N) rangenprandomrandint traceback format_excloggerdebugtimesleep) arraykeyZcatch max_retriesZ initial_delaynZ base_delayZ next_delaymsgrrrrobust_getitems"    rfc@s eZdZdZdddddZdS) BackendArrayrNznp.typing.DTypeLike)dtypecCsttdf|j}||Sr4)r Z BasicIndexerslicendim)selfrhrbrrrget_duck_arrayszBackendArray.get_duck_array)N)__name__ __module__ __qualname__ __slots__rlrrrrrgsrgc@sPeZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ dS)AbstractDataStorercCs tdSr4NotImplementedErrorrkrrrget_dimensionssz AbstractDataStore.get_dimensionscCs tdSr4rrrtrrr get_attrsszAbstractDataStore.get_attrscCs tdSr4rrrtrrr get_variablesszAbstractDataStore.get_variablescCsiSr4rrtrrr get_encodingszAbstractDataStore.get_encodingcCs.tdd|D}t|}||fS)a, This loads the variables and attributes simultaneously. A centralized loading function makes it easier to create data stores that do automatic encoding/decoding. For example:: class SuffixAppendingDataStore(AbstractDataStore): def load(self): variables, attributes = AbstractDataStore.load(self) variables = {'%s_suffix' % k: v for k, v in variables.items()} attributes = {'%s_suffix' % k: v for k, v in attributes.items()} return variables, attributes This function will be called anytime variables or attributes are requested, so care should be taken to make sure its fast. css|]\}}t||fVqdSr4)r:r&kvrrr sz)AbstractDataStore.load..)r rwrNrvrk variables attributesrrrloads   zAbstractDataStore.loadcCsdSr4rrtrrrcloseszAbstractDataStore.closecCs|Sr4rrtrrr __enter__szAbstractDataStore.__enter__cCs |dSr4)r)rkZexception_typeZexception_valuer[rrr__exit__szAbstractDataStore.__exit__N) rmrnrorprurvrwrxrrrrrrrrrqsrqc@s.eZdZdZd ddZd ddZd dd ZdS) ArrayWritersourcestargetsregionslockNcCsg|_g|_g|_||_dSr4r)rkrrrr__init__ szArrayWriter.__init__cCsHt|r.|j||j||j|n|r<|||<n||d<dS)N.)rrappendrr)rksourcetargetZregionrrradds   zArrayWriter.addTcCs\|jrXt|j}|duri}|j|j|jf|j|d|jd|}g|_g|_g|_|SdS)NT)rcomputeflushr)rr storerrr)rkrZchunkmanager_store_kwargsZ chunkmanagerZ delayed_storerrrsyncs&  zArrayWriter.sync)N)N)TN)rmrnrorprrrrrrrr s  rc@sveZdZdZddZddZddZdd Zd d Zd d Z ddZ e ddfddZ ddZ dddZdddZdS)AbstractWritableDataStorercs4fdd|D}fdd|D}||fS)a Encode the variables and attributes in this store Parameters ---------- variables : dict-like Dictionary of key/value (variable name / xr.Variable) pairs attributes : dict-like Dictionary of key/value (attribute name / attribute) pairs Returns ------- variables : dict-like attributes : dict-like csi|]\}}||qSrencode_variableryrtrr Lr*z4AbstractWritableDataStore.encode..csi|]\}}||qSrencode_attributeryrtrrrMr*)rNr}rrtrencode;sz AbstractWritableDataStore.encodecCs|S)zencode one variabler)rkr{rrrrPsz)AbstractWritableDataStore.encode_variablecCs|S)zencode one attributer)rkarrrrTsz*AbstractWritableDataStore.encode_attributecCs tdSr4rr)rkdimlengthrrr set_dimensionXsz'AbstractWritableDataStore.set_dimensioncCs tdSr4rrrkrzr{rrr set_attribute[sz'AbstractWritableDataStore.set_attributecCs tdSr4rrrrrr set_variable^sz&AbstractWritableDataStore.set_variablecCs|||jdS)z in stores, variables are all variables AND coordinates in xarray.Dataset variables are variables NOT coordinates, so here we pass the whole dataset in instead of doing dataset.variables N)rattrs)rkZdatasetrrr store_datasetasz'AbstractWritableDataStore.store_datasetNcCsL|durt}|||\}}|||j||d|j||||ddS)a Top level method for putting data on this store, this method: - encodes variables/attributes - sets dimensions - sets variables Parameters ---------- variables : dict-like Dictionary of key/value (variable name / xr.Variable) pairs attributes : dict-like Dictionary of key/value (attribute name / attribute) pairs check_encoding_set : list-like List of variables that should be checked for invalid encoding values writer : ArrayWriter unlimited_dims : list-like List of dimension names that should be treated as unlimited dimensions. Nunlimited_dims)rrset_attributesset_dimensions set_variables)rkr~rcheck_encoding_setwriterrrrrrjs zAbstractWritableDataStore.storecCs"|D]\}}|||qdS)z This provides a centralized method to set the dataset attributes on the data store. Parameters ---------- attributes : dict-like Dictionary of key/value (attribute name / attribute) pairs N)rNr)rkrrzr{rrrrs z(AbstractWritableDataStore.set_attributesc CsH|D]:\}}t|}||v}|j||||d\} } || | qdS)a This provides a centralized method to set the variables on the data store. Parameters ---------- variables : dict-like Dictionary of key/value (variable name / xr.Variable) pairs check_encoding_set : list-like List of variables that should be checked for invalid encoding values writer : ArrayWriter unlimited_dims : list-like List of dimension names that should be treated as unlimited dimensions. rN)rNr9Zprepare_variabler) rkr~rrrZvnr{r8checkrrrrrrs z'AbstractWritableDataStore.set_variablesc Cs|durt}|}i}|D] }d||<q|D]}|tt|j|jq4|D]\\}}||vr|||krt d|d|d||dqZ||vrZ||v}| |||qZdS)au This provides a centralized method to set the dimensions on the data store. Parameters ---------- variables : dict-like Dictionary of key/value (variable name / xr.Variable) pairs unlimited_dims : list-like List of dimension names that should be treated as unlimited dimensions. Nz,Unable to update size for existing dimensionz (z != )) setruvaluesupdatedictzipdimsshaperNr.r) rkr~rZ existing_dimsrr{rrZ is_unlimitedrrrrs,   z(AbstractWritableDataStore.set_dimensions)N)N)rmrnrorprrrrrrr frozensetrrrrrrrrr8s  ' rc@seZdZdZddZdS)WritableCFDataStorercsBt||\}}fdd|D}fdd|D}||fS)Ncsi|]\}}||qSrrryrtrrrr*z.WritableCFDataStore.encode..csi|]\}}||qSrrryrtrrrr*)rrNr}rrtrrszWritableCFDataStore.encodeN)rmrnrorprrrrrrsrc@seZdZUdZdZded<dZded<dZded<d d d d Zdd dddddddZ dddddZ ddddddZ dS)BackendEntrypointa ``BackendEntrypoint`` is a class container and it is the main interface for the backend plugins, see :ref:`RST backend_entrypoint`. It shall implement: - ``open_dataset`` method: it shall implement reading from file, variables decoding and it returns an instance of :py:class:`~xarray.Dataset`. It shall take in input at least ``filename_or_obj`` argument and ``drop_variables`` keyword argument. For more details see :ref:`RST open_dataset`. - ``guess_can_open`` method: it shall return ``True`` if the backend is able to open ``filename_or_obj``, ``False`` otherwise. The implementation of this method is not mandatory. - ``open_datatree`` method: it shall implement reading from file, variables decoding and it returns an instance of :py:class:`~datatree.DataTree`. It shall take in input at least ``filename_or_obj`` argument. The implementation of this method is not mandatory. For more details see . Attributes ---------- open_dataset_parameters : tuple, default: None A list of ``open_dataset`` method parameters. The setting of this attribute is not mandatory. description : str, default: "" A short string describing the engine. The setting of this attribute is not mandatory. url : str, default: "" A string with the URL to the backend's documentation. The setting of this attribute is not mandatory. NzClassVar[tuple | None]open_dataset_parametersz ClassVar[str] descriptionurlr)r cCsBdt|jd}|jr(|d|j7}|jr>|d|j7}|S)N<>z z Learn more at )typermrr)rktxtrrr__repr__s zBackendEntrypoint.__repr__)drop_variablesr;zstr | Iterable[str] | Nonerr)r=rr0r cKs tdS)` Backend open_dataset method used by Xarray in :py:func:`~xarray.open_dataset`. Nrr)rkr=rr0rrrr>s zBackendEntrypoint.open_datasetbool)r=r cCsdS)rFr)rkr=rrrguess_can_open#sz BackendEntrypoint.guess_can_openr)r=r0r cKs tdS)zb Backend open_datatree method used by Xarray in :py:func:`~xarray.open_datatree`. Nrr)rkr=r0rrr open_datatree-s zBackendEntrypoint.open_datatree) rmrnro__doc__r__annotations__rrrr>rrrrrrrs !      rz5dict[str, tuple[str | None, type[BackendEntrypoint]]]BACKEND_ENTRYPOINTS)rA)= __future__rloggingrr_r[collections.abcrrtypingrrrnumpyrXZxarray.conventionsrZ xarray.corer Zxarray.core.utilsr r r Z xarray.namedarray.parallelcompatr Zxarray.namedarray.pycompatriorZh5netcdf.legacyapirZncDatasetLegacyH5ZnetCDF4r<Zxarray.core.datasetrErZxarray.core.typesr getLoggerrmr]r6rr3r9r:rKrHrS ExceptionrfZExplicitlyIndexedrgrqrrrrrrrrrrsL             :  4.% R