Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why is urlpath passed as a list to open_files if it can be already a list? #1773

Open
wachsylon opened this issue Jan 16, 2025 · 0 comments
Open

Comments

@wachsylon
Copy link

wachsylon commented Jan 16, 2025

The fsspec.open command allows urlpath to be a list. But a few lines later, urlpath is passed as a list. If it is already a list, it becomes a list of lists. I ran into problems with this:

File /work/bm1344/conda-envs/py_312/lib/python3.12/site-packages/fsspec/core.py:487, in open(urlpath, mode, compression, encoding, errors, protocol, newline, expand, **kwargs)
    429 """Given a path or paths, return one ``OpenFile`` object.
    430 
    431 Parameters
   (...)
    484   https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations
    485 """
    486 expand = DEFAULT_EXPAND if expand is None else expand
--> 487 out = open_files(
    488     urlpath=[urlpath],
    489     mode=mode,
    490     compression=compression,
    491     encoding=encoding,
    492     errors=errors,
    493     protocol=protocol,
    494     newline=newline,
    495     expand=expand,
    496     **kwargs,
    497 )
    498 if not out:
    499     raise FileNotFoundError(urlpath)

File /work/bm1344/conda-envs/py_312/lib/python3.12/site-packages/fsspec/core.py:295, in open_files(urlpath, mode, compression, encoding, errors, name_function, num, protocol, newline, auto_mkdir, expand, **kwargs)
    216 def open_files(
    217     urlpath,
    218     mode="rb",
   (...)
    228     **kwargs,
    229 ):
    230     """Given a path or paths, return a list of ``OpenFile`` objects.
    231 
    232     For writing, a str path must contain the "*" character, which will be filled
   (...)
    293       https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations
    294     """
--> 295     fs, fs_token, paths = get_fs_token_paths(
    296         urlpath,
    297         mode,
    298         num=num,
    299         name_function=name_function,
    300         storage_options=kwargs,
    301         protocol=protocol,
    302         expand=expand,
    303     )
    304     if fs.protocol == "file":
    305         fs.auto_mkdir = auto_mkdir

File /work/bm1344/conda-envs/py_312/lib/python3.12/site-packages/fsspec/core.py:651, in get_fs_token_paths(urlpath, mode, num, name_function, storage_options, protocol, expand)
    649 if protocol:
    650     storage_options["protocol"] = protocol
--> 651 chain = _un_chain(urlpath0, storage_options or {})
    652 inkwargs = {}
    653 # Reverse iterate the chain, creating a nested target_* structure

File /work/bm1344/conda-envs/py_312/lib/python3.12/site-packages/fsspec/core.py:343, in _un_chain(path, kwargs)
    341 kwargs = kwargs.copy()
    342 for bit in reversed(bits):
--> 343     protocol = kwargs.pop("protocol", None) or split_protocol(bit)[0] or "file"
    344     cls = get_filesystem_class(protocol)
    345     extra_kwargs = cls._get_kwargs_from_urls(bit)

File /work/bm1344/conda-envs/py_312/lib/python3.12/site-packages/fsspec/core.py:552, in split_protocol(urlpath)
    549     if len(protocol) > 1:
    550         # excludes Windows paths
    551         return protocol, path
--> 552 if urlpath.startswith("data:"):
    553     return urlpath.split(":", 1)
    554 return None, urlpath

AttributeError: 'list' object has no attribute 'startswith'
@wachsylon wachsylon changed the title Why is urlpath passed as a list to open_files if it can be already a list? Why is urlpath passed as a list to open_files if it can be already a list? Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant