You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For kedro-datasets plugin, it is not possible to resolve wildcards in local filepaths. For example, I want to read an arbitrary JSON file my_file.json by passing data/01_raw/*.json as filepath and setting expand parameter for fs_args to true. For cloud storage, this works fine, but if a local filepath is passed, I get a DatasetError. The reason is that self._storage_options, where fs_args are stored, won't be passed during loading as long as self._protocol is file.
Steps to Reproduce
Create arbitrary JSON file and add the following to your catalog.yml:
The filepath is resolved properly and no DatasetError is raised.
Actual Result
DatasetError: Failed while loading data from dataset
JSONDataset(filepath=C:/Users/***/Documents/Projects/***/Code/data-pipeline/data/01_raw/*.json,
load_args={'orient': records}, protocol=file, save_args={}).
File
C:/Users/***/Documents/Projects/***/Code/data-pipeline/data/01_raw/*.json does not exist
Your Environment
Kedro version used (pip show kedro or kedro -V): 0.19.10
Kedro plugin and kedro plugin version used (pip show kedro-airflow): 5.1.0
Python version used (python -V): 3.12
Operating system and version: Windows 10
The text was updated successfully, but these errors were encountered:
Description
For kedro-datasets plugin, it is not possible to resolve wildcards in local filepaths. For example, I want to read an arbitrary JSON file
my_file.json
by passingdata/01_raw/*.json
as filepath and settingexpand
parameter forfs_args
totrue
. For cloud storage, this works fine, but if a local filepath is passed, I get aDatasetError
. The reason is thatself._storage_options
, where fs_args are stored, won't be passed during loading as long asself._protocol
is file.Steps to Reproduce
catalog.yml
:my_data_raw
in your pipeline.Expected Result
The filepath is resolved properly and no
DatasetError
is raised.Actual Result
Your Environment
pip show kedro
orkedro -V
): 0.19.10pip show kedro-airflow
): 5.1.0python -V
): 3.12The text was updated successfully, but these errors were encountered: