-
Notifications
You must be signed in to change notification settings - Fork 185
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Cloud-compatibility tests for Text Components (use fsspec, not os)
Goal
Ensure all Text Components use fsspec for paths/IO. Ban os.path, pathlib.Path, glob, shutil for URI work.
Why
os/pathlib/glob assume local FS and break on s3://, gs://, abfs://, http(s)://.
What breaks → What to use
Exists/stat: os.path.exists("s3://…") → False
➜ fs.exists(p), fs.info(p)
Open/write: open("s3://…") fails
➜ with fs.open(p, "wb") as f: …
Join/normalize: os.path.normpath("s3://b/a/../c") mangles protocol
➜ posixpath.join(root, "a", "b") (after url_to_fs)
List/walk: os.listdir, os.walk fail
➜ fs.ls(root), fs.walk(root)
Glob: glob.glob("s3://…") → []
➜ fs.glob("s3://bucket/**/*.json")
Mk/rm/mv/cp: os.makedirs, shutil.rmtree/move
➜ fs.makedirs, fs.rm, fs.mv, fs.cp
What works:
- os.path.join → Allow this as this works (and we dont care about windows support)
Refactors stuff like:
- open(uri, mode) → fs.open(p, mode)
- os.listdir/os.walk → fs.ls/fs.walk
- glob.glob → fs.glob
- os.makedirs/os.remove/shutil.* → fs.makedirs/fs.rm/fs.mv/fs.cp
Copilot
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request