Skip to content

[FEA] Cloud-compatibility tests for Text Components (use fsspec, not os) #984

@VibhuJawa

Description

@VibhuJawa

Cloud-compatibility tests for Text Components (use fsspec, not os)

Goal

Ensure all Text Components use fsspec for paths/IO. Ban os.path, pathlib.Path, glob, shutil for URI work.

Why

os/pathlib/glob assume local FS and break on s3://, gs://, abfs://, http(s)://.

What breaks → What to use

Exists/stat: os.path.exists("s3://…") → False
➜ fs.exists(p), fs.info(p)

Open/write: open("s3://…") fails
➜ with fs.open(p, "wb") as f: …

Join/normalize: os.path.normpath("s3://b/a/../c") mangles protocol
➜ posixpath.join(root, "a", "b") (after url_to_fs)

List/walk: os.listdir, os.walk fail
➜ fs.ls(root), fs.walk(root)

Glob: glob.glob("s3://…") → []
➜ fs.glob("s3://bucket/**/*.json")

Mk/rm/mv/cp: os.makedirs, shutil.rmtree/move
➜ fs.makedirs, fs.rm, fs.mv, fs.cp

What works:

  • os.path.join → Allow this as this works (and we dont care about windows support)

Refactors stuff like:

  • open(uri, mode) → fs.open(p, mode)
  • os.listdir/os.walk → fs.ls/fs.walk
  • glob.glob → fs.glob
  • os.makedirs/os.remove/shutil.* → fs.makedirs/fs.rm/fs.mv/fs.cp

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions