Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Nodes] Add Prebatch setting to ParallelMapper #1417

Merged
merged 7 commits into from
Jan 2, 2025
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
fix test
andrewkho committed Dec 30, 2024
commit 6a999171896121c77f50a16d5cc4e487aa8a5b69
2 changes: 1 addition & 1 deletion torchdata/nodes/map.py
Original file line number Diff line number Diff line change
@@ -416,7 +416,7 @@ def __init__(
if prebatch <= 0:
raise ValueError(f"{prebatch=} must be a positive integer!")
self.map_fn = MapOverBatch(map_fn=map_fn) # type: ignore[assignment]
self.source = Batcher(self.source, batch_size=prebatch, drop_last=False) # type: ignore[assignment]
self.source = Batcher(source, batch_size=prebatch, drop_last=False) # type: ignore[assignment]

_it = _ParallelMapperImpl(
source=self.source,