Skip to content

Commit

Permalink
Remove prefetch from pubsub source
Browse files Browse the repository at this point in the history
The `prefetchN` in the pubsub source was complicated because:

1. The preceding `parEvalMapUnordered` effectively does a pre-fetch
   already
2. PrefetchN works in chunks, so there was a danger our Source was
   pre-fetching more batches from pubsub than expected.
3. It gave a double-meaning to the `parallelPullCount` config param,
   because the same param got used in two places.  Which is just
   confusing.

Moreover, I found it just wasn't needed.  The input rate of this Source
is high enough without the prefetch.
  • Loading branch information
istreeter committed Jan 10, 2025
1 parent d49080a commit 4173065
Showing 1 changed file with 0 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ object PubsubSource {
} yield Stream
.fixedRateStartImmediately(config.debounceRequests, dampen = true)
.parEvalMapUnordered(parallelPullCount)(_ => pullAndManageState(config, stub, refStates))
.prefetchN(parallelPullCount)
.concurrently(extendDeadlines(config, stub, refStates))
.onFinalize(nackRefStatesForShutdown(config, stub, refStates))

Expand Down

0 comments on commit 4173065

Please sign in to comment.