Java Stream Gatherers: Build Stateful Intermediate Operations

A stream gatherer is a customizable intermediate operation: it consumes upstream elements and can emit zero, one, or many downstream elements while maintaining state. Use Stream.gather when the transformation belongs in the middle of a pipeline—windowing, prefix scans, bounded concurrent mapping, or a custom state machine. Use a collector when the operation is terminal and produces the pipeline’s final container or summary.

The Gatherer API became final in JDK 24 after previews in JDK 22 and 23. This article uses the final JDK 24–25 API; earlier preview code requires preview flags and may not match the final contract.

Gatherers keep producing a stream

import java.util.stream.Gatherers;
import java.util.stream.IntStream;

public class WindowedTotals {
    public static void main(String[] args) {
        var totals = IntStream.rangeClosed(1, 8)
            .boxed()
            .gather(Gatherers.windowFixed(3))
            .map(window -> window.stream().mapToInt(Integer::intValue).sum())
            .toList();

        System.out.println(totals);
    }
}

Compile and run on JDK 24 or later:

javac WindowedTotals.java
java WindowedTotals

It prints [6, 15, 15]. windowFixed(3) emits two full windows and one shorter final window, then the ordinary map and toList stages continue the pipeline. The JDK 25 built-in gatherer documentation specifies that window lists are unmodifiable and that a positive window size is required.

windowSliding(3) instead emits overlapping encounter-ordered windows. If the input is shorter than the requested window, it emits one short window. Both window implementations may allocate eagerly, so an enormous window size can waste memory even for a small stream.

Choose among the five built-ins

Gatherers provides five operations:

• windowFixed(n) groups non-overlapping batches and can emit a short final batch.
• windowSliding(n) emits overlapping windows, advancing by one element.
• scan(initial, function) emits every successive accumulated value.
• fold(initial, function) emits one final accumulated value after upstream ends.
• mapConcurrent(limit, mapper) applies the mapper on virtual threads with at most the requested concurrency while preserving encounter order.

A scan and a reduction have different shapes:

var running = Stream.of(2, 3, 5)
    .gather(Gatherers.scan(() -> 0, Integer::sum))
    .toList();                    // [2, 5, 10]

var total = Stream.of(2, 3, 5)
    .gather(Gatherers.fold(() -> 0, Integer::sum))
    .findFirst();                 // Optional[10]

Use scan when downstream stages need intermediate states. For a conventional associative final reduction, reduce or collect is usually clearer and can have better parallel behavior than an intrinsically ordered fold.

mapConcurrent is useful for blocking mappers, not as a way to accelerate arbitrary CPU work:

var pages = urls.stream()
    .gather(Gatherers.mapConcurrent(20, client::loadPage))
    .filter(Page::isValid)
    .toList();

The limit bounds mapper concurrency, results retain stream encounter order, and an exception is rethrown as a RuntimeException when that result reaches downstream. Remaining tasks are then canceled on a best-effort basis. Side effects can still occur before cancellation, so make concurrent mappers safe to retry or compensate.

A custom gatherer has four lifecycle parts

The Java 25 Gatherer contract defines an initializer, integrator, optional combiner, and optional finisher. This sequential gatherer emits only values up to and including the first negative input:

import java.util.stream.Gatherer;
import java.util.stream.Stream;

public class TakeThroughNegative {
    static <T> Gatherer<T, ?, T> takeThrough(
            java.util.function.Predicate<? super T> stop) {
        return Gatherer.ofSequential(
            (unused, element, downstream) -> {
                boolean accepted = downstream.push(element);
                return accepted && !stop.test(element);
            }
        );
    }

    public static void main(String[] args) {
        var result = Stream.of(3, 2, -1, 9, 8)
            .gather(takeThrough(n -> n < 0))
            .toList();
        System.out.println(result);
    }
}

It prints [3, 2, -1]. The integrator first respects downstream.push—which can return false when downstream no longer wants elements—then returns false after the stop condition. Returning false tells the gathering implementation to send no more upstream elements to this integrator.

Use ofSequential when a correct combiner is unavailable. A stream may be parallel upstream, but a gatherer with the default combiner must be evaluated sequentially. Claiming parallel support with an invalid combiner can produce results that depend on partition boundaries.

Parallelism requires an algebra, not a flag

For a parallelizable stateful gatherer, each partition receives isolated state. The combiner must merge two completed partition states into a result equivalent to processing their inputs in encounter order. The finisher then emits any buffered end-of-stream output.

This is easy for some transformations and fundamentally awkward for others. A state machine whose decision for the first item in the right partition depends on the last item in the left partition may need boundary metadata and careful output reconciliation. If that proof is difficult, keep the gatherer sequential and document the tradeoff.

Gatherer implementations must not retain their state or Downstream reference beyond the callback invocation that receives it, and must not expose those references to other threads. Violating that lifecycle makes behavior race-prone and breaks the stream implementation’s freedom to partition or short-circuit.

Composition does not remove buffering costs

first.andThen(second) composes two gatherers, and ordinary stream operations can appear before and after gather. Short-circuiting terminal operations such as findFirst can stop a custom integrator that propagates downstream rejection correctly.

Laziness does not mean zero storage. Windows buffer elements, fold retains aggregate state until upstream ends, and an ill-designed custom operation can retain the entire input. For infinite streams, use a gatherer that emits incrementally or short-circuits; a fold cannot emit its single result while upstream never finishes. Pick the operation by output shape, state size, and parallel law—not simply because a gatherer can express it.