Building Iterator Pipelines with itertools

Intermediate

Python already provides for loops, comprehensions, and generators. The itertools module adds reusable building blocks for common iteration patterns. Most tools return iterators, so values can be produced as they are requested instead of building an intermediate list.

Keep iteration lazy and combine sources

Use chain.from_iterable() when you have an iterable of iterables and want to process their items one after another. It flattens one level of nesting in sequence; it does not recursively flatten nested structures. Because it yields items as they are requested, the grouped input does not need to be copied into one large list first.

from itertools import chain
groups = [[1, 2], [3]]
print(list(chain.from_iterable(groups)))?

Take a bounded part of an iterator

islice() selects positions from an iterable using start, stop, and step values similar to slicing. Unlike sequence slicing, it also works with iterators that do not support indexing. It consumes the source as it advances, and the stop position is excluded. Use a finite stop when taking values from an infinite iterator such as count().

from itertools import count, islice
print(list(islice(count(3), 4)))?

Group an iterable into batches

batched(iterable, n) yields tuples containing up to n items each. The final tuple can be shorter when the input length is not divisible by n. The strict=True option, available since Python 3.13, raises ValueError instead when a full final batch cannot be formed. Batching is useful for chunked processing without manually managing an index.

from itertools import batched
print(list(batched([1, 2, 3, 4, 5], 2)))?

Compare neighboring values with pairwise

pairwise() yields overlapping pairs of adjacent items: the first and second, then the second and third, and so on. It is useful for differences between consecutive readings or checking transitions. It does not produce every possible pair; use combinations() when every unordered pair is needed.

from itertools import pairwise
print(list(pairwise(["a", "b", "c", "d"])))?

Calculate running results with accumulate

accumulate() yields each intermediate accumulated result, not just the final one. With its default operation it produces running totals; with a two-argument function such as max it can produce a running maximum. This differs from sum(), which returns only one final total.

Choose between products, combinations, and permutations

product() generates a Cartesian product, such as every size-and-color pairing. combinations() chooses positions without considering order; permutations() chooses ordered arrangements. These tools operate on positions, so equal input values in different positions can still produce repeated-looking outputs. The number of results can grow quickly, and product() stores input pools, so use these tools with finite inputs and check the size of the result space first.

from itertools import combinations
print(list(combinations(["A", "B", "C"], 2)))?

Keep the choice tied to the task: chain combines streams, islice bounds consumption, batched groups items, pairwise compares neighbors, accumulate exposes running results, and the combinatorial tools generate defined sets of selections. These are building blocks for iteration, not replacements for every loop or comprehension.

Group consecutive values with groupby

groupby(iterable, key=...) yields each key together with an iterator for the consecutive items that have that key. It starts a new group whenever the key changes; it does not collect matching values from separate parts of the input the way a database GROUP BY operation does.

If all records with the same key should be together, sort the records by the same key function before calling groupby(). Each group is an iterator that shares the source with the outer groupby iterator, so consume a group inside the loop or convert it to a list when you need to keep its values.

from itertools import groupby
print([(key, list(group)) for key, group in groupby("ABA")])?