Using generators for working with paginated APIs

In this post, I will examine a solid use case for JavaScript async generators: working with paginated APIs.

Generator functions produce an iterable object that can be used to abstract the code handling the pagination from the code that is consuming the data. In the context of handling data from network requests, we will be working with the async counterparts.

Consider the following example API which returns a JSON array of objects:

GET /records?count=X&skip=Y

The count parameter controls the size of the set and skip controls how many preceding items are skipped. In combination, these provide pagination of data, allowing the consumer to comfortably work with potentially very large data sets.

Note that this is a simplified example and requires a stable data set. In production, you would be more likely to use a safer cursor-based API, but the generator pattern will be equally applicable there.

Consider a scenario that involves running some sort of business logic across every record returned by the API. Fetching the data involves repeatedly calling the API until all data has been fetched.

const PAGE_SIZE = 10;

async function getRecordsPage(count = 100, skip = 0) {
  const res = await fetch(`/records?count=${count}&skip=${skip}`);
  if (!res.ok) {
    throw new Error('Error response');
  }

  const body = await res.json();
  if (!Array.isArray(body)) {
    throw new Error('Bad response');
  }

  return body;
}

let skip = 0;
let page = await getRecordsPage(PAGE_SIZE, skip);

while (page.length) {
  for (const record of page) {
    doSomething(record);
  }

  skip += page.length;
  page = await getRecordsPage(PAGE_SIZE, skip);
}

This approach works but maintainability suffers with the pagination plumbing mixed with the business logic. We can improve this by using a generator:

async function* getRecords() {
  let skip = 0;
  let page = await getRecordsPage(PAGE_SIZE, skip);

  while (page.length) {
    for (const record of page) {
      yield record;
    }

    skip += page.length;
    page = await getRecordsPage(PAGE_SIZE, skip);
  }
}

for await (const record of getRecords()) {
  doSomething(record);
}

Note the use of the * symbol to indicate a generator. Because fetching pages is asynchronous, we use for await...of to consume the data.

The result is a clean separation of network request, pagination handling, and business logic.

If needed, a further improvement could be to abstract the pagination logic and make it reusable:

async function* getIterable<T>(
  count: number,
  getPage: (count: number, skip: number) => Promise<T[]>
) {
  let skip = 0;
  let page = await getPage(count, skip);

  while (page.length) {
    for (const item of page) {
      yield item;
    }

    skip += page.length;
    page = await getPage(count, skip);
  }
}

function getRecords() {
  return getIterable(PAGE_SIZE, getRecordsPage);
}

In summary, consuming paginated APIs is one of the use cases where generators are a perfect fit and allow for an elegant separation of concerns between API request code and application business logic.