Generators and stream processing – Part 1

Generators seem to be a rarely used but quite powerful tool in Javascript. I was recently inspired at work by how useful they can be. I decided to assemble my playground and try to uncover their superpowers. In this article, I will explore its application for iterators. In the following series of posts, my goal is to display more advanced use cases and present asynchronous generators.

Data iteration – one might laugh – there is nothing new to discover. Right? While or for loops. Big of a deal, right? Turns out – wrong! How many times as a developer, when you had to iterate over an object’s values, you wrote code like this in order to access key-value pairs: 

const obj = { name: "Tarjei", age:1123};

const keys = Object.keys(obj);
for(const k of keys){
    console.log(k, obj[k])
}

And that’s okay, but it’s a bit wordy way of expressing that we want to iterate over key-value pairs. Wouldn’t it be much better if we could write our code like this?

const obj { name: "Tarjei", age:1123};

for(const [key, value] of pairs(obj)){
    console.log(key, value)
}

And here is where generators come into play. For-loops in JS is capable of iteration, as long as they are provided with an iterator. An object becomes an iterator when it implements a next() method. That is basically what the generator call is returning. Before we start writing an iterator, it’s worth mentioning one essential syntax keyword of the generator – yield. According to documentation: 

The yield keyword is used to pause and resume a generator function.

Let’s try exploring pairs iterator implementation to understand this a bit better

function* pairs(obj) { 
  const keys = Object.keys(obj);
  while (keys.length > 0) {
    const k = keys.splice(0, 1)[0];
    yield [k, obj[k]];
  }
}

 Starting from the top, generators are defined like regular functions, with an exception of * after the function keyword to indicate this is a generator. Within the body of the generator, yield keyword becomes available. In the body of the generator, we are invoking Object keys to construct key-value pairs. 

Next, we iterate for as long as there are still keys in the array, and we yield the array containing 2 elements [key, value]. This array will be later on visible on iterating variables from the previous snippet. Note that while loop is not finished on yield. The generated value is returned to the caller. Then caller (for-loop) invokes implicitly next() on the iterator and the loop continues until the while-loop inside generator is over.

A fair question might be – why do I need this? Cannot I simply map an array? And here is a key difference. The map function needs to store the keys array as well as the mapped array. Generators process and yield elements one at a time. It will also become clear when we get into unbounded streams, where we don’t have all elements available for mapping.

Getting back to pairs iterator. What about deeply nested object? Can we improve this generator, to print keys path and values? Let’s give it a try.

function* nestedPairs(obj, prefix = null) {
  const keys = Object.keys(obj);
  while (keys.length > 0) {
    const k = keys.splice(0, 1)[0];
    const type = typeof obj[k];
    const wholeKey = prefix ? `${prefix}.${k}` : k;
    if (type === 'object') {
      yield* nestedPairs(obj[k], wholeKey);
    } else {
      yield [wholeKey, obj[k]];
    }
  }
}

There is another elephant in a room! yield*! what it does, instead of generating new value, it’s giving control to a generator that is called. In this case when we have object value, we invoke same generator, with encountered value and we give control to this generator, util it finish its job. We can run this to see, it will iterate over nested objects, and print corresponding path-value pairs.

const obj = {
  name :"test",
  address: {
      line1: "First line of address",
      line2: "Second line of address"
    }
}

for( const [k,v] of nestedPairs(obj)){
  console.log(k,v)
}

The last fun example I came up with is processing an array in a window. Doesn’t seem to be very useful in synchronous processing, but I will elaborate more on it when discussing asynchronous generators.

 

function* inWindow(arr, n = 1, step = n) {
  let i = 0;
  while (arr.length > i) {
    yield arr.slice(i, i + n);
    i += step;
  }
}

This generator takes an array, window size, and step size as arguments. It will yield sliced “windows” when iterated and later on available in loop iteration variables. Have fun exploring these examples. In the next post I will discuss asynchronous generators, and inspiration about sequences I got from Kotlin! Stay tuned.

Adrian Jutrowski

Software Developer