Yos Riady software craftsman 🌱

Python Generators and the Yield keyword

When we call a normal Python function, execution starts at function’s first line and continues until a return statement, exception, or the end of the function (which is seen as an implicit return None) is encountered. Once a function returns control to its caller, that’s it - any work done by the function and stored in local variables is lost. A new call to the function creates everything from scratch.

There are times, though, when it’s beneficial to have the ability to create a “function” which, instead of simply returning a single value, is able to yield a series of values. To do so, such a function would need to be able to “save its work,” so to speak. Python’s yield keyword allows you to di exactly this: temporarily transfer control to another code block.

The key to understanding the yield keyword is to understand that yield triggers a lazy evaluation of your return values (using generators.)

Take a look at the following code:

def createGenerator():
    mylist = range(3)
    for i in mylist:
        yield i*i

mygenerator = createGenerator() # create a generator

print(mygenerator) # mygenerator is an object!

for i in mygenerator:
    print(i)

Let’s first define a few terms:

Iteration is a process implying iterables (implementing the __iter__() method) and iterators (implementing the __next__() method).

Iterables are any objects you can get an iterator from. Iterators are objects that let you iterate on iterables.

A generator is an iterable which can only be iterated through once because the values in a generator are lazily computed on the fly. Let’s see an example:

By using yield instead of return in a for loop, we can run the code until it hits yield, then it’ll return the first value of the loop. Then, each other call will run the loop you have written in the function one more time, and return the next value, until there is no value to return. The generator is considered empty once the function runs but does not hit yield anymore. It can be because the loop had come to an end, or because you do not satisfy a “if/else” anymore.

Similarly, in a for x in mylist loop, Python does two things:

  • Retrieves an iterator for my list, an object with a next() method

  • Uses the iterator to loop over its elements, by calling next() until there are no more values in the iterator

Many Python objects implements the iterator protocol. In a user defined class, you can implement the __iter__() method to make instances of your class iterable. This method should return an iterator. An iterator is an object with a next() method.

Furthermore, yield works with the for..in syntax, because the for..in syntax expects an iterable, and a generator is an iterable.

There are a few key ideas I hope you take away from this post:

  • generators are used to yield a series of values
  • yield saves the “index” of a generator
  • A generator is just a special type of iterator
  • Like iterators, we can get the next value from a generator using next() for gets values by calling next() implicitly

Author

Yos is a software craftsman based in Singapore.

📬 Subscribe to my newsletter

Get notified of my latest articles by providing your email below.


Going Serverless book

Interested to find out more about serverless? Going Serverless teaches you how to build scalable applications with the Serverless framework and AWS Lambda. You'll learn how to design, develop, test, deploy, and secure Serverless applications from planning to production.

Learn More →