Chapter 8: Iterators and Generators
This article is part of the General-Purpose Python Programming tutorial series, which has been fully completed. Its goal is to take you from zero to mastery of the Python programming language. This tutorial does not just skim over Python’s content; it explains it in a refined and in-depth way, across 5 stages and 25 chapters. So you need to study patiently to truly gain something. Although it does not involve using any frameworks, it gives a comprehensive explanation of operating systems and network communication, and even rewrites some open-source modules and servers. After finishing it, you will not only have mastered the Python programming language, but also gained the ability to learn other programming languages quickly, the ability to read all Python source code without obstacles, and a comprehensive understanding of computers and networks. For complete beginners, it is an excellent textbook for getting into the field of computing and mastering a programming language. For those who already have some Python foundation, I believe this tutorial will take your skills to the next level.
1 Iterators
1. What Is an Iterator
An iterator is a tool for iteration. Iteration is a repetitive process, and each repetition is based on the result of the previous one.
# This is an iteration process: although it repeats, the result is different each time
dict1 = {'x': 1, 'y': 2}
n = 0
for i in dict1:
if n < len(dict1):
print(dict1[i])
n += 1
# This is not an iteration process: it keeps repeating without any change
while True:
print('=------->')2. Iterable Objects
To understand what an iterator really is, you must first be clear about one concept: what is an iterable object? In Python, any object that has a built-in iter method is an iterable object.
# This is not an iterable object
num = 1
# The following are all iterable objects
str1 = 'hello'
list1 = [1, 2, 3]
tup1 = (1, 2, 3)
dict1 = {'x': 1}
set1 = {'a', 'b', 'c'}
file1 = open('a.txt', 'w', encoding='utf-8')3. How to Use Iterators
The return value obtained by calling the iter method of an iterable object is an iterator object.
dict1 = {'x': 1, 'y': 2, 'z': 3}
iter_dict1 = dict1.__iter__() # Names starting and ending with double underscores are called magic methods in Python; they will be covered in detail in the object-oriented chapters. Curious students can look them up on their own
print(iter_dict1.__next__())
print(iter_dict1.__next__())
print(iter_dict1.__next__())
# print(iter_dict1.__next__()) # Stop iteration
set1 = {'a', 'b', 'c'}
iter_set1 = set1.__iter__()
print(iter_set1.__next__())
print(iter_set1.__next__())
print(iter_set1.__next__())
# print(iter_set1.__next__()) # Stop iteration
list1 = [1, 2, 3]
iter_list1 = list1.__iter__()
print(iter_list1.__next__())
print(iter_list1.__next__())
print(iter_list1.__next__())4. Iterable Objects vs. Iterator Objects
(1) Iterable Objects
Iterable objects don’t need to be obtained: Python’s built-in str, list, tuple, dict, set and file are all iterable objects. Their characteristic is that they have a built-in iter method, and calling it returns an iterator object.
(2) Iterator Objects
A file object is itself both an iterable object and an iterator object. Calling the iter method of an iterable object returns an iterator object. The characteristic of an iterator object is that it has a built-in next method, and calling it gets one value from the iterator object; an iterator object also has a built-in iter method, and calling it returns the iterator itself.
str1 = 'hello' # Iterable object
iter_str1 = str1.__iter__() # Iterator object
print(iter_str1.__next__()) # Get one value from the iterator object
print(iter_str1.__iter__() is iter_str1)
print(iter_str1.__iter__().__iter__() is iter_str1)
print(iter_str1.__iter__().__iter__().__iter__() is iter_str1)
# A file itself is both an iterator object and an iterable object
f = open('a.txt', 'r', encoding='utf-8')
print(f.__iter__() is f)
print(f.__next__())
print(f.__next__())
print(f.__next__())
print(f.__next__())
print(f.__next__())5. Pros and Cons of Iterators
(1) Pros
<1> They provide a way to get values without relying on indexes
Suppose you haven’t learned the for loop yet. For iterable objects without indexes, such as set, dict or file, how would you get each value out of them individually?
# Set
set1 = {1, 2, 3, 4, 5, }
iter_set1 = set1.__iter__()
while True:
# try and except are covered in the last chapter of Stage 3 (object-oriented programming); we just use them briefly here
try: # Monitor whether the code block under try raises an exception
print(iter_set1.__next__())
except StopIteration: # Equivalent to an if check: if the exception raised is StopIteration
break
# Dictionary
dict1 = {'x': 1, 'y': 2, 'z': 3}
iter_dict1 = dict1.__iter__()
while True:
try:
print(iter_dict1.__next__())
except StopIteration:
break
# File
# The file has five lines, each containing one number: 1, 2, 3, 4, 5
file1 = open('a.txt', 'r', encoding='utf-8')
iter_file1 = file1.__iter__()
while True:
try:
print(iter_file1.__next__())
except StopIteration:
break
file1.close()
# List
# Iterable objects with indexes can naturally use this too
list1 = [1, 2, 3, 4, 4, 5, 5, 6, 6, 6]
iter_list1 = list1.__iter__()
while True:
try:
print(iter_list1.__next__())
except StopIteration:
break<2> Iterators save more memory
# How range is used
for i in range(0, 10):
print(i)
# item is an iterable object representing all numbers from 0 to 100000000000000000000000
item = range(0, 100000000000000000000000)
iter_item = item.__iter__() # Iterator object
while True:
try:
print(iter_item.__next__())
except StopIteration:
break(2) Cons
<1> Getting values has limitations
Getting values is cumbersome: you can only get them one by one, only moving forward, and only once.
x = [1, 2, 3]
iter_x = x.__iter__()
while True:
try:
print(iter_x.__next__())
except StopIteration:
break
print('第二次=================================>')
# iter_x = x.__iter__() # With this line commented out, nothing is retrieved the second time; like a child on a slide, you have to climb back up
while True:
try:
print(iter_x.__next__())
except StopIteration:
break<2> You can’t get the length with len
Until an iterator object has yielded its last value, you can never know its length.
Life is like a box of assorted chocolates: you never know what flavor your next one will be.
x = [1, 2, 3]
iter_x = x.__iter__()
# print(len(iter_x)) # There is no method to get the length6. How the for Loop Works
The for loop is called an iterator loop. What follows in must be an iterable object. The for loop calls the iter method of the object after in to get an iterator object, then calls the iterator object’s next method to get a return value and assigns it to a variable, over and over, until all values have been retrieved; the for loop then detects the exception and ends the loop automatically.
file1 = open('a.txt', 'r', encoding='utf-8')
for line in file1: # iter_file1=file1.__iter__()
print(line)
for item in {'x': 1, 'y': 2}:
print(item)2 Generators
1. What Is a Generator
We can think of the iterator discussed above as an old hen. In theory, an old hen can have an infinite number of eggs in its belly, but it has to lay them one at a time. Python provides us with several built-in “old hen” data types.
A generator is essentially an iterator; or rather, a generator is a special kind of iterator, because a generator is an iterator that we make ourselves.
2. Two Uses of yield
- yield gives us a way to define custom iterators: you can use the yield keyword inside a function, and the result of calling the function is a generator.
- yield can be used to return values like return. The difference is that return can only return a value once, while yield can return many times, because yield can save the function’s execution state.
Comparing yield and return
# yield
def test_yield():
print('=======>first')
yield 1
print('=======>second')
yield 2
print('=======>third')
yield 3
# When returning with yield, calling the function does not execute the function body; the return value is a generator object
res = test_yield()
print(res) # <generator object test_yield at 0x1078f7660>
print(res.__iter__() is res)
print(res.__next__())
print(res.__next__())
print(res.__next__())
# return
def test_return():
print('=======>first')
return 1 # When returning with return, the function finishes executing
print('=======>second')
return 2
print('=======>third')
return 3
res = test_return()3. Constructing a Generator
If a function contains the yield keyword, calling the function does not execute the function body; the return value is a generator object.
def chicken():
print('=====>first')
yield 1
print('=====>second')
yield 2
print('=====>third')
yield 3
obj = chicken()
print(obj)
print(obj.__iter__() is obj)
print(obj.__next__())
print(obj.__next__())
print(obj.__next__())4. A Custom range
(1) How range Is Used
for i in range(1, 10, 1):
"""
range最多可以接收三个参数，第一个是起始位置，默认值为0，
第二个是结束位置，无默认值，必须指定，
第三个是步长，默认值为1，
如果只传一个位置参数，那就是指的结束位置，
如果传两个位置参数，第一个为起始位置，
第二个为结束位置，
range第一个能取到，最后一个取不到，顾头不顾尾
"""
print(i)(2) A Custom range
# A simple version of range: it only accepts two or three positional arguments, and the start position has no default value
def show_my_range(start, stop, step=1):
n = start
while n < stop:
yield n
n += step
for item in show_my_range(1, 10, 3):
print(item)5. yield Expressions
yield can pause a function, so naturally it can save the function’s running state. We can use the yield expression form to do some interesting things.
def eat(name):
print('【1】%s is ready for eating' % name)
while True:
food = yield # This is the yield expression form; yield can be assigned to a variable
print('【2】%s starts to eat %s' % (name, food))
person1 = eat('Albert')
# The function pauses at the line food = yield
person1.__next__()
# Continue executing the code; since yield has no value, i.e. yield = None, food = None
person1.__next__()yield certainly can’t stay empty forever; there must be a way to pass a value to yield, and that way is send.
def eat(name):
print('【1】%s is ready for eating' % name)
while True:
food = yield
print('【2】%s starts to eat %s' % (name, food))
person1 = eat('Albert')
"""
对于表达式形式的yield，在使用前必先初始化
即第一次必须传None，或者用__next__方法
"""
# person1.send(None) # Initialization, same effect as the next line of code
person1.__next__()
person1.send('蒸羊羔') # send has two functions: 1 passing a value, 2 initialization
person1.send('蒸鹿茸')
person1.send('蒸熊掌')
person1.send('烧素鸭')
person1.close() # After closing, nothing more can be eaten, and no doggy bag either
# person1.send('烧素鹅')
# person1.send('烧鹿尾')We already know that yield can return values, so can it be combined with the yield expression form? If we need to keep a record of what has been eaten, we need this usage.
def eat(name):
print('%s is ready for eating' % name)
food_list = []
while True:
food = yield food_list
print('%s starts to eat %s' % (name, food))
food_list.append(food)
name = 'Albert'
person1 = eat(name)
person1.send(None)
# person1.__next__()
res1 = person1.send('蒸羊羔')
print('%s has eaten %s' % (name, res1))
res2 = person1.send('蒸鹿茸')
print('%s has eaten %s' % (name, res2))
res3 = person1.send('蒸熊掌')
print('%s has eaten %s' % (name, res3))
res4 = person1.send('烧素鸭')
print('%s has eaten %s' % (name, res4))
person1.close() # After closing, nothing more can be eaten, and no doggy bag either
# person1.send('烧素鹅')
# person1.send('烧鹿尾')The approach above helps you better understand how yield executes, but it is obviously a bit verbose. To achieve the same functionality, there is a more concise way to write it.
def eat(name):
print('%s is ready for eating' % name)
food_list = []
while True:
food = yield food_list
print('%s starts to eat %s' % (name, food))
food_list.append(food)
print('%s has eaten %s' % (name, food_list))
person1 = eat('Albert')
person1.send(None)
person1.send('蒸羊羔')
person1.send('蒸鹿茸')
person1.send('蒸熊掌')
person1.send('烧素鸭')
person1.close()3 Procedural Programming
(1) Programming Paradigms
Procedural programming is a way of thinking about programming, also called a programming paradigm. What is explained at the end of this article is one of the programming paradigms, namely procedural programming.
Programming paradigms are like the different schools of martial arts: none is superior to another. For example, I’m from the Ming Cult and you’re all from Emei; it’s not necessarily true that the Ming Cult’s martial arts are better than Emei’s. What makes the difference is the person using them, or the scenario.
The core of the procedural programming mindset is the word 过程 (process). A process is the set of steps for solving a problem: what to do first, what to do next, and what to do last. Writing programs procedurally is like designing an assembly line; it is a mechanical way of thinking. In this tutorial, all the examples we have explained so far have been procedural programming.
(2) Pros and Cons of Procedural Programming
- Pros: complex problems are turned into a flow, and thereby simplified.
- Cons: when you modify one stage, the other stages may also need to be modified; pulling one hair moves the whole body, i.e. poor extensibility.
For most people, the programs they write need constant iteration (whenever requirements change, they have to be modified). If you used procedural programming exclusively, extending the program would become impractical, but for some very small features procedural programming is certainly simpler. Besides that, the counterpart of procedural programming is object-oriented programming, which we will explain later; using object-oriented programming makes program design more complex. In some scenarios, when a project’s design is very large and doesn’t need to change often, perhaps being updated to a new generation only every two or three years, procedural programming might be adopted to keep the program design concise, but object-oriented thinking is still needed. To date, the Windows operating system has roughly seventy million lines of code (you might not write that much code in your entire career), using a large amount of C++ and a small amount of assembly language. When writing programs we often need to combine the two, but in the final analysis, looking at the whole, program design is still procedural: there must be a clear order of steps. You will understand this more deeply once you learn object-oriented programming.