Day 16–101 Days of DevOps — Boto3 Concepts(Waiter, Meta, and Paginator)

Welcome to Day 16 of 101 Days of DevOps. The topic for today is Boto3 Concepts(Waiter, Meta, and Paginator). On Day 15 you have learned the basics of Boto3. https://www.101daysofdevops.com/courses/101-days-of-devops/lessons/day-15-introduction-to-boto3/ . Let’s solidify that idea by looking at some other Boto3 concepts.

To view the complete course, please check the below url.

101 Days of DevOps

For more info, register via the below link

Create an Account

YouTube Channel link

Prashant Lakhera

Boto3 Meta

Boto3 Paginator

Boto3 Waiters

Waiters: In the case of Boto3, there are some requests which are not instant. One such example is when you try to start or stop any instance. For such kinds of requests, you can initiate those requests and check back at some later time. But in cases such as EC2, we need to wait for the requests to be complete before we move on to the next part. So AWS SDK provides many waiters that allow you to block the code until the process is complete.

A typical ec2 instance start script with waiter look like this

import boto3
ec2 = boto3.resource("ec2")
instance_id=input("Please enter the instance id: ")
instance=ec2.Instance(instance_id)
print("Starting EC2 instance")
instance.start()
instance.wait_until_running()
print("Your instance is up and running")

In this script, we use a waiter wait_until_running, which runs in the background and loops over 40 times every 5 seconds. The other available waiters are for EC2

wait_until_exists 
wait_until_running 
wait_until_stopped 
wait_until_terminated'

The above code is written using resources. Similarly, we have several waiters available for the client.EC2 — Boto3 Docs 1.17.94 documentation

import boto3
ec2 = boto3.client("ec2")
instance_id=input("Please enter the instance id: ")
print("Starting EC2 instance")
instance=ec2.start_instances(InstanceIds=[instance_id])
waiter = ec2.get_waiter('instance_running')
waiter.wait(InstanceIds=[instance_id])
print("Your instance is up and running")

In the case of a client, the waiter will run 15 loops in the background over the period of 40 seconds.

We can also mix and match where we can use the resource object to start the instance but use client waiters.

import boto3
ec2 = boto3.resource("ec2")
ec2_cli = boto3.client("ec2")
instance_id=input("Please enter the instance id: ")
instance=ec2.Instance(instance_id)
print("Starting EC2 instance")
instance.start()
waiter = ec2_cli.get_waiter('instance_running')
waiter.wait(InstanceIds=[instance_id])
print("Your instance is up and running")

Meta: As discussed in Introduction to Boto3 blog, Boto3 client support all the operation whereas resource doesn’t. Suppose we have the existing code written using the resource, and now if there is a requirement where we need to modify the code to use the operation which is not supported by the resource. Do we need to re-write the code to use the client? The answer is no; we can use meta to access the client directly via resource. In the below use case, we want to list out all the regions supported by EC2, and this operation is not available for resources; hence, in this case, we can access client operation using meta.

>>> import boto3
>>> ec2 = boto3.resource("ec2")
>>> for region in ec2.meta.client.describe_regions()['Regions']:
...     print(region['RegionName'])
... 
eu-north-1
ap-south-1
eu-west-3
eu-west-2
eu-west-1
ap-northeast-3
ap-northeast-2
ap-northeast-1
sa-east-1
ca-central-1
ap-southeast-1
ap-southeast-2
eu-central-1
us-east-1
us-east-2
us-west-1
us-west-2

Paginator: In the case of AWS clients, certain operations return the incomplete result, requiring subsequent requests to obtain the entire result set. The process of sending the subsequent requests to continue where a previous request left off is called pagination. E.g., in the case of the S3 list_object operation, it only returns 1000 objects, so we need to send the subsequent requests with the appropriate marker to retrieve the next page of results.

Advantage: The advantage of the pagination approach is you will get the results more quickly rather than waiting for the entire result.

Let understand this with the help of IAM.

Let try to print the list of IAM user present in my AWS account using the resource.

import boto3
iam_res = boto3.resource("iam")
count=1
for user in iam_res.users.all():
    print(count, user.name)
    count=count+1
python3 test.py |wc -l
     145

As you can see in the output of the above script, I have 145 users present in my account. Let’s re-write the same code using the client.

import boto3
iam_client = boto3.client("iam")
count=1
for user in iam_client.list_users()['Users']:
    print(count, user['UserName'])
    count=count+1
python3 test.py |wc -l
     100

As you can see in the above code, it only returns the first 100 results. If you use the paginator called list_users and paginate through the response, you will see all the IAM users. By default, the page size is set to 100, so it will return the result in two different pages that are 100 on the first page and the remaining user on the next page.

import boto3
iam_client = boto3.client("iam")
paginator = iam_client.get_paginator('list_users')
page_iterator=paginator.paginate()
count=1
for user in page_iterator:
    for username in user['Users']:
        print(count, username['UserName'])
        count+=1
python3 test.py  |wc -l
     145

NOTE: For most of the AWS resources, it returns 50 or 100 results except S3 and the paginator is not available for all AWS services in those cases you need to write your own custom paginator.

So the bottom line if you are trying to retrieve more than one page of results, you need to use a paginator to issue multiple API requests on your behalf.

Please join me with my journey by following any of the below links

• Website: https://101daysofdevops.com/
• Twitter: @100daysofdevops OR @lakhera2015
• Facebook: https://www.facebook.com/groups/795382630808645/
• Medium: https://medium.com/@devopslearning
• GitHub: https://github.com/100daysofdevops/100daysofdevops
• YouTube Channel: https://www.youtube.com/user/laprashant/videos?view_as=subscriber
• Slack: https://join.slack.com/t/100daysofdevops/shared_invite/zt-au03logz-YfDUp_FJF4rAUeDEbgWmsg
• Reddit: r/101DaysofDevops
• Meetup: https://www.meetup.com/100daysofdevops/