Boto3 Concepts(Waiter, Meta, and Paginator)

To read the complete blog

Boto3 Concepts(Waiter, Meta, and Paginator)

In the last blog, you understand the concept of Boto3. Let’s solidify that idea by looking at some other Boto3 concepts.

Introduction to AWS Boto3

Boto3 Meta

Boto3 Paginator

Boto3 Waiters

Waiters: In the case of Boto3, there are some requests which are not instant. One such example is when you try to start or stop any instance. For such kinds of requests, you can initiate those requests and check back at some later time. But in cases such as EC2, we need to wait for the requests to be complete before we move on to the next part. So AWS SDK provides many waiters that allow you to block the code until the process is complete.

A typical ec2 instance start script with waiter look like this

import boto3

ec2 = boto3.resource("ec2")

instance_id=input("Please enter the instance id: ")
instance=ec2.Instance(instance_id)
print("Starting EC2 instance")
instance.start()
instance.wait_until_running()
print("Your instance is up and running")

In this script, we use a waiter wait_until_running, which runs in the background and loops over 40 times every 5 seconds. The other available waiters are for EC2

wait_until_exists 
wait_until_running 
wait_until_stopped 
wait_until_terminated'

The above code is written using resources. Similarly, we have several waiters available for the client.

EC2 - Boto3 Docs 1.17.94 documentation

import boto3

ec2 = boto3.client("ec2")

instance_id=input("Please enter the instance id: ")
print("Starting EC2 instance")
instance=ec2.start_instances(InstanceIds=[instance_id])
waiter = ec2.get_waiter('instance_running')
waiter.wait(InstanceIds=[instance_id])
print("Your instance is up and running")

In the case of a client, the waiter will run 15 loops in the background over the period of 40 seconds.

We can also mix and match where we can use the resource object to start the instance but use client waiters.

import boto3

ec2 = boto3.resource("ec2")
ec2_cli = boto3.client("ec2")

instance_id=input("Please enter the instance id: ")
instance=ec2.Instance(instance_id)
print("Starting EC2 instance")
instance.start()
waiter = ec2_cli.get_waiter('instance_running')
waiter.wait(InstanceIds=[instance_id])
print("Your instance is up and running")

Meta: As discussed in Introduction to Boto3 blog, Boto3 client support all the operation whereas resource doesn’t. Suppose we have the existing code written using the resource, and now if there is a requirement where we need to modify the code to use the operation which is not supported by the resource. Do we need to re-write the code to use the client? The answer is no; we can use meta to access the client directly via resource. In the below use case, we want to list out all the regions supported by EC2, and this operation is not available for resources; hence, in this case, we can access client operation using meta.

>>> import boto3
>>> ec2 = boto3.resource("ec2")
>>> for region in ec2.meta.client.describe_regions()['Regions']:
...     print(region['RegionName'])
... 
eu-north-1
ap-south-1
eu-west-3
eu-west-2
eu-west-1
ap-northeast-3
ap-northeast-2
ap-northeast-1
sa-east-1
ca-central-1
ap-southeast-1
ap-southeast-2
eu-central-1
us-east-1
us-east-2
us-west-1
us-west-2

Paginator: In the case of AWS clients, certain operations return the incomplete result, requiring subsequent requests to obtain the entire result set. The process of sending the subsequent requests to continue where a previous request left off is called pagination. E.g., in the case of the S3 list_object operation, it only returns 1000 objects, so we need to send the subsequent requests with the appropriate marker to retrieve the next page of results.

GitHub

100daysofdevops/100daysofdevops

Please join me with my journey by following any of the below links

• Website: https://101daysofdevops.com/
• Twitter: @100daysofdevops OR @lakhera2015
• Facebook: https://www.facebook.com/groups/795382630808645/
• Medium: https://medium.com/@devopslearning
• GitHub: https://github.com/100daysofdevops/100daysofdevops
• Slack: https://join.slack.com/t/100daysofdevops/shared_invite/enQtNzg1MjUzMzQzMzgxLWM4Yjk0ZWJiMjY4ZWE3ODBjZjgyYTllZmUxNzFkNTgxZjQ4NDlmZjkzODAwNDczOTYwOTM2MzlhZDNkM2FkMDA
• YouTube Channel: https://www.youtube.com/user/laprashant/videos?view_as=subscriber