Introduction
In the field of web development, the ability to retrieve data from APIs plays a vital role in creating dynamic and interactive applications. However, the traditional synchronous API calls, which execute one request at a time, may cause delays and hinder the performance of your application when managing multiple requests or large data sets.
This is where asynchronous API fetching comes into play. With asynchronous API fetching, also referred to as non-blocking or concurrent API fetching, you can handle multiple API requests simultaneously without interfering with the main thread, allowing your application to remain responsive, even when handling real-time data or large amounts of data. This approach can significantly improve your application’s efficiency and overall performance.
Prerequisites
To follow this tutorial, you’ll need a basic understanding of Python, including data structures and functions. Additionally, you’ll need to install the aiohttp library, which provides an asynchronous HTTP client for Python. You can install it using the command pip install aiohttp.
Understanding jsonplaceholder.typicode.com
The tutorial you are following is based on the jsonplaceholder.typicode.com API, a web service that offers a comprehensive collection of mock JSON data. This API is specifically designed to help developers test their applications and learn how to integrate APIs in their projects. The API provides endpoints for various resources, including users, posts, and todos, allowing developers to access a diverse range of data.
For instance, the users endpoint returns a list of users with their details such as name, email, and phone number. By accessing this endpoint, developers can retrieve information about specific users and use it in their applications. Similarly, the posts endpoint provides a list of posts with their title, body, and author. This endpoint is beneficial for developers who want to retrieve blog posts or create their blogging platforms.
The todos endpoint is another endpoint that developers can use to retrieve a list of tasks with their title, status, and user ID. This endpoint is ideal for developers who want to create a to-do list application or integrate tasks into their existing applications. By using this API, developers can focus on building their applications without worrying about the data source.
Overall, the jsonplaceholder.typicode.com API is an excellent resource for developers who want to learn how to use APIs in their applications. It provides a rich collection of data and endpoints, making it easier for developers to test their applications and learn how to use APIs effectively.
Code Breakdown
import aiohttp
import asyncio
# Function to fetch posts for a user
async def get_posts(session, user_id):
url = f"https://jsonplaceholder.typicode.com/users/{user_id}/posts"
async with session.get(url) as response:
return await response.json()
# Function to fetch todos for a user
async def get_todos(session, user_id):
url = f"https://jsonplaceholder.typicode.com/users/{user_id}/todos"
async with session.get(url) as response:
return await response.json()
# Function to fetch all users
async def get_users():
try:
# Fetch users' data from the API
async with aiohttp.ClientSession() as session:
async with session.get('https://jsonplaceholder.typicode.com/users') as response:
users = await response.json()
# Map users data to the desired format
formatted_users = [
{
'id': user['id'],
'username': user['username'],
'email': user['email'],
'phone': user['phone']
}
for user in users
]
# Fetch and add posts and todos for each user
users_with_posts_and_todos = []
for user in formatted_users:
posts = await get_posts(session, user['id'])
todos = await get_todos(session, user['id'])
# Map posts and todos to the desired format
formatted_posts = []
for post in posts:
formatted_posts.append({
'title': post['title'],
'body': post['body']
})
formatted_todos = []
for todo in todos:
formatted_todos.append({
'title': todo['title']
})
# Add posts and todos to the user object
user_with_posts_and_todos = {
**user,
'posts': formatted_posts,
'todos': formatted_todos
}
users_with_posts_and_todos.append(user_with_posts_and_todos)
# return all users with their posts and todos in a list
return users_with_posts_and_todos
except Exception as error:
print('Error:', error)
raise error
# Function to print users' data to the console
async def print_users_data():
try:
users = await get_users()
print(users)
except Exception as error:
print('Error:', error)
# Main function
async def main():
await print_users_data()
# Entry point
if __name__ == '__main__':
asyncio.run(main())The code in this tutorial consists of three main functions: get_posts, get_todos, and get_users. The get_posts and get_todos functions fetch posts and todos for a given user ID, respectively. The get_users function fetches all users from the API and returns an array of users with their corresponding posts and todos.
Asynchronous Functions in Python
Python provides a way to define and handle asynchronous functions using the async and await keywords. Asynchronous functions are designed to be non-blocking, meaning they can run concurrently without hindering the main thread.
To define an asynchronous function in Python, we use the async keyword to create a coroutine. Coroutines are asynchronous functions that can be run concurrently with other coroutines. Inside a coroutine, we use the await keyword to pause the execution of the current coroutine until the awaited expression is complete. This allows other coroutines to be executed in the meantime without blocking the main thread.
The await keyword can be used with any awaitable object, which is an object that has an await() method that returns an iterator. This includes coroutines, generators, and other objects implementing the await() method.
In summary, using asynchronous functions and the async and await keywords in Python can significantly enhance the performance of programs that require concurrent execution of multiple tasks.
aiohttp: An Asynchronous HTTP Client for Python
The aiohttp library is a powerful tool for making asynchronous HTTP requests in Python. It provides a high-level abstraction for creating and managing asynchronous client sessions, making HTTP requests, and handling asynchronous responses. It offers several advantages over the standard requests library, including:
- Concurrent Request Handling:
aiohttpcan handle multiple HTTP requests simultaneously, improving application performance and responsiveness. - Non-blocking I/O:
aiohttprelies on asynchronous I/O mechanisms, allowing the main thread to continue processing while waiting for network responses. - Efficient Error Handling:
aiohttpprovides robust error handling mechanisms to gracefully handle network errors and unexpected conditions. - Coroutine-Based Programming:
aiohttpintegrates seamlessly with Python’sasynciolibrary, enabling coroutine-based asynchronous programming.
If you’re working with asynchronous API fetching in Python, the aiohttp library is an essential tool that can significantly enhance your application’s performance and responsiveness.
Check docs.aiohttp.org for more information.
Retrieving Users’ Data, using aiohttp
The get_users function is a useful piece of code that allows users to retrieve data from an API. It uses the aiohttp library, which is a powerful tool for making HTTP requests asynchronously.
When the function is called, it creates an asynchronous client session, which is a way of handling multiple HTTP requests simultaneously. The session sends an HTTP request to the API endpoint for all users, which returns a response containing the data in JSON format.
The function then parses the JSON response to extract the user data and stores it in a list. This list can then be used to access and manipulate the user data in various ways, depending on the needs of the user.
Overall, this solution is a great reuseable way to fetch data from an API quickly and efficiently, making it an indispensable tool for developers and data analysts alike.
Retrieving Posts and Todos
The get_posts and get_todos functions are similar to the get_users function. They create an asynchronous client session, make HTTP requests to the appropriate API endpoints, and parse the responses as JSON. The post and todo data is then stored in separate lists for each user.
How to Map Data in Python
Mapping is a typical programming operation involving transforming data from one format to another. In the context of the get_users function, mapping converts the raw user data retrieved from the API into a structured format that’s easier to work with.
# Map users data to the desired format
formatted_users = [
{
'id': user['id'],
'username': user['username'],
'email': user['email'],
'phone': user['phone']
}
for user in users
]The formatted_users variable represents a list of user objects. Each user object contains key-value pairs representing the user’s ID, username, email, and phone. The code iterates through the list of raw user data and creates a new user object for each user. The respective key-value pairs from the raw data are extracted and assigned to the corresponding keys in the new user object for each user.
Extracting key-value pairs and assigning them to corresponding keys in a new object is essentially a mapping operation. It involves transforming the raw data into a structured format that’s more organized and easier to manipulate.
# Add posts and todos to the user object
user_with_posts_and_todos = {
**user,
'posts': formatted_posts,
'todos': formatted_todos
}
users_with_posts_and_todos.append(user_with_posts_and_todos)The ** operator plays a crucial role in this mapping operation. It’s used to unpack the user dictionary into keyword arguments when creating a new user object. This allows the user’s data to seamlessly incorporate into the new object without explicitly copying the entire dictionary.
In summary, mapping is a fundamental data manipulation technique that allows us to transform data from one format to another. It’s beneficial in API fetching scenarios where we need to convert raw data into a structured format for further processing.
Handling Errors with try
The try block is used to handle potential errors that may occur during API calls or data manipulation. The except block catches the errors and prints the error message.
asyncio, How to use it and Why
asyncio is a crucial tool for asynchronous programming in Python, providing features to create asynchronous functions, manage coroutines, and handle asynchronous operations like network communication and file I/O.
The primary benefits of using asyncio include:
- Enhanced Performance: Asynchronous programming enables multiple tasks to run concurrently, reducing the time it takes to complete a series of operations.
- Improved Responsiveness: By handling multiple requests simultaneously, asynchronous applications can respond to user interactions more quickly, providing a better user experience.
- Resource Optimization: Asynchronous programming allows the main thread to focus on other tasks while waiting for asynchronous operations to complete, improving resource utilization.
- Scalability: Asynchronous applications can handle increasing workloads more efficiently as they can handle more concurrent requests without blocking the main thread.
In summary, asyncio is an indispensable tool for building efficient, responsive, and scalable applications in Python. It empowers developers to handle multiple tasks simultaneously, enhance application performance, and improve user experience.
Why not requests?
The requests library is a popular choice for making synchronous HTTP requests in Python. However, for asynchronous API fetching, using asynchronous requests is generally preferred over synchronous requests.
Synchronous requests are executed one at a time, which can lead to slower performance and responsiveness. When making multiple requests, synchronous requests can block the main thread, preventing other tasks from being executed. This can cause problems with user interactions and overall application performance.
In the context of the given code, using asynchronous requests allows the get_users function to fetch all users from the API and their corresponding posts and todos simultaneously. This prevents the main thread from being blocked and ensures a more responsive and performant application.
Conclusion
Asynchronous API fetching is a powerful technique for improving the performance and responsiveness of your Python applications. By leveraging asynchronous functions and libraries like aiohttp and asyncio, you can handle multiple API requests simultaneously, reducing the time it takes to fetch and process data, and enhancing the user experience.
The complete code is available on GitHub: https://github.com/rezabs/fetch-api-async-python
