️Hiroshi Takeuchi 📟
Asynchronous HTTP request using Tokio and Reqwest in Rust

Asynchronous HTTP request using Tokio and Reqwest in Rust


Rust

As a newcomer to Rust, I’ve completed the Rust book and tackle a LeetCode daily challenge almost every day. Now, it’s time for me to dive into Rust and get my hands dirty! However, due to my limited availability, I don’t have enough bandwidth to dedicate to a large or even medium-sized project at the moment. Therefore, I’m considering starting with something smaller, such as a TODO REST API.

In this post, I’ll be discussing one of the most fundamental yet practical topics in Rust - asynchronous HTTP requests.

Sample code

Tokio and Reqwest

Tokio serves as an asynchronous runtime for Rust. If you have a background in JavaScript, you’ll find it relatively easy to grasp how it works. One notable distinction between Tokio and JavaScript runtime environments, like Node.js, lies in the fact that JavaScript primarily operates within a single thread (although there are exceptions, as explained here . Conversely, Tokio stands out as a multi-threaded asynchronous runtime, enabling exceptional speed and performance.

(By the way multi-threaded node.js programming is also an interesting topic. It might just become a subject of my another blog post in the near future.)

Reqwest is a widely used HTTP client library for Rust. It comes bundled with numerous convenient features. Through my explorations, I’ve discovered that the following features are particularly impressive:

  • JSON serialization/deserialization
  • HTTPS support

Furthermore, Reqwest also works on top of Tokio. Armed with this powerful combination I will explore concurrent HTTP request techniques.

Here is an example of a single HTTP request using reqwest with tokio:

use hyper::StatusCode;

pub async fn fetch(url: &str) -> Result<StatusCode, Box<dyn std::error::Error>> {
    let status_code = reqwest::get(url).await?.status();
    Ok(status_code)
}

And you can perform it as follows:

use std::env;
use std::time::Instant;

use http_client::fetch;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let url = match env::args().nth(1) {
        Some(url) => url,
        None => {
            // Display a friendly message
            println!("Usage: client <url>");
            return Ok(());
        }
    };

    println!("Making an HTTP request to {url}...");
    let start = Instant::now();
    let response = fetch(&url).await?;
    println!(
        "Received a response from {} - Status code: {}",
        url, response
    );
    println!("(The request took {:?})", start.elapsed());
    Ok(())
}

And here is a result:

$ cargo run https://httpbin.org/ip
:
Making an HTTP request to https://httpbin.org/ip...
Received a response from https://httpbin.org/ip - Status code: 200 OK
(The request took 1.938816541s)

Concurrent HTTP Request with tokio::spawn

tokio::spawn is a basic API provided by tokio to create a new asynchronous task. The code below is an example of sending multiple HTTP requests concurrently:

#[tokio::main]
async fn main___() -> Result<(), Box<dyn std::error::Error>> {
    let urls = vec!["https://www.google.com", "https://www.rust-lang.org"];
    let mut join_handles = vec![];

    for url in urls {
        println!("Making an HTTP request to {url}");
        join_handles.push(tokio::spawn(async move {
            match fetch(&url).await {
                Ok(status_code) => {
                    println!("Received a response from {url} - Status code: {status_code}")
                }
                Err(err) => println!("ERROR: {:?}", err),
            }
        }));
    }

    // Ensure that all async tasks have been completed before proceeding
    for handle in join_handles {
        handle.await?;
    }
    Ok(())
}

Restrict Concurrent HTTP Requests with a Semaphore

When it comes to limiting the number of concurrent HTTP requests, the use of a semaphore becomes crucial. Let’s delve into a practical example to illustrate its effectiveness. In the modified code snippet below, a semaphore is employed to ensure that only two requests occur simultaneously:

#[tokio::main]
async fn main_() -> Result<(), Box<dyn std::error::Error>> {
    let semaphore = Arc::new(Semaphore::new(2)); // Limit the number of concurrent requests to 2
    let urls = vec![
        "https://www.google.com",
        "https://www.rust-lang.org",
        "https://www.facebook.com",
        "https://www.linkedin.com",
        "https://www.microsoft.com",
    ];
    let mut join_handles = vec![];

    for url in urls {
        let permit = semaphore.clone().acquire_owned().await?;
        println!("Acquired a permission for {url}");
        join_handles.push(tokio::spawn(async move {
            sleep(Duration::from_secs(2)).await;
            if let Ok(code) = fetch(&url.to_string()).await {
                println!("Received a response from {} - Status code: {}", url, code);
            } else {
                println!("Failed to get a response from {url}");
            }
            // Need to explicitly drop the permission
            drop(permit);
        }));
    }

    // Ensure that all async tasks have been completed before proceeding
    for handle in join_handles {
        handle.await?;
    }
    Ok(())
}

Conclusion

If you want more granular control over HTTP request, check out Hyper . It’s a great library for building custom HTTP clients. I also found a great tutorial on Tokio’s web site. It shows you how to use it by creating a Redis-like client and server.I’ll definitely check it out later.

Thanks for reading ✌️

© 2024 Hiro