Is there a way to form batched requests using python requests module ? I want to send multiple homog-CodePudding

Is there a way to form batched requests using python requests module ? I want to send multiple homogenous http API requests in a single POST request. I am trying to use the GCP documentation https://cloud.google.com/dns/docs/reference/batch?hl=en_US&_ga=2.138235123.-2126794010.1660759555 to create multiple DNS records in a single POST request. Can anyone help with a simple example of how this could be achieved using python requests module ?

This is how the sample POST request will look like:

POST /batch/farm/v1 HTTP/1.1
Authorization: Bearer your_auth_token
Host: www.googleapis.com
Content-Type: multipart/mixed; boundary=batch_foobarbaz
Content-Length: total_content_length

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item1:[email protected]>

GET /farm/v1/animals/pony

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item2:[email protected]>

PUT /farm/v1/animals/sheep
Content-Type: application/json
Content-Length: part_content_length
If-Match: "etag/sheep"

{
  "animalName": "sheep",
  "animalAge": "5"
  "peltColor": "green",
}

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item3:[email protected]>

GET /farm/v1/animals
If-None-Match: "etag/animals"

--batch_foobarbaz--

Basically; the main intention here is to not overload the remote API with multiple http requests causing the rate limit throttling but instead use batched http requests so that the remote API gets only a single batched request embedded with multiple requests in the form of parts.

CodePudding user response：

No. HTTP doesn't work that way. You can send multiple requests simultaneously using threads, but you can't send multiple POSTs through a single request.

CodePudding user response：

According to the doc, the individual batch HTTP requests are supposed to go in the body of the request. You can try building the requests up manually. Like so:

import requests

body = """
--batch_foobarbaz
Content-Type: application/http
Content-ID: <item1:[email protected]>

GET /farm/v1/animals/pony

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item2:[email protected]>

PUT /farm/v1/animals/sheep
Content-Type: application/json
Content-Length: part_content_length
If-Match: "etag/sheep"

{
  "animalName": "sheep",
  "animalAge": "5"
  "peltColor": "green",
}

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item3:[email protected]>

GET /farm/v1/animals
If-None-Match: "etag/animals"

--batch_foobarbaz--
"""

response = requests.post(
    "https://www.googleapis.com/batch/API/VERSION/batch/form/v1",
    data=body,
    headers={
        'Authorization': 'Bearer your_auth_token',
        'Host': 'www.googleapis.com',
        'Content-Type': 'multipart/mixed; boundary=batch_foobarbaz'
    }
)

Of course, you'd have to build the individual requests manually:

Content-Type: application/http
Content-ID: <item1:[email protected]>

GET /farm/v1/animals/pony

Unless you can find a library that can construct HTTP requests according to the HTTP/1.1 RFC. I can't think of one off the top of my head.