How to Send a POST with Python Requests?

HTTP POST requests play a crucial role in web development, allowing clients to send data to servers efficiently. Unlike GET requests, which retrieve data, POST requests are primarily used for tasks like submitting form data, uploading files, and interacting with APIs that require user input. Understanding how to send a Python session POST request using Python's requests library is essential for developers working with web applications, web services, APIs, and automation scripts.

Python requests library provides a simple and intuitive way to handle HTTP requests, making it a popular choice among developers. In this guide, we'll explore how to construct and send POST requests using the requests module, inspect responses, and handle JSON data effectively. Additionally, we'll discuss using sessions for maintaining connections and how proxies can improve performance.

How to build JSON POST request with Requests

Set the Request method to POST

To send a POST request using the Python requests library, you need to use the requests.post() method. This method allows you to send data to a server or API and is useful for tasks such as submitting form data or uploading files.

Set the POST data

POST requests typically include data in the request body. This data can be sent as form-encoded key-value pairs or JSON. Here’s an example of sending form data with the correct headers:

import requests

url = "https://httpbin.org/post"
data = {"username": "testuser", "password": "securepassword"}
headers = {"Content-Type": "application/x-www-form-urlencoded"}

response = requests.post(url, data=data, headers=headers)
print("Form Data Response:\n", response.text)

Setting the Content-Type header ensures the server correctly interprets the request as form data.

Set the POST headers

Headers provide additional information about the request, such as content type, authentication, and user agent. Setting the correct custom headers ensures proper communication with the server. 

Here's how you can add headers to your POST request:

headers = {"Content-Type": "application/x-www-form-urlencoded"}

response = requests.post(url, data=data, headers=headers)

In the example above, we specify Content-Type: application/x-www-form-urlencoded for form data. If you’re sending JSON data, you can leave requests to handle the header automatically when you use the json parameter.

POST JSON data

When sending JSON data, use the json parameter instead of data to ensure the request body is formatted correctly:

json_data = {"name": "John", "age": 30}

response = requests.post(url, json=json_data)

print("JSON Data Response:\n", response.text)

Using json= automatically sets Content-Type: application/json, so you don’t need to set it manually.

Handling JSON data

When working with APIs, handling JSON format responses is crucial. Here’s how to inspect and parse JSON responses while ensuring proper error handling:

try:

    response = requests.post(url, json=json_data)

    response.raise_for_status()  # Raises HTTPError for 4xx, 5xx responses

    json_response = response.json()  # Parses JSON response

    print("Parsed JSON Response:\n", json_response)


except requests.exceptions.HTTPError as http_err:

    print(f"HTTP error occurred: {http_err}")

except requests.exceptions.RequestException as req_err:

    print(f"Request error: {req_err}")

except ValueError:

    print("Response is not valid JSON")

Using raise_for_status() helps catch failed requests early, while except ValueError ensures your script doesn’t crash if the response isn’t valid JSON.

Using sessions in Python Requests

Using a session allows you to persist parameters across multiple requests, reducing redundant authentication and improving efficiency. Here’s an example:

with requests.Session() as session:

    # Set persistent headers
    session.headers.update({"Authorization": "Bearer YOUR_TOKEN"})


    # Add cookies that will persist across requests
    session.cookies.update({"session_id": "abc123"})


    # First request with initial setup
    response_1 = session.post(url, json=json_data)
    print("Session Response 1:\n", response_1.json(), "\n")


    # Update the user agent for subsequent requests
    session.headers.update({"User-Agent": "custom_user_agent"})


    # Second request automatically uses all headers and cookies
    response_2 = session.post(url, json=json_data)
    print("Session Response 2:\n", response_2.json())

Sessions are particularly useful when dealing with APIs that require authentication, as they maintain cookies and headers across multiple requests. Using a with block ensures the session is properly closed after use.

Enhancing performance with proxies

A proxy acts as an intermediary between your request and the target server. Instead of sending requests directly from your IP, a proxy routes them through another IP, providing anonymity and bypassing restrictions. They can help to avoid rate limits as well as help you access geo-restricted content. Additionally, proxies hide your real IP address, reducing the risk of being blocked or flagged by the server.

To use a residential proxy with the requests library, define the proxy settings as follows:

import requests

USER = "proxy_username"

PASS = "proxy_password"

proxies = {

    "http": f"https://customer-{USER}:{PASS}@pr.oxylabs.io:7777",

    "https": f"https://customer-{USER}:{PASS}@pr.oxylabs.io:7777"

}

response = requests.post(

    "https://httpbin.org/post",

    proxies=proxies,

    json={"name": "John", "age": 30}

)

print(response.text)

There are different types of proxy servers available, each serving a unique purpose. Datacenter proxies are known for their speed but can be easily detected. On the other hand, a residential proxy provides real IPs from ISPs, making them much harder to identify. While some users look for free proxies, they can vary in quality – though some reliable providers do offer them as a trial or limited service. For the best performance and security, choosing the best proxy provider ensures a balance of speed, anonymity, and reliability.

Using a good proxy service ensures smooth and uninterrupted API interactions, especially for automation and data extraction tasks.

Conclusion

Mastering HTTP POST requests using Python’s requests library is essential for web developers and API consumers. We’ve covered the key aspects, including setting up POST requests, sending JSON data, handling responses, and maintaining sessions for efficient communication.

Whether you're scraping websites, working with APIs, or automating form submissions, understanding these techniques will make your applications more reliable and efficient. 

Take a look at our other blog posts to learn more about web scraping as a whole – whether you’re looking for how to run a Python script, how to run Python on Windows, or how to rotate proxies on Python, we’ve got you covered.