Implementing Rotating Proxy IPs in Python: A Best Practice Guide

In scenarios such as web crawlers, data scraping, and API requests, the use of proxy IP is very important. Through proxy IP, we can hide the real IP address, avoid being banned by the target server, and improve the success rate of requests.

However, the use of a single proxy IP is often subject to various restrictions, such as request frequency, number of requests, etc. Therefore, rotating proxy IP becomes an effective solution. This article will introduce how to use Python to implement proxy IP rotation and give best practice guidelines.

1. Acquisition and management of proxy IP

Before implementing proxy IP rotation, we first need to obtain a certain number of proxy IPs and manage them. Proxy IP can be obtained through purchase, free acquisition, etc.

Purchasing a proxy IP usually results in a more stable and efficient proxy service, but the cost is relatively high; a free proxy IP costs less, but may not be as stable and efficient as a paid proxy.

After obtaining the proxy IP, we need to store them for subsequent use. A simple way is to store the list of proxy IPs in a text file, one proxy IP address and port number per line. Another way is to use a database for storage, which makes it easier to query, add, delete and other operations.

2. Rotation strategy of proxy IP

The rotation strategy of proxy IP is the core of realizing proxy IP rotation. Common rotation strategies include the following:

Random rotation: randomly select a proxy IP from the proxy IP list for use. This strategy is simple and easy to implement, but may result in some proxy IPs being overused, while others are rarely used.

Sequential rotation: Use each proxy IP in sequence in the order of the proxy IP list. This strategy ensures that each proxy IP is used evenly, but may cause the request to be recognized and blocked by the target server.

Weight-based rotation: Set a weight for each proxy IP and rotate based on the weight. The weight can be set based on factors such as the quality and stability of the proxy IP. This strategy allows for more flexible control over the use of proxy IPs.

Rotation based on the number of failures: When a proxy IP request fails, increase its failure count. When the number of failures reaches a certain threshold, it will be removed from the proxy IP list or its weight will be reduced. This strategy can reduce request failures caused by proxy IP quality issues.

3. Python implements proxy IP rotation

In Python, we can use the requests library to send HTTP requests and use the proxy IP by setting the proxies parameter. Below is a simple sample code that demonstrates how to use a random rotation strategy to implement proxy IP rotation:

import requests

import random

# Read proxy IP list

with open('proxies.txt', 'r') as f:

     proxies = [line.strip().split(':') for line in f.readlines()]

# Randomly select a proxy IP

def get_random_proxy():

     return random.choice(proxies)

#Send HTTP request

def send_request(url):

     proxy = get_random_proxy()

     proxies = {

         'http': f'http://{proxy[0]}:{proxy[1]}',

         'https': f'https://{proxy[0]}:{proxy[1]}'

     }

     try:

         response = requests.get(url, proxies=proxies)

         return response.text

     except requests.RequestException as e:

         print(f'Request failed: {e}')

         return None

# Usage example

url = 'http://example.com'

response = send_request(url)

print(response)

In the above code, we first read the proxy IP list file and parse it into a list containing the proxy IP address and port number. Then, we defined a get_random_proxy function to randomly select a proxy IP from the proxy IP list.

In the send_request function, we call the get_random_proxy function to obtain a proxy IP and set it as the proxies parameter of the requests library. We then send an HTTP request and return the response content. If the request fails, we print an error message and return None.

4. Best Practice Guidelines

Regularly check the validity of the proxy IP: The proxy IP may become invalid due to various reasons, such as being blocked by the target server, expired, etc. Therefore, we need to regularly check the validity of proxy IPs, promptly remove invalid proxy IPs, and add new valid proxy IPs.

Set a reasonable request interval and frequency: too frequent requests may cause the proxy IP to be blocked by the target server. Therefore, we need to set a reasonable request interval and frequency to avoid excessive pressure on the target server.

Exception handling and retry mechanism: During network requests, you may encounter various abnormal situations, such as connection timeout, request failure, etc. We need to handle these exceptions and set up a retry mechanism to improve the success rate of requests.

Use multiple proxy IP providers: Relying on a single proxy IP provider can be risky. To reduce risk, we can use multiple proxy IP providers and rotate between them.