Use connection pool in Python Requests


Requests is a great library for performing HTTP related request and response.
As it’s description, it supports connection pool automatically.

Requests takes all of the work out of Python HTTP/1.1 — making your integration with web services seamless. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, powered by urllib3, which is embedded within Requests.

Actually, the following snippet will not use connection pool provided from urllib3.

requests.get('http://developer.github.com/v3/')
requests.get('http://developer.github.com/v3/media/')

Requests use urllib3.PoolManager in requests.adapters.HTTPAdapter. Which is mounted in requests.Session.__init__. Thus, for each request

def request(method, url, **kwargs):
    session = sessions.Session()
    return session.request(method=method, url=url, **kwargs)

will create new session, which make requests.request won’t gain any benefit from connection pool. Alternatively, you can use requests.Session to make Requests use connection pool powered by urllib3.

session = requests.Session()
session.get('http://developer.github.com/v3/')
session.get('http://developer.github.com/v3/media/')

I rewrite a benchmark_request.py from urllib3 benchmark.py.

The result

Completed requests_session_get in 7.067s
Completed requests_get in 9.162s

,

Leave a Reply

Your email address will not be published. Required fields are marked *