Best practice with retries with requests

19 April 2017   38 comments   Python

tl;dr; I have a lot of code that does response = requests.get(...) in various Python projects. This is nice and simple but the problem is that networks are unreliable. So it's a good idea to wrap these network calls with retries. Here's one such implementation.

The First Hack

import time
import requests


def get(url):
        return requests.get(url)
    except Exception:
        # sleep for a bit in case that helps
        # try again
        return get(url)

This, above, is a terrible solution. It might fail for sooo many reasons. For example SSL errors due to missing Python libraries. Or the URL might have a typo in it, like get('http:/').

Also, perhaps it did work but the response is a 500 error from the server and you know that if you just tried again, the problem would go away.


while True:
    response = get('')
    if response.status_code != 500:
        # Hope it won't 500 a little later

What we need is a solution that does this right. Both for 500 errors and for various network errors.

The Solution

Here's what I propose:

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

def requests_retry_session(
    status_forcelist=(500, 502, 504),
    session = session or requests.Session()
    retry = Retry(
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

Usage example...

response = requests_retry_session().get('')

s = requests.Session()
s.auth = ('user', 'pass')
s.headers.update({'x-test': 'true'})

response = requests_retry_session(session=s).get(

It's an opinionated solution but by its existence it demonstrates how it works so you can copy and modify it.

Testing The Solution

Suppose you try to connect to a URL that will definitely never work, like this:

t0 = time.time()
    response = requests_retry_session().get(
except Exception as x:
    print('It failed :(', x.__class__.__name__)
    print('It eventually worked', response.status_code)
    t1 = time.time()
    print('Took', t1 - t0, 'seconds')

There is no server running in :9999 here on localhost. So the outcome of this is...

It failed :( ConnectionError
Took 1.8215010166168213 seconds


1.8 = 0 + 0.6 + 1.2

The algorithm for that backoff is documented here and it says:

A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). urllib3 will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) seconds. If the backoff_factor is 0.1, then sleep() will sleep for [0.0s, 0.2s, 0.4s, ...] between retries. It will never be longer than Retry.BACKOFF_MAX. By default, backoff is disabled (set to 0).

It does 3 retry attempts, after the first failure, with a backoff sleep escalation of: 0.6s, 1.2s.
So if the server never responds at all, after a total of ~1.8 seconds it will raise an error:

In this example, the simulation is matching the expectations (1.82 seconds) because my laptop's DNS lookup is near instant for localhost. If it had to do a DNS lookup, it'd potentially be slightly more on the first failure.

Works In Conjunction With timeout

Timeout configuration is not something you set up in the session. It's done on a per-request basis. httpbin makes this easy to test. With a sleep delay of 10 seconds it will never work (with a timeout of 5 seconds) but it does use the timeout this time. Same code as above but with a 5 second timeout:

t0 = time.time()
    response = requests_retry_session().get(
except Exception as x:
    print('It failed :(', x.__class__.__name__)
    print('It eventually worked', response.status_code)
    t1 = time.time()
    print('Took', t1 - t0, 'seconds')

And the output of this is:

It failed :( ConnectionError
Took 21.829053163528442 seconds

That makes sense. Same backoff algorithm as before but now with 5 seconds for each attempt:

21.8 = 5 + 0 + 5 + 0.6 + 5 + 1.2 + 5

Works For 500ish Errors Too

This time, let's run into a 500 error:

t0 = time.time()
    response = requests_retry_session().get(
except Exception as x:
    print('It failed :(', x.__class__.__name__)
    print('It eventually worked', response.status_code)
    t1 = time.time()
    print('Took', t1 - t0, 'seconds')

The output becomes:

It failed :( RetryError
Took 2.353440046310425 seconds

Here, the reason the total time is 2.35 seconds and not the expected 1.8 is because there's a delay between my laptop and I tested with a local Flask server to do the same thing and then it took a total of 1.8 seconds.


Yes, this suggested implementation is very opinionated. But when you've understood how it works, understood your choices and have the documentation at hand you can easily implement your own solution.

Personally, I'm trying to replace all my requests.get(...) with requests_retry_session().get(...) and when I'm making this change I make sure I set a timeout on the .get() too.

The choice to consider a 500, 502 and 504 errors "retry'able" is actually very arbitrary. It totally depends on what kind of service you're reaching for. Some services only return 500'ish errors if something really is broken and is likely to stay like that for a long time. But this day and age, with load balancers protecting a cluster of web heads, a lot of 500 errors are just temporary. Obivously, if you're trying to do something very specific like requests_retry_session().post(...) with very specific parameters you probably don't want to retry on 5xx errors.



Your first ( ostensibly "horrible") solution works the best for me, the rest is too verbose.


"Obivously, if you're trying to do something very specific like requests_retry_session().post(...) with very specific parameters you probably don't want to retry on 5xx errors."

Actually, this wouldn't work with the current solution, retry is not applied for POST by default - it needs to be specifically white listed if it's wanted (bite my ass ;) )

Otherwise, thanks for the great article!


I had to make more than 8 000 requests. My script had been stumbling after several hundreds requests. Your solution—requests_retry_session()—saved my day. Thanks!


really cool thx !


nice, thank you!


This is awesome! Thank you!

Felipe Dornelas

Networks are unreliable, but TCP is fault-tolerant. The problem is that application servers are unreliable.


How do I also handle 404 errors with this?


Exactly, this hangs when hitting 404 errors...


There is no need to set connect retries, or read retries, total retries takes precedent over the rest of the retries, so set it once there and it works for read, redirect, connect, status retries


Awesome! This resolved my issue!


Pretty cool! Thank you!!!


You set status_forcelist, but status kwarg is set to None as default (according to urllib3.util.retry.Retry docs), so retries on bad-statuses-reason will never be made.
Should we specify connect=retries or I have misunderstanding?
P.S. sorry for my english


came across this review because I'm getting this problem, how to solve it?


I decided so but I'm not sure if it's right because get an error:


err: requests.exceptions.RetryError: HTTPConnectionPool(host='', port=80): Max retries exceeded with url: /status/504 (Caused by ResponseError('too many 504 error responses',))


Did you ever sort this?


Good job!!! Thank you!!!

Miro Hrončok

Googled this and it worked like a charm! Thank You.


how to use proxy?


Love it!

However, is there a way to print/log all reponses?
E.g. When it retries 3 times, print the status code of all three requests?

Peter Bengtsson

I doubt it but requests uses logging. You just need to configure your logging to turn it up so you can see these kinds of things happening.


Thanks! made my code much more reliable. Thanks for posting this for everyone to use.


there is a typo - "sesssion"


trying the time out i get
NameError: global name 'time' is not defined

Peter Bengtsson

You need to inject ‘import time’ first.


How do you propose dealing with this situation?

I can't seem to get anyone to respond, and my script is totally broken at the moment.

Anthony Camilo

Did you ever get an answer, i'm on the same boat.

Brikend Rama

Thank you for your code snippet. Works great

Jeff Walters

Excellent solution. Thank you for posting this article/solution! I had no idea that the HTTPAdapters existed. You just saved me a few hours of my life.

DEnilson Grupp Fernandes

Excellent. Thanks for that


Does not work if requests fails to read a chunked response :(


The following will setup an HTTP server to repro (set the sleep to be greater than your read timeout):
import ssl
from time import sleep
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer

PORT = 8001

class CustomHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        print "SLEEP"
        self.send_header('Transfer-Encoding', 'chunked')
        self.send_header('Content-type', 'text/plain')
        self.wfile.write(', world!')
        print "WAKE"

httpd = HTTPServer(("", PORT), CustomHandler)
httpd.socket = ssl.wrap_socket(httpd.socket, keyfile='/home/local/ANT/schdavid/tmp/key.pem', certfile='/home/local/ANT/schdavid/tmp/cert.pem')

except KeyboardInterrupt:
    print 'Goodbye'


Thanks... it worked well for me. Good Article...


It was simple but elegant. It covered almost everything. Keep up the good work!

Guillermo Chussir

Very good idea. I'll try this on my scripts. Thanks!


How to do mock unit testing on request_retry_session?

Peter Bengtsson

Do you have to? Also, doesn't that depend greatly on how you mock `requests`?

Your email will never ever be published

Related posts

Public Class Fields saves sooo many keystrokes in React code 14 April 2017
Web Console trick to get all URLs into your clipboard 27 April 2017
Related by Keyword:
hashin 0.14.0 with --update-all and a bunch of other features 13 November 2018
Fancy linkifying of text with Bleach and domain checks (with Python) 10 October 2018
Fastest way to download a file from S3 29 March 2017
Cope with JSONDecodeError in requests.get().json() in Python 2 and 3 16 November 2016
How to track Google Analytics pageviews on non-web requests (with Python) 03 May 2016