#Example: HTTP Client Retrying HTTP Requests Safely

HTTP requests can fail for transient reasons: a network connection drops, a downstream service returns 503 Service Unavailable, or a rate limiter responds with 429 Too Many Requests.

While retrying can recover from temporary failures, retrying the wrong request can duplicate a state-changing operation. The hard part is not the loop. It is deciding whether a retry is safe and bounded.

#1. Retry Is a Policy, Not Just a Loop

A naive retry loop answers none of the safety questions required in production systems.

Request
  │
  ├── Success ───────────────────────► Return Response
  │
  └── Failure
        │
        ├── Not Transient ───────────► Return Error
        │
        ├── Unsafe to Repeat ────────► Return Error
        │
        ├── Deadline Exceeded ───────► Return Error
        │
        └── Retryable
              └── Backoff + Jitter ──► Next Attempt

A retry policy should answer:

1. Is the failure likely to be temporary?
2. Is repeating the operation safe?
3. Can the request body be replayed?
4. How long should we wait?
5. When must we stop?

These are application decisions.

#2. Retry Transient Failures

Some failures are commonly temporary.

Typical HTTP status codes include:

• 429 Too Many Requests
• 502 Bad Gateway
• 503 Service Unavailable
• 504 Gateway Timeout

A network error may also be temporary:

resp, err := client.Do(req)
if err != nil {
	// The connection may have failed temporarily.
}

But neither an HTTP status nor a network error automatically means that another attempt is appropriate.

For example, retrying 400 Bad Request usually does not fix the request.

Likewise, retrying 401 Unauthorized or 403 Forbidden does not normally fix authentication or authorization.

A useful default is:

This is a starting point, not a universal rule. The API's documented semantics should take precedence.

#3. Safe to Retry Is a Separate Question

A transient failure can still be unsafe to repeat.

Consider:

POST /payments

The client sends the request.

The server processes the payment.

The connection then fails before the client receives the response.

The client sees:

network error

The client does not know whether the payment succeeded.

Retrying the request may create a second payment.

This leads to one of the most important rules for HTTP retries:

A timeout or connection failure does not mean the server did nothing.

The client often cannot distinguish:

request was never received

from:

request was received and processed,
but the response was lost

For read-only operations such as GET, repeating the request is normally safe.

For state-changing operations, safety depends on the API.

#4. Idempotency Makes Retries Safer

An API can provide an application-level idempotency mechanism.

For example:

POST /payments
Idempotency-Key: 8f6d4a...

The client uses the same key when retrying the same logical operation.

The server can then recognize that the operation has already been processed and avoid performing it twice.

The exact semantics depend on the API.

The important distinction is:

network retry
    ≠
application-level idempotency

A retry mechanism cannot manufacture idempotency that the server does not provide.

For an operation with important side effects, determine the API's retry and idempotency semantics before adding automatic retries.

#5. Go's Transport Already Retries Some Requests

Application-level retry is not the only retry mechanism in net/http.

Go's http.Transport can automatically retry certain network failures. Its retry behavior is deliberately conservative and depends on conditions such as whether the request is considered idempotent and whether a request body can be replayed.

This is not the same as an application retry policy.

For example, Transport-level retry does not mean:

503 → wait → retry

three times with exponential backoff.

Think of the two mechanisms separately:

http.Transport
    │
    └── limited automatic retries
        for certain network failures

application
    │
    └── retry policy
        ├── HTTP status
        ├── network errors
        ├── backoff
        ├── idempotency
        └── deadline

Do not rely on Transport retries to define the reliability policy for your application.

#6. A Request Body Must Be Replayable

Retrying a request with a body requires a fresh body for each attempt.

This is easy to miss.

body := bytes.NewReader(payload)

req, err := http.NewRequest(
	http.MethodPost,
	url,
	body,
)

The first attempt consumes the reader.

The second attempt cannot assume that the reader is still positioned at the beginning.

For common in-memory readers, http.NewRequest can populate Request.GetBody so the body can be recreated:

payload := []byte(`{"name":"Alice"}`)

req, err := http.NewRequest(
	http.MethodPost,
	url,
	bytes.NewReader(payload),
)
if err != nil {
	return err
}

fmt.Println(req.GetBody != nil)

For a file or another streaming source, the application must deliberately provide a way to create a fresh body.

For example:

getBody := func() (io.ReadCloser, error) {
	return os.Open("request.json")
}

Each call opens a new stream starting at byte zero.

Seekable does not automatically mean replayable.

Replayability means that another attempt can obtain a fresh body with the same request data.

#7. Close the Response Before Retrying

A retryable HTTP response still has a response body that must be handled.

This is incomplete:

if resp.StatusCode == http.StatusServiceUnavailable {
	// retry
}

The response body remains open.

For a small response, a limited drain followed by Close can help connection reuse:

if resp.StatusCode == http.StatusServiceUnavailable {
	_, _ = io.CopyN(io.Discard, resp.Body, 4<<10)
	resp.Body.Close()

	// retry
}

Do not blindly drain an untrusted or potentially huge error response.

The detailed rules for response-body limits, draining, and connection reuse belong in the Read Response Safely example.

The retry loop should simply make sure each response is properly finished before the next attempt.

#8. Use Exponential Backoff

Immediate retries can amplify an outage.

Imagine 10,000 clients all receiving 503:

server overloaded
      │
      ▼
10,000 clients retry immediately
      │
      ▼
server receives another 10,000 requests
      │
      ▼
more overload

This is a retry storm.

A common approach is exponential backoff:

attempt 1 → 200 ms
attempt 2 → 400 ms
attempt 3 → 800 ms

Add jitter so clients do not all retry at exactly the same time.

For example:

delay := time.Duration(1<<attempt) * 200 * time.Millisecond

jitter := time.Duration(
	rand.Int63n(int64(delay / 2)),
)

delay += jitter

The exact values should be based on the service's latency, capacity, and failure behavior.

There is no universal "correct" retry delay.

#9. Respect Retry-After

A server may explicitly tell the client when to retry:

HTTP/1.1 429 Too Many Requests
Retry-After: 5

When the API provides meaningful Retry-After semantics, the retry policy should consider them.

For example, a local policy of:

200 ms

should not blindly override a server response asking clients to wait several seconds.

This is particularly important for rate-limited APIs.

The goal of retrying is recovery, not increasing the load on an already overloaded service.

#10. Keep Retries Inside the Original Deadline

Retry attempts should not reset the entire timeout.

This is dangerous:

attempt 1 → 10 seconds
attempt 2 → 10 seconds
attempt 3 → 10 seconds

An operation that was supposed to finish within 10 seconds could now take almost 30 seconds.

Instead, use one request context for the entire operation:

ctx, cancel := context.WithTimeout(
	context.Background(),
	10*time.Second,
)
defer cancel()

Each attempt uses the same context:

req, err := http.NewRequestWithContext(
	ctx,
	http.MethodGet,
	url,
	nil,
)

The deadline therefore applies to the whole retry operation.

10-second overall deadline
        │
        ├── attempt 1
        ├── backoff
        ├── attempt 2
        ├── backoff
        └── attempt 3
                │
                ▼
          deadline reached

A retry that starts after the caller's deadline has expired is not a retry. It is wasted work.

#11. A Timeout Does Not Prove Failure at the Server

This deserves special attention.

Suppose the client has a 5-second deadline:

client ───── request ─────► server
       ◄──── response ?

After five seconds:

context deadline exceeded

The client knows that it did not receive a completed result within its deadline.

It does not necessarily know what happened on the server.

The server may have:

• rejected the request,
• never received it,
• received it but not started processing,
• processed it successfully,
• processed it and sent a response that was lost.

For GET, retrying is often reasonable.

For a side-effecting operation, the result may be fundamentally unknown.

That is why idempotency is not merely an optimization for retries. It is part of the correctness model for distributed operations.

#12. Network Timeout Errors Need Classification

A network failure can be examined using net.Error:

var netErr net.Error

if errors.As(err, &netErr) && netErr.Timeout() {
	// The operation timed out.
}

Context cancellation and deadlines can be checked separately:

if errors.Is(err, context.DeadlineExceeded) {
	// The request context deadline expired.
}

These checks answer different questions.

context.DeadlineExceeded tells you that the context deadline was exceeded.

net.Error.Timeout() identifies an error with timeout semantics at the network layer.

Do not classify every error as retryable merely because it came from client.Do.

The actual error and the operation's semantics still matter.

#13. Do Not Retry Forever

A retry policy needs a hard upper bound.

For example:

const maxAttempts = 3

But an attempt limit alone is not enough.

You also need a time boundary:

maximum attempts
        +
overall deadline
        +
backoff

The deadline is especially important because a retry may spend most of its budget waiting for a slow server or a slow connection.

A useful policy might therefore look like:

max attempts: 3
overall deadline: 10 seconds
backoff: exponential
jitter: enabled

These are policy parameters, not net/http defaults.

Choose them according to the operation's SLO and failure behavior.

#14. A Small Retry Helper

For a retryable operation, the retry loop can remain small:

func doWithRetry(
	ctx context.Context,
	client *http.Client,
	method string,
	url string,
	getBody func() (io.ReadCloser, error),
) (*http.Response, error) {
	const maxAttempts = 3

	for attempt := 0; attempt < maxAttempts; attempt++ {
		var body io.ReadCloser

		if getBody != nil {
			var err error

			body, err = getBody()
			if err != nil {
				return nil, err
			}
		}

		req, err := http.NewRequestWithContext(
			ctx,
			method,
			url,
			body,
		)
		if err != nil {
			if body != nil {
				body.Close()
			}
			return nil, err
		}

		resp, err := client.Do(req)

		if err == nil {
			if !isRetryableStatus(resp.StatusCode) ||
				attempt == maxAttempts-1 {
				return resp, nil
			}

			_, _ = io.CopyN(
				io.Discard,
				resp.Body,
				4<<10,
			)
			resp.Body.Close()
		} else if attempt == maxAttempts-1 {
			return nil, err
		}

		delay := time.Duration(1<<attempt) * 200 * time.Millisecond

		timer := time.NewTimer(delay)

		select {
		case <-ctx.Done():
			timer.Stop()
			return nil, ctx.Err()
		case <-timer.C:
		}
	}

	return nil, fmt.Errorf("retry attempts exhausted")
}

func isRetryableStatus(status int) bool {
	switch status {
	case http.StatusTooManyRequests,
		http.StatusBadGateway,
		http.StatusServiceUnavailable,
		http.StatusGatewayTimeout:
		return true
	default:
		return false
	}
}

The important part is not the helper itself.

The important part is the contract around it:

• the caller decides whether the operation is safe to repeat;
• every attempt gets a fresh body;
• attempts share the caller's context;
• retryable responses are closed;
• attempts are bounded;
• backoff prevents immediate retry storms.

For production code, the retry policy should also account for Retry-After and jitter.

#15. Retry GET and Retry POST Are Different Problems

A simple GET retry can often be straightforward:

GET
 │
 ├── 503
 │
 ├── backoff
 │
 └── GET again

A POST may require an entirely different decision:

POST /orders
       │
       ├── request processed?
       │       │
       │       ├── yes
       │       └── unknown
       │
       └── response received?
               │
               ├── yes
               └── no

If the operation has side effects, the question is no longer simply:

"Did the request fail?"

It becomes:

"Can this logical operation safely be repeated if the outcome is unknown?"

That answer belongs to the API contract.

#16. Reuse the HTTP Client

The retry loop should reuse the same http.Client:

client := &http.Client{
	Timeout: 10 * time.Second,
}

Do not create a new client and Transport for every attempt.

Clients and Transports are designed to be reused and are safe for concurrent use.

Recreating them for every retry prevents effective connection pooling and adds unnecessary connection-management overhead.

The retry policy should control attempts, while the HTTP client continues to manage connections.

#17. Common Mistakes

#Retrying every error

for {
	_, err := client.Do(req)
	if err == nil {
		break
	}
}

This can become an infinite retry loop.

Always bound attempts and total time.

#Retrying every 5xx

Not every 5xx response necessarily means that repeating the operation is useful.

Use the API's semantics and retry policy.

#Retrying a timed-out POST

A timeout does not prove that the server did not process the request.

For important side effects, use an API-level idempotency mechanism when available.

#Reusing an already-consumed body

attempt 1 → body consumed
attempt 2 → body starts at the wrong position

Create a fresh body for every attempt.

#Retrying without backoff

Immediate retries can turn a temporary failure into a larger outage.

Use bounded exponential backoff with jitter.

#Ignoring Retry-After

A rate-limited server may explicitly tell clients to wait.

Do not blindly retry faster.

#Resetting the timeout on every attempt

The retry operation should remain inside the caller's original deadline.

#Creating a new http.Client for every attempt

This defeats connection pooling and adds unnecessary resource pressure.

Reuse the client.

#Assuming Go automatically retries everything

http.Transport performs only a limited class of automatic retries.

Application-level retry policy is a separate concern.

#18. Rule of Thumb

Retry only when the failure is plausibly transient, the operation is safe to repeat, and the retry still fits inside the caller's deadline.

For production code:

Transient failure
       +
Safe to repeat
       +
Fresh request body
       +
Bounded attempts
       +
Backoff + jitter
       +
Caller deadline
       │
       ▼
     Retry

If one of these conditions is missing, returning the error may be safer than retrying.

#Key Takeaways

• Retry is a policy, not just a loop.
• A transient failure does not automatically make a retry safe.
• A timeout or connection failure does not prove that the server did nothing.
• GET is normally safe to repeat; state-changing operations require more care.
• Use application-level idempotency when an operation can safely be repeated.
• A request body must be replayable before it can be retried.
• Close retryable responses before starting the next attempt.
• Use bounded exponential backoff with jitter.
• Respect Retry-After when the API provides it.
• Keep all attempts inside the caller's original deadline.
• Reuse the same http.Client and Transport.
• Go's http.Transport already performs limited automatic retries, but that is not an application retry policy.