#Example: HTTP Client Stream Large Responses

Large HTTP responses should usually be streamed, not loaded into memory.

The simplest mistake is:

data, err := io.ReadAll(resp.Body)

For a small API response, that is fine. For a large download, it makes memory usage proportional to the response size.

This example shows how to stream a response directly to a file while enforcing a maximum size.

#Quick Example

Stream the response body directly to a file:

package main

import (
	"fmt"
	"io"
	"net/http"
	"os"
)

func main() {
	resp, err := http.Get("https://example.com/large-file.zip")
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
		// Best-effort drain so the transport may reuse the connection.
		_, _ = io.Copy(io.Discard, resp.Body)
		panic(fmt.Sprintf("unexpected HTTP status: %s", resp.Status))
	}

	file, err := os.Create("large-file.zip")
	if err != nil {
		panic(err)
	}
	defer file.Close()

	if _, err := io.Copy(file, resp.Body); err != nil {
		panic(err)
	}
}

The important part is:

io.Copy(file, resp.Body)

The response is consumed incrementally instead of creating a byte slice containing the entire response.

For production code, also set an explicit request timeout and handle file cleanup carefully.

#Why Stream the Response?

Suppose a server returns a 2 GB file.

With:

data, err := io.ReadAll(resp.Body)

the application must hold the response data in memory.

With:

io.Copy(file, resp.Body)

memory usage stays roughly independent of the response size.

The data flow becomes:

HTTP response
      │
      ▼
resp.Body
      │
      ▼
   io.Copy
      │
      ▼
     file

There is no need to materialize the entire response.

This matters for:

• large file downloads
• backups
• database exports
• object storage
• compressed archives
• media files
• large API responses

#io.Copy Is More Than a 32 KiB Loop

io.Copy is not simply equivalent to writing your own fixed-size Read loop.

It first checks whether the source implements io.WriterTo. If not, it checks whether the destination implements io.ReaderFrom:

source.WriterTo
      ↓
destination.ReaderFrom
      ↓
generic buffered copy

This allows concrete types to provide more efficient transfer paths. The io package documents this dispatch explicitly.

For example, *os.File implements io.ReaderFrom. On Linux, its implementation can use platform-specific mechanisms such as copy_file_range and splice when the source and destination support them, reducing userspace copies.

Do not, however, assume that every HTTP download automatically becomes a Linux zero-copy transfer. net/http wraps the underlying network connection, and the zero-copy path depends on the concrete Reader exposed to os.File.ReadFrom.

The useful engineering rule is simpler:

Prefer io.Copy over a hand-written buffer loop unless you actually need per-chunk logic.

Let the source and destination choose their optimized path.

#Enforce a Maximum Response Size

Streaming prevents a large response from consuming all available memory, but it does not prevent an unexpectedly large file from consuming disk space.

Use io.LimitReader when the application has a maximum acceptable size.

const maxSize = 100 << 20 // 100 MiB

limited := io.LimitReader(resp.Body, maxSize+1)

n, err := io.Copy(file, limited)
if err != nil {
	return err
}

if n > maxSize {
	return fmt.Errorf("response exceeds %d bytes", maxSize)
}

The +1 is important.

If the limit is exactly 100 MiB, reading only:

io.LimitReader(resp.Body, maxSize)

cannot distinguish:

exactly 100 MiB

from:

more than 100 MiB

Reading one extra byte lets the application detect an oversized response.

A complete helper might look like this:

func download(
	client *http.Client,
	url string,
	dst string,
	maxSize int64,
) error {
	resp, err := client.Get(url)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
		return fmt.Errorf("unexpected HTTP status: %s", resp.Status)
	}

	file, err := os.Create(dst)
	if err != nil {
		return err
	}

	n, err := io.Copy(file, io.LimitReader(resp.Body, maxSize+1))
	if err != nil {
		_ = file.Close()
		return err
	}

	if n > maxSize {
		_ = file.Close()
		return fmt.Errorf("response exceeds %d bytes", maxSize)
	}

	if err := file.Close(); err != nil {
		return err
	}

	return nil
}

The limit is enforced while reading, not after the complete response has already been written.

#Content-Length Is Only an Early Check

If the server provides a useful Content-Length, it can reject obviously oversized responses before downloading them:

if resp.ContentLength > maxSize {
	return fmt.Errorf("response is too large")
}

But this is only an early rejection.

A response may have:

resp.ContentLength == -1

when its length is unknown.

The application should still enforce the limit while reading:

limited := io.LimitReader(resp.Body, maxSize+1)

Do not make the application's size guarantee depend on Content-Length.

Compression makes this distinction even more important.

When net/http.Transport automatically requests gzip and transparently decompresses the response, the resulting Response has:

resp.Uncompressed == true
resp.ContentLength == -1
resp.Header.Get("Content-Length") == ""
resp.Header.Get("Content-Encoding") == ""

and resp.Body yields the decompressed bytes. This is deliberate behavior in the standard library.

Therefore, Content-Length is not an application-level decompressed-size limit.

For a hard payload limit, enforce the bound on the stream you actually consume.

#Write to a Temporary File First

For downloads that must not leave a partial destination file, write to a temporary file and rename it only after the download succeeds.

Create the temporary file in the destination directory:

tmp, err := os.CreateTemp(filepath.Dir(dst), ".download-*.tmp")

This matters because os.Rename generally requires the source and destination to be on the same filesystem. Creating the temporary file in the system temporary directory can therefore cause a cross-device rename failure when dst is on another mount.

A complete implementation:

package download

import (
	"fmt"
	"io"
	"net/http"
	"os"
	"path/filepath"
)

func Download(
	client *http.Client,
	url string,
	dst string,
	maxSize int64,
) error {
	resp, err := client.Get(url)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
		return fmt.Errorf("unexpected HTTP status: %s", resp.Status)
	}

	tmp, err := os.CreateTemp(filepath.Dir(dst), ".download-*.tmp")
	if err != nil {
		return err
	}

	tmpPath := tmp.Name()
	keep := false

	defer func() {
		if !keep {
			_ = os.Remove(tmpPath)
		}
	}()

	n, err := io.Copy(tmp, io.LimitReader(resp.Body, maxSize+1))
	if err != nil {
		_ = tmp.Close()
		return err
	}

	if n > maxSize {
		_ = tmp.Close()
		return fmt.Errorf("response exceeds %d bytes", maxSize)
	}

	if err := tmp.Sync(); err != nil {
		_ = tmp.Close()
		return err
	}

	if err := tmp.Close(); err != nil {
		return err
	}

	if err := os.Rename(tmpPath, dst); err != nil {
		return err
	}

	keep = true
	return nil
}

The important sequence is:

create temporary file in destination directory
        ↓
stream response
        ↓
enforce size limit
        ↓
Sync
        ↓
Close
        ↓
Rename

This prevents a failed download from replacing the existing destination with a partial file.

The cleanup defer intentionally ignores a second Close or Remove error after an earlier failure. If every filesystem error must be reported, use an explicit cleanup path or a named-return pattern instead.

For downloads where durability across sudden power loss matters, Sync is relevant. For ordinary caching, it may be unnecessary overhead.

#Reuse the HTTP Connection After an Early Exit

If the application stops reading a response early, it may leave unread response data behind.

For example, after detecting an oversized response:

n, err := io.Copy(file, io.LimitReader(resp.Body, maxSize+1))
if err != nil {
	return err
}

if n > maxSize {
	return fmt.Errorf("response too large")
}

the body may still contain additional data.

If the remaining response is small, an application can make a bounded best-effort drain before returning:

const maxDrainBytes = 256 << 10 // 256 KiB

_, _ = io.Copy(
	io.Discard,
	io.LimitReader(resp.Body, maxDrainBytes),
)

Then resp.Body.Close() is still required.

The important point is that draining should itself be bounded. Never turn an oversized-response error path into an attempt to consume an arbitrarily large response.

Modern Go HTTP/1 clients also perform limited automatic draining when Response.Body.Close is called, so explicit draining is not universally required for connection reuse. An explicit bounded drain is useful when the application wants a predictable, best-effort policy rather than relying entirely on the transport's internal behavior.

#Do Not Use io.ReadAll Just to Check the Size

This is tempting:

data, err := io.ReadAll(resp.Body)
if err != nil {
	return err
}

if int64(len(data)) > maxSize {
	return fmt.Errorf("response too large")
}

It defeats the purpose.

The application has already allocated memory proportional to the response size before checking the limit.

If the requirement is:

Never allow this response to consume more than approximately maxSize bytes of application-level buffering.

then enforce the bound during the read.

#Stream Through Multiple Destinations

Sometimes the response needs to be written to more than one destination.

For example:

file, err := os.Create("download.bin")
if err != nil {
	return err
}
defer file.Close()

writer := io.MultiWriter(file, hash)

_, err = io.Copy(writer, resp.Body)
if err != nil {
	return err
}

io.MultiWriter writes each chunk to its destinations sequentially.

If any destination blocks, the copy blocks.

If one destination returns an error, the copy stops.

This is useful for operations such as:

HTTP response
      │
      ▼
  io.MultiWriter
      ├──► file
      └──► hash

It does not create independent asynchronous consumers.

If one destination is slow, the HTTP response consumption is also slowed.

#Stream to a Consumer with io.Pipe

Sometimes the response should not be stored locally at all.

For example, an application may download data and immediately feed it into another processing pipeline.

A pipe can connect the response reader to another component:

pr, pw := io.Pipe()

go func() {
	defer pw.Close()

	if _, err := io.Copy(pw, resp.Body); err != nil {
		_ = pw.CloseWithError(err)
	}
}()

processErr := process(pr)

io.Pipe provides synchronous streaming between a writer and reader. It does not provide an application-level buffer that can absorb an arbitrarily large response.

The producer can therefore be slowed by the consumer, providing backpressure.

This is useful when the downstream operation should process data as it arrives rather than waiting for the complete response.

Lifecycle management matters: if the consumer stops early, the producer must have a way to stop as well. In more complex pipelines, use context cancellation and make sure every goroutine has a defined exit path.

#JSON Responses Can Also Be Streamed

Streaming is not limited to files.

For a large JSON response, use json.Decoder rather than loading the entire response first:

const maxJSONSize = 50 << 20 // 50 MiB

limited := io.LimitReader(resp.Body, maxJSONSize+1)
decoder := json.NewDecoder(limited)

for decoder.More() {
	var item Item

	if err := decoder.Decode(&item); err != nil {
		return err
	}

	process(item)
}

json.Decoder is streaming, but it still maintains internal buffering, and decoding a very large individual JSON value can require substantial memory.

Putting io.LimitReader in front of the decoder provides an overall input bound:

HTTP response
      │
      ▼
io.LimitReader
      │
      ▼
json.Decoder
      │
      ▼
application objects

The limit protects the amount of JSON input consumed. It does not guarantee that every individual decoded value has constant memory usage.

For protocols containing extremely large individual objects or strings, impose additional application-level bounds on those values.

#Compression Changes the Size Problem

HTTP clients commonly negotiate gzip automatically.

That creates two different sizes:

wire bytes
    ↓
decompression
    ↓
application bytes

A compressed response can be much smaller on the network than after decompression.

When net/http performs automatic decompression, resp.Body exposes the decompressed stream and the response metadata is adjusted accordingly. In particular, ContentLength is set to -1 and the corresponding Content-Length header is removed.

Therefore:

io.LimitReader(resp.Body, maxSize+1)

limits the data actually exposed to the application.

If the application must protect itself against decompression expansion, enforce the limit at the point where decompressed data enters the application.

For particularly sensitive endpoints, consider the relationship between:

• compressed response size
• decompressed size
• CPU cost
• disk usage
• downstream processing limits

A network-level size limit and an application-level payload limit are different controls.

#Streaming Does Not Mean Unlimited

A streaming implementation can still fail badly if it has no bounds.

Consider:

io.Copy(file, resp.Body)

This protects memory usage, but a malicious or broken server could send data indefinitely.

The application may need several independent limits:

HTTP request lifetime
        │
        ├── maximum response size
        │
        ├── maximum disk usage
        │
        └── maximum processing time

For example:

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()

req, err := http.NewRequestWithContext(
	ctx,
	http.MethodGet,
	url,
	nil,
)
if err != nil {
	return err
}

resp, err := client.Do(req)
if err != nil {
	return err
}
defer resp.Body.Close()

The context limits the lifetime of the request.

io.LimitReader limits the amount of response data consumed.

They solve different problems.

#Client.Timeout and Streaming

An HTTP client timeout also applies while reading the response body.

For example:

client := &http.Client{
	Timeout: 2 * time.Minute,
}

This is useful for ordinary downloads with a known upper bound on total request duration.

For long-lived streaming responses, however, an overall timeout may be the wrong policy.

A stream that is expected to remain open for hours should not use a two-minute total timeout simply because ordinary API requests do.

Use a request context or a more appropriate transport-level timeout policy when the application needs different semantics.

The important distinction is:

Client.Timeout
    → total HTTP operation lifetime

io.LimitReader
    → maximum bytes consumed

ResponseHeaderTimeout
    → time waiting for response headers

Do not treat them as interchangeable.

#A Manual Read Loop

io.Copy is usually the right abstraction.

A manual loop can make the streaming behavior explicit:

buf := make([]byte, 32*1024)

for {
	n, err := resp.Body.Read(buf)

	if n > 0 {
		if _, writeErr := file.Write(buf[:n]); writeErr != nil {
			return writeErr
		}
	}

	if err == io.EOF {
		break
	}

	if err != nil {
		return err
	}
}

The important rule is:

Process n > 0 before handling err.

A reader is allowed to return data and an error in the same call.

The 32 KiB buffer here is illustrative. io.Copy may use different buffering or a WriterTo / ReaderFrom fast path.

Use a manual loop when you need per-chunk logic such as:

• progress reporting
• checksums
• rate limiting
• custom framing
• application-specific accounting

Otherwise, prefer io.Copy.

#What Happens When the Response Fails?

A streaming download can fail after receiving a perfectly valid beginning of the response.

For example:

200 OK
↓
100 MB received
↓
connection lost

The application now has a partial file.

That is why the temporary-file pattern is often preferable:

temporary file
     ↓
download succeeds
     ↓
rename

rather than:

destination file
     ↓
download fails
     ↓
partial file remains

If the protocol supports resumable downloads, the application can instead preserve partial state deliberately and use mechanisms such as HTTP range requests.

That is a different design from treating a partial download as a successful file.

#Choosing the Implementation

#Small response

Use:

data, err := io.ReadAll(resp.Body)

when the response is known to be small and bounded.

Examples:

• small JSON API responses
• configuration documents
• short text responses

#Large response

Use:

io.Copy(dst, resp.Body)

when the response should be streamed.

#Large response with a hard size limit

Use:

io.Copy(dst, io.LimitReader(resp.Body, maxSize+1))

when the application must reject oversized responses.

#Durable download

Use:

temporary file in destination directory
        ↓
Sync
        ↓
Close
        ↓
Rename

when a partial destination file is unacceptable.

#Large JSON stream

Use:

json.Decoder

with an appropriately bounded input stream when the protocol permits incremental decoding.

#Code Review

When reviewing large-response handling, ask:

Memory

• Is io.ReadAll being used on an unbounded response?
• Is the maximum in-memory payload known?

Copy path

• Is io.Copy being used instead of an unnecessary manual buffer loop?
• Could WriterTo or ReaderFrom provide a more efficient path?

Size

• Is Content-Length being treated as the only size check?
• Is the actual stream bounded with io.LimitReader when necessary?
• Is the maxSize+1 pattern used when oversized responses must be detected?

Files

• Is the temporary file created in the destination directory?
• Can a failed download leave a partial destination?
• Should a temporary file be used?
• Are Sync, Close, and Rename errors handled according to the durability requirements?

Timeouts

• Can the server keep the connection open indefinitely?
• Is the request context appropriate for the expected download duration?
• Is Client.Timeout being confused with a body-size limit?

Compression

• Is the application limiting the data it actually processes?
• Does automatic decompression change the meaning of the available response metadata?

Connection reuse

• If the application stops reading early, does it have a bounded drain policy where useful?
• Is an unbounded drain being avoided?

Concurrency

• If io.Pipe is used, can every producer and consumer exit?
• If multiple writers are used, can one slow destination stall the entire pipeline?

Partial results

• What happens if the connection fails halfway through?
• Is a partial file distinguishable from a completed file?

#Engineering Rule

For large HTTP responses, think in terms of streams and bounds, not byte slices.

HTTP response
     │
     ▼
  resp.Body
     │
     ▼
  bounded stream
     │
     ├──► file
     ├──► decoder
     └──► processing pipeline

The core pattern is simple:

io.Copy(dst, io.LimitReader(resp.Body, maxSize+1))

But production correctness comes from the surrounding decisions:

• bound the data you accept
• bound the request lifetime
• use io.Copy so the standard library can select an appropriate copy path
• handle partial results
• close resources explicitly
• create temporary files on the destination filesystem
• use temporary files when necessary
• understand decompression
• preserve backpressure
• avoid loading large responses into memory

For more production-oriented Go HTTP patterns, see the HTTP Client examples on gobase.net.