#Example: HTTP Client Streaming Multipart File Uploads

Multipart uploads are easy when the file is small:

package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"mime/multipart"
	"net/http"
	"os"
	"path/filepath"
)

func uploadSmall(ctx context.Context, client *http.Client, url, path string) error {
	file, err := os.Open(path)
	if err != nil {
		return err
	}
	defer file.Close()

	var body bytes.Buffer
	writer := multipart.NewWriter(&body)

	part, err := writer.CreateFormFile("file", filepath.Base(path))
	if err != nil {
		return err
	}

	if _, err := io.Copy(part, file); err != nil {
		return err
	}

	if err := writer.Close(); err != nil {
		return err
	}

	req, err := http.NewRequestWithContext(
		ctx,
		http.MethodPost,
		url,
		&body,
	)
	if err != nil {
		return err
	}

	req.Header.Set("Content-Type", writer.FormDataContentType())

	resp, err := client.Do(req)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
		return fmt.Errorf("upload failed: %s", resp.Status)
	}

	return nil
}

This is often the right implementation for small files because it is straightforward and replayable.

The problem appears when the body becomes large.

For large uploads, the goal is not simply to avoid buffering. The implementation must also handle backpressure, cancellation, goroutine lifetime, replayability, and retry semantics correctly.

#Stream Large Files with io.Pipe

io.Pipe lets the multipart encoder write directly into the HTTP request body without first materializing the complete request in memory.

The producer writes multipart data:

file → multipart.Writer → PipeWriter

The HTTP transport reads it:

PipeReader → Transport → network

io.Pipe itself does not maintain an application-level in-memory buffer. A write blocks until the reader consumes the data or the pipe is closed.

The transport and operating system may have their own buffering. That is separate from io.Pipe's synchronization semantics.

This gives the upload natural backpressure: if the transport stops consuming the body, the multipart producer cannot run arbitrarily far ahead.

#A complete streaming body

type uploadBody struct {
	*io.PipeReader
}

func newUploadBody(
	ctx context.Context,
	path string,
	boundary string,
) (*uploadBody, error) {
	pr, pw := io.Pipe()

	done := make(chan struct{})

	go func() {
		defer close(done)

		file, err := os.Open(path)
		if err != nil {
			_ = pw.CloseWithError(err)
			return
		}
		defer file.Close()

		writer := multipart.NewWriter(pw)

		if err := writer.SetBoundary(boundary); err != nil {
			_ = pw.CloseWithError(err)
			return
		}

		part, err := writer.CreateFormFile("file", filepath.Base(path))
		if err != nil {
			_ = pw.CloseWithError(err)
			return
		}

		if _, err := io.Copy(part, file); err != nil {
			_ = pw.CloseWithError(err)
			return
		}

		if err := writer.Close(); err != nil {
			_ = pw.CloseWithError(err)
			return
		}

		// PipeWriter.Close closes the write side with EOF.
		// In the standard library implementation, Close always returns nil.
		_ = pw.Close()
	}()

	go func() {
		select {
		case <-ctx.Done():
			_ = pr.CloseWithError(ctx.Err())
		case <-done:
		}
	}()

	// The wrapper makes the ownership of the PipeReader explicit.
	// It currently adds no behavior beyond io.ReadCloser.
	return &uploadBody{PipeReader: pr}, nil
}

The wrapper is deliberately small. io.PipeReader already implements io.ReadCloser; the wrapper gives the application a place to add logging, metrics, or lifecycle instrumentation later.

The producer goroutine must be able to terminate when either:

1. the multipart body finishes normally, or
2. the request context is canceled.

The context watcher exists to prevent the producer from remaining blocked on pw.Write after the HTTP request has stopped consuming the body.

io.Pipe supports concurrent reads, writes, and closes. The important design issue is therefore not that concurrent close is unsafe. It is that the streaming code should have a clear lifecycle: normal completion closes the writer, while cancellation closes the reader side so a blocked producer can wake up with an error.

Cancellation and normal completion can happen close together. The pipe implementation safely serializes those operations, but the resulting error path depends on which side closes the pipe first. When debugging unusual cancellation failures, keep that lifecycle in mind.

#Generate the Multipart Boundary Once

A multipart boundary must remain consistent between the Content-Type header and every replay of the request body.

The standard library already generates valid boundaries:

var boundarySource bytes.Buffer
boundaryWriter := multipart.NewWriter(&boundarySource)

boundary := boundaryWriter.Boundary()
contentType := boundaryWriter.FormDataContentType()

The buffer is only used to construct the multipart.Writer and obtain a valid boundary. It does not receive any upload payload.

For an actual request body, create a new multipart.Writer over the pipe and set the same boundary:

writer := multipart.NewWriter(pw)

if err := writer.SetBoundary(boundary); err != nil {
	return err
}

Do not manually construct:

multipart/form-data; boundary=...

Use FormDataContentType() so the header follows the multipart package's boundary formatting rules.

#Replay the Body with GetBody

A request body can be replayed only if the application can create an equivalent new body.

For example:

req.GetBody = func() (io.ReadCloser, error) {
	return newUploadBody(ctx, path, boundary)
}

This matters for redirects that preserve the request body.

For 307 and 308 redirects, net/http preserves the method and body only when Request.GetBody is available. http.NewRequest automatically supplies GetBody for common in-memory body types such as bytes.Buffer, bytes.Reader, and strings.Reader; a streaming pipe body needs its own implementation.

#Replayability is not retry safety

These are different questions:

Can I create the body again?
        ↓
      GetBody

Should I send the request again?
        ↓
      retry policy

A body may be perfectly replayable while the operation itself is unsafe to repeat.

For example:

POST /payments

may create a charge every time it is received.

By contrast, an upload endpoint might be designed to safely retry the same object.

The application must decide whether the operation is idempotent or protected by an idempotency mechanism.

#Request Context Is Per Request Lifecycle

The context used by newUploadBody belongs to the request that owns that body.

That matters when implementing manual retries.

Do not treat this:

req.GetBody = func() (io.ReadCloser, error) {
	return newUploadBody(ctx, path, boundary)
}

as a general-purpose retry factory whose context can outlive the original request.

GetBody has no context parameter:

func() (io.ReadCloser, error)

It is designed to recreate the body for the request lifecycle managed by net/http.

#Automatic redirects

For an automatic 307 or 308 redirect, net/http creates the redirected request from the original request. The redirected request therefore retains the original request's context.

That is intentional.

If the original context has already been canceled or its deadline has expired, the redirect does not receive a fresh timeout merely because GetBody can recreate the upload body.

In other words:

original request
    │
    ├── context
    │
    └── GetBody()
          │
          └── replayed body

GetBody makes the body replayable. It does not create a new request lifetime.

If the original request context expires, a 307/308 redirect cannot turn that expired request into a fresh attempt.

#Manual retries

Manual retries are different.

If your retry policy creates a new request with a new timeout or cancellation policy, create a new request and a new GetBody closure for that attempt:

attemptCtx, cancel := context.WithTimeout(parent, 30*time.Second)
defer cancel()

req, err := http.NewRequestWithContext(
	attemptCtx,
	http.MethodPost,
	url,
	body,
)
if err != nil {
	return err
}

req.GetBody = func() (io.ReadCloser, error) {
	return newUploadBody(attemptCtx, path, boundary)
}

The important distinction is:

Stable upload state
    ├── file path
    └── multipart boundary

Per-attempt state
    └── request context

Do not accidentally make a canceled request context the lifetime of future manual retry attempts.

#Source File Changes Are a Separate Problem

A replayable body must represent the same logical request body.

Consider:

req.GetBody = func() (io.ReadCloser, error) {
	return newUploadBody(ctx, path, boundary)
}

If another goroutine modifies the file between attempts, the second body may differ from the first.

For uploads where consistency matters:

• treat the source file as immutable during the request;
• upload from a stable snapshot;
• or use a content-addressed object/version.

GetBody means “create another body,” not “guarantee that the source has not changed.”

#Content-Length

Streaming does not automatically mean chunked transfer encoding.

If the exact multipart request size is known, ContentLength can be set.

For a simple multipart body containing one file, the size is approximately:

preamble
+ multipart headers
+ file size
+ CRLF
+ closing boundary

But calculating it correctly requires accounting for the exact multipart encoding.

Do not guess.

If the exact size is not available, leaving ContentLength unknown lets net/http choose the appropriate transfer behavior.

The important distinction is:

streaming body ≠ unknown length

A body can be streamed while still having a known content length.

#HTTP/2 Flow Control Still Applies

io.Pipe provides backpressure between the multipart producer and the HTTP transport.

HTTP/2 adds another layer of flow control.

A simplified path is:

file
  ↓
multipart.Writer
  ↓
io.Pipe
  ↓
HTTP transport
  ↓
HTTP/2 flow control
  ↓
network

If the receiver is slow or the HTTP/2 flow-control window prevents more data from being sent, the transport stops consuming the pipe quickly enough. The producer eventually blocks on the pipe.

That is desirable.

The pipe should not turn a slow network into unbounded application-level memory growth.

The exact buffering and scheduling behavior belongs to the transport; io.Pipe itself remains a synchronous handoff between the producer and consumer.

#Cancellation and Timeouts

A streaming upload needs a request-scoped cancellation policy.

For example:

ctx, cancel := context.WithTimeout(
	context.Background(),
	5*time.Minute,
)
defer cancel()

body, err := newUploadBody(ctx, path, boundary)
if err != nil {
	return err
}

req, err := http.NewRequestWithContext(
	ctx,
	http.MethodPost,
	url,
	body,
)
if err != nil {
	return err
}

There are two related mechanisms here:

request context
    ↓
cancel this request and its body production

http.Client.Timeout
    ↓
overall client-side request deadline

The request context belongs to one request attempt. A client's timeout applies to the client's request lifecycle, including reading the response body.

For long-running uploads, choose these limits deliberately. A timeout that is appropriate for a 1 MB API request may be completely inappropriate for a 5 GB upload.

#MaxConnsPerHost Is Not an Upload Rate Limit

Large uploads can occupy connections for a long time.

For example:

transport := http.DefaultTransport.(*http.Transport).Clone()

transport.MaxConnsPerHost = 16

client := &http.Client{
	Transport: transport,
}

MaxConnsPerHost limits the total number of connections per host, including connections that are dialing, active, or idle.

It does not limit:

• bytes per second;
• individual upload size;
• request duration.

A large number of concurrent uploads can therefore consume the available connection capacity for other requests.

If uploads and latency-sensitive API calls share a client, consider whether they should share the same connection pool.

Note: the type assertion above assumes http.DefaultTransport is a *http.Transport. DefaultTransport is exposed as a RoundTripper, so code that replaces it with another implementation would make this assertion panic. This is particularly easy to encounter in tests that replace http.DefaultTransport. If you own the transport configuration, constructing or retaining an explicit *http.Transport is often clearer.

#Testing Cancellation

Streaming bugs often appear only when the consumer stops reading while the producer is blocked.

A useful test deliberately cancels the request while the upload is in progress.

The important property is not merely:

request returns an error

It is:

request returns
    ↓
producer terminates
    ↓
context watcher terminates
    ↓
no goroutine remains blocked on the pipe

Go 1.27 adds the goroutineleak profile to runtime/pprof. It can detect a class of goroutines permanently blocked on synchronization primitives, but it is not a general detector for every kind of goroutine leak. In particular, a goroutine blocked in network I/O is not what this profile is designed to detect.

That makes it useful for this kind of pipe/channel lifecycle testing, but not a replacement for inspecting goroutine behavior as a whole.

#Benchmark the Right Thing

Do not benchmark only:

bytes.Buffer
vs
io.Pipe

The useful questions are:

1. How much memory is retained?
2. How does peak memory change with file size?
3. How many concurrent uploads can the process sustain?
4. What happens when the receiver is slow?
5. How quickly does cancellation release the producer?
6. Does the implementation create goroutines that survive completed requests?

A realistic benchmark can use an httptest.Server with a deliberately slow handler to simulate a slow receiver:

func BenchmarkUploadBody(b *testing.B) {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// Read the request body slowly to exercise backpressure.
	}))
	defer server.Close()

	for i := 0; i < b.N; i++ {
		// Create a representative body.
		// Upload it to the slow test server.
		// Measure allocations and completion time.
	}
}

For meaningful results, keep the following constant:

• file size;
• destination behavior;
• concurrency;
• network conditions;
• request headers;
• timeout policy.

Report the actual measurements rather than inventing a fixed “streaming is X% faster” claim. For uploads, memory behavior and concurrency capacity are often more important than raw throughput.

#Response Handling

The request body and response body have independent lifecycles.

Always close the response body:

resp, err := client.Do(req)
if err != nil {
	return err
}
defer resp.Body.Close()

if resp.StatusCode < 200 || resp.StatusCode >= 300 {
	const maxErrorBody = 64 << 10

	// Read one extra byte so we can distinguish exactly 64 KiB
	// from a response that exceeds the limit.
	data, err := io.ReadAll(
		io.LimitReader(resp.Body, maxErrorBody+1),
	)
	if err != nil {
		return err
	}

	if len(data) > maxErrorBody {
		return fmt.Errorf("upload failed: response body too large")
	}

	return fmt.Errorf(
		"upload failed: %s: %s",
		resp.Status,
		data,
	)
}

Do not assume that a successful upload means the server returned a small response.

If the response body can be large, apply the same bounded-reading discipline to the response.

#Choosing the Implementation

#Buffer the multipart body

Use bytes.Buffer when:

• files are small;
• memory usage is predictable;
• replayability is important;
• simplicity matters more than streaming.

Advantages:

• simple;
• replayable;
• easy to test;
• easy to calculate Content-Length.

Cost:

• memory grows with request size.

#Stream with io.Pipe

Use io.Pipe when:

• files can be large;
• memory usage must remain bounded;
• the server accepts streaming uploads;
• backpressure is useful.

Advantages:

• low application-level memory overhead;
• natural backpressure;
• works with large files.

Costs:

• producer goroutine;
• cancellation lifecycle;
• more complicated replay;
• more complicated testing.

Streaming is not automatically better. It is better when the request size makes buffering undesirable.

#Code Review: What to Look For

When reviewing a streaming multipart upload, ask:

#Body lifecycle

• Is every opened file closed?
• Is the pipe reader closed when the request ends?
• Can the producer remain blocked after cancellation?
• Does the producer terminate on every error path?

#Context

• Does the upload have an explicit cancellation or timeout policy?
• Is the request context treated as per-attempt state?
• Is a canceled context accidentally reused by manual retries?
• Is it clear that automatic 307/308 redirects remain within the original request context?

#Replay

• Is GetBody implemented when the request may need body replay?
• Does it recreate the complete multipart body?
• Does every replay use the same multipart boundary?
• Is replayability being confused with retry safety?

#Memory

• Is the complete file accidentally buffered?
• Does the implementation maintain bounded application-level memory under slow receivers?
• Are response bodies bounded as well?

#Concurrency

• Can cancellation race with normal producer completion without leaving a blocked goroutine?
• Are goroutine lifetimes tied to request lifetimes?
• Has the cancellation path been tested under a blocked writer?

#HTTP behavior

• Are large uploads competing with latency-sensitive requests for the same connection pool?
• Is MaxConnsPerHost being confused with a bandwidth limit?
• Are retries appropriate for the operation?

These questions usually expose more production problems than checking whether the multipart syntax itself is correct.

#Engineering Trade-off

The core design is simple:

small body
    ↓
bytes.Buffer
    ↓
simple and replayable

large body
    ↓
multipart.Writer
    ↓
io.Pipe
    ↓
HTTP transport
    ↓
bounded memory + backpressure

But streaming introduces another responsibility:

streaming
    +
cancellation
    +
replayability
    +
goroutine lifecycle

The difficult part is not making a file upload stream.

The difficult part is making the stream stop correctly, replay correctly, and remain bounded when the network does not cooperate.

That is the part worth getting right in production Go code.