The S3 Upload Mystery: When Your Files Play Hide and Seek

• 6 min read
AWS S3 Debugging Node.js JavaScript Modules

There's something uniquely frustrating about bugs that work "most of the time." You know the type—everything's green in staging, the first few uploads sail through, and then suddenly... radio silence. Files just vanishing into the void.

This is the story of how a quirky incompatibility between JavaScript module systems nearly cost us a week of sanity—and how we fixed it without rewriting half the codebase.

The Setup

We had a fairly standard flow: users upload multiple documents, our backend pipes them to S3, everyone's happy. Except they weren't. Support tickets started trickling in—"my second document didn't upload," "only the first file saved," and my personal favourite, "it worked yesterday."

The symptoms were maddening:

  • Single file uploads? Perfect.
  • Multiple files in sequence? First one uploads, the rest fail silently.
  • Retry the exact same files? Sometimes it works.

Classic "works on my machine" energy, except it was more like "works on the first try."

Digging In

Our stack runs TypeScript with ESM (ECMAScript Modules), but we're using AWS SDK v2—which is CommonJS. Usually, this isn't a big deal. You require() the package, wire it up, move on with your life.

But S3 uploads with streams? That's where things get spicy.

When you pass a stream to s3.upload(), the SDK internally does an instanceof check against its own stream classes. Here's the problem: in an ESM context, the stream module you import and the one AWS SDK v2 uses internally aren't the same reference. They're like identical twins who refuse to acknowledge each other.

// What we see
passThrough instanceof Readable // true

// What AWS SDK sees
passThrough instanceof AWS.util.stream.Readable // false

The first upload works because Node.js caches module references. Subsequent uploads in the same process? The references drift, the instanceof check fails, and AWS SDK quietly gives up on your stream like a cat losing interest in a toy.

The "Proper" Fix (That We Couldn't Do)

The textbook answer is simple: upgrade to AWS SDK v3.

SDK v3 is modular, ESM-native, and doesn't have this issue. It's also a significant rewrite. Our S3 utilities are shared across a dozen microservices, each with their own upload patterns, error handling, and quirks. We estimated the migration at several sprints of careful refactoring and testing.

We had a production issue affecting users now. We needed a fix today.

The Pragmatic Fix

Instead of fighting the module system, we decided to make peace with it. Three changes made all the difference:

1. Patch the SDK's Stream References

Before each upload, we explicitly tell AWS SDK to use our stream module:

function ensureAWSStreamPatched() {
  if (AWS.util?.stream) {
    AWS.util.stream.Stream = streamModule.Stream
    AWS.util.stream.Readable = streamModule.Readable
    AWS.util.stream.PassThrough = streamModule.PassThrough
  }
}

Is it elegant? Not really. Does it work? Absolutely.

2. Fresh S3 Clients Per Upload

AWS SDK v2 has some internal state that gets weird across multiple sequential operations. Instead of a singleton S3 client, we spin up a fresh instance for each upload:

const freshS3Client = new AWS.S3({
  ...getS3Config(),
  httpOptions: {
    timeout: 300000,
    connectTimeout: 30000,
  },
})

A few extra bytes of memory per upload is a small price for reliability.

3. Buffer Fallback

When the stream instanceof check fails (and we now detect this explicitly), we fall back to buffering the file in memory before upload. Not ideal for massive files, but for documents under 100MB? Works like a charm.

if (!willPassCheck) {
  // Stream won't work, buffer the whole thing
  const chunks = []
  for await (const chunk of incomingStream) {
    chunks.push(chunk)
  }
  uploadBody = Buffer.concat(chunks)
}

The Takeaway

Sometimes the "right" solution isn't the right now solution. We could have spent weeks on a proper SDK migration while users couldn't upload their documents. Instead, we spent an afternoon understanding the root cause and implementing a targeted fix.

The AWS SDK v3 migration is still on our roadmap—it's cleaner, more maintainable, and future-proof. But shipping a working fix today beats shipping a perfect fix next month.

Software engineering is full of these trade-offs. The trick is knowing when to fight the battle and when to patch around it. This time, we patched. And honestly? I'd do it again.

For the curious: AWS SDK v3 splits the monolithic aws-sdk package into modular clients (@aws-sdk/client-s3, etc.) with proper ESM support. If you're starting a new project, just use v3. If you're maintaining a large codebase on v2, well... now you know how to make streams behave.