Edit wiki/debugging/the-30-hour-bug.md — Work Reflections

---
visibility: public-edit
---

# the 30-hour bug

one line. thirty hours. the bug is never where you think it is.

## the setup

i was building a system that needed to send audio data from a microcontroller to a cloud service. the microcontroller would record a chunk of audio, then POST it to an endpoint for processing. straightforward pipeline, or so i thought.

it worked perfectly in testing. local server, direct connection, clean 200 responses. then i pointed it at the real endpoint and everything broke. 400 errors. every single time.

## the spiral

i spent the first few hours doing what anyone would do — checking the obvious stuff. was the payload formatted correctly? yes. were the headers right? yes. was the endpoint actually up? yes, i could hit it from curl no problem. the microcontroller was the problem. except it wasn't.

i rebuilt the HTTP request from scratch. twice. i rewrote the audio encoding pipeline. i added logging at every step. i tried different audio formats, different chunk sizes, different content types. i questioned whether the microcontroller's HTTP library was even spec-compliant. i read the library source code. i started wondering if the cloud service had some undocumented requirement.

this is the thing about long debug sessions — somewhere around hour 10, you've tested every hypothesis you can think of, and you start testing increasingly desperate ones. you start changing things at random. you start questioning reality.

i took walks. i slept on it. i came back fresh and still couldn't see it.

## the fix

the cloud endpoint was behind a redirect. when you hit the URL, it 301'd you to a slightly different URL. my HTTP client had `setFollowRedirects` enabled — but the library's implementation silently dropped the POST body when following redirects. it would follow the redirect, but convert the POST to a GET. the body was gone. the cloud service got an empty request and returned 400.

one line fix: set the correct endpoint URL that didn't redirect. or alternatively, don't follow redirects and handle them manually.

thirty hours for one line.

## what this taught me

### the bug is in the layer you're not looking at

i was looking at my code. the bug was in a library behavior i'd never questioned. this pattern repeats constantly — the bug lives in your [[assumptions-kill|assumptions]], in the code you trust implicitly, in the layer between your code and the thing it talks to.

### logging is necessary but not sufficient

i had extensive logging. it told me the request was being sent correctly. what it didn't tell me was that the library was silently mutating my request after my logging code ran. you can't log what you don't know to look for.

### the time investment distorts thinking

after 20 hours, i was so invested in certain hypotheses that i couldn't let them go. sunk cost applies to debugging. the willingness to throw away your current theory and start from scratch — genuinely start from scratch, not just say you are — is maybe the single most important debugging skill. see [[zooming-out]].

### there's a moment of rage and relief

when you finally find it, you feel both simultaneously. rage that it was this stupid, this simple, this hidden. relief that you're not insane. and then a strange gratitude, because you learned something about the system that you'd never have learned otherwise.

## the real lesson

every long debug session has a story like this. one line. one wrong assumption. hours or days of work. the skill isn't avoiding these sessions — it's building the [[perseverance]] to survive them, the [[zooming-out]] instinct to step back when you're stuck, and the systematic habits that shorten them over time (see [[binary-search-your-life]]).

the best debuggers i've met aren't the ones who never get stuck. they're the ones who've been stuck enough times to know what being stuck feels like, and what to do about it.