debugging hardware systems

debugging software is annoying. debugging hardware is a completely different kind of pain. at the signal-processing-workflow, i learned this the hard way across multiple EEG headsets, sensor arrays, and experimental setups.

the headset saga

we went through two EEG headsets over the summer. both had problems. the first one:

random packet loss (probably wifi interference)
some channels were dead or railed
accelerometer always reading something even when stationary
important channels railed even with conductive gel

the second one:

getting alpha waves to work well was "really hard"
heartbeat was definitely being picked up
sensitivity to everything: movement, environmental noise, electrical interference

"things are breaking down — channels are being railed, packets are being lost."

the debugging process

binary search

"binary search for finding issues in hardware — literally do whatever." the most transferable debugging concept: isolate variables. when everything is broken, start cutting things in half until you find the problem.

concretely:

test each channel independently
swap out the board (we tried a new cyton board)
move to different rooms to isolate electrical noise
test on different people
test with and without gel
check if the problem exists in the raw data or only after processing

the environment matters

problems with current testing: "extremely sensitive. movement. environmental noise."

things that affected signal quality:

wifi and bluetooth interference ("pretty chilling, have done it before" — meaning it could be worked around)
footsteps from other people in the room
electrical devices nearby
brightness of the room
how well the electrodes were seated

fixing the environment often fixed the "hardware problem."

validate before trusting

the most important lesson: validate your setup before running real experiments.

alpha band: close your eyes, you should see 8-13Hz activity. if you can't, something is wrong.
flash test: flash yourself and look for VEPs in the data. if you can't see them on yourself, you won't see them on anyone.
known signal: send a known electrical signal through the system and verify it comes out the other side correctly.

"does have good alpha band testing" vs "does not have good alpha band testing" was the difference between a functional and a broken headset.

ask the symptoms, not the fix

"ways to fix: ask LLM the symptoms, recheck wiring, common issues with EEG headsets, ask LLM for more ways to fix."

the instinct is to google the fix. the better approach is to describe exactly what you're seeing (symptoms) and work backward to causes. "the accelerometer reads non-zero when stationary" is more useful than "EEG headset not working."

the multi-sensor setup problem

for one experiment, we needed to put EEG, heart rate, pupilometry, temperature, and other sensors on a person simultaneously. new debugging challenges:

how do all the sensors fit on the person?
how to make them stable on the person?
how to put them all on quickly?
data sync: each device has its own clock. getting timestamps aligned is its own project.

"can't get the data — whoop not connected, oura only nighttime, empatica not connected for some reason." consumer wearables are especially frustrating because you can't control their firmware or data export.

transferable lessons

hardware problems are usually environment problems. the device is fine; the setup is wrong.
binary search everything. when in doubt, cut the problem space in half.
validate the simple case first. if you can't measure a strong, well-known signal, you definitely can't measure a subtle one.
patience is a technical skill. software iteration takes seconds. hardware iteration takes hours (set up, run, wait, analyze, tear down, modify, repeat).
document obsessively. which channels, which settings, which room, which time. when you finally get it working, you need to know what changed.