Pathway to acceptance engineering development limitations of change request NN testing open loop testing sims track testing shadow mode real world testing wave1 user problems rollout metrics why end to end system? human values is hard to codify the interfaces betteen observation and action isn’t wel defined can scale to handle long tail problems homogeneous compute + known latency how to support data collection for corner cases? small NNs that trigger data collection for specific problems user intervention large change in state space Two Problems with Alignment underspecification: specs are non-exhaustive overgeneralization: specs are non-restrictive