Talk by Lisa Soder AI Regulation EU GPAI Code of Practice: makes fronter model providers track, document, and report AI incidents SB 53 Large Developer: identify and respond to safety incidents + internal governance Problem: define what it means to “track, document, and report”? Methods Goals help companies with compliance help regulators for assessing compliance technical tooling Approach legal analysis best practices in AI best practices in other domains What is an AI incident? “In house evaluation is not enough.” proactive monitoring is important for things that don’t have external signals (i.e. cyber espionage) methods activation probes output classifiers LLM-as-judge human labeling for certain cases clio (https://arxiv.org/abs/2412.13678) Good Rood-Cause Analysis Similar outcomes can come from very different causes, so you need to good data to contextualize. …but data retention processes don’t support incident analysis (because people don’t retend data for more than 30 days; models maybe self hosted)