In a keynote, Christine Yen, co-founder and CEO of Honeycomb, discusses the challenges that large language models (LLMs) present to traditional software development practices like testing and debugging. She argues that the inherent unpredictability and non-deterministic nature of LLMs necessitate a shift towards observability, which focuses on understanding software behavior in production by observing what is actually happening with live users. Yen emphasizes the importance of combining observability with evaluation tools (evals) to define and capture expected and unexpected LLM behaviors, creating feedback loops for continuous improvement. By systematically tracking inputs and outputs and leveraging existing observability practices, teams can navigate the complexities of building reliable systems in the age of AI.
Conference Video – Observability in the Age of LLMs – Christine Yen
