А вот и постмортем подоспел.

Summary: On May 29, 2025, a new feature was added to Service Control for additional quota policy checks. This code change and binary release went through our region by region rollout, but the code path that failed was never exercised during this rollout due to needing a policy change that would trigger the code. As a safety precaution, this code change came with a red-button to turn off that particular policy serving path. The issue with this change was that it did not have appropriate error handling nor was it feature flag protected. Without the appropriate error handling, the null pointer caused the binary to crash. Feature flags are used to gradually enable the feature region by region per project, starting with internal projects, to enable us to catch issues. If this had been flag protected, the issue would have been caught in staging.

Ну в общем классика - запушили недотестированное говно в прод, забыв добавить фичафлаг.

Разумеется, такого больше не повторится (наверное) Разумеется, они сделают выводы (наверное) Может ещё на конференции какой про это расскажут ;)

@snakeslair #thereisnoclouds