CircleCI January 2023: One Laptop, Every Customer's Secrets
Threat Intelligence
An engineer's compromised laptop turned into a session cookie, and the session cookie turned into the ability to read environment variables and project keys across customers. The mass-rotation event that followed taught every CI/CD team what their attack surface really is.
By Arjun Raghavan, Security & Systems Lead, BIPI · April 15, 2024 · 9 min read
On January 4, 2023 CircleCI told its entire customer base to rotate every secret stored in the platform. Not a subset. Every secret. The cause was a single engineer's laptop, infected with malware that stole a session cookie. That cookie bypassed SSO and MFA because the user was already authenticated, and from there the actor moved into a tool with internal privileges to read customer encryption keys.
Timeline
- December 16, 2022: malware lands on a CircleCI engineer's laptop. The endpoint product on the laptop does not detect the specific stealer in time.
- December 19: the actor uses a stolen SSO session cookie to authenticate to a subset of CircleCI's internal systems as the engineer.
- December 22: the actor accesses customer environment variables and tokens. Encryption keys for those secrets are also exfiltrated, which is the part that forced the universal rotation.
- December 29: CircleCI is notified by a customer of suspicious access in their environment using a credential that had been stored in CircleCI.
- January 4, 2023: public advisory issued. CircleCI tells customers to rotate everything and provides log-export tools to identify what changed.
- January 13, 2023: CircleCI publishes the post-incident report with full timeline and IOCs.
Root cause
Session cookies that survived the MFA wall. Once the cookie was on the attacker's machine, the SSO IdP could not distinguish between the original session and the stolen one. The blast-radius driver was internal tooling that, with the engineer's role, could read customer-encrypted secrets in the clear.
MFA at login is necessary but it ends at login. Session theft is the modern bypass.
Attacker actions
After loading the cookie the actor used legitimate CircleCI internal tools to query customer environments. Outside CircleCI, downstream activity included probing of customer cloud accounts using exfiltrated credentials, with at least one customer reporting attempts to enumerate S3 and access Heroku. The volume of secrets exposed across customers meant the actor had options for weeks afterward, which is why CircleCI's rotation guidance was so urgent.
Detection signals
- Stealer-malware indicators on developer endpoints: persistence in scheduled tasks, Discord or Telegram exfil channels, browser data folder enumeration.
- Sessions reused from a new device fingerprint or IP without a corresponding re-authentication event in the IdP. Some IdPs surface this; many require explicit configuration.
- Programmatic access to internal admin tooling using a session that was issued during an interactive login from a different ASN.
- Customer-side: unfamiliar callers using long-lived credentials with names that match CircleCI naming conventions.
Lessons
- Inventory every CI-stored secret. If you cannot rotate everything in a weekend, you cannot survive a CircleCI-class incident.
- Migrate from long-lived cloud keys to OIDC-issued short-lived credentials. Workload identity federation closes a large slice of this blast radius.
- Implement session binding for internal admin tooling. Device-bound cookies, sensor checks, or recent re-auth requirements all help.
- Run the rotation drill annually. Pick a Friday, rotate every CI secret, verify nothing breaks. The drill is the only way to know the runbook works.
CircleCI handled the disclosure unusually well: detailed timeline, public IOCs, and clear customer guidance. That does not undo the rotation work for thousands of teams, and it does not change the structural lesson. Every CI/CD platform is a high-value target because it concentrates production secrets. The defense is to concentrate less, rotate faster, and assume the next CircleCI is already in someone's quarterly plan.
Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.