Before AI security agents, Atlassian put 44 billion daily events into one schema. That is the order that works
Atlassian normalized its security telemetry to OCSF on Databricks and reports 80% lower ingestion cost. Its agent plans sit on top of that foundation, and most are still roadmap.
Listen to this article · 5 min
AI-generated narration of the full article.
Atlassian’s security team processes about 44 billion events a day. Before it could think seriously about AI agents for threat hunting, it had a more basic problem: two decades of growth and acquisitions had left it with logs in many formats, each describing a user, a host or a connection in its own way. An agent asked to correlate a login with a DNS query would first have to work out that the two records were talking about the same person.
The company’s answer, described in a Databricks post on October 15 co-written with Atlassian engineers, was to normalize everything to the Open Cybersecurity Schema Framework (OCSF) in a security lakehouse on Databricks. The results it reports are about cost and speed, not AI: an 80% reduction in ingestion cost, more than 500 production threat detection signals on one schema, a 60% increase in analyst productivity and queries up to 95% faster, from about 45 seconds to 2 or 3. These figures are Atlassian’s, from its own data estate and starting point.
What is built, and what is planned on top
Planned
- Reusable agent skills from analyst work
- Agent-orchestrated threat hunts
- Contextual SQL
- Context graphs
Roadmap
Live: detection and triage
- 500+ detection signals
- Alerts clustered on standard fields
Live: one schema
- OCSF for all telemetry
- Schema enforced in typed Go code
- Agentic mapping of new logs, gated
Sources
- Authentication, network, DNS, scanners
- Formats from decades of acquisitions
Why the schema comes first
The value of a common schema is not elegance. It is that one field means one thing everywhere. In Atlassian’s design, a username lives in the same standardized field whether the event came from an authentication system or a network sensor, so clustering, enrichment and correlation can key on one value.
That is exactly what a security agent needs and usually lacks. Block’s security team showed this week why the best AI investigations depend on joining evidence across products’ logs. An agent can only make that join reliably if the identifiers agree. Without normalization, the model spends tokens and judgment on reconciling formats, and its errors look like insight.
Where AI already helps
The agentic piece that is live is modest and instructive. Atlassian uses an agentic workflow to map new log sources to the schema: analysts select logs, and after a gated confirmation the workflow generates mappings and pipeline code, tests them and deploys through canary and production stages. The authors say the rigid schema reduced hallucinations and token usage.
Note the direction. AI is used first to build and maintain the data foundation, under a gate, with tests and staged deployment. It is the kind of work we described when we said an agent’s first lasting deliverable is often cleaner data. Engineering choices matter too: the team chose to transform data in flight and embeds the schema in Go structs, so invalid output fails at compile time rather than at 3 a.m.
What is still roadmap
The agent capabilities people will want to read about are listed as next steps: turning analysts’ field usage into reusable agent skills, agent orchestration for threat hunts, contextual SQL for questions such as tracing DNS command-and-control traffic, and context graphs on top of OCSF. None of that means agents investigate 44 billion events today. And OCSF adoption alone does not guarantee accurate detection or entity resolution; the post describes consistent fields, not a resolution method.
The lesson for security AI programs
If your security team is evaluating AI agents for the SOC, ask what they will read before asking what they will do:
- How many schemas describe a user, a device and a connection in your telemetry today? Each one is a reconciliation the agent will attempt, and sometimes get wrong.
- Can you query a year of history in seconds? Agents investigate by querying. Slow queries become slow, expensive investigations.
- Is the mapping of new sources governed? Every new log source is a chance to break the schema the agents depend on.
Atlassian’s sequence, schema first, detections second, agents third, is not the fastest path to an AI demo. It is the one where the agents, when they arrive, have something consistent to reason over.