Writing Custom Sigma Pipelines for Your Log Schema

Every custom Sigma pipeline exists to solve the same problem: a Sigma rule speaks an abstract field taxonomy, and your SIEM does not. A rule that says TargetObject and Image has to become RegistryKey and InitiatingProcessFileName before it runs against your data. This post — the capstone of the Sigma series — shows how to write a processing pipeline in YAML that maps Sigma's field names onto your log schema, scopes the mapping to the right rules, and survives the moment your schema inevitably differs from the bundled pipelines.

Key Takeaways

• A custom Sigma pipeline is a YAML file of ordered transformations that rewrite a rule's abstract fields into the field names your SIEM actually uses during sigma convert.
• The two transformations you reach for most are field_name_mapping (rename Sigma fields to your schema) and add_condition (scope a query to the right table or index).
• rule_conditions with a logsource match keep a transformation from firing on every rule, so a registry mapping only touches registry rules.
• priority controls ordering when you stack your pipeline on top of a backend's bundled one — lower priority runs first.
• Pipelines are how you adapt community Sigma rules to a non-standard schema without editing the rules themselves, which keeps the rules portable and updatable.

Environment

• sigma-cli 1.0+ with a backend installed — examples here use the kusto backend for Microsoft Sentinel and Defender XDR.
• A handful of Sigma rules to convert — the rules from earlier in this series are good test subjects.
• Knowledge of your own log schema: the exact field names your data uses, which you get from the SIEM's schema reference or a sample event.
• A text editor — a pipeline is just a YAML file you pass to sigma convert -p.

The Problem

Sigma rules are written against an abstract taxonomy on purpose. A rule for registry persistence uses TargetObject and Details because those are Sigma's generic names, not because any SIEM stores them that way. When you convert that rule for Microsoft Sentinel, TargetObject needs to become RegistryKey; for a different platform it becomes something else again. That translation is exactly what a processing pipeline does, and the major backends ship with pipelines that handle the common cases — the sysmon pipeline I leaned on throughout this series is one of them.

The bundled pipelines stop being enough the moment your data does not match the assumptions they were written against: a custom log source, a renamed field, an in-house table, or events forwarded through the Windows Event Forwarding setup into a schema nobody upstream anticipated. At that point you either hand-edit every converted query — which throws away the portability that made Sigma worth using — or you write a pipeline once and let it translate every rule automatically. The second option is the whole point of the format.

The Solution — Writing a Custom Sigma Pipeline

Step 1 — Understand the three parts of a pipeline file

A pipeline is a YAML document with a name, a priority, and a list of transformations. Each transformation has an id, a type, and the parameters that type needs. The minimal shape looks like this:

name: My Schema Pipeline
priority: 100
transformations:
  - id: rename_registry_key
    type: field_name_mapping
    mapping:
      TargetObject: RegistryKey
    rule_conditions:
      - type: logsource
        category: registry_set

That single transformation renames Sigma's TargetObject to RegistryKey, but only for rules whose log source is registry_set — the rule_conditions block is what scopes it. Without that condition, the mapping would try to rename a field on every rule you convert, including ones that have no TargetObject at all.

Step 2 — Map every field the rule uses

A field mapping is only useful if it covers all the fields the rule references. The registry Run-key rule from earlier in this series uses TargetObject, Details, and Image, so the mapping has to translate all three to whatever your schema calls them:

  - id: registry_field_map
    type: field_name_mapping
    mapping:
      TargetObject: RegistryKey
      Details: RegistryValueData
      Image: InitiatingProcessFileName
    rule_conditions:
      - type: logsource
        category: registry_set

The values on the right are the field names from the Microsoft Sentinel DeviceRegistryEvents schema. Get one wrong and the converted query references a column that does not exist, so the query runs but matches nothing — which is why validating against a known-good event, as I stressed in the SIEM correlation post, is not optional.

Step 3 — Scope the query to the right table with add_condition

Renaming fields is half the job; the query also has to hit the right table. The add_condition transformation injects an extra condition into the rule before conversion — useful for pinning a table name or an index that the rule itself does not specify:

  - id: scope_to_registry_table
    type: add_condition
    conditions:
      Type: DeviceRegistryEvents
    rule_conditions:
      - type: logsource
        category: registry_set

Whether you need this depends on the backend — the kusto backend already resolves a table from the log source, so for Sentinel you often do not. It matters more on backends that key off an index or a sourcetype, where the rule has no other way to know which data to search. Reach for it when the converted query is correct but pointed at the wrong place.

Step 4 — Control ordering with priority

You rarely run your pipeline alone — you stack it on top of a backend's bundled pipeline. Ordering then matters, because if the bundled sysmon pipeline has already renamed a field, your mapping has to account for the name it produced, not the original. priority sets the order: lower numbers run first.

sigma convert -t kusto -p sysmon -p my_schema.yml runkey_persistence.yml

Pipelines passed to sigma convert apply in priority order regardless of the order you list them on the command line. Give your pipeline a higher priority number than the bundled one so it runs afterward and operates on the already-transformed field names — get this backwards and your mapping silently does nothing because it is matching field names that no longer exist by the time it runs.

Step 5 — Test against one rule before trusting it on all of them

A pipeline is code, and it deserves the same caution. Convert a single rule, read the output, and confirm every field name resolved to a real column before you run the pipeline across your whole rule set:

sigma convert -t kusto -p sysmon -p my_schema.yml \
  --without-pipeline scheduled_task.yml

Read the generated query the way you would read any detection — does it reference columns that exist, does it point at the right table, would it match a known-good test event? This is the same validate-before-you-trust discipline that runs through the whole series, from the Sysmon configuration that produces the events to the rules that consume them. A pipeline that converts cleanly but maps to the wrong field is worse than no pipeline, because it looks like it works.

Frequently Asked Questions

When do I need a custom Sigma pipeline instead of a bundled one?

When your log schema does not match what the bundled pipeline assumes — a renamed field, a custom table, an in-house log source, or data forwarded into a schema the backend authors never saw. The bundled pipelines handle standard Sysmon and common platforms; anything non-standard is what a custom pipeline is for.

What is the difference between field_name_mapping and add_condition?

field_name_mapping renames the fields a rule already references — Sigma's TargetObject becomes your RegistryKey. add_condition injects a new condition the rule did not have, typically to scope the query to a specific table or index. One translates existing fields; the other adds a constraint.

Why does my converted query match nothing after I add a pipeline?

Almost always a field name that mapped to a column that does not exist, or a pipeline ordering problem where your mapping ran before the bundled one and matched field names that were about to change. Check the generated query against your real schema, and confirm your pipeline's priority runs it after the bundled pipeline.

How do rule_conditions keep a transformation from firing everywhere?

A transformation with no conditions applies to every rule converted. A rule_conditions block with a logsource match — for example category: registry_set — restricts it to rules with that log source, so a registry field mapping never touches a process-creation rule. It is how one pipeline can safely hold mappings for many different log sources.

Conclusion

A custom Sigma pipeline is the piece that makes the whole portability promise real. Community rules and the rules from this series are written against an abstract taxonomy; a pipeline is the YAML that bends that taxonomy onto the field names your SIEM actually stores, without ever editing the rules. field_name_mapping renames, add_condition scopes, rule_conditions restrict, and priority orders — four ideas that cover the large majority of real cases.

The limitation is the one that bites everyone once: a pipeline that converts without error is not the same as a pipeline that is correct. A wrong field mapping produces a query that runs and matches nothing, which is the worst kind of failure because it is silent. Test against one rule and one known-good event before you trust it across the set. Do that, and you can pull rules from the community, point them at a schema nobody upstream anticipated, and have them run — which is the entire reason to write detections in Sigma in the first place.

Related Posts

• Sysmon Configuration for Windows Security Monitoring — the events your converted rules ultimately run against.
• Windows Event Forwarding Setup for Centralised Security Logs — how logs reach the schema a pipeline has to map.
• From Logs to Threats: SIEM Correlation Rules for Real Attacks — validating a query against real data before trusting it.

Authoritative reference: the pySigma Processing Pipelines documentation and the SigmaHQ rule repository.

Editorial note: posts on this blog are drafted with AI assistance and then reviewed, edited, and tested against a real environment before publishing. Commands, output, and screenshots come from systems I actually ran the work on.