observability-pipelines

Configure and manage Datadog Observability Pipelines for log collection, processing, and routing.

5|2|Updated Mar 9, 2026
One-click install
npx skills add https://github.com/lauhon/pi --skill observability-pipelines-lauhon
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: observability-pipelines
Source: https://github.com/lauhon/pi/tree/main/skills/observability-pipelines
Command: npx skills add https://github.com/lauhon/pi --skill observability-pipelines-lauhon

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Managing log pipelines across diverse sources, processors, and destinations requires deep knowledge of the Datadog Observability Pipelines API and its component schemas. This Skill provides the operational instructions and configuration patterns needed to design, validate, deploy, and maintain pipelines without memorizing the API surface. ## Core Features & Use Cases - Full Pipeline Lifecycle: List, create, update, validate, and delete pipelines via the Datadog API v2 remote config endpoints. - Component Library: Covers 15+ sources (Kafka, S3, Splunk, Pub/Sub), 17+ processors (filter, grok parsing, PII scanning, quota, sampling), and 17+ destinations (Datadog Logs, S3, Splunk HEC, Microsoft Sentinel, Elasticsearch). - Use Case: Build a compliance pipeline that ingests logs from the Datadog Agent, redacts credit card numbers with the sensitive_data_scanner processor, and routes sanitized logs to both Datadog and an S3 archive. ## Quick Start Ask the agent to create a pipeline that collects logs from the Datadog Agent, filters for errors, and sends them to Datadog Logs.

Frequently Asked Questions about observability-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a Datadog Observability Pipeline?

Send a POST request to /api/v2/remote_config/products/obs_pipelines/pipelines with a JSON body defining sources, processors, and destinations. Each component needs an id, type, and inputs array referencing upstream component IDs.

How do I validate a pipeline configuration before deploying?

POST the pipeline configuration to the /pipelines/validate endpoint with your DD-API-KEY and DD-APPLICATION-KEY headers. The API returns validation errors such as missing required fields or invalid input references without creating the pipeline.

What log sources does Datadog Observability Pipelines support?

It supports 15+ source types including datadog_agent, kafka, splunk_tcp, splunk_hec, amazon_s3, google_pubsub, fluentd, http_server, sumo_logic, rsyslog, syslog_ng, logstash, and generic socket listeners.

Can I redact PII from logs in a pipeline?

Yes, the sensitive_data_scanner processor detects and redacts PII using library patterns like credit_card and us_ssn or custom regex rules. It supports redact, partial_redact, and hash actions scoped to specific fields.

Why does my pipeline fail validation with invalid input reference errors?

This error occurs when a processor or destination lists an inputs ID that does not exist in the pipeline. Verify every ID referenced in inputs arrays matches a defined source or processor group, and avoid circular dependencies between components.

What credentials are required to use the Observability Pipelines API?

You need DD_API_KEY, DD_APP_KEY, and DD_SITE environment variables. The API is in Preview, so access requires filling out a Datadog form, and appropriate permissions like observability_pipelines_read and observability_pipelines_deploy are required.