data-engineer

Building clean data pipelines involves implementing data models, I/O adapters, and processing services from approved architecture designs while adhering to clean coding standards. This approach ensures maintainable, production-ready data engineering components with proper error handling and test coverage.

Reviewing data engineering code requires analyzing existing pipelines against clean coding rules to generate severity-rated violation reports. This identifies overly long functions, unclear naming, and missing edge cases, providing actionable suggested fixes for maintainable codebases.

Data pipeline implementations should adhere to pytest, mypy, and ruff quality gates to ensure code consistency and testability. Meeting these quality standards eliminates unmaintainable implementations and enforces clean coding best practices across the data engineering codebase.

Yes, this approach supports implementing data models, I/O adapters, processing services, and orchestrators. It applies to building new data pipeline components from approved architecture designs and reviewing existing data code against clean coding and testing standards.

The best way to ensure test coverage for data ingestion pipelines is to include comprehensive unit and end-to-end tests during implementation. Building components like CSV readers and database writers with proper error handling satisfies requirements for clean, testable data engineering work.