Next platform deadline: Amazon EKS Nov 26, 2026  

All 14 deadlines

Data Engineering & Analytics

Data Pipelines & ELT Engineering

Pipelines fail quietly: a late job, a renamed column, a dashboard still showing yesterday. We rebuild ingestion, transformation and orchestration as tested, version-controlled code, so data arrives on time and someone knows when it does not.

Sound familiar?

  • Hundreds of scheduled jobs, stored procedures and scripts that only one person understands.
  • Dashboards show stale or wrong numbers and the business notices before the data team does.
  • Connector or warehouse bills grew faster than the data, and nobody knows which pipeline drives the cost.
  • dbt or Airflow was adopted, but without tests, environments, code review or owners.

What you receive

  • Pipeline inventory with cost and failure data
  • dbt project with tests and documentation
  • Orchestration code and schedules
  • CI/CD in your repository
  • Alerting and runbooks

Book a free 30-minute review

What we deliver

Scope is confirmed in writing in the assessment, with anything out of scope listed too.

  1. 01

    Pipeline inventory

    Sources, jobs, schedules, run times, failures and cost per pipeline, in one sheet you keep.

  2. 02

    Ingestion

    Managed connectors such as Fivetran or Airbyte where they pay off, and code where they do not, including Azure Data Factory, AWS Glue and Matillion.

  3. 03

    Transformation as code

    dbt models with tests, documentation and lineage, organised into staging, intermediate and mart layers.

  4. 04

    Orchestration

    Airflow, Dagster, Databricks Workflows or Fabric pipelines with retries, freshness targets, backfills and dependency-aware scheduling.

  5. 05

    CI/CD and environments

    Pull-request checks, separate development and production targets, and automated deployments.

  6. 06

    Alerting and runbooks

    Failure and freshness alerts routed to an owner, with runbooks for the common failures.

Fixed scope and a fixed fee, agreed in writing.

Every engagement is scoped after a free 30-minute review. The scope and fee are confirmed in writing before work starts, and the assessment fee is credited toward the project.

Step 01

Pipeline health check

Fixed fee, scoped after the free review

Inventory of pipelines with run times, failures and cost, the hot spots ranked, and a fixed quote for the fixes. Credited toward delivery.

Step 02

Pipeline modernization

Fixed fee, scoped after the free review

Rebuilt pipelines delivered domain by domain, scoped from the health check.

From first call to delivery

You can stop after any step and keep everything we have written.

  1. 01

    Free

    Free 30-minute review

    We look at what you run, what is urgent and whether a fixed-fee engagement fits. You get a short written summary.

  2. 02

    1–2 weeks · credited

    Pipeline health check

    Inventory, failure and cost hot spots, and a fixed quote for the fixes.

  3. 03

    Typically 4–10 weeks

    Pipeline modernization

    Rebuilt pipelines delivered domain by domain, with tests and alerts.

Technologies

  • dbt
  • Fivetran
  • Airbyte
  • Matillion
  • Apache Airflow
  • Dagster
  • Azure Data Factory
  • AWS Glue
  • Apache Spark
  • Snowflake
  • Databricks
  • Google BigQuery
  • GitHub Actions

Names are trademarks of their owners. Listing does not imply partnership.

Questions about this service

Straight answers on scope, fees and how we work. Anything else, ask us.

Ask a question

Should we buy connectors or build them?

Managed connectors are usually cheaper than engineering time for common SaaS sources; high-volume databases and unusual APIs are often cheaper to build. The health check compares both for your sources.

Do you only work with dbt?

dbt is our default for SQL transformation because it brings tests, documentation and lineage, but we also work with Spark, Databricks notebooks, Fabric and stored procedures where they fit better.

Airflow or Dagster?

Both work well. Airflow is the most widely known; Dagster models data assets directly. We usually keep what you run and fix how it is used, unless there is a clear reason to switch.

Can you take over pipelines a previous contractor built?

Yes. That is a common starting point, and the health check documents what exists before anything changes.

Start with a free 30-minute review.

Databases, cloud or AI: talk to a senior engineer about what you run and what is coming. We reply within one business day and sign your NDA on request.