Modernising data transformation with DBT

The project addressed several key issues:

• Talend didn’t allow for efficient collaborative working (only two simultaneous connections possible).
• The Snowflake views accumulated over time were poorly organised, creating maintenance and reliability problems.
• Undetected discrepancies occurred due to a lack of data quality testing.
• The pipelines were managed solely by the Data Engineers, which blocked the analysts’ autonomy.

The objective: migrate to dbt to improve data quality, team collaboration and processing scalability.

The migration project was divided into four main stages:

1. Rethinking architecture

A new dbt architecture has been designed around best practices:

• Clear separation of layers (staging, intermediate, mart)
• SQL modularity
• Integrated documentation
• Versioning via Git
  This architecture was co-constructed with the customer’s data team to take account of specific business requirements, notably the complex flows of retail in Asia.

2. Upgrade teams’ skills on dbt

A progressive training program has been set up to guarantee autonomy:

• Hands-on workshops on dbt fundamentals (testing, documentation, jinja…)
• Peer-programming sessions on concrete cases
• Internal guides adapted to customer context (naming, structure, tests, etc.)
• Conduct regular code reviews to embed best practices.

3. Technical migration from Talend to dbt

Talend pipelines were :

• Analysed and simplified
• Rewritten in reusable SQL in dbt
• Completed with existing Snowflake views transformed into tested and documented dbt models
• Equipped with automatic tests (uniqueness, presence, consistency)

4. Validate migration and bring teams on board

Running the process twice enabled us to compare the results of Talend and dbt, correct any discrepancies, reassure the business teams and adjust the workflows.
Meanwhile, Tableau analysts familiarised themselves with the dbt tool, learning how to understand models, navigate documentation and perform transformations.

Measurable gains

• Reduce pipeline development and maintenance time
• Significant improvement in data quality thanks to automatic testing
• Complete traceability with dbt’s integrated data lineage
• Eliminate dependency on a proprietary tool (Talend) and costs reduction

Better teamwork

• No more silos between Data Engineers and Analysts: everyone can contribute to the models
• Clear documentation, rapid error identification, easy onboarding
• More agile processes with faster deliveries and better quality control

Very positive feedback

• Tableau analysts praise models’ legibility
• Developers appreciate debugging speed and ease of use
• Business units have reported an improvement in the reliability of data, particularly in Asia.

A solid outlook

Today, the dbt architecture is in production across the customer’s entire data perimeter, with :

• A collaborative interface via dbt Cloud
• Centralised monitoring
• A transformation platform ready for self-service

Next steps include :

• Reinforced governance through structured versioning and data contracts
• Extending tests to business quality rules
• Better integration with analysis and observability tools

This project marks a key stage in the modernization of the customer’s data ecosystem, laying the foundations for a robust, agile and collaborative platform.