Sign in to view linked content

Free Resources


Free Apache Iceberg Course


Free Copy of “Apache Iceberg: The Definitive Guide”


Free Copy of “Apache Polaris: The Definitive Guide”


2025 Apache Iceberg Architecture Guide


How to Join the Iceberg Community


Iceberg Lakehouse Engineering Video Playlist


Ultimate Apache Iceberg Resource Guide
Once data has been ingested into your system, the next step is to prepare it for actual use. This typically involves cleaning, transforming, and storing the data in a way that supports analysis, reporting, or further processing. This is where data pipelines come in, and at the center of pipeline design are two common strategies: ETL and ELT.Although they may look similar at first glance, ETL and ELT represent fundamentally different approaches to handling data transformations, and each has its strengths and trade-offs depending on the context in which it’s used.


What is ETL?
ETL stands for Extract, Transform, Load. It’s the traditional method used in many enterprise environments for years. The process starts by extracting data from source systems such as databases, APIs, or flat files. This raw data is then transformed—typically on a separate processing server or ETL engine—before it is finally loaded into a data warehouse or other destination system.For example, imagine a retail company collecting daily sales data from multiple stores. In an ETL workflow, the system might extract those records at the end of the day, standardize formats, filter out corrupted rows, aggregate sales by region, and then load the clean, transformed dataset into a reporting warehouse like Snowflake or Redshift.One of the key advantages of ETL is that it allows you to load only clean, verified data into your warehouse. That often means smaller storage footprints and potentially better performance on downstream queries.However, this approach also has limitations. Because the transformation happens before loading, you must decide upfront how the data should be shaped. If business rules change or additional use cases emerge, you may need to go back and reprocess the data.


What is ELT?
ELT reverses the order of the last two steps: Extract, Load, Transform. In this model, raw data is extracted from the source and immediately loaded into the target system—usually a cloud data warehouse that can scale horizontally. Once the data is in place, transformations are performed within the warehouse using SQL or warehouse-native tools.This approach takes advantage of the high compute power and scalability of modern cloud platforms. Instead of bottlenecking on a dedicated ETL server, the warehouse can handle complex joins, aggregations, and transformations at scale.Let’s go back to the retail example. With ELT, all sales data is loaded as-is into the warehouse. Analysts or data engineers can then write transformation scripts to reshape the data for various use cases—trend analysis, regional comparisons, or fraud detection—all without having to re-ingest or reload the source data.ELT offers more flexibility for evolving requirements, supports broader self-service analytics, and enables faster time-to-insight. The trade-off is that it requires strong governance and monitoring. Because raw data is stored in the warehouse, the risk of exposing inconsistent or unclean data is higher if transformation logic isn’t managed carefully.


Choosing Between ETL and ELT
The decision to use ETL or ELT often depends on your stack, performance needs, and organizational practices.ETL still makes sense in environments with strict data governance, limited warehouse compute resources, or scenarios where only clean data should be retained. It’s also common in legacy systems and on-premise architectures.ELT shines in modern cloud-native environments where scalability and agility are top priorities. It’s often used with platforms like Snowflake, BigQuery, or Redshift, which are built to handle large volumes of raw data and complex SQL-based transformations efficiently.In practice, many organizations use a hybrid approach. Critical data may go through an ETL flow, while experimental or rapidly evolving datasets follow an ELT pattern.


The Bigger Picture
ETL and ELT are just different roads to the same destination: getting data ready for use. As the modern data stack evolves, so do the tools and best practices for managing these flows. Whether you choose one approach or blend both, what matters most is building pipelines that are reliable, maintainable, and aligned with your organization’s goals.In the next post, we’ll focus on batch processing—the traditional foundation of many ETL workflows—and discuss how data engineers design, schedule, and optimize these processes for scale.

AI Runner is a desktop application for Linux and Windows that allows you to easily streamline AI workflows, use RAG on your own private data, have conversations with offline chatbots, and generate AI art and videos from images for free.It uses a permissive Apache 2 license and can be easily extended to create your own applications.This is one of the few desktop applications (perhaps the only) to offer a multi-modal experience to non-technical users as they are able to download a packaged version from itch.io and run a setup wizard to get up and running without any technical barriers.Take a look at my Github repo and let me know what you think. And if you like what you see, be sure to leave me a star to support the project.