Free Resources


Free Apache Iceberg Course


Free Copy of “Apache Iceberg: The Definitive Guide”


Free Copy of “Apache Polaris: The Definitive Guide”


2025 Apache Iceberg Architecture Guide


How to Join the Iceberg Community


Iceberg Lakehouse Engineering Video Playlist


Ultimate Apache Iceberg Resource Guide
In contrast to batch processing, where data is collected and processed in chunks, streaming data processing deals with data in motion. Instead of waiting for data to accumulate before running transformations, streaming pipelines ingest and process each piece of data as it arrives. This model enables organizations to respond to events in real time, a capability that’s becoming increasingly essential in domains like finance, security, and customer experience.In this post, we’ll unpack the core ideas behind streaming, how it works in practice, and the challenges it presents compared to traditional batch systems.


What is Streaming Data?
Streaming data refers to data that is continuously generated by various sources—website clicks, IoT sensors, user interactions, system logs—and transmitted in real time or near-real time. This data typically arrives in small payloads, often as individual events, and needs to be processed with minimal delay.The goal of a streaming pipeline is to capture this data as it’s generated, perform necessary transformations, and deliver it to its destination with as little latency as possible.A simple example would be a ride-sharing app that tracks vehicle locations in real time. As each car moves, GPS data is streamed to a backend system that updates the user interface and helps dispatch rides based on current conditions.


How Streaming Systems Work
Unlike batch jobs that execute on a schedule, streaming systems run continuously. They consume data from a source, process it incrementally, and push it to a sink—all without waiting for a dataset to be complete.At the heart of a streaming system is a message broker or event queue, which acts as a buffer between data producers and consumers. Apache Kafka is a popular choice here. It allows producers to publish events to topics, and consumers to read from those topics independently, often with strong guarantees around ordering and durability.Once events are ingested, a processing engine takes over. Tools like Apache Flink, Spark Structured Streaming, and Apache Beam allow developers to apply transformations on a per-record basis or over time-based windows. This is where operations like filtering, aggregating, joining, and enriching occur.These transformations must be designed to handle data that may arrive late, out of order, or in bursts. As such, streaming systems often implement complex logic to manage time—distinguishing between event time (when the event occurred) and processing time (when it was received)—to ensure results are accurate.


Use Cases and Business Impact
The appeal of streaming pipelines lies in their ability to power real-time applications. Fraud detection systems can flag suspicious transactions as they happen. E-commerce platforms can recommend products based on live browsing behavior. Logistics companies can monitor fleet activity and adjust routes on the fly.In operational analytics, dashboards fed by streaming data provide up-to-the-minute visibility, allowing teams to make informed decisions in response to changing conditions.Streaming is also a foundational component of event-driven architectures. When services communicate via events, streaming systems act as the glue that ties the application together, enabling asynchronous, decoupled interactions.


Challenges in Streaming Systems
Despite its power, streaming introduces complexity that shouldn’t be underestimated. Handling late or out-of-order data is a major concern. If an event shows up ten minutes after it was supposed to be processed, the system must be smart enough to either incorporate it correctly or account for the gap.State management is another critical factor. When a pipeline needs to remember information across multiple events—like keeping a running total or maintaining a session—it must manage that state reliably, often across distributed systems.There’s also the issue of fault tolerance. Streaming systems must be able to recover from crashes or network issues without duplicating results or losing data. This requires sophisticated checkpointing, replay, and exactly-once processing semantics, which tools like Flink and Beam are designed to provide.Finally, testing and debugging streaming pipelines can be more difficult than batch jobs. Because they run continuously and deal with time-sensitive data, reproducing issues often requires specialized tooling or replay mechanisms.


When to Choose Streaming
Streaming makes sense when low-latency data processing is essential to the business. This could mean operational decision-making, customer experience personalization, or complex event processing in a microservices architecture.It’s not always the right tool for the job, though. For workloads that don’t require immediate insights—or where simplicity and reliability matter more—batch processing remains the better choice.As data engineers, the key is to understand the trade-offs and choose the right pattern for each use case.In the next post, we’ll shift gears and look at how data is modeled for analytics. Understanding the differences between OLTP and OLAP systems, as well as the pros and cons of different schema designs, is critical to building pipelines that serve real business needs.

Free Resources


Free Apache Iceberg Course


Free Copy of “Apache Iceberg: The Definitive Guide”


Free Copy of “Apache Polaris: The Definitive Guide”


2025 Apache Iceberg Architecture Guide


How to Join the Iceberg Community


Iceberg Lakehouse Engineering Video Playlist


Ultimate Apache Iceberg Resource Guide
For many data engineering tasks, real-time insights aren’t necessary. In fact, a large portion of the data processed across organizations happens in scheduled intervals—daily sales reports, weekly data refreshes, monthly billing cycles. This is where batch processing comes in, and despite the growing popularity of streaming, batch remains the backbone of many data-driven workflows.In this post, we’ll explore what batch processing is, how it works under the hood, and why it’s still a critical technique in the data engineer’s toolbox.


What is Batch Processing?
Batch processing is the execution of data workflows on a predefined schedule or in response to specific triggers. Instead of processing data as it arrives, the system collects a set of data over a period of time, then processes that set as a single unit.This approach is particularly useful when data arrives in large quantities but doesn’t need to be acted on immediately. For example, processing daily transactions from a point-of-sale system or generating overnight reports for executive dashboards.Batch jobs are often triggered at set times—say, every night at 2 a.m.—and are designed to run until completion, often without user interaction. They can run for seconds, minutes, or even hours depending on the volume of data and complexity of the transformations.


Under the Hood: How Batch Jobs Work
The anatomy of a batch job usually includes several stages. First, the job identifies the data it needs to process. This might involve querying a database for all records created in the last 24 hours or scanning a specific folder in object storage for new files.Next comes the transformation phase. This is where data is cleaned, filtered, joined with other datasets, and reshaped to fit its target structure. This phase can include tasks like date formatting, currency conversion, null value imputation, or the calculation of derived fields.Finally, the job writes the transformed data to its destination—often a data warehouse, data lake, or downstream reporting system.To manage all of this, engineers rely on workflow orchestration tools. These tools provide scheduling, error handling, and logging capabilities to ensure that jobs run in the right order and can recover gracefully from failure.


Tools and Technologies
Several tools have become staples in batch-oriented workflows. Apache Airflow is one of the most widely used. It allows engineers to define complex workflows as Directed Acyclic Graphs (DAGs), where each node represents a task and dependencies are explicitly declared.Other tools like Luigi and Oozie offer similar functionality, though they are less commonly used in newer stacks. Cloud-native platforms such as AWS Glue and Google Cloud Composer provide managed orchestration services that integrate tightly with the respective cloud ecosystems.In addition to orchestration, batch jobs often depend on distributed processing engines like Apache Spark. Spark allows massive datasets to be processed in parallel across a cluster of machines, reducing processing times dramatically compared to traditional single-node tools.


Strengths and Limitations
One of the biggest advantages of batch processing is its simplicity. Since data is processed in chunks, you can apply robust validation and error-handling routines before moving data downstream. It's also easier to track and audit, which is especially important for regulated industries.Batch jobs are also cost-efficient when working with large volumes of data that don’t require immediate availability. Processing once per day means you can spin up compute resources only when needed, rather than keeping systems running continuously.However, the main limitation is latency. If something happens in your business—say, a spike in fraudulent transactions—you won’t know about it until after the next batch job runs. For use cases that require faster insights or real-time responsiveness, batch processing isn’t sufficient.There’s also the issue of windowing and completeness. Since batch jobs process data in slices, late-arriving records can fall outside the intended window unless carefully managed. This adds complexity to pipeline design and requires thoughtful handling of time-based logic.


Where Batch Still Shines
Despite its limitations, batch processing remains ideal for a wide range of use cases. Financial reconciliations, data archival, slow-changing dimensional data updates, and long-running analytics workloads are just a few examples where batch continues to dominate.As a data engineer, understanding how to design efficient and reliable batch workflows is an essential skill, especially in environments where consistency and auditability are critical.In the next post, we’ll explore the counterpart to batch: streaming data processing. We’ll look at what it means to process data in real time, how it differs from batch, and what patterns and tools make it work.

What is the optimization in react:
Typically optimization in react refers to improve the user experience by reducing the load time and minimizing the unnecessary re-rendering in the DOM.
Developers have handled optimization manually by applying techniques
like lazy loading , React.memo and using optimization hooks such as useMemo and useCallBack.
Before diving into the new feature of React compiler presented in react 19 version, let us understand the techniques developers have been using to improve performance and optimization.1. Performance Hooks:- UseMemo :In JavaScript, objects and arrays are passed by reference. This means that during a re-render, even if their content doesn't change, a new reference is created.
In the context of react , re-rendering will create new reference for the object/array, so we need to cache the result of a calculation between re-renders.
UseMemo come to address this issue and returns a memoized value.
let us understand by the example below when a user object will be memoized and will refer to a new reference only when one of its dependencies changes.
⛶import React, { useState, useMemo } from "react";

const UserProfile = () = {
const [username, setUsername] = useState("");
const [age, setAge] = useState(20);

// user object is stable unless username or age changes
const user = useMemo(() = {
return {
name: username,
age: age,
};
}, [username, age]);

return (
div
input
placeholder="Enter name"
value={username}
onChange={(e) = setUsername(e.target.value)}
/
p{user.name} is {user.age} years old/p
/div
);
};-useCallBack:The useCallback is similar to useMemo Hooks. The main difference is that useCallback returns a memoized function instead of memoized value.
let's understand the example below from w3school.com
without useCallbackat each time countchange Appcomponent re-rendering so a new reference of addTodofunction will be created and then Todoscomponent will be re-rendered (as addTodo one of its props).
⛶import { useState, useCallback } from "react";
import ReactDOM from "react-dom/client";
import Todos from "./Todos";

const App = () = {
const [count, setCount] = useState(0);
const [todos, setTodos] = useState([]);

const increment = () = {
setCount((c) = c + 1);
};
const addTodo = useCallback(() = {
setTodos((t) = [...t, "New Todo"]);
}, [todos]);

return (

Todos todos={todos} addTodo={addTodo} /
hr /
div
Count: {count}
button onClick={increment}+/button
/div
/
);
};

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(App /);
//Todo component
import { memo } from "react";

const Todos = ({ todos, addTodo }) = {
console.log("child render");
return (

h2My Todos/h2
{todos.map((todo, index) = {
return p key={index}{todo}/p;
})}
button onClick={addTodo}Add Todo/button
/
);
};

export default memo(Todos);2. React memo:By default a child component will be re-rendered if its parent component re-renders , using memo will cause React to skip rendering a component if its props have not changed.
The example used in useCallBack example already used React.memo
to avoid re-rendering when count state changed (countstate will only cause re-rendering in the Appcomponent , not in the child component Todos.3. Lazy loading:It is used for code splitting , in React, it is a technique that allows you to load components, modules, or assets asynchronously, improving the loading time of your application. It can be achieved by using the built-in React.lazy() method and Suspense component.
the example below show the way to implement it ,Loadingcomponent here is the UI to display when the loading is processing , it could be simply
⛶p....Loadingp⛶import { lazy, Suspense } from 'react';

const MarkdownPreview = lazy(() = import('./MarkdownPreview'));

function App() {
return (
Suspense fallback={Loading /}
h2Preview/h2
MarkdownPreview /
/Suspense
);
}4. React 19: the react compiler RC:So now as we are familiar with the manual way of fixing optimization and implementing the necessary techniques that address it.
The react compiler presented in the 19th version comes essentially to enhance the performance and optimization without the intervention of developers.
And as it states its documentation that it’s easy to forget applying memoization or applying them incorrectly can lead to inefficient updates as React has to check parts of your UI that don’t have any meaningful changes.
The compiler uses its knowledge of JavaScript and React’s rules to automatically memoize values or groups of values within your components and hooks.


Conclusion:
Should we handle optimization manually ? short answer, No, React compiler will do it for us.
Is still worth to understand techniques of optimization in react?
*Yes * it worths as majoriy of projects and apps used by companies still need the manual way, in addition, according to react documentation RC isn't stable yet even it was tested by Meta in production.