When working with Mongoose and TypeScript, two helper types make your life much easier:
⛶/**
* Extracts the “plain” shape of your schema—
* just the fields you defined, without Mongoose’s built-in methods or `_id`.
*/
export type User = InferSchemaTypetypeof userSchema;

/**
* Represents a fully “hydrated” Mongoose document:
* your fields plus all of Mongoose’s methods and metadata
* (e.g. `_id`, `save()`, `populate()`, etc.).
*/
export type UserDocument = HydratedDocumentUser;

export const userModel = modelUserDocument("user", userSchema);


InferSchemaType
• Produces a pure TypeScript type from your schema definition.• Use it whenever you need just the data shape (e.g. DTOs, service inputs/outputs).


HydratedDocument
• Wraps your base type T with Mongoose’s document helpers.• Use it for any function that deals with real, database-backed
documents (e.g. returns from find, create, save).For example, in a repository interface you might write:
⛶export interface IUserRepository {
findOneByEmail(email: string): PromiseUserDocument;
findById(id: Types.ObjectId): PromiseUserDocument;
create(
createUserDto: PickCreateUserDto, 'email' | 'password',
): PromiseUserDocument;
}Here, each method clearly promises a “live” Mongoose document (with built-in methods) while elsewhere you can rely on User for pure data shapes—keeping your boundaries and types crystal clear.Let’s connect!!: ?LinkedIn
GitHub

Free Resources


Free Apache Iceberg Course


Free Copy of “Apache Iceberg: The Definitive Guide”


Free Copy of “Apache Polaris: The Definitive Guide”


2025 Apache Iceberg Architecture Guide


How to Join the Iceberg Community


Iceberg Lakehouse Engineering Video Playlist


Ultimate Apache Iceberg Resource Guide
In contrast to batch processing, where data is collected and processed in chunks, streaming data processing deals with data in motion. Instead of waiting for data to accumulate before running transformations, streaming pipelines ingest and process each piece of data as it arrives. This model enables organizations to respond to events in real time, a capability that’s becoming increasingly essential in domains like finance, security, and customer experience.In this post, we’ll unpack the core ideas behind streaming, how it works in practice, and the challenges it presents compared to traditional batch systems.


What is Streaming Data?
Streaming data refers to data that is continuously generated by various sources—website clicks, IoT sensors, user interactions, system logs—and transmitted in real time or near-real time. This data typically arrives in small payloads, often as individual events, and needs to be processed with minimal delay.The goal of a streaming pipeline is to capture this data as it’s generated, perform necessary transformations, and deliver it to its destination with as little latency as possible.A simple example would be a ride-sharing app that tracks vehicle locations in real time. As each car moves, GPS data is streamed to a backend system that updates the user interface and helps dispatch rides based on current conditions.


How Streaming Systems Work
Unlike batch jobs that execute on a schedule, streaming systems run continuously. They consume data from a source, process it incrementally, and push it to a sink—all without waiting for a dataset to be complete.At the heart of a streaming system is a message broker or event queue, which acts as a buffer between data producers and consumers. Apache Kafka is a popular choice here. It allows producers to publish events to topics, and consumers to read from those topics independently, often with strong guarantees around ordering and durability.Once events are ingested, a processing engine takes over. Tools like Apache Flink, Spark Structured Streaming, and Apache Beam allow developers to apply transformations on a per-record basis or over time-based windows. This is where operations like filtering, aggregating, joining, and enriching occur.These transformations must be designed to handle data that may arrive late, out of order, or in bursts. As such, streaming systems often implement complex logic to manage time—distinguishing between event time (when the event occurred) and processing time (when it was received)—to ensure results are accurate.


Use Cases and Business Impact
The appeal of streaming pipelines lies in their ability to power real-time applications. Fraud detection systems can flag suspicious transactions as they happen. E-commerce platforms can recommend products based on live browsing behavior. Logistics companies can monitor fleet activity and adjust routes on the fly.In operational analytics, dashboards fed by streaming data provide up-to-the-minute visibility, allowing teams to make informed decisions in response to changing conditions.Streaming is also a foundational component of event-driven architectures. When services communicate via events, streaming systems act as the glue that ties the application together, enabling asynchronous, decoupled interactions.


Challenges in Streaming Systems
Despite its power, streaming introduces complexity that shouldn’t be underestimated. Handling late or out-of-order data is a major concern. If an event shows up ten minutes after it was supposed to be processed, the system must be smart enough to either incorporate it correctly or account for the gap.State management is another critical factor. When a pipeline needs to remember information across multiple events—like keeping a running total or maintaining a session—it must manage that state reliably, often across distributed systems.There’s also the issue of fault tolerance. Streaming systems must be able to recover from crashes or network issues without duplicating results or losing data. This requires sophisticated checkpointing, replay, and exactly-once processing semantics, which tools like Flink and Beam are designed to provide.Finally, testing and debugging streaming pipelines can be more difficult than batch jobs. Because they run continuously and deal with time-sensitive data, reproducing issues often requires specialized tooling or replay mechanisms.


When to Choose Streaming
Streaming makes sense when low-latency data processing is essential to the business. This could mean operational decision-making, customer experience personalization, or complex event processing in a microservices architecture.It’s not always the right tool for the job, though. For workloads that don’t require immediate insights—or where simplicity and reliability matter more—batch processing remains the better choice.As data engineers, the key is to understand the trade-offs and choose the right pattern for each use case.In the next post, we’ll shift gears and look at how data is modeled for analytics. Understanding the differences between OLTP and OLAP systems, as well as the pros and cons of different schema designs, is critical to building pipelines that serve real business needs.

Free Resources


Free Apache Iceberg Course


Free Copy of “Apache Iceberg: The Definitive Guide”


Free Copy of “Apache Polaris: The Definitive Guide”


2025 Apache Iceberg Architecture Guide


How to Join the Iceberg Community


Iceberg Lakehouse Engineering Video Playlist


Ultimate Apache Iceberg Resource Guide
For many data engineering tasks, real-time insights aren’t necessary. In fact, a large portion of the data processed across organizations happens in scheduled intervals—daily sales reports, weekly data refreshes, monthly billing cycles. This is where batch processing comes in, and despite the growing popularity of streaming, batch remains the backbone of many data-driven workflows.In this post, we’ll explore what batch processing is, how it works under the hood, and why it’s still a critical technique in the data engineer’s toolbox.


What is Batch Processing?
Batch processing is the execution of data workflows on a predefined schedule or in response to specific triggers. Instead of processing data as it arrives, the system collects a set of data over a period of time, then processes that set as a single unit.This approach is particularly useful when data arrives in large quantities but doesn’t need to be acted on immediately. For example, processing daily transactions from a point-of-sale system or generating overnight reports for executive dashboards.Batch jobs are often triggered at set times—say, every night at 2 a.m.—and are designed to run until completion, often without user interaction. They can run for seconds, minutes, or even hours depending on the volume of data and complexity of the transformations.


Under the Hood: How Batch Jobs Work
The anatomy of a batch job usually includes several stages. First, the job identifies the data it needs to process. This might involve querying a database for all records created in the last 24 hours or scanning a specific folder in object storage for new files.Next comes the transformation phase. This is where data is cleaned, filtered, joined with other datasets, and reshaped to fit its target structure. This phase can include tasks like date formatting, currency conversion, null value imputation, or the calculation of derived fields.Finally, the job writes the transformed data to its destination—often a data warehouse, data lake, or downstream reporting system.To manage all of this, engineers rely on workflow orchestration tools. These tools provide scheduling, error handling, and logging capabilities to ensure that jobs run in the right order and can recover gracefully from failure.


Tools and Technologies
Several tools have become staples in batch-oriented workflows. Apache Airflow is one of the most widely used. It allows engineers to define complex workflows as Directed Acyclic Graphs (DAGs), where each node represents a task and dependencies are explicitly declared.Other tools like Luigi and Oozie offer similar functionality, though they are less commonly used in newer stacks. Cloud-native platforms such as AWS Glue and Google Cloud Composer provide managed orchestration services that integrate tightly with the respective cloud ecosystems.In addition to orchestration, batch jobs often depend on distributed processing engines like Apache Spark. Spark allows massive datasets to be processed in parallel across a cluster of machines, reducing processing times dramatically compared to traditional single-node tools.


Strengths and Limitations
One of the biggest advantages of batch processing is its simplicity. Since data is processed in chunks, you can apply robust validation and error-handling routines before moving data downstream. It's also easier to track and audit, which is especially important for regulated industries.Batch jobs are also cost-efficient when working with large volumes of data that don’t require immediate availability. Processing once per day means you can spin up compute resources only when needed, rather than keeping systems running continuously.However, the main limitation is latency. If something happens in your business—say, a spike in fraudulent transactions—you won’t know about it until after the next batch job runs. For use cases that require faster insights or real-time responsiveness, batch processing isn’t sufficient.There’s also the issue of windowing and completeness. Since batch jobs process data in slices, late-arriving records can fall outside the intended window unless carefully managed. This adds complexity to pipeline design and requires thoughtful handling of time-based logic.


Where Batch Still Shines
Despite its limitations, batch processing remains ideal for a wide range of use cases. Financial reconciliations, data archival, slow-changing dimensional data updates, and long-running analytics workloads are just a few examples where batch continues to dominate.As a data engineer, understanding how to design efficient and reliable batch workflows is an essential skill, especially in environments where consistency and auditability are critical.In the next post, we’ll explore the counterpart to batch: streaming data processing. We’ll look at what it means to process data in real time, how it differs from batch, and what patterns and tools make it work.