What Is a Machine Learning Pipeline? Stages, Architecture, and What Breaks in Production

Every machine learning project I have seen fail in production failed for the same reason: the model was fine, and everything around it was improvised.

A machine learning pipeline is the automated, versioned sequence of steps that takes raw data through validation, feature engineering, training, evaluation, and deployment, and keeps the deployed model monitored and retrained - so that the same code path runs in training and in production. The last clause is the whole point. Everything else in this guide is detail.

This is written for engineers and technical founders who have a model that works in a notebook and need it to work on Tuesday at 3 a.m. with nobody watching. I will walk through the seven stages, the architecture that connects them, how to pick an orchestrator without a vendor telling you, one real example with code, what changes for deep learning, and the failure modes that vendor documentation never mentions because vendors do not run your pipeline.

What is a machine learning pipeline, and what it is not

The word pipeline in machine learning gets used for three different things, and the confusion costs real money.

An ML pipeline in the sense of this article is the end-to-end system: data in, monitored model out, repeatable without a human.

A scikit-learn Pipeline object is a much smaller thing: a chain of transformers and an estimator inside one process. It is a building block of stage three and four, not the whole system. Half of the tutorials ranking for this term describe only this.

A data pipeline moves and transforms data for any consumer - warehouses, dashboards, models. A machine learning data pipeline is the subset that feeds features to training and serving. If your company already has data engineers, the ML pipeline starts where their pipeline ends.

MLOps is the practice of running ML pipelines reliably: versioning, CI/CD for models, monitoring. The pipeline is the artifact. MLOps is the discipline.

Why bother with a pipeline at all? Because a model retrained by hand once a quarter is a model that silently degrades for two of those three months. Machine learning pipelines exist to make retraining boring. A model somebody has to nudge by hand every quarter is not intelligent automation, it is a recurring chore with a deadline.

Machine learning pipeline steps: the seven stages

Different sources count differently. IBM describes three stages with sixteen sub-steps. Google describes four pipelines. The AI Overview on this query lists six. I use seven because six always hides monitoring inside deployment, and monitoring is where production pipelines actually live.

Machine learning pipelines differ in tooling far more than in structure: the stages below are the same whether the orchestrator is Airflow or a cron job. For each stage: what it does, what artifact it produces, and how it fails.

1. Data ingestion

Pull raw data from sources - databases, event streams, files, third-party APIs - into a place the pipeline owns, usually a table in a warehouse or a partition in a data lake. Data lakes hold the raw and semi-structured material, data warehouses hold the modeled tables; a pipeline normally reads from both. The artifact is a versioned snapshot: this run trained on this data, and you can prove it.

Fails when: the upstream schema changes and nobody told you. A renamed column in a source table is the most common cause of a pipeline that “worked yesterday.”

2. Validation and preprocessing

Check the snapshot against expectations before spending compute on it: row counts, null rates, value ranges, category sets. Then clean: deduplicate, fix types, handle missing values with a rule you wrote down. The artifact is a validation report and a clean dataset.

Fails when: validation is skipped because “the data has always been fine.” It was fine until a bug upstream started writing zeros into a price column, and the model learned that everything is free.

3. Feature engineering

Turn clean records into the numbers the model sees: encodings, aggregations, time windows, embeddings. The artifact is a feature set plus the code that produced it - and that code has to be importable by the serving system. This is the stage where training-serving skew is born or prevented.

Fails when: features are computed with pandas in training and re-implemented in Java for serving. Two implementations of “average order value over 30 days” will disagree, and the model will be wrong in a way no test catches.

4. Model training

Fit the model on a training split. The stage looks the same whether the task is supervised learning on labeled rows, unsupervised clustering, or reinforcement learning against a simulator - what differs is where the training signal comes from. Record hyperparameters, random seeds, data version, code version, and environment. The artifact is a model file plus the metadata that makes the run reproducible.

Fails when: the winning experiment cannot be reproduced because the seed was not set, or the notebook was edited after the run, or the training data was a local CSV nobody saved.

5. Evaluation

Score the model on a held-out set that was split before any feature engineering, and compare against two thresholds: the previous production model, and the business number you agreed on before the project started. The artifact is an evaluation report and a go/no-go decision.

Fails when: the held-out set leaks. Temporal leakage - training on data from after the prediction time - is the classic. The model looks excellent offline and useless online.

6. Deployment and serving

Package the model and the feature code, register the version, and serve it - as a batch job that writes predictions to a table, or as a real-time API. The artifact is a registered, addressable model version and a serving endpoint with a rollback path.

Fails when: deployment is a manual copy of a pickle file to a server. No version, no rollback, no idea which model answered which request.

7. Monitoring and retraining

Watch three things: the inputs (feature drift), the outputs (prediction distribution), and, when labels arrive, the accuracy. Retrain on a schedule or a trigger, but only promote the new model if it passes stage five. The artifact is a dashboard, an alert, and a retraining run that can be traced back to what caused it.

Fails when: retraining is automatic and promotion is automatic. A bad batch of data trains a bad model that replaces a good one at 2 a.m. Gate the promotion.

Machine learning pipeline architecture

Here is the architecture we use as the default for tabular and ranking problems. The machine learning pipeline diagram below is simplified, but every box exists in production.

The shape to notice: two paths, one code package.

Training path and serving path

The training path runs on a schedule or a trigger. It reads historical data, validates it, computes features, trains, evaluates against the current production model, and registers the winner.

The serving path runs on every request or every batch. It reads live data, computes features, loads the registered model, and returns predictions. It also logs every prediction with its features and model version, because those logs are next month’s training data.

Both paths import the same feature package. Not the same logic rewritten twice. The same code, the same version. This one decision prevents most of the skew problems in the failure section below.

Feature store, model registry, metadata store

You do not need to buy all three on day one, but you need the function of each.

• A feature store is a table keyed by entity and time where features are written once and read by both paths. Early on, a warehouse table with a timestamp column does the job.
• A model registry maps a model name to versions, each with its metrics and the data and code versions that produced it. MLflow does this well and free.
• A metadata store records every run: who, when, which inputs, which outputs. It is what lets you answer “why did predictions change on the 14th?”

Batch vs real-time

A batch pipeline scores everything once a day and writes to a table. A real-time pipeline scores one record per request under a latency budget. Most machine learning pipelines in production are batch, and most business problems are batch problems dressed up as real-time ones. Ask what decision needs the prediction and when. If the answer is “in the morning report,” a nightly job beats a 50 ms API by a wide margin in cost and complexity. An end-to-end machine learning pipeline for a real-time case is the same seven stages with a serving layer that has to be engineered for latency and failover.

Machine learning pipeline orchestration tools

Machine learning pipelines need something to run the stages in order, on a schedule, with retries and logs. That is the orchestrator. Pick it last, after the stages exist as functions with clear inputs and outputs. Here is how I would choose in 2026.

Tool	What it is	Pick it when	Skip it when
Airflow	General-purpose DAG scheduler	You already run it for data pipelines; batch ML fits in	You need Kubernetes-native isolation per step or you have no ops team
Kubeflow Pipelines	Kubernetes-native ML workflow engine	You run on Kubernetes and need GPU steps, isolation, and artifact tracking	You have three models and no Kubernetes
Vertex AI / SageMaker / Azure ML	Managed pipelines from the cloud vendor	You are all-in on one cloud and want the fastest path to a running pipeline	You need portability, or the pricing surprises you at scale
Prefect / Dagster	Modern Python-first orchestrators	You want typed assets and a better developer experience than Airflow	Your platform team has standardized on something else
Metaflow / ZenML	Opinionated ML-specific frameworks	A small team wants versioning and deployment conventions out of the box	You need to bend the framework to an existing platform
MLflow	Tracking, registry, and packaging (not an orchestrator)	Always, for experiment tracking and the model registry	Never; just do not expect it to schedule anything
scikit-learn Pipeline	In-process preprocess-and-fit chain	Always, inside stage three and four, to keep train and serve identical	As a substitute for any of the above

The pattern I see work for teams under ten engineers: MLflow for tracking and registry, one Python orchestrator (Prefect or Dagster, or Airflow if it is already there), a warehouse table as the feature store, and a plain container for serving. Kubeflow and full managed platforms earn their complexity when you have many models and a platform team to own them.

A machine learning pipeline example

A worked example with the details simplified. It is the shape we build for staffing-tech clients like the one in our AI sourcing agent case study, where the search layer already runs at roughly a thousand requests per second and the ranking model behind it has to be retrained without anyone noticing.

The problem. A recruiting platform needed to rank candidates for open roles. The first model was built in a notebook by one data scientist and scored candidates once a week by hand. Recruiters stopped trusting it within a month because the scores changed in ways nobody could explain.

The pipeline we built, stage by stage:

1. Ingestion. Nightly snapshot of candidates, roles, and recruiter actions from the production database into a warehouse, with a snapshot_date on every row.
2. Validation. Great Expectations checks on row counts and null rates; a failed check stops the run and posts to Slack instead of training on broken data.
3. Features. One Python package, features/, with functions that take a candidate-role pair and a timestamp and return a feature vector. Training and serving both import it.
4. Training. A gradient-boosted ranker, trained on a rolling window of recruiter actions - long enough to cover a full hiring cycle, short enough that last year’s market does not dominate - with the seed, data snapshot, and git commit logged to MLflow.
5. Evaluation. NDCG at 10 on a time-based split, compared against the current production model. Promotion only if it beats production by a margin we agreed with the client.
6. Serving. A FastAPI container that loads the registered model version and computes features from the same package. Every response logged with features and model version.
7. Monitoring. Daily drift report on the top twenty features, a weekly accuracy check once recruiter outcomes arrive, and a retraining trigger that fires when drift on those features crosses the threshold agreed at launch or when the weekly check falls below the previous model.

What it changed. Recruiters get a score that is the same on Friday as on Monday unless the data changed, and a log that says why when it did. The part I care about most: retraining stopped being an event. It runs on its trigger, passes the gate or does not, and nobody is paged either way.

The parity trick in code. The single most valuable pattern is the one below: the feature preprocessing lives inside the pipeline object that gets saved, so serving cannot drift from training.

# features/build.py - imported by BOTH the training job and the serving API
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

NUMERIC = ["years_experience", "skills_overlap", "days_since_active"]
CATEGORICAL = ["seniority", "location_bucket"]

def build_pipeline(model):
    preprocess = ColumnTransformer([
        ("num", StandardScaler(), NUMERIC),
        ("cat", OneHotEncoder(handle_unknown="ignore"), CATEGORICAL),
    ])
    return Pipeline([("features", preprocess), ("model", model)])

# training job
# pipe = build_pipeline(LGBMRanker(...)); pipe.fit(X_train, y_train)
# mlflow.sklearn.log_model(pipe, "ranker")

# serving API
# pipe = mlflow.sklearn.load_model("models:/ranker/Production")
# pipe.predict(live_dataframe)   # same transforms, same encoder categories

The encoder categories, the scaler means, and the model weights travel together as one artifact. If you take one thing from this article, take that.

Deep learning pipelines: what changes

Deep learning pipelines follow the same seven stages, but four things get harder.

Data loading becomes the bottleneck. Images, audio, and long documents do not fit in memory. The pipeline needs streaming loaders, sharded datasets, and a preprocessing step that runs once and caches, not on every epoch.

Training needs GPU scheduling. A training step that needs four GPUs for six hours has to reserve them, checkpoint every N steps, and resume from the last checkpoint when the spot instance disappears. The orchestrator has to understand that a step can fail halfway and should not restart from zero.

Labels have their own pipeline. For most vision and speech projects, the label pipeline - collecting, reviewing, versioning annotations - is bigger than the model pipeline. Treat it as stage one, not as a prerequisite someone else handles. When labels are the bottleneck, the techniques in our guide to dealing with a lack of data apply.

Evaluation gets expensive. Running the validation set through a large model takes real time and money. Sample it for the fast gate, run the full set nightly.

Everything else - the shared feature code, the registry, the gated promotion - applies unchanged. If anything, it matters more, because a deep learning retrain costs thousands of dollars and you do not want to run it twice.

Where machine learning pipelines break in production

None of this is in the vendor documentation, because vendors do not operate your pipeline. From ours:

Training-serving skew. The model sees features computed one way in training and another in serving. Cause: two implementations. Fix: one package, imported by both, tested with the same inputs.

Silent schema changes upstream. A column gets renamed, a unit changes from cents to dollars, an enum gains a value. Nothing crashes. Predictions quietly go wrong. Fix: validation at ingestion with hard failures.

Label leakage. A feature that is only known after the outcome sneaks into training. Offline metrics look superb. Fix: point-in-time correctness in feature computation, and an engineer who is paranoid about it.

Feature drift vs concept drift. Inputs change distribution (drift), or the relationship between inputs and outputs changes (concept drift). Different problems: the first often needs no retrain, the second always does. Fix: monitor both separately.

Non-reproducible training. The model in production cannot be rebuilt because the seed, the data version, or the dependency versions were not recorded. Fix: log all three on every run, and rebuild the production model once a quarter to prove you can.

Retraining that overwrites a good model. Automatic retrain plus automatic promotion equals an incident at 2 a.m. Fix: evaluation gate before promotion, and a rollback that takes one command.

GPU quota blocks the retrain. The scheduled retrain cannot get the instances it needs and silently skips. Fix: alert on “did not run,” not just on “failed.”

Works in the notebook, dies in the orchestrator. Different Python version, different library pins, a file path that only exists on one laptop. Fix: the training step runs in the same container image locally and in the orchestrator.

When you don’t need a pipeline

One model, retrained by hand every quarter, feeding a monthly report? A notebook and a calendar reminder are fine. The pipeline earns its cost when retraining is frequent, when predictions drive automated decisions, or when more than one person has to be able to run it. If you are deciding whether your data science team builds this in-house or you bring in an outside one, the data science consulting buyer’s guide covers the trade-offs and the prices, and our data science consulting services page has the engagement formats. Build the pipeline when the cost of the model being silently wrong exceeds the cost of building it. For most production systems that day comes sooner than the team expects.

FAQ

What is a machine learning pipeline?

A machine learning pipeline is the automated, versioned sequence of steps that takes raw data through validation, feature engineering, training, evaluation, and deployment, then keeps the deployed model monitored and retrained. The defining property is that training and serving share the same code path. In short, a pipeline in machine learning is the system, not the scikit-learn object.

What are the stages of a machine learning pipeline?

Seven, in practice: data ingestion, validation and preprocessing, feature engineering, training, evaluation, deployment and serving, and monitoring with retraining. Vendor docs collapse some into six or expand them into a dozen sub-steps; the seven cover every one of them.

How do I build a machine learning pipeline?

Start with a notebook that trains one model end to end. Move each stage into a function with explicit inputs and outputs. Put the feature code in a package that both training and serving import. Add data validation before training and an evaluation gate before deployment. Only then choose an orchestrator.

What are the 7 stages of machine learning model development?

Problem framing and data ingestion, data validation and cleaning, feature engineering, model training, evaluation against a held-out set and a business threshold, deployment to a serving system, and monitoring with a retraining trigger. Stage seven is the one most projects skip.

What is the architecture of an ML pipeline?

Two paths that share code. The training path reads historical data, computes features, trains, evaluates, and registers a model. The serving path computes the same features on live data and calls the registered model. A feature store, a model registry, and a metadata store connect the two, and a monitoring loop feeds drift signals back into retraining.

The short version

Machine learning pipelines are not schedulers with models in them. It is the discipline that makes training and serving the same system, makes every model rebuildable, and makes retraining boring. Build the seven stages as functions first, share the feature code, gate every promotion, and pick the orchestrator last.

If you have a model that works in a notebook and a product that needs it to work every night, that gap is most of what we do. Our custom AI development engagements start with a fixed-price pilot that ends with the pipeline running in your infrastructure, and the case studies show what that looked like for other teams.