So you've mastered scraping a single page. But what about scraping an entire blog or news site with dozens, or even hundreds, of pages? The moment you need to click "Next," the complexity skyrockets.This is where most web scraping projects get messy. You start writing custom logic to find and follow pagination links, creating a fragile system that breaks the moment a website's layout changes.What if you could bypass that entire headache? In this guide, we'll build a robust script that scrapes every article from a blog and saves it to a CSV, all by leveraging an AI-powered feature that handles the hard parts for you.We're going to ustilise the AutoExtract part of the Zyte API. This returns us JSON data with the information we need, with no messing around.You'll need an API Key to start, head over here and you'll get generous free credits to try this and the rest of our Web Scraping API.


Getting Your Script Ready
First, the essentials. We'll use the requests library to communicate with the Zyte API, os to securely load our API key, and csv to save our structured data.Remember, the golden rule of credentials is never hardcode your API key. Storing it as an environment variable is the professional standard for keeping your keys safe and your code portable.
⛶import os
import requests
import csv

APIKEY = os.getenv("ZYTE_API_KEY")
if APIKEY is None:
raise Exception("No API key found. Please set the ZYTE_API_KEY environment variable.")With our environment secure, we can focus on the scraping logic.


Using articleNavigation
Here’s where we replace lines and lines of tedious code with a single parameter. We'll create a function that makes one smart request to the Zyte API. Instead of just asking for raw HTML, we set articleNavigation to True.This single instruction tells the API's machine learning model to perform a series of complex tasks automatically:
Render the page in a real browser to handle any JavaScript-loaded content.
Identify the main list of articles on the page.
Extract key details for each article (URL, headline, date, etc.) into a clean structure.
Locate the "Next Page" link to enable seamless pagination.
⛶def request_list(url):
"""
Sends a request to the Zyte API to extract article navigation data.
"""
api_response = requests.post(
"https://api.zyte.com/v1/extract",
auth=(APIKEY, ""),
json={
"url": url,
"articleNavigation": True,
# This is crucial for sites that load content with JavaScript
"articleNavigationOptions": {"extractFrom":"browserHtml"},
},
)
return api_response


Why This Crushes Manual Parsing
Let's be clear about what this one parameter replaces. Without it, you'd be stuck doing this the hard way:

The Manual Approach:


Fetch the page's HTML using a library like requests.
Realise the content is loaded by JavaScript. Now you need to bring in a heavy tool like Selenium or Playwright to control a browser instance.
Open your browser's developer tools and painstakingly inspect the HTML to find the right CSS selectors or XPath for the article list (e.g., soup.find_all('div', class_='blog-post-item')).
Write more selectors to extract the headline, URL, and date from within each list item.
Hunt down the selector for the "Next Page" button (e.g., soup.find('a', {'aria-label': 'Next'})).
Write logic to handle cases where the button might be disabled or absent on the last page.
Repeat this entire process for every website you want to scrape.


This manual process is not only time-consuming but incredibly brittle. The moment a developer changes a class name from blog-post-item to post-preview, your scraper breaks. You become a full-time maintenance engineer, constantly fixing broken selectors.The articleNavigation feature, powered by AI, understands page structure contextually. It's not looking for a specific class name; it's looking for what looks like a list of articles and a pagination link, making it vastly more resilient to minor website updates.


The Loop: Crawling from Page to Page
With our smart request function ready, we just need a loop to keep it going. A while loop is the perfect tool for the job.We give it a starting URL and let it run. In each iteration, it calls our function, adds the extracted articles to a master list, and then looks for the nextPage URL in the API response. This URL becomes the target for the next loop.The try...except block is an elegant and robust way to stop the process. When the API determines there are no more pages, the nextPage key will be missing from its response. This causes a KeyError, which we catch to cleanly exit the loop. No more complex logic to check for disabled or missing buttons!
⛶def main():
articles = []
nextPage = "https://zyte.com/learn" # Our starting point

while True:
print(f"Scraping page: {nextPage}")
resp = request_list(nextPage)

# Add the found articles to our list
for item in resp.json()["articleNavigation"]["items"]:
articles.append(item)

# Try to find the next page; if not found, we're done!
try:
nextPage = resp.json()["articleNavigation"]["nextPage"]["url"]
except KeyError:
print("Last page reached. Breaking loop.")
break


Saving Your Data to CSV
After the loop completes, we have a clean list of dictionaries, with each dictionary representing an article. The final step is saving this valuable data. Python's built-in csv library is perfect for this.The DictWriter is especially useful because it automatically uses the dictionary keys (like headline and url from the API response) as the column headers in your CSV file. This ensures your output is always well-structured and ready for analysis.
⛶def save_to_csv(articles):
"""
Saves a list of article dictionaries to a CSV file.
"""
keys = articles[0].keys() # Get headers from the first article

with open('articles.csv', 'w', newline='', encoding='utf-8') as output_file:
dict_writer = csv.DictWriter(output_file, keys)
dict_writer.writeheader()
dict_writer.writerows(articles)

print(f"
Successfully saved {len(articles)} articles to articles.csv!")And that's it. You've built a powerful, resilient, and scalable scraper that handles one of the most tedious tasks in web scraping automatically. You've saved hours of development time and future-proofed your code against trivial website changes.


Complete Code
Here is the full, commented script. Grab it, set your API key, and start pulling data the smart way.
⛶import os
import requests
import csv

# Load API key from environment variables for security
APIKEY = os.getenv("ZYTE_API_KEY")
if APIKEY is None:
raise Exception("No API key found. Please set the ZYTE_API_KEY environment variable.")

def request_list(url):
"""
Sends a request to the Zyte API to extract article navigation data.
This one function replaces manual parsing and pagination logic.
"""
print(f"Requesting data for: {url}")
api_response = requests.post(
"https://api.zyte.com/v1/extract",
auth=(APIKEY, ""),
json={
"url": url,
"articleNavigation": True,
# Ensure JS-rendered content is seen by the AI extractor
"articleNavigationOptions": {"extractFrom": "browserHtml"},
},
)
api_response.raise_for_status() # Raise an exception for bad status codes
return api_response

def save_to_csv(articles):
"""
Saves a list of article dictionaries to a CSV file.
"""
if not articles:
print("No articles to save.")
return

# Use the keys from the first article as the CSV headers
keys = articles[0].keys()

with open('articles.csv', 'w', newline='', encoding='utf-8') as output_file:
dict_writer = csv.DictWriter(output_file, keys)
dict_writer.writeheader()
dict_writer.writerows(articles)

print(f"
Successfully saved {len(articles)} articles to articles.csv!")

def main():
"""
Main function to orchestrate the scraping and saving process.
"""
articles = []
# The first page of the blog we want to scrape
nextPage = "https://zyte.com/learn"

while True:
resp = request_list(nextPage)
json_response = resp.json()

# Add the articles found on the current page to our master list
found_items = json_response.get("articleNavigation", {}).get("items", [])
if found_items:
articles.extend(found_items)

# Check for the next page URL. If it doesn't exist, break the loop.
# This is far more reliable than checking for a disabled button selector.
try:
nextPage = json_response["articleNavigation"]["nextPage"]["url"]
except (KeyError, TypeError):
print("Last page reached. Scraping complete.")
break

# Save all the collected articles to a CSV file
save_to_csv(articles)

if __name__ == "__main__":
main()

Q: Flatten a mixed array⚡ Concepts tested:
• Recursion
• Array flattening logic
• Handling mixed data types? Questions + Solutions:
? https://replit.com/@318097/DMG-R1-flatten#index.js

Hey fellow developers! ? I've been wrestling with Express.js middleware for years, and I finally put together something that doesn't make me want to pull my hair out every time I start a new project. Let me share what I've learned.You know that feeling when you're starting a new Express.js project and you're like, "Alright, time to set up middleware... again"? And then you spend the next 3 hours googling "express middleware best practices" for the hundredth time, copying random snippets from Stack Overflow, and hoping they play nice together?Yeah, I was there too. Until I got fed up and decided to build a middleware system that actually makes sense and works consistently across projects. Today, I'm sharing exactly how I did it – and trust me, your future self will thank you.


Why This Matters (And Why Most Middleware Sucks)
Here's the thing: most Express.js tutorials show you cute little middleware examples that work great in isolation but fall apart the moment you try to use them in a real application. You'll see things like:
⛶// This is what tutorials show you
app.use((req, res, next) = {
console.log('Hello World!');
next();
});Cool, but what about error handling? What about validation? What about authentication that doesn't break when you look at it wrong? What about middleware that actually helps you build something production-ready?That's where this guide comes in. I've built middleware that:

Actually handles errors properly (shocking, I know)
Validates data without making you cry
Handles authentication like a grown-up application
Plays nice with other middleware
Doesn't mysteriously break in production



The Foundation: Request Logging That Actually Helps
Let's start with something simple but incredibly useful – request logging that tells you what's actually happening:
⛶export const requestLogger = (req, res, next) = {
const start = Date.now();

console.log(`${new Date().toISOString()} - ${req.method} ${req.path}`);

// Here's the magic: override res.end to capture response time
const originalEnd = res.end;
res.end = function(...args) {
const duration = Date.now() - start;
console.log(`${req.method} ${req.path} - ${res.statusCode} - ${duration}ms`);
originalEnd.apply(this, args);
};

next();
};Why this rocks: Instead of just logging when requests come in, this tells you how long they took and what status code they returned. When something's running slow at 3 AM, you'll know exactly which endpoint is the culprit.The trick here is overriding res.end() – that's the final method Express calls when sending a response, so we can measure the total time accurately.


Error Handling That Doesn't Suck
Here's where most people mess up. They either don't handle errors at all, or they have some janky error handler that sometimes works. Here's what actually works:
⛶export const errorHandler = (err, req, res, next) = {
console.error(`Error: ${err.message}`);
console.error(err.stack);

let statusCode = 500;
let message = 'Internal Server Error';

// Handle different types of errors properly
if (err.name === 'ValidationError') {
statusCode = 400;
message = 'Invalid input data';
} else if (err.name === 'UnauthorizedError') {
statusCode = 401;
message = 'Unauthorized access';
} else if (err.statusCode) {
statusCode = err.statusCode;
message = err.message;
}

res.status(statusCode).json({
error: {
message,
// Only show stack trace in development
...(process.env.NODE_ENV === 'development' { stack: err.stack })
}
});
};Why this works: It recognizes different error types and responds appropriately. No more generic 500 errors that tell you nothing. Your API clients will actually know what went wrong.The key insight here is having a consistent error format and being smart about what information you expose in different environments.


Validation That Doesn't Make You Want to Quit
I've seen so many validation approaches that are either overly complex or completely inadequate. Here's a middle-ground approach that actually works:
⛶export const createValidator = (rules) = {
return (data) = {
const errors = [];

for (const [field, rule] of Object.entries(rules)) {
const value = data[field];

if (rule.required (value === undefined || value === null || value === '')) {
errors.push(`${field} is required`);
continue;
}

if (value !== undefined rule.type typeof value !== rule.type) {
errors.push(`${field} must be a ${rule.type}`);
}

if (value rule.minLength value.length rule.minLength) {
errors.push(`${field} must be at least ${rule.minLength} characters`);
}

if (value rule.pattern !rule.pattern.test(value)) {
errors.push(`${field} format is invalid`);
}
}

return {
isValid: errors.length === 0,
errors,
data
};
};
};And here's how you use it:
⛶const userSchema = createValidator({
username: { required: true, type: 'string', minLength: 3 },
email: {
required: true,
type: 'string',
pattern: /^[^\s@]+@[^\s@]+\.[^\s@]+$/
},
age: { type: 'number' }
});

app.post('/users', validateBody(userSchema), (req, res) = {
// req.validatedBody contains clean, validated data
res.json({ message: 'User created', data: req.validatedBody });
});Why I love this: It's simple enough to understand at a glance, flexible enough to handle most validation needs, and gives you clear error messages. No need to learn a whole validation library for basic use cases.


Authentication That Actually Secures Things
Authentication middleware is where things usually get messy. Here's a clean approach:
⛶export const authenticate = (req, res, next) = {
const authHeader = req.headers.authorization;

if (!authHeader || !authHeader.startsWith('Bearer ')) {
return res.status(401).json({
error: { message: 'No valid authentication token provided' }
});
}

const token = authHeader.substring(7);

// Replace this with your actual token validation
if (validateToken(token)) {
req.user = getUserFromToken(token);
next();
} else {
res.status(401).json({
error: { message: 'Invalid authentication token' }
});
}
};And for authorization:
⛶export const authorize = (...roles) = {
return (req, res, next) = {
if (!req.user) {
return res.status(401).json({
error: { message: 'Authentication required' }
});
}

if (roles.length !roles.includes(req.user.role)) {
return res.status(403).json({
error: { message: 'Insufficient permissions' }
});
}

next();
};
};Usage example:
⛶// Protected route
app.get('/profile', authenticate, (req, res) = {
res.json({ user: req.user });
});

// Admin-only route
app.delete('/users/:id', authenticate, authorize('admin'), (req, res) = {
res.json({ message: 'User deleted' });
});What makes this work: Clear separation between authentication (who are you?) and authorization (what can you do?). The middleware decorates the request with user info that downstream handlers can use.


Rate Limiting That Actually Prevents Abuse
Most rate limiting examples you see are either too simplistic or require Redis. Here's a practical in-memory solution that works great for most applications:
⛶const requestCounts = new Map();

export const rateLimitMiddleware = (options = {}) = {
const {
windowMs = 15 * 60 * 1000, // 15 minutes
max = 100,
message = 'Too many requests, please try again later'
} = options;

return (req, res, next) = {
const key = req.ip || req.connection.remoteAddress;
const now = Date.now();

// Clean up old entries (prevents memory leaks)
for (const [ip, data] of requestCounts.entries()) {
if (now - data.resetTime windowMs) {
requestCounts.delete(ip);
}
}

if (!requestCounts.has(key)) {
requestCounts.set(key, { count: 0, resetTime: now });
}

const counter = requestCounts.get(key);

if (now - counter.resetTime windowMs) {
counter.count = 0;
counter.resetTime = now;
}

counter.count++;

// Set standard rate limit headers
res.set({
'X-RateLimit-Limit': max,
'X-RateLimit-Remaining': Math.max(0, max - counter.count),
'X-RateLimit-Reset': new Date(counter.resetTime + windowMs)
});

if (counter.count max) {
return res.status(429).json({
error: {
message,
retryAfter: Math.ceil((counter.resetTime + windowMs - now) / 1000)
}
});
}

next();
};
};Why this approach works: It's stateless (no external dependencies), automatically cleans up old entries, follows HTTP standards for rate limiting headers, and gives clients clear information about when they can try again.


Putting It All Together
Here's how you'd use all of this in a real application:
⛶import express from 'express';
import {
setupMiddleware,
errorHandler,
notFoundHandler,
authenticate,
authorize,
validateBody,
createValidator,
rateLimitMiddleware
} from './middleware/index.js';

const app = express();

// Basic middleware setup
setupMiddleware(app);

// Public endpoint
app.get('/api/health', (req, res) = {
res.json({ status: 'ok', timestamp: new Date().toISOString() });
});

// Validated endpoint
const userSchema = createValidator({
username: { required: true, type: 'string', minLength: 3 },
email: { required: true, type: 'string', pattern: /^[^\s@]+@[^\s@]+\.[^\s@]+$/ }
});

app.post('/api/users', validateBody(userSchema), (req, res) = {
res.json({ message: 'User created', data: req.validatedBody });
});

// Protected endpoint
app.get('/api/profile', authenticate, (req, res) = {
res.json({ user: req.user });
});

// Admin endpoint with extra rate limiting
app.get('/api/admin/users',
authenticate,
authorize('admin'),
rateLimitMiddleware({ windowMs: 10 * 60 * 1000, max: 10 }),
(req, res) = {
res.json({ message: 'Admin data' });
}
);

// Error handling (always last!)
app.use(notFoundHandler);
app.use(errorHandler);

app.listen(3000, () = {
console.log('Server running on port 3000');
});


The Secret Sauce: Middleware Ordering
Here's something that trips up a lot of developers – order matters. A lot. Here's the order that actually works:

Security stuff first (CORS, security headers)

Rate limiting (before parsing, so you don't waste CPU on bad requests)

Body parsing (so other middleware can access req.body)

Logging (after parsing, so you can log request data)

Authentication (before routes that need it)
Your routes
404 handler

Error handler (always last)
Get this wrong, and you'll spend hours debugging why your middleware isn't working.


Real Talk: What This Gets You
After implementing this system across several projects, here's what I've noticed:✅ Debugging is actually possible – When something breaks, the logs tell you exactly what happened and where.✅ Onboarding new developers is smoother – The middleware is self-documenting and follows predictable patterns.✅ Security is built-in – Authentication, authorization, and rate limiting are consistent across all endpoints.✅ Testing is straightforward – Each middleware has a single responsibility and can be tested in isolation.✅ Production deployment is less scary – Error handling is consistent, logging is comprehensive, and rate limiting prevents most abuse.


Where to Go From Here
This system handles about 80% of what most applications need. As you grow, you might want to add:

Database-backed rate limiting (Redis) for multi-instance deployments

JWT token validation instead of simple token checking


Request correlation IDs for tracking requests across services

Metric collection for monitoring and alerting

Content compression and caching middleware

But honestly? Start with this. It's production-ready, well-tested, and will serve you well until you have specific reasons to add complexity.The best part? This isn't some framework-specific magic. It's just good old Express.js middleware done right. No dependencies, no vendor lock-in, just solid fundamentals that will work for years to come.What do you think? Have you built similar middleware systems? What patterns have worked (or failed spectacularly) for you? Drop a comment – I'd love to hear about your experiences!And if this helped you out, give it a clap ? and share it with your fellow developers. We've all wasted too much time on middleware that doesn't work properly.Happy coding! ?P.S. – If you're working on a team, seriously consider standardizing on something like this. Future you (and your teammates) will thank you when you're not debugging middleware interactions at 2 AM.