This article contains affiliate links. If you buy through them, we may earn a commission at no extra cost to you.
n8n Basics: How to Scrape Any Website — Step by Step
Web scraping used to mean writing Python scripts, managing proxies, and debugging broken selectors at midnight. Not anymore. With n8n, you can extract data from almost any website using a visual workflow — no code required. This guide walks you through exactly how to do it, from zero to a working scraper.
Why Use n8n for Web Scraping?
n8n is an open-source workflow automation tool that connects apps, APIs, and data sources through a drag-and-drop interface. Unlike dedicated scraping tools, n8n lets you immediately do something useful with the data you collect — send it to a spreadsheet, trigger an email, store it in a database, or pipe it into an AI model.
- No code required for most scraping tasks
- Built-in HTTP node fetches raw HTML from any public URL
- HTML Extract node parses content using CSS selectors
- Instant integrations with Google Sheets, Notion, Airtable, Slack, and more
- Self-hostable — your data stays on your server
What You Will Need
Before you start, make sure you have the following:
- A running n8n instance (cloud or self-hosted via Docker)
- The URL of the website you want to scrape
- Basic knowledge of CSS selectors (you can look these up in your browser's DevTools)
Step 1 — Create a New Workflow in n8n
Open your n8n dashboard and click New Workflow. Give it a descriptive name like Website Scraper — Product Prices. This keeps your workspace organised as your automation library grows.
Step 2 — Add a Trigger Node
Every n8n workflow starts with a trigger. For scraping, you have two common choices:
- Manual Trigger — runs the workflow when you click a button. Great for testing.
- Schedule Trigger — runs the workflow automatically at set intervals (e.g. every morning at 8:00).
Start with the Manual Trigger while building, then switch to Schedule Trigger once everything works.
Step 3 — Fetch the Page with the HTTP Request Node
Add an HTTP Request node and connect it to your trigger. Configure it like this:
- Method: GET
- URL: the full address of the page you want to scrape
- Response Format: String (this returns the raw HTML)
Click Execute Node to test it. You should see a wall of HTML in the output panel — that means it's working. The n8n HTTP node supports headers, authentication, and query parameters, so you can reach protected or paginated pages too.
Step 4 — Extract Data with the HTML Extract Node
This is where the magic happens. Add an HTML Extract node after the HTTP Request node. This node uses CSS selectors to pull specific pieces of content out of the raw HTML.
For example, if you are scraping product names from an e-commerce page, you might configure it like this:
- Source Data: Binary Data (or the data field from the previous node)
- Key:
productName - CSS Selector:
h2.product-title - Return Value: Text
- Return All Matches: Enabled (to get every product on the page)
To find the right CSS selector, right-click the element in your browser and choose Inspect. Look at the class names and tag structure, then build your selector accordingly.
Multiple Fields at Once
The HTML Extract node in n8n lets you define multiple extraction rules in one step. You can pull the product name, price, image URL, and rating all from a single node execution. Each gets its own key name that flows into the next node cleanly.
Step 5 — Clean and Transform the Data
Raw scraped data is rarely perfect. Use the Set node or Code node in n8n to:
- Trim whitespace from text fields
- Convert price strings like "€29,99" into numbers
- Add a timestamp for when the data was scraped
- Filter out items that don't match your criteria
Step 6 — Send the Data Somewhere Useful
This is where n8n outshines dedicated scraping tools. Once you have clean data, connect it to any downstream node:
- Google Sheets node — append rows to a spreadsheet automatically
- Notion node — create or update database entries
- Email node — send yourself a daily digest of scraped results
- Webhook node — push data to another system in real time
- AI / OpenAI node — summarise or classify the scraped content instantly
Handling Pagination
Many websites spread their content across multiple pages. In n8n, you can handle this with a loop. Use the Split In Batches node or a recursive workflow call to iterate through page numbers. Generate the URL for each page dynamically using an expression like https://example.com/products?page={{ $json.pageNumber }}.
Important: Respect Robots.txt and Terms of Service
Always check a website's robots.txt file and terms of service before scraping. Some sites explicitly forbid automated access. Use scraping responsibly — for personal research, price monitoring of your own competitors, or internal data aggregation. The n8n community is full of legitimate use cases that stay well within ethical boundaries.
Real Use Cases for n8n Web Scraping
- Price monitoring: track competitor prices and get alerts when they drop
- Job listing aggregation: collect job posts from multiple boards into one sheet
- News monitoring: scrape headlines from industry blogs and summarise them with AI
- Lead generation: extract contact information from public directories
- Content research: gather data for SEO analysis or market research
Troubleshooting Common Issues
The HTML Extract node returns nothing
Double-check your CSS selector in the browser first. Also make sure the Response Format in the HTTP Request node is set to String, not JSON.
The website blocks your requests
Some sites detect and block automated requests. Try adding a User-Agent header in the HTTP Request node to mimic a real browser. For heavily protected sites, consider using a headless browser integration.
The data format is inconsistent
Use the IF node in n8n to branch logic based on whether a field exists or matches a pattern. This makes your workflow resilient to minor page structure changes.
Final Thoughts
Web scraping with n8n is genuinely accessible to anyone willing to spend an afternoon learning CSS selectors. You get the power of custom data extraction combined with the integration ecosystem of a full automation platform. Whether you are monitoring prices, aggregating news, or building a data pipeline for your business, n8n gives you a clean, visual way to make it happen — without writing a single line of backend code.
Start simple: pick one website, extract one piece of data, send it to a Google Sheet. Once that works, expand. The n8n workflow system scales with your ambition.
This post was created with tools we use and recommend: n8n for workflow automation, Turbotic as an AI-native automation alternative, ElevenLabs for AI voiceover, Placid for visual content creation, and Hostinger for reliable VPS hosting. Some links are affiliate links.