How to Write a Script That Finds Every Parameterized URL Variant on Your Site

Before you can fix duplicate content caused by URL parameters, you need to know the actual scope. Guessing from a few examples in Search Console misses most of the problem. This walks through a small Python script that parses server logs and groups requests by their base path, so you can see exactly how many parameter variants each route is generating.

Why log parsing beats Search Console alone

Search Console tells you which duplicate issues Google noticed, but it samples and aggregates, so smaller-volume parameter combinations don't always surface individually. Your own server logs (or a CDN's access logs) contain every request that actually happened, which gives you the real count.

Setting up the parser

This assumes a standard combined log format, common to Apache and nginx by default:

import re
from collections import defaultdict
from urllib.parse import urlparse, parse_qs
LOG_PATTERN = re.compile(
    r'"(?:GET|POST) (?P<path>[^\s]+) HTTP'
)
def parse_log_line(line):
    match = LOG_PATTERN.search(line)
    if not match:
        return None
    return match.group("path")
def group_by_base_path(log_file):
    groups = defaultdict(set)
    with open(log_file, "r", encoding="utf-8", errors="ignore") as f:
        for line in f:
            path = parse_log_line(line)
            if not path:
                continue
            parsed = urlparse(path)
            base = parsed.path
            query = parse_qs(parsed.query)
            if query:
                param_signature = tuple(sorted(query.keys()))
                groups[base].add(param_signature)
    return groups

Reading the results

Once you've grouped requests, sort by the number of distinct parameter signatures per base path to find your worst offenders:

groups = group_by_base_path("access.log")
ranked = sorted(groups.items(), key=lambda kv: len(kv[1]), reverse=True)
for base_path, signatures in ranked[:20]:
    print(f"{base_path}: {len(signatures)} distinct parameter combinations")

A route showing forty or fifty distinct parameter combinations against a handful of actual product variations is a clear candidate for canonical tag cleanup, and probably a robots.txt rule for anything that's pure tracking noise.

Extending it to check canonical tags directly

Once you have a list of the worst offending base paths, cross-reference a sample of each parameter variant's actual canonical tag using a lightweight request:

import requests
from bs4 import BeautifulSoup
def get_canonical(url):
    resp = requests.get(url, timeout=10)
    soup = BeautifulSoup(resp.text, "html.parser")
    tag = soup.find("link", rel="canonical")
    return tag["href"] if tag else None

Run this against ten or so parameter variants per base path and confirm they all resolve to the same canonical URL. Inconsistent results here point to a templating bug rather than a missing rule, which is a different fix.

Filtering out bot traffic that isn't Googlebot

Not every request in your logs represents crawl budget you care about optimizing. Scrapers, monitoring services, and other bots hit the same URLs, and including them in your analysis inflates the apparent problem or, worse, hides the real Googlebot pattern underneath noise from unrelated traffic. Filter by user agent before counting:

def is_googlebot(line):
    return "Googlebot" in line
def group_googlebot_only(log_file):
    groups = defaultdict(set)
    with open_log(log_file) as f:
        for line in f:
            if not is_googlebot(line):
                continue
            path = parse_log_line(line)
            if not path:
                continue
            parsed = urlparse(path)
            query = parse_qs(parsed.query)
            if query:
                signature = tuple(sorted(query.keys()))
                groups[parsed.path].add(signature)
    return groups

A simple user agent string match is a reasonable starting filter, though be aware that user agent strings can be spoofed. If you need to confirm requests are genuinely from Google's crawlers rather than something pretending to be, a reverse DNS lookup against the requesting IP, checked against Google's published IP ranges, is the more rigorous approach, but for a first-pass audit the string match is usually good enough to get directionally accurate numbers.

Handling gzipped and rotated logs

Most production setups rotate and compress logs after a day or two, so a real version of this script needs to handle .gz files transparently:

import gzip
def open_log(path):
    if path.endswith(".gz"):
        return gzip.open(path, "rt", encoding="utf-8", errors="ignore")
    return open(path, "r", encoding="utf-8", errors="ignore")

Swap this into group_by_base_path in place of the plain open() call, and loop over every rotated file in your log directory rather than a single file, so you get a representative sample across at least a week rather than a few hours that might not reflect a weekly cron job or a scheduled crawl pattern.

Adding a simple CLI wrapper for repeated runs

If you expect to run this audit more than once, wrapping it as a small command-line tool with argparse saves time on every subsequent run:

import argparse
def main():
    parser = argparse.ArgumentParser(description="Find parameterized URL variants from access logs")
    parser.add_argument("log_dir", help="Directory containing log files (supports .gz)")
    parser.add_argument("--top", type=int, default=20, help="Number of worst offenders to show")
    parser.add_argument("--googlebot-only", action="store_true", help="Filter to Googlebot requests only")
    args = parser.parse_args()
    # wire up group_by_base_path / group_googlebot_only based on args here
    print(f"Analyzing logs in {args.log_dir}, showing top {args.top} offenders")
if __name__ == "__main__":
    main()

This turns a one-off analysis script into something a whole team can rerun monthly as part of a standing technical SEO check, rather than something only one person remembers how to invoke correctly.

Exporting results for non-technical teammates

The raw script output is fine for an engineer, but an SEO specialist or product manager on the same team will get more value from a CSV they can sort and filter themselves. Adding an export step takes a few lines:

import csv
def export_to_csv(ranked_results, output_path="parameter_audit.csv"):
    with open(output_path, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["base_path", "distinct_combinations", "total_requests"])
        for base_path, data in ranked_results:
            writer.writerow([base_path, len(data), sum(data.values()) if isinstance(data, dict) else ""])

Sharing this file alongside the raw numbers turns a script only one engineer can run into a shared artifact the whole team can reference when deciding which templates to prioritize fixing first.

Turning the output into a prioritized list

Raw counts of parameter combinations are useful, but pairing them with actual crawl frequency data makes the prioritization sharper. Extend the script to also track how many total requests hit each base path, not just how many distinct combinations exist:

def group_with_counts(log_file):
    groups = defaultdict(lambda: defaultdict(int))
    with open_log(log_file) as f:
        for line in f:
            path = parse_log_line(line)
            if not path:
                continue
            parsed = urlparse(path)
            base = parsed.path
            query = parse_qs(parsed.query)
            signature = tuple(sorted(query.keys())) if query else ()
            groups[base][signature] += 1
    return groups

Now you can rank base paths not just by how many distinct combinations exist, but by total wasted request volume, which is the number that actually matters for crawl budget. A path with fifteen combinations getting hit constantly is a bigger problem than one with sixty combinations that barely get crawled at all.

Where this fits in the bigger fix

This script gets you the diagnostic data. The actual remediation, classifying parameters and applying canonical tags correctly, is covered step by step in 137foundry.com's guide to stopping URL parameters from creating duplicate content. Running this kind of audit before touching any template code saves you from fixing the wrong ten pages while the real offenders keep piling up, since intuition about which routes are the worst offenders is frequently wrong until you actually measure it.

The Python urllib.parse documentation covers the parsing utilities used here in more depth if you need to handle edge cases like array-style parameters or encoded query strings. If you're working with a larger log volume than fits comfortably in memory, the pandas documentation covers chunked reading patterns that scale better than the dictionary-based approach shown here once you're past a few million log lines. For confirming which IP ranges genuinely belong to Google's crawlers before trusting a user-agent-based filter for anything beyond a rough first pass, Google's published crawler IP list documents the verification process directly.