Duplicate content: the technical causes and how to find them

"Duplicate content" sounds like a content problem, something about copied articles or thin pages. On most sites it is mainly a technical problem. The same page is reachable at several URLs because of how the server, the CMS, the analytics setup or the navigation builds links. Search engines notice, group those URLs together, and choose one of them to index. Sometimes they choose well. Sometimes the URL in results is the one with a tracking parameter, or the HTTP version, or the print view.

There is no penalty for this kind of duplication. The costs are quieter: links and other signals split across several URLs, crawling spent on copies instead of new pages, and reports that are hard to read because one page appears under five addresses. This guide lists the causes, how to detect each, and how to fix them at the root.

The usual suspects

Cause	Example	Best fix
Protocol and host variants	http:// vs https://, www vs bare domain	301 redirects to one version
Trailing slash	/shoes and /shoes/ both return 200	Redirect one form to the other, site-wide
Letter case	/Shoes/ and /shoes/	Redirect to lowercase; link in lowercase
Index files	/ and /index.php or /index.html	Redirect the file name to the directory
Tracking parameters	?utm_source=newsletter, ?ref=, ?fbclid=	Self canonical on the clean URL; never use in internal links
Session IDs	?sid=a8f3…	Use cookies instead; canonical as fallback
Sort and filter parameters	?sort=price	See faceted navigation
Print or AMP-style alternate views	/article/print/	Remove, or canonical to the main version
Products in several categories	/men/boots/x and /sale/x	One product URL independent of category path
Pagination of comments or reviews	?replytocom=, /comment-page-2/	Disable or canonicalise; avoid crawlable reply links
Staging and development copies	staging.example.com indexed	Password-protect; remove from index
Spare domains	Old brand domain serving the same site	301 to the main domain

How to find them

Ask your server directly

Many causes can be checked in two minutes with curl. Take one real page and request its variants, looking only at the status code and redirect target:

for u in http://example.com/shoes/ https://example.com/shoes/ https://www.example.com/shoes \
         https://www.example.com/Shoes/ https://www.example.com/shoes/index.html; do
  curl -s -o /dev/null -w "%{http_code} %{redirect_url}  $u\n" "$u"; done

Exactly one variant should return 200. Every other one should return a 301 or 308 directly to it. Any variant that returns 200 is a duplicate. The redirects guide explains how to choose and enforce the main host.

Crawl and compare

A site crawl finds duplicates your own navigation creates. Look for:

• Pages with identical titles or identical H1s at different URLs.
• Pages with identical content hashes, which most crawlers can calculate.
• URLs that differ only in parameters, case or trailing slash.
• Canonical tags that point somewhere other than the URL itself, which often reveals the template that creates variants.

Read what Google chose

Search Console's Page indexing report has three statuses that are all about duplicates:

• Alternate page with proper canonical tag: the duplicate points to a canonical, and Google agrees. This is fine.
• Duplicate without user-selected canonical: Google found duplicates and you gave no hint. It picked one itself.
• Duplicate, Google chose different canonical than user: you declared a canonical, but Google decided another URL is the main one. This is the one to investigate, because Google is telling you your signals disagree.

Use URL Inspection on samples to see which URL Google selected as canonical for each.

When Google overrides your canonical, it is usually because other signals point elsewhere. The most common reasons: internal links mostly point to the other URL; the sitemap lists the other URL; the "canonical" page redirects, returns an error or is noindexed; or the two pages are not actually equivalent, for example because one has noticeably less content. Line up every signal for the URL you want (links, sitemap, redirects, hreflang, the canonical tag itself) and Google will usually follow within a few crawls.

Search for your own text

Copy a distinctive sentence from an important page and search for it in quotes. If other domains appear, someone has copied or syndicated the content. If your own staging domain or an old domain appears, you have found a different kind of leak.

Fix at the source, not with tags

Canonical tags are useful, but they are hints, and they do not stop crawling. The most robust fixes remove the duplicate URL from existence or from every link:

1. Server-level normalisation. One rule each for protocol, host, trailing slash, case and index files, applied before the application runs. These rules fix thousands of URLs at once.
2. Clean internal links. Templates should only ever output the canonical form of a URL. If your navigation links to /Shoes and your sitemap lists /shoes/, you are sending mixed signals no tag can fully cancel.
3. Category-independent product URLs. A product reachable at /product-name/ no matter which category the user came from removes a whole class of duplicates. Breadcrumbs can still reflect the path.
4. Canonical tags for the rest. Parameters you cannot avoid, such as campaign tracking, get a self-referencing canonical on the clean URL. The canonical tags guide covers the rules.
5. Sitemaps that list only canonical URLs. A sitemap is one more signal; make it agree with the others.

When duplication is intentional

Some duplication exists for good reasons, and the job is to label it rather than remove it.

• Regional versions of the same page in the same language, such as English for the US, UK and Australia. These are not treated as duplicates if they are marked with hreflang and genuinely localised; see hreflang implementation. Pages that differ only by currency symbol may still be clustered.
• Syndicated articles on partner sites. Ask for a link and a canonical back, or for the partner to noindex the copy.
• Product descriptions from manufacturers used by hundreds of retailers. These are not penalised, but there is nothing to rank you above the others. Adding your own content, such as fit notes, real photos, comparison and answers to common questions, is what separates the page.
• Boilerplate repeated across pages, such as shipping information or legal notices. This is normal and expected. Search engines identify the main content and largely ignore repeated template text.

Staging sites: the duplicate that keeps coming back

A staging or development copy that gets indexed is the most embarrassing duplicate, and it happens constantly. Someone removes the password to show a client, a robots.txt block is lifted for testing, or a copy of the live database is deployed to a new subdomain with no protection at all. The staging copy then competes with the live site, sometimes outranking it for its own brand name.

The reliable prevention is HTTP authentication on every non-production environment, so nothing is reachable without a password. A noindex header is a good second layer. A robots.txt block alone is not enough, because blocked URLs can still be indexed from links, and the block itself is often copied into production by accident. If a staging site is already indexed, protect it with a password, then use the Removals tool in Search Console on its property to hide it quickly while Google drops it.

Keeping it fixed

Duplicate content problems return after redesigns, CMS updates, new plugins and new marketing tools. A short recurring check helps: the curl loop above on one URL of each template, a monthly glance at the three duplicate statuses in Search Console, and a scheduled check of canonical and redirect setup. The SEO checker reports host and protocol redirects and the canonical on each page it checks, so a change that suddenly makes the www and non-www versions both return 200 shows up in the next run rather than months later.