Content Clustering with Embeddings for Duplicate Page Cleanup

In some websites I’ve worked with, the term duplicate page cleanup shows up after a small drift in editorial habits. Unintended overlaps creep in, such as similar articles, rehashed summaries, two pages covering the same topic but with different wording. If you’ve experienced this, you know the frustration of deciding which content to keep, merge or retire. One approach that’s been especially helpful is using content clustering with embeddings.

What is Content Clustering with Embeddings and How Does it Work

Let’s start inside the term. “Embeddings” are simply mathematical representations of text. Think of each page as a point in a high-dimensional space, where similar means closer points.

When you use an embedding model, you convert each page’s cleaned content into a vector. Then you can compare those vectors: the closer two vectors are, the more semantically similar the pages.

Content clustering on the other hand means grouping pages whose vectors are very close together. If you see a cluster of pages around the same meaning space, you’re likely looking at content that’s overlapping in topic or value. That gives you the signal you need for cleanup: maybe do a redirect, merge content, canonicalise, or rewrite to distinguish.

Documentation often drifts, multiple internal pages will describe the same security check with slightly different wording.

Embedding-based clustering is a practical way to tackle that. It trims content bloat, restores sensible internal links, and makes information easier to find.

It sometimes it misses weird corner cases or surfaces surprising groupings, but it translates well to public website content too.

Choosing the Right Strategy for Duplicate Page Cleanup

You’ll want to treat two types of duplication differently, one is literal or near-literal duplication (same content reused); the other is semantic duplication (different wording but same underlying topic).

If two pages are nearly identical in text, you might catch them with basic string matching or a tool like MinHash + LSH. But when they’re different in wording yet still overlapping, embeddings bring the insight.

Visualization: Red lines show near-duplicate pages (MinHash). Blue lines show semantic similarity.

So the first step is deciding your scope. Are you dealing with thousands of pages, dozens, or just a blog section? Then pick a chunk-strategy: full page embeddings (simple) or section/paragraph embeddings (more nuanced). If your pages are long and cover multiple topics, sectioning can help such that you don’t accidentally group pages just because they both mention “security audit” somewhere far down.

Once you decide chunking, you’ll generate embeddings using a consistent model, this is key. If you switch models mid-run you won’t be comparing apples to apples. Then you build a similarity index (for example using an ANN library like FAISS, Milvus or a managed vector database) so you can quickly ask: “What pages are most similar to this one?”

Finally, you run a clustering algorithm on the vectors (or pre-filtered pairs) to get candidate groups of near-duplicates. Clustering with HDBSCAN works well when you don’t know how many clusters you’ll get; you just need to surface meaningful groups without forcing every page into a cluster.

Interpreting Clusters and Planning Your Clean-Up

Once clusters are formed, don’t rush to act based just on the algorithm’s score. I had a project where two pages clustered at 0.92 similarity, but one had unique conversion metrics and a slightly different audience. If we’d auto-merged without review we would’ve thrown away a functioning page. So human review is part of the workflow.

Here’s a rough decision-framework (feel free to tailor it):

• If two pages are very similar (say vector similarity above 0.95) and one has low traffic/engagement, consider redirecting to the stronger one or canonicalising it.
• If similarity is around 0.9–0.95 and both pages have traffic, review the content manually. Maybe merge into a single stronger asset or rewrite one to target a distinct angle.
• If similarity is lower (say 0.8–0.9) then often you’re better off rewriting one of the two to target a separate sub-topic (especially if both get traffic).
• If similarity is below 0.8, usually they’re different enough that you can leave both, unless business logic demands otherwise.

In one cleaning I did, we had a cluster of five pages each covering “secure coding practices for web apps.” Two were low-traffic, three had decent engagement, but all three basically repeated the same list of bullet points. We kept the best one, merged content from the others, redirected them, and rewrote one to focus on “secure coding for mobile apps” instead. After six weeks we saw clearer user paths and fewer duplicate search queries. (Yes, Google did fewer “which page do I go to” moments.)

Technical Essentials: Tools, Models, Thresholds

Models & embedding generation

Pick an embedding model (e.g., one from the Sentence-Transformers family or an API like OpenAI’s embeddings) and stick with it for the entire run. If you switch, similarity scores shift and comparisons become unreliable.

Indexing & nearest neighbour retrieval

For anything more than a few hundred pages, you’ll want a system like FAISS or HNSWILS (or a managed vector database such as Milvus or Pinecone). These let you ask for the top-k most similar pages for each page in your site.

Clustering

Once you have similarity pairs/neighbours, you need to form clusters. I lean toward HDBSCAN in many cases, because you don’t have to predetermine how many groups exist and it handles variable cluster density. Typical alternatives include Agglomerative clustering or DBSCAN.

Thresholds

Here are some starting points (but test on your specific content):

• Similarity ≥ 0.95: very likely redundant.
• 0.9 ≤ similarity < 0.95: high risk of redundant or overlapping content.
• 0.8 ≤ similarity < 0.9: potential for overlap, but may still be distinct enough if intent differs.
• < 0.8: likely different topics.

Workflow Recap

1. 1. Crawl or extract pages plus content and metadata (traffic, queries).
   2. Clean out boilerplate (nav/footers) so you’re comparing the meaningful text.
   3. Chunk if necessary.
   4. Generate embeddings, normalise if required.

1. Build index, run nearest-neighbour search.
2. Cluster or group pages based on similarity.
3. Sample clusters, human review.
4. Decide action (redirect, merge, rewrite, leave).
5. Monitor metrics (traffic, engagement, search ranking) after changes.

Challenges and How to Handle Them

You’ll probably hit some of these:

• Model drift and consistency: If you routinely add pages or re-embed with a different model/version, your earlier similarity scores won’t align. This is how to fix it, version your model, store embeddings and track model version.
• False positives: Two pages might cluster because they share language (e.g., “This guide explains…”) but the audience or intent differs. Always sample manually before executing mass actions.
• Multilingual content: If your site covers multiple languages, embeddings may mix them if your model isn’t language-aware. You might need to split by language first.
• Merge vs separate decision: Sometimes two pages are very similar but still serve slightly different personas. The algorithm can’t decide for you. Use analytics and business context.
• Tracking impact: After you apply redirects or canonical tags, traffic might drop suddenly, not always because you messed up, but because the algorithm distilled many pages into one, and you need to track user flows accordingly.

When this kind of cleanup pays off

This approach brings value in several scenarios:

• You have lots of pages, and you suspect content cannibalisation (many pages fighting for the same query).
• Your editorial team writes many variations of the same topic because they weren’t coordinating.
• You manage a large site where sections overlap (for example “help docs” and “blog posts” plus “knowledge base” all covering similar topics).
• You want to prepare your site for growth, then know that fewer duplicate/overlapping pages often means clearer signals to search engines, better internal linking, and improved user experience.

What to Watch After You Act

Cleanup isn’t done when you click “redirect” or “canonicalise.” You’ll want to monitor what happens next. Key indicators include:

• Does organic traffic to the canonicalised/merged page increase or stay stable?
• Do bounce rates or time-on-page change significantly (for better or worse)?
• Are there crawl errors or soft-404s from old URLs?
• In search console (or your equivalent), does query ranking improve (or stay stable) for the merged page?
• Do user flows change? Maybe a formerly under-used page was part of a funnel, if you remove it, did that impact conversions?

Final Thoughts

Using content clustering with embeddings gives you a structured way to sift through content overlap, consolidate pages thoughtfully, and steer your site toward clarity. It doesn’t replace editorial judgment, rather, it gives you a map of where your content is dense, overlapping, or duplicated. From that map you make informed decisions.