Remove Duplicate Lines

Removing duplicate lines — also called deduplication or dedup — is one of the most common text cleanup tasks. Whether you merged two files, exported a database column, scraped URLs, or copied log output, repeated lines waste space and can break downstream tools (email senders, importers, analyzers).

SuperTextTools deduplicates text entirely in your browser. Paste your list, pick how to treat duplicates, and get a clean result with live stats — no terminal, no upload, no account.

Common scenarios

• Email lists — avoid sending the same address twice in a campaign
• URL lists — clean web-scraping or sitemap exports
• Log files — collapse repeated error lines to see distinct issues
• Merged files — combine exports where rows overlap
• CSV single columns — dedupe one field before re-import
• Keyword lists — SEO and research tools often emit duplicates

The Unix equivalent

On the command line, sort | uniq -c does similar work. This tool offers the same power with clickable modes: keep first or last occurrence, show only lines that repeated, or show only lines that appeared once — plus optional sort by frequency or alphabetically.

Choosing a mode

• Keep first — order matters; trust the earliest row
• Keep last — later rows win (e.g. newest record)
• Only duplicates — audit which values repeated
• Only unique — find outliers that appear exactly once

Case and whitespace

Real data is messy. Email domains are often case-insensitive; URL paths are usually case-sensitive. Toggle case sensitive to match your system. Trim whitespace catches hello vs hello without changing how the surviving line looks in output.

Performance

Deduplication is O(n) using hash maps. The tool handles tens of thousands of lines in the browser without freezing — tested on large lists so you can clean exports in one pass.