How to Remove Duplicate Lines From a List (5 Easy Ways)

By HSP · Published October 5, 2026 · 5 min read · Writing & Text

Duplicate lines creep into email lists, keyword lists, product codes, survey answers and log files whenever data is merged from several sources. Removing them by hand is tedious and easy to get wrong. Here are five quick methods, from a one-click online tool to spreadsheet and command-line options, plus the settings that decide what counts as a duplicate.

First, decide what counts as a duplicate

Computers compare text exactly, so these lines are all different unless you tell the tool otherwise:

• apple
• Apple (different capital letter)
• apple (trailing space)
• apple (leading space)

In most real lists, such as email addresses or names, you want these treated as the same. That means trimming spaces and ignoring case before comparing. You should also decide whether to keep the original order (first occurrence wins) or sort the list, and whether to remove empty lines.

Method 1: Online duplicate line remover

1. Open the remove duplicate lines tool.
2. Paste your list, one item per line.
3. Choose the options: trim spaces, ignore case, remove empty lines, and sort A–Z or Z–A (or keep the original order).
4. Copy the cleaned list.

The tool shows how many lines were removed, and everything runs in your browser, so private lists such as customer emails are not uploaded. This is the quickest option for lists you copy from emails, documents or web pages.

Method 2: Excel

1. Select the column or range.
2. Go to Data › Remove Duplicates (in the Data Tools group).
3. Tick the columns that must match for a row to count as a duplicate, then click OK.

Excel deletes the duplicate rows and tells you how many were removed. It keeps the first occurrence. Because this permanently changes your data, work on a copy. To see duplicates before deleting them, use Home › Conditional Formatting › Highlight Cells Rules › Duplicate Values. In newer versions of Excel, =UNIQUE(A2:A100) returns a de-duplicated list in a new place without touching the original.

Excel’s comparison ignores case, but not extra spaces. Clean those first with =TRIM(A2).

Method 3: Google Sheets

• Formula: =UNIQUE(A2:A) in an empty column lists each value once, in the original order.
• Menu: select the range and choose Data › Data cleanup › Remove duplicates, then choose the columns to compare.
• Count duplicates: =COUNTIF(A:A, A2) next to each row shows how many times a value appears.

Sheets also has Data cleanup › Trim whitespace to remove stray spaces first.

Method 4: Notepad++ and code editors

In Notepad++, choose Edit › Line Operations › Remove Duplicate Lines. There is also “Remove Consecutive Duplicate Lines”, which only removes repeats that sit next to each other, so sort first if you use that option. In Visual Studio Code, you can sort lines with the “Sort Lines Ascending” command and then remove adjacent duplicates, or use an extension.

Method 5: The command line

On macOS, Linux or Windows Subsystem for Linux:

Add > clean.txt to the end of a command to save the result in a new file.

Worked example

Input:

banana
Apple
apple

cherry
banana

With trim spaces, ignore case and remove empty lines turned on, and the original order kept, the output is banana, Apple, cherry: three lines instead of six. Without trimming and ignoring case, “Apple” and “apple ” would both survive.

Tips for email and contact lists

Email lists are the most common reason people remove duplicates, and they have a few quirks. The domain part of an address is not case-sensitive, and in practice most providers treat the whole address that way, so ignoring case is usually safe. Watch for copies of the same address with different labels, such as “Jane Doe <[email protected]>” and “[email protected]”: strip the names first so only the addresses remain. Finally, keep a record of people who unsubscribed, so that merging lists does not add them back by accident.

After removing duplicates

• Compare two versions: paste the before and after lists into the text diff checker to see exactly which lines were removed.
• Fix capitalisation: make a list consistent with the case converter; see how to change uppercase to lowercase.
• Find repeated words in prose: the word frequency counter shows which words you use most often.
• Run a fair draw: removing duplicate entries is the first step before picking a random winner.

Common mistakes

• Forgetting about trailing spaces and capital letters, so duplicates slip through.
• Using uniq without sorting first.
• Removing duplicates in the original file without a backup.
• In multi-column data, comparing only one column and deleting rows that differ elsewhere.