Tool to deduplicate a list. Doubles or duplicates are a repeated redundant information present twice (or more) in a list, usually useless.
Duplicates in a List - dCode
Tag(s) : Data Processing
dCode is free and its tools are a valuable help in games, maths, geocaching, puzzles and problems to solve every day!
A suggestion ? a feedback ? a bug ? an idea ? Write to dCode!
Duplicates in a List
Duplicates Detector
Answers to Questions (FAQ)
What is a duplicate? (Definition)
A duplicate is a redundant element (double or triple or more) in a set of elements (a list).
How to remove doubles in a list?
Indicate / copy-paste items in the List field. dCode detects if the list is a list of terms (one per line) or a list of words (with separator) and finds redundant items repeated more than once in order to un-duplicate.
Example: A,B,C,D,A,B,C,A,B,A can be deduplicated as A,B,C,D (repeated terms were removed)
Be carefull to take into account all writing variants of a same element: ignore accents and diacritics allow to treat words like item and ìtém as a same word twice. Also, ignoring uppercase and lowercase allow to detect/find item and ITEM as a duplication of the same word and delete them.
How to count doubles in a list?
By listing items, dCode detector counts the number of appearance of each, the list can be sorted to detect the most duplicated ones.
dCode counts duplicates/redundancies, so 2 identical items is counted 1 duplicate. N identical elements is counted N-1.
Why counting doubles?
There are many applications where the presence of duplicates can be a problem:
— detect and delete duplicate files to free up disk space
— deduplicate data from a database, retaining only one occurrence of each unique/distinct record.
— sending mailings by checking if a list of email addresses contains duplicates in order to filter the list and delete identical emails.
Source code
dCode retains ownership of the "Duplicates in a List" source code. Any algorithm for the "Duplicates in a List" algorithm, applet or snippet or script (converter, solver, encryption / decryption, encoding / decoding, ciphering / deciphering, breaker, translator), or any "Duplicates in a List" functions (calculate, convert, solve, decrypt / encrypt, decipher / cipher, decode / encode, translate) written in any informatic language (Python, Java, PHP, C#, Javascript, Matlab, etc.) or any database download or API access for "Duplicates in a List" or any other element are not public (except explicit open source licence like Creative Commons). Same with the download for offline use on PC, mobile, tablet, iPhone or Android app.
Reminder: dCode is an educational and teaching resource, accessible online for free and for everyone.
Cite dCode
The content of the page "Duplicates in a List" and its results may be freely copied and reused, including for commercial purposes, provided that dCode.fr is cited as the source.
Exporting the results is free and can be done simply by clicking on the export icons ⤓ (.csv or .txt format) or ⧉ (copy and paste).
To cite dCode.fr on another website, use the link:
In a scientific article or book, the recommended bibliographic citation is: Duplicates in a List on dCode.fr [online website], retrieved on 2025-04-15,
