Remove Duplicates from a List in Python

There are several approaches to remove duplicates from a list in Python. Here are the 3 most common methods:

(1) Using set() (Order not preserved)

The simplest and the most efficient way to remove duplicates from a list in Python is by using a set().

A set is an unordered collection of unique elements, so converting a list into a set automatically removes duplicate values. However, note that this method does not preserve the original order of elements in the list.

my_list = [5, 10, 5, 15, 2, 30, 2, 20]

unique_list = list(set(my_list))
print(unique_list) # Output: [2, 5, 10, 15, 20, 30]

(2) Using dict.fromkeys() (Order preserved)

dict.fromkeys() method creates a dictionary using the elements of a list as keys. Since dictionary keys are unique and maintain insertion order (from Python 3.7 and later), this removes duplicates while preserving the original order.

my_list = [5, 10, 5, 15, 2, 30, 2, 20]

unique_list = list(dict.fromkeys(my_list))
print(unique_list) # Output: [5, 10, 15, 2, 30, 20]

If you are using a version before Python 3.7, you should use collections.OrderedDict instead, as it maintains order while removing duplicates.

from collections import OrderedDict

my_list = [5, 10, 5, 15, 2, 30, 2, 20]
unique_list = list(OrderedDict.fromkeys(my_list))
print(unique_list)  # Output: [5, 10, 15, 2, 30, 20]

(3) Using list comprehension

To remove duplicates using list comprehension, you can iterate through the original list and add elements to a new list only if they are not already present. This method preserves the order.

my_list = [5, 10, 5, 15, 2, 30, 2, 20]

# Initialize an empty list to store unique elements
unique_list = []

[unique_list.append(x) for x in my_list if x not in unique_list]
print(unique_list) # Output: [5, 10, 15, 2, 30, 20]

Choosing the right method

Use set() if you don’t need to preserve the order and want the fastest way to remove duplicates.

Use dict.fromkeys() if you need to preserve the original order while removing duplicates (works in Python 3.7 and later). For earlier versions of Python, use collections.OrderedDict instead, as it also maintains order.

Use list comprehension if you prefer a clear and readable approach that maintains order. It is less efficient for larger lists due to repeatedly checking for existing elements.