How to remove duplicates from a list
Short answer
distinct() in a stream keeps the first occurrence of each value and preserves the order:
void main() {
var numbers = List.of(3, 1, 3, 2, 1);
IO.println(numbers.stream().distinct().toList());
}Output:
[3, 1, 2]With a set
A set stores only unique values. LinkedHashSet also remembers the insertion order:
void main() {
var tags = List.of("java", "sql", "java", "git");
var unique = new ArrayList<>(new LinkedHashSet<>(tags));
IO.println(unique);
}Output:
[java, sql, git]Uniqueness by a field
To keep one object for each value of a field, collect them into a Map keyed by that field:
record User(String email, String name) {}
void main() {
var users = List.of(
new User("anna@site.com", "Anna"),
new User("oleg@site.com", "Oleg"),
new User("anna@site.com", "Anna Smith"));
var unique = users.stream()
.collect(Collectors.toMap(User::email, u -> u, (first, second) -> first, LinkedHashMap::new))
.values();
IO.println(unique);
}Output:
[User[email=anna@site.com, name=Anna], User[email=oleg@site.com, name=Oleg]]Ignoring case
A TreeSet with a case-insensitive comparator treats Java and java as equal:
void main() {
var set = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
set.addAll(List.of("Java", "java", "SQL", "sql", "Git"));
IO.println(set);
}Output:
[Git, Java, SQL]Common mistake
Using HashSet when order matters. It does not keep the insertion order:
void main() {
var words = List.of("pear", "apple", "kiwi", "pear", "banana");
IO.println(new HashSet<>(words));
}Output:
[banana, apple, pear, kiwi]The order of a HashSet depends on hash codes and may differ from the original. If order matters, use LinkedHashSet or distinct().
Functions used in this recipe
- Stream.distinct() — Removes duplicate elements from a stream.
- LinkedHashSet — A set that remembers the insertion order of elements.
- HashSet — A set without duplicates backed by a hash table; order is not guaranteed.
- Collectors.toMap() — A collector that builds a map from stream elements.