I have a folder with the same photos but different names and I want to remove the duplicates (doesn't matter which ones) using stream, to which I'm fairly new.
I tried to use this method, but of course it's not that simple and it's just deleting all files.
File directory = new File("D:\\Photos\\Test");
List<File> files = Arrays.asList(Objects.requireNonNull(directory.listFiles()));
files.stream().distinct().forEach(file -> Files.delete(file.toPath()));
I also tried to convert all files to byte array and use dictinct function on byte stream, but it didn't find any duplicates...
Is there a way to make this happen by using only stream?
CodePudding user response：
A while ago I made an android app that compares two images and checks if the image is a duplicate or not, so I was having the same problem like this and after searching for a while I found the answer on StackOverflow but currently I don't have the answer link saved so i am sharing the code, maybe it gives you some idea or help.
public class Main {
public static void main(String[] args) throws IOException {
ImageChecker i = new ImageChecker();
BufferedImage one = ImageIO.read(new File("img1.jpg"));
BufferedImage two = ImageIO.read(new File("img2.jpg"));
if(one.getWidth() one.getHeight() >= two.getWidth() two.getHeight()) {
i.setOne(one);
i.setTwo(two);
} else {
i.setOne(two);
i.setTwo(one);
}
System.out.println(i.compareImages());
}
}
public class ImageChecker {
private BufferedImage one;
private BufferedImage two;
private double difference = 0;
private int x = 0;
private int y = 0;
public ImageChecker() {
}
public boolean compareImages() {
int f = 20;
int w1 = Math.min(50, one.getWidth() - two.getWidth());
int h1 = Math.min(50, one.getHeight() - two.getHeight());
int w2 = Math.min(5, one.getWidth() - two.getWidth());
int h2 = Math.min(5, one.getHeight() - two.getHeight());
for (int i = 0; i <= one.getWidth() - two.getWidth(); i = f) {
for (int j = 0; j <= one.getHeight() - two.getHeight(); j = f) {
compareSubset(i, j, f);
}
}
one = one.getSubimage(Math.max(0, x - w1), Math.max(0, y - h1),
Math.min(two.getWidth() w1, one.getWidth() - x w1),
Math.min(two.getHeight() h1, one.getHeight() - y h1));
x = 0;
y = 0;
difference = 0;
f = 5;
for (int i = 0; i <= one.getWidth() - two.getWidth(); i = f) {
for (int j = 0; j <= one.getHeight() - two.getHeight(); j = f) {
compareSubset(i, j, f);
}
}
one = one.getSubimage(Math.max(0, x - w2), Math.max(0, y - h2),
Math.min(two.getWidth() w2, one.getWidth() - x w2),
Math.min(two.getHeight() h2, one.getHeight() - y h2));
f = 1;
for (int i = 0; i <= one.getWidth() - two.getWidth(); i = f) {
for (int j = 0; j <= one.getHeight() - two.getHeight(); j = f) {
compareSubset(i, j, f);
}
}
System.out.println(difference);
return difference < 0.1;
}
public void compareSubset(int a, int b, int f) {
double diff = 0;
for (int i = 0; i < two.getWidth(); i = f) {
for (int j = 0; j < two.getHeight(); j = f) {
int onepx = one.getRGB(i a, j b);
int twopx = two.getRGB(i, j);
int r1 = (onepx >> 16);
int g1 = (onepx >> 8) & 0xff;
int b1 = (onepx) & 0xff;
int r2 = (twopx >> 16);
int g2 = (twopx >> 8) & 0xff;
int b2 = (twopx) & 0xff;
diff = (Math.abs(r1 - r2) Math.abs(g1 - g2) Math.abs(b1
- b2)) / 3.0 / 255.0;
}
}
double percentDiff = diff * f * f / (two.getWidth() * two.getHeight());
if (percentDiff < difference || difference == 0) {
difference = percentDiff;
x = a;
y = b;
}
}
public BufferedImage getOne() {
return one;
}
public void setOne(BufferedImage one) {
this.one = one;
}
public BufferedImage getTwo() {
return two;
}
public void setTwo(BufferedImage two) {
this.two = two;
}
}
this code first compares the height and width of the image because the image may have different sizes and after that, it compares them pixel by pixel using the RGB code and returns the result.
Note:- all the code credit goes to the original writer but I don't remember the name, so if you are the writer plz tell me I will update your name and answer the link.
CodePudding user response：
Try this one. You can do the clean up but I think this will work for you. I took the reference from this link.
How to compare images for similarity using java
public class FileCompare {
public static void main(String[] args) {
File directory = new File("D:\\Photos\\Test");
List<File> filesToBeDeleted = new ArrayList<>();
List<File> files = Arrays.asList(Objects.requireNonNull(directory.listFiles()));
IntStream.range(0, files.size() - 1).forEach(i -> {
boolean bool = compareImage(files.get(i), files.get(i 1));
if (bool) {
filesToBeDeleted.add(files.get(i 1));
}
});
filesToBeDeleted.stream().forEach(file -> {
try {
Files.delete(file.toPath());
} catch (IOException e) {
e.printStackTrace();
}
});
}
public static boolean compareImage(File fileA, File fileB) {
try {
BufferedImage biA = ImageIO.read(fileA);
DataBuffer dbA = biA.getData().getDataBuffer();
int sizeA = dbA.getSize();
BufferedImage biB = ImageIO.read(fileB);
DataBuffer dbB = biB.getData().getDataBuffer();
int sizeB = dbB.getSize();
if (sizeA == sizeB) {
for (int i = 0; i < sizeA; i ) {
if (dbA.getElem(i) != dbB.getElem(i)) {
return false;
}
}
return true;
} else {
return false;
}
} catch (Exception e) {
return false;
}
}
}