【Feature Request】Improve Multi-threading & GPU Acceleration for Faster Duplicate Scanning

[askdddc]

While using Duplicate Cleaner (Video Mode) to scan for duplicate files, I’ve noticed that overall hardware utilization is quite low, which significantly impacts performance. The main observations are:

During stages such as metadata reading, hash calculation, and duplicate matching, CPU and GPU usage typically remains below 10%
The software does not appear to fully utilize multi-core CPUs or GPU acceleration
Scanning large volumes of video files takes considerably longer than expected, resulting in low efficiency

Based on this experience, I would like to propose the following improvements:

Enhanced Multi-threading Support (Including Matching Stage)
Introduce more comprehensive multi-threaded parallel processing across all stages, including file scanning, metadata extraction, hash calculation, and especially the duplicate matching phase. This would improve CPU core utilization and eliminate single-thread bottlenecks during matching.
Optional GPU Acceleration
For compute-intensive tasks such as video processing, hash calculation, and duplicate matching, consider adding GPU acceleration support (e.g., CUDA / OpenCL) to further improve performance.
Task Scheduling Optimization
Improve task distribution and scheduling mechanisms to avoid single-thread limitations and ensure more balanced resource utilization, especially when processing large datasets.

Overall, Duplicate Cleaner is an excellent tool in terms of functionality, but there is significant room for improvement in performance utilization. Optimizing resource usage would greatly enhance scanning speed and overall user experience.

Thank you to the development team for your continued work and improvements!

[DigitalVolcano]

Thanks for the suggestions. Duplicate Cleaner 6 is in development and most of it is now multithreaded with speeds of 10x faster in places. Metadata caching is also in place which will speed up repeat scans.