Request: Faster saving to cache, or better indication of time and other copy of filename options

The best solution for finding and removing duplicate files.
mjdalways
Posts: 6
Joined: Tue Jan 26, 2021 12:42 am

Request: Faster saving to cache, or better indication of time and other copy of filename options

Post by mjdalways »

The saving hashes to cache takes a long time, for example I have a job running now, it took around 15 minutes to run everything and it is currently still saving hashes to cache after 25 minutes. The progress bar just keeps scrolling it give no indication as to where it is in the process.

Additionally sometimes I have files that are duplicates but dont say copy of, they might say (1), (2) etc or maybe abc Copy. Can we add an option to check first x characters of name or maybe exclude last x of name when matching? I know these can be filtered with the marking system to select them, but given that you added copy of I think the other options are a good extension of that for matching duplicates.
mjdalways
Posts: 6
Joined: Tue Jan 26, 2021 12:42 am

Re: Request: Faster saving to cache, or better indication of time and other copy of filename options

Post by mjdalways »

For a more concrete example, below you can see a scan that introduced just 6 new hashes. The scan until Saving to cache took 1:18, to save the 6 extra hashes to the cache took over 13 minutes! This is to an internal drive. Note that the -MD5.data file is 4,798,236KB so I understand this is very large, but it read the file relatively quickly, it was just saving/writing that took all the time.

NOTE:
I have a 512GB SSD for my main drive (this is where the log is stored),
an 8TB 7200RPM HD for my document/programs drive (where the cache is stored)
and I was reading from a 24TB 7200RPM drive.

12/10/2025 9:22:11 PM
Scan complete
Total time taken:00:14:22
211,271 file(s) scanned in 14,873 folder(s) (12.2 TB)
7,959 group(s) of duplicates
28,139 file(s) have duplicates (8.20 GB)
Hashes calculated: 6
Quick Hashes calculated: 6
Useful Quick Hashes: 0
Hash Errors: 0
User avatar
DigitalVolcano
Site Admin
Posts: 1910
Joined: Thu Jun 09, 2011 10:04 am

Re: Request: Faster saving to cache, or better indication of time and other copy of filename options

Post by DigitalVolcano »

It's usually fairly quick at writing to the cache. Perhaps there's a tweak to be do to increase the speed. (it may be re-building the indexes). Currently it doesn't remove deleted files from the cache (on the to-do list), so you may be better off deleting/renaming the cache file and taking the initial hit for better performance.
mjdalways
Posts: 6
Joined: Tue Jan 26, 2021 12:42 am

Re: Request: Faster saving to cache, or better indication of time and other copy of filename options

Post by mjdalways »

deleting cache made a huge difference thanks
Post Reply