时间戳

首页 > 解决方案 > 在文件中查找包含重复数据的行

batch-file - 在文件中查找包含重复数据的行

问题描述

我试图弄清楚如何在文件中查找包含重复数据的行。在这里，我加载一行并将其分解为原始变量，然后分别为每个变量搜索相同的数据文件，列出与新文件匹配的行以供进一步处理。

@echo off
:: reset temp files
set "db=%temp%\lbdb"
del "%db%"
copy /y nul "%db%" >nul
set "dupes=%temp%\dupes"
del "%dupes%"
copy /y nul "%dupes%" >nul

:: rebuild database (full,dura,size,strict,id)
:: maybe add option to keep database if not too old?
setlocal enabledelayedexpansion
for /f "delims=" %%v in ('dir "d:\videos" /b /s /a-d 2^>nul') do @(
    for /f "tokens=2 delims=," %%i in ('ffprobe -v quiet -show_entries "format=duration" -of csv "%%v"') do @(
        @set "full=%%v" & @call reader >nul
        echo %%v/%%i/%%~zv/!strict!/!id!
    )
)>>"%db%"

:: for each line in database, read variables and call for processing
for /f "tokens=1,2,3,4,5 delims=/" %%a in ('sort "%db%"') do if exist "%%a" call :checks "%%a" "%%b" "%%c" "%%d" "%%e"
exit /b

:checks
:: full is unique, compare %dura%, %size%, %strict% and %id% separately
set "full=%~1" & set "dura=%~2" & set "size=%~3" & set "strict=%~4" & set "id=%~5"

:: find each type of match and list possibly same videos
:: how to exclude the current loaded line from results?
:: findstr /c:"/%dura%/" "%db%" >>"%dupes%"
:: findstr /c:"/%size%/" "%db%" >>"%dupes%"
:: findstr /c:"/%strict%/" "%db%" >>"%dupes%"
findstr /c:"/%id%" "%db%" >>"%dupes%"

exit /b

问题是每个“加载”行（用于进行搜索）显然会在该文件中找到自己。如果只有一行包含应该被忽略的匹配项，并且应该收集双打或更多，以便可以处理它们。加载该行时，我必须从文件中删除整行吗？我希望有一个更快的解决方案。

编辑：

示例%db%（看起来不错）

d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E01 Pilot [Comedy] r10.0 720p x264 AC3 tt1612578.mkv/1272.772000/586337002/SHIT?MY?DAD?SAYS?S01E01/showShitMyDadSaysS01E01
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E02 Wi-Fight [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1274.400000/586061951/SHIT?MY?DAD?SAYS?S01E02/showShitMyDadSaysS01E02
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E03 The Truth About Dads and Moms [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1230.000000/587071468/SHIT?MY?DAD?SAYS?S01E03/showShitMyDadSaysS01E03
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04 Code Ed [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04

我希望结果%dupes%是：

d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04 Code Ed [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04

最后，由于数据在%db%. 请注意，我已经评论了结果使用的搜索。注释实际上并未出现在文件中

:: searching showShitMyDadSaysS01E01 
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E01 Pilot [Comedy] r10.0 720p x264 AC3 tt1612578.mkv/1272.772000/586337002/SHIT?MY?DAD?SAYS?S01E01/showShitMyDadSaysS01E01
:: searching showShitMyDadSaysS01E02 
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E02 Wi-Fight [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1274.400000/586061951/SHIT?MY?DAD?SAYS?S01E02/showShitMyDadSaysS01E02
:: searching showShitMyDadSaysS01E03 
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E03 The Truth About Dads and Moms [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1230.000000/587071468/SHIT?MY?DAD?SAYS?S01E03/showShitMyDadSaysS01E03
:: searching showShitMyDadSaysS01E04 
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04 Code Ed [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04

唷，这很难解释

标签： batch-filebatch-processing

解决方案

我终于想通了。简单的解决方案太简单而无法注意到的那些时刻之一。这轻而易举地通过了 27.000 个文件，只需要 1 小时 5 分钟，而我删除的旧版本需要大约 10 小时才能完成。最重要的是，它保留了相关数据，因此每周扫描只需约 20 分钟。

@echo off
echo %time:~0,2%:%time:~3,2% updating video information...
setLocal enableDelayedExpansion
for /f "delims=" %%v in ('dir "d:\videos" /b /s /a-d 2^>nul') do (
    if not exist "c:%%~pv" md "c:%%~pv" && attrib +h "c:\videos && echo first time takes a while (~400 videos/minute), please be patient...
    if not exist "c:%%~pnxv" for /f "tokens=2 delims=," %%i in ('ffprobe -v quiet -show_entries "format=duration" -of csv "%%v"') do (
        set "full=%%v" & call reader >nul
        echo %%v/%%i/%%~zv/!id!/!strict!>"c:%%~pnxv"
    )
)

echo %time:~0,2%:%time:~3,2% gathering database...
del "%temp%\db" 2>nul
for /f "delims=" %%v in ('dir "c:\videos" /b /s /a-d 2^>nul') do (
    rem if video is missing remove the data
    if not exist "d:%%~pnxv" del "c:%%~pnxv"
    rem is it too much to load the line here and check the fast properties to make sure it's valid?
    if exist "c:%%~pnxv" type "c:%%~pnxv" >>"%temp%\db"
)

echo %time:~0,2%:%time:~3,2% comparing by identity and size+duration data...
del "%temp%\dupes" 2>nul
:: for each line in database, read variables and call for processing. sorting is useless, is it safe tor emove?
for /f "tokens=1,2,3,4,5 delims=/" %%a in ("%temp%\db") do if exist "%%a" call :compare "%%a" "%%b" "%%c" "%%d" "%%e"
:: if we have collected dupes list, execute the job for the dupe files
if exist "%temp%\dupes" (echo %time:~0,2%:%time:~3,2% moving aside...) else (echo %time:~0,2%:%time:~3,2% no duplicates found.)
if exist "%temp%\dupes" for /f "tokens=1,2,3,4,5 delims=/" %%a in ("%temp%\dupes") do if exist "%%a" call :execute "%%a"
if exist "%temp%\dupes" echo %time:~0,2%:%time:~3,2% finished.
exit /b

:compare
:: full is unique, compare %dura%, %size%, %strict% and %id%
set "full=%~1" & set "dura=%~2" & set "size=%~3" & set "id=%~4" & set "strict=%~5"

if "%dura%"=="" echo duration missing %1 %2 %3 %4 %5 & pause
if "%id%"=="" echo id missing %1 %2 %3 %4 %5 & pause

:: find all lines matching properties (incl. self line) w/temp storage
findstr /c:"%id%" "%temp%\db" >"%temp%\analysis"
:: check for both identical duration and size. %size% can be shortened for some flexibility
findstr /c:"/%dura%/%size%/" "%temp%\db" >>"%temp%\analysis"
:: collect all lines except the one containing the current file
findstr /v /c:"%full%" "%temp%\analysis" >>"%temp%\dupes"
exit /b

:execute
:: move to wherever user wants them
set "full=%~1"
call report %~n0 locate duplicates
exit /b

推荐阅读