Why comparison of two md5sum files is not working properly?-CodePudding

I have 2 lists with files with their md5sum checks and the lists have different paths for the same files.

Example of content in first file with check sums (server.list):

2c03ff18a643a1437ec0cf051b8b7b9d  /tmp/fastq1_L001_R1_001.fastq.gz
c430f587aba1aa9f4fdf69aeb4526621  /tmp/fastq1_L001_R2_001.fastq.gz

Example of content in two file with check sums (downloaded.list):

2c03ff18a643a1437ec0cf051b8b7b9d  /home/projects/fastq1_L001_R1_001.fastq.gz
c430f587aba1aa9f4fdf69aeb4526621  /home/projects/fastq1_L001_R2_001.fastq.gz

When I run the following line, I got the following line:

awk -F"/" 'FNR==NR{filearray[$1]=$NF; next }!($1 in filearray){printf "%s has a different md5sum\n",$NF}' downloaded.list server.list

fastq1_L001_R1_001.fastq.gz has a different md5sum
fastq1_L001_R2_001.fastq.gz has a different md5sum

Why I am getting this message since the first column is the same in both files? Can someone enlighten me on this issue?

Edit:

If I remove the path and leave only the file name, it works just fine.

CodePudding user response：

Assumptions:

filename (sans path) and md5sum have to match
filenames may not be listed in the same order
filenames may not exist in both files

Sample data:

$ head downloaded.list server.list
==> downloaded.list <==
2c03ff18a643a1437ec0cf051b8b7b9d  /home/projects/fastq1_L001_R1_001.fastq.gz   # match
YYYYf587aba1aa9f4fdf69aeb4526621  /home/projects/fastq1_L001_R5_911.fastq.gz   # different md5sum
c430f587aba1aa9f4fdf69aeb4526621  /home/projects/fastq1_L001_R2_001.fastq.gz   # match
MNOPf587aba1aa9f4fdf69aeb4526621  /home/projects/fastq1_L001_R8_abc.fastq.gz   # filename does not exist in other file
ABCDf587aba1aa9f4fdf69aeb4526621  /home/projects/fastq1_L001_R9_004.fastq.gz   # different filename but matching md5sum (vs last line of other file)

==> server.list <==
2c03ff18a643a1437ec0cf051b8b7b9d  /tmp/fastq1_L001_R1_001.fastq.gz             # match
c430f587aba1aa9f4fdf69aeb4526621  /tmp/fastq1_L001_R2_001.fastq.gz             # match
XXXXf587aba1aa9f4fdf69aeb4526621  /tmp/fastq1_L001_R5_911.fastq.gz             # different md5sum
TUVWff18a643a1437ec0cf051b8b7b9d  /tmp/fastq1_L999_R6_922.fastq.gz             # filename does not exist in other file
ABCDf587aba1aa9f4fdf69aeb4526621  /tmp/fastq1_L001_R7_933.fastq.gz             # different filename but matching md5sum (vs last line of other file)

One awk idea to address white space issues as well as verifying filename matches:

awk '                                    # stick with default field delimiter of white space but ...
{ md5sum=$1
  n=split($2,arr,"/")                    # split 2nd field on "/" delimiter
  fname=arr[n]

  if (FNR==NR)
     filearray[fname]=md5sum
  else {
     if (fname in filearray && filearray[fname] == $1)
        next
     printf "%s has a different md5sum\n",fname
  }
}
' downloaded.list server.list

This generates:

fastq1_L001_R5_911.fastq.gz has a different md5sum
fastq1_L999_R6_922.fastq.gz has a different md5sum
fastq1_L001_R7_933.fastq.gz has a different md5sum

CodePudding user response：

The whitespace on $1 used as an array key is causing problems. Removing it:

awk -F"/" '{gsub(/ /, "", $1)}; FNR==NR{filearray[ $1]=$NF; next }!($1 in filearray){printf "%s has a different md5sum\n",$NF}' list1.txt list2.txt