Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download only new hash files #242

Merged
merged 2 commits into from
May 23, 2018
Merged

Download only new hash files #242

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2d3ff6d
check for existing hashfile downloads
ktneely May 4, 2018
7044976
Merge pull request #1 from ktneely/getHashes-patch-1
ktneely May 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions analyzers/Virusshare/getHashes.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,16 @@ if [ ! -d $1 ]; then
fi

cd $1
for u in `curl https://virusshare.com/hashes.4n6 | grep -E "hashes\/VirusShare_[0-9]{5}\.md5" | cut -d\" -f2`
for u in `curl https://virusshare.com/hashes.4n6 | grep -E "VirusShare_[0-9]{5}\.md5" | c\
ut -d\" -f2 | cut -d\/ -f2`
do
echo $u
wget https://virusshare.com/$u
if [ -e $1/$u ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you've disabled commits for members, I cannot change that line. You have to delete $1/ as the scripts cds into the directory before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script does cd into $1/, but when I'm checking the file names, the update performs two cut commands, lopping off the directory assigned to $1. A little convoluted, but it makes the comparison easier.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not what I wanted to say. You cutsomething off, that's right but it only affects the filename, after that you still have that commands:

  • cd $1
  • if [ -e $1/$u ]

That leads to "being in" directory $1 and checks if there's an directory $1 with file $u in it. Or I'm completely blind and oversee something. :D

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prior to the cd, the script cuts the full path twice, making u only the file name as pulled from the virusshare website, which it then compares to the filenames in directory $1.

See, e.g. | cut -d\" -f2 | cut -d\/ -f2 in the update as opposed to only cut -d\" -f2 in the original.

The first cuts down to e.g. "hashes/VirusShare_00049.md5", the second lops off the directory on the hosting server and leaves only e.g. "VirusShare_00049.md5", which is then compared to the filename in $1.

echo "File already downloaded"
else
wget https://virusshare.com/hashes/$u
sleep 3
fi

done | tee -a ../$0.log
cd ..