This script works well in finding what I need, but there are occassions where a 404 error just kills everything.
#!/bin/sh set +e exec 7<foo.txt exec 8<bar.tmp echo "Retrieving data" while read line1 <&7 && read line2 <&8 do echo "beginning... retrieving files from d list" echo "this WILL take a while" echo $line1 echo $line2 wget -e robots=off -t1 -r -p -Q20k --wait=30 --random-wait --limit-rate=200k -np -U "$line1" http://$line2/page.html cp /home/user/testing/*.html /home/user/production echo "done" done exec 7<&- exec 8<&-
I want to continue the script because even though this site, known as $line2 has a 404, the others don't.
I have done the "set +e", and even ran the script with "|| true", all stopping after the error. Because of the 404, there are no files to copy - and then it fails to go onto the next site.