Bug 11567

Summary: updxlrator - simultaneous downloads of the same file occur frequently for Windows
Product: IPFire Reporter: jluth
Component: ---Assignee: jluth
Status: CLOSED FIXED QA Contact:
Severity: - Unknown -    
Priority: - Unknown - CC: jluth
Version: 2   
Hardware: unspecified   
OS: Unspecified   
See Also: https://bugzilla.ipfire.org/show_bug.cgi?id=11558
Attachments: reduceSimultaneousDownloads.patch

Description jluth 2017-12-16 10:16:15 UTC
Created attachment 547 [details]

Recently, Microsoft has started blasting multiple (threaded?) requests for different sections of the same, huge, windowsupdates file. This can easily result in the same url being downloaded by multiple wget commands. I have seen 4 or 5 identical downloads starting within 1 second. This results in a corrupt file, that is identifiable as the file grows to a larger than 100% size since multiple wgets are dumping data onto the same file.

(We have a slow internet connection, so between the time when the "lock" file is deleted and wget actually started downloading data allowed multiple threads to start. A ps ax | grep <url> shows multiple wget and download commands running.)

I have attached the modifications I have made to reduce the problem. I'll know better how effective it is after the next patch Tuesday.  Perhaps a better programmer could figure out how to effectively use flock in this situation to make it fool-proof...