Bug 11567

Summary: updxlrator - simultaneous downloads of the same file occur frequently for Windows
Product: IPFire Reporter: jluth
Component: ---Assignee: jluth
Status: CLOSED FIXED QA Contact:
Severity: - Unknown -    
Priority: - Unknown - CC: jluth
Version: 2   
Hardware: unspecified   
OS: Unspecified   
See Also: https://bugzilla.ipfire.org/show_bug.cgi?id=11558
Attachments: reduceSimultaneousDownloads.patch

Description jluth 2017-12-16 10:16:15 UTC
Created attachment 547 [details]
reduceSimultaneousDownloads.patch

Recently, Microsoft has started blasting multiple (threaded?) requests for different sections of the same, huge, windowsupdates file. This can easily result in the same url being downloaded by multiple wget commands. I have seen 4 or 5 identical downloads starting within 1 second. This results in a corrupt file, that is identifiable as the file grows to a larger than 100% size since multiple wgets are dumping data onto the same file.

(We have a slow internet connection, so between the time when the "lock" file is deleted and wget actually started downloading data allowed multiple threads to start. A ps ax | grep <url> shows multiple wget and download commands running.)

I have attached the modifications I have made to reduce the problem. I'll know better how effective it is after the next patch Tuesday.  Perhaps a better programmer could figure out how to effectively use flock in this situation to make it fool-proof...