Comments on: Downloading a bunch of files in parallel using Clojure Agents

By: Steve Miner

Steve Miner — Mon, 16 Aug 2010 19:38:26 +0000

If you don’t want to accidentally drop any elements, use partition-all (in Clojure 1.2) instead of partition.

(partition-all 3 (range 7)) ==> ((0 1 2) (3 4 5) (6))

Also, in Clojure 1.2, partition can take additional args to specify the step and pad so this will work:

(partition 3 3 nil (range 7)) ==> ((0 1 2) (3 4 5) (6))

By: David Cabana

David Cabana — Tue, 12 Jan 2010 05:16:30 +0000

Something to watch out for, by example. Evaluate (partition 3 (range 7)) and you will see that is it is ((0 1 2) (3 4 5)). Notice the missing element 6.

Your example partitions the urls into blocks of 15. If the number of urls to be downloaded is not evenly divisible by 15 you will miss some.

By: Baishampayan

Baishampayan — Thu, 31 Dec 2009 12:01:15 +0000

Daniel,
Thanks for stopping by I agree with you. I could have used fill-queue for this, could then download a new file as soon as one has finished instead of waiting for the whole bunch to be done.

I will probably post a new version which uses fill-queue instead.

And Happy New Year to you too!

By: Daniel Werner

Daniel Werner — Wed, 30 Dec 2009 17:46:17 +0000

Nice, I could have used exactly such a download tool just this week. Thanks for sharing.

One thing that immediately hit me upon seeing “partition” is that your code does download files of the same partition in parallel, but (apply await agnts) will force all downloads to complete before the next partition is even considered.

Tom Faulhaber showed how to use fill-queue in a doseq to process HTTP requests in a queue without blocking newly incoming requests. You might be able to adapt this pattern to your needs:

http://infolace.blogspot.com/2009/08/simple-webhooks-with-clojure-and-ring.html

Oh, and have a happy 2010!