Multithreading a for loop?

Paramount · 6 Jan 2011 at 22:21

I've got a for loop that, in sequence, downloads a number of webpages - basically it pulls a URL from a database, grabs the html using a httpwebrequest, and saves a part of it back into the database. Now over 1-10 pages this is no big deal, but upwards of 40 it starts taking a while (from one-by-one contacting the server and downloading it's contents).

This isn't a processor limiting loop, so is there a way I can speed it up by opening multiple connections at the same time?

I'm using c# and am relatively new to programming. Just a point in the right direction would be a great help.

brainchylde · 7 Jan 2011 at 01:01

I'm assuming this is a windows forms application?

If so, have a look at background worker. It's basically an overlay of the more complex system.threading. It depends of course what you're after and the complexity. The more complex you get, the more you're going to be looking at pure threading.

I often use the background worker where I want to carry out a simple operation like running a tableadapter.fill, but which takes a while. The background worker ensures that the user interface doesn't grey out and become unresponsive. I often show a loading circle or other progress indicator when I'm doing this.

mk17 · 7 Jan 2011 at 12:32

If you're on VS2010, you can use a Parallel.For instruction to automate this.

PsychoDuck · 7 Jan 2011 at 13:58

Or if you are using VS2008 with the parallel extensions installed.

brainchylde · 7 Jan 2011 at 14:24

mk17 said:
If you're on VS2010, you can use a Parallel.For instruction to automate this.

This is very neat - I didn't know about this!

Paramount · 8 Jan 2011 at 17:22

Brill, thanks

topbanana · 8 Jan 2011 at 19:20

As you say, the task is not compute bound but I/O bound. It's not a job for threads, but I/O completion ports.

The standard async functions of HttpWebRequest will operate in this manner without needing to spin up any threads.