I've got a question about threads in C#. I'm writing a web crawler to look for broken links on web sites. Say I want to run it on 200 sites. I thought that blocked threads would not consume CPU time, and since downloading a web page (with HttpWebRequest.GetResponse) is a "blocking" call, I thought I could run 200 threads just fine, because each thread would "spend" a vast majority of its time (real time) waiting for web responses. However, it did not work out as I had hoped. When I ran 100 threads, the CPU usage was at 10%, so I thought 200 threads would work, and even though the CPU usage was around 15% when trying 200 threads, I noticed that some of the threads didn't even get a chance to start (maybe this was the case even for 100, but I didn't notice it then).
I asked around in IRC, and was directed to this resource on threads in C#. However, that didn't really help me. It even said:
Multithreading is useful when a thread is awaiting a response from another computer or piece of hardware. While one thread is blocked while performing the task, other threads can take advantage of the otherwise unburdened computer.
Which is what I thought from the beginning.
I made a test program that just added and subtraced numbers in one thread, and had N other threads (N ranging from 0 to 1000) waiting for a lock, being blocked. The one thread that was not blocked performed just as many additions and subtractions however many other threads was created, but being blocked by a lock.
So, threads being blocked by a lock does not take any time on the CPU, but threads waiting for web responses do? What's the difference?
by tufflax via /r/csharp