We all know about provider consumer model. My question is about optimal resources utilization in this model.
For example, take a large 1 mln record import of CSV file and try to import it into a database table. One can optimize code to run with parallelism 4 on a dual core machine. That's great. What if the production code runs on a 16-core server that's busy doing other staff. We would like to import data with optimal load on the system resources. How do we do that?
It is best if the system is self-balancing. It spawns more consumers until the benefit is no longer there. Is it something TPL in .NET supports already? Is there a standard algorithm I am missing? Do you have a good 3rd party library that can do that?
by DarkRye via /r/csharp