How to process input records from the file Asynchronously using multiple tasks in C#

I need to process every record from an input file asynchronously using batch concept. Example – say my input file has 100 records and my batch size is set to 10, I need to process the records in 10 batches (10 records per batch) and all these batches should be processed asynchronously. The batch size 10 is not fixed and it might vary.

I have a read function which reads each record from the file and if 10 records are read, I need to call an async method which process these records using a task. But the actual thread should continue reading next set of records and fill the next set of batch (read next 10 records) and once they are read, call the same async method which process these records using another task and this should continue until all records are read.

Right now, I am able to read the record and fill in the batch and then process each batch one after the other, but I wanted to do it asynchronously.

I am providing a snippet of my code below:

public async Task ProcessRecordAsync(InputFile)

{

public int recordCount = 0;

List<Task> TaskList = new List<Task>();

While (condition to check if records present)

{

Object getRecordVal = ReadInputRecord();

if(++recordCount >= 10)

{

var LastTask = new Task(async () => await ProcessRecordAsync());

LastTask.Start();

TaskList.Add(LastTask);

}

}

Task.WaitAll(TaskList.ToArray());

}

ProcessRecordAsync() –> This is the function which processes the input record

I think I am going wrong somewhere in calling the task incorrectly. Once the batch is filled, I wanted to call the ProcessRecordAsync function using a task and the main thread should still continue to read the records and fill the next set of batches. With this code, I am getting an exception.

I am getting below error:

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.

at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)

at System.Collections.Generic.List`1.Enumerator.MoveNextRare()

Could someone help me here? Is this the right way handling multiple tasks?

submitted by /u/sriramchander89
[link] [comments]

Leave a Reply