What’s the right way to expose data in this scenario?

I have a program that does some things over time. While it’s doing things, I’d like to expose that data some how in a manner that is fast for me, the producer, fast enough for the consumer.

Obviously I can fire events. They’re doing the job right now, especially when async event listeners. But I am generating 100,000 events and I want to be able to scale to 1m or 10m events so the event object might be wrong in this case. Perhaps I need to build some type of stream?

I just need to get the objects exposed in memory to third parties, and get my side of it garbage collectable as soon as possible.

Is this a good use case for something like

https://blog.marcgravell.com/2018/07/pipe-dreams-part-1.html

https://blogs.msdn.microsoft.com/dotnet/2018/07/09/system-io-pipelines-high-performance-io-in-net/

Or perhaps TPL DataFlow?

Thank you very much. .Net core or framework are both viable options here.

submitted by /u/yog-sothos
[link] [comments]

Leave a Reply