Advice on the architecture of a Async heavy database sync utility console app

Hi Reddit,

I’m getting started on creating a console app that one-way syncs some tables and fields from a mysql database to an Oracle database. The app runs continuously polling an activity table which is populated via triggers (monitor mode). It also get’s called with some arguments to force a resync of each record in one or all table(s) (rebuild mode).

I originally wrote this in PHP and it’s been super reliable. I’m looking to rewrite in C# dot.net core to take advantage of async and, for the learning experience.

Monitor Mode

Long running process that polls the source db activity table for rows of distinct id, tableName then iterates each result (SyncRecord())-

  • Check of the source record exists otherwise delete from destination db
  • Compare source record against destination record and updates/creates destination record if needed
  • Removes the rows from the activity table

Most of the time there is a few records in the activity table but it’s not unusual for there to be hundreds of thousands to process. I’m thinking that each call to SyncRecord would be async. Does that make sense?

Rebuild Mode –

Called on the console e.g. ./DataAsync.exe –table=all –since=2019-01-01 –progress –config=tables.yml

Loops through each table and calls SyncTable() which loops through every record and calls SyncRecord().

Is is safe to have both SyncTable and SyncRecord async? There is 10s of million records to sync so would dot.net manage the number of processes by it self or would I have to manage the number of concurrent operations my self?

Thanks!

submitted by /u/trcullen
[link] [comments]

Leave a Reply