My team is tasked with comparing the rows in approx. 1300 tables that are involved in a CDC process between DB2 and SQL Server. We've split them up by table prefix into 41 groups - not equal sized. Some groupings have 5 or fewer tables, while others have several dozen. There are three of us working on this and we are each creating a "Master" automation. The "Master" automations execute their tasks in a serial manner. We've come to the conclusion that any grouping that contains a single table (or multiple tables) which require several millions of rows being compared should not be in the Master flows and are scheduled in intervals of 30 or 60 minutes apart.
My Master automation consists of 8 smaller automations.
A second Master automation currently consists of 6 smaller ones.
A third Master automation currently consists of 3 smaller ones (he's been on vacation).
These three Master automations are all scheduled for 1:00 AM on Sundays.
On our local workstations, when the data compares themselves complete successfully, we see the temp directories being deleted. When the data compares either crash our TDP or cause it to freeze up and be exited from Task Manager, the temp directories require manual cleanup.
I haven't submitted the large Master automation on my workstation to see what happens in my temp directory.