The memory issue I mentioned is not a problem on your machine but in the product to correctly detect if there is enough memory. So it would be a case that the product misunderstood that there is not enough memory and hence terminated your optimization. To check if there is a termination occurred, you may check the Messages window to see if there is a line saying “User Cancel”. And when you contact Support, you may quote Case 714970 or simply ask for escalating your problem to the dev.
Regarding setting the options to terminate, both the original and alternative SQL will be affected by the options. A workaround would be to execute the original in one setting and then execute the other alternatives in another settings. However, I will suggest you start by executing only the alternatives by setting a 5 mins termination time. That is, skip running the original SQL, as you already know its run time. Doing this would help shorten your overall execution time. After finding an alternative that runs in 5 mins, you may then run the original SQL to compare. And in case you don’t find any alternative running faster than 5 mins, you can change the termination time to 10 mins (or 30 mins) to run the alternatives again.