I have an automation script I am testing that imports some rather large files. When I run the automation the progress bar in the background processes view will sometimes stop. TDP (Toad Data Point) still opens and closes menus etc, but the import is dead. At this point I have to close the TDP via Windows task manager and restart TDP. I am not getting any error messages or other indication of what is happening.
I’m not convinced from your description that anything is truly hung or stopped processing as opposed to our background messages are not tell you that we are still working.
We could narrow this down a bit. Please try the following;
-
Run the import template outside of automation from the tools menu. How long does this take to complete. When if finishes click through to the “View Event Log”. This will show you the steps occurring with time stamps.
-
Then go to the automation script and in the settings activity turn the logging to Verbose. Now run the script. You should get the same logging level from import as the view event log in step one. Let it run for twice as long as it took in step one. I think the log file is not written to the automation log until the very end of its processing. This would give the impression of being hung.
-
One additional but possibly useful step is to schedule this script to be run. Use the Scheduler button on bottom right of window wizard bar. This will have TDP run in another instance. I would make sure to have email set up for successful completion and on error. Set it up so both emails include the log file.
This should get us some info to see better what is occurring.
I did some testing and have the same issue with the import template outside of automation, running the automation while its open, or using the scheduler. Over the weekend I tried each of these methods giving them double digit hours to complete and they never do.
I am importing a folder containing 13 excel files with over 250k rows in each (11-15 Mb each). I can get each file to import on its own and they take less than 10 minutes each, but occasionally even the individual will hang. I don’t know if this may have something to do with system resources or toad’s memory management etc, but I have not found examples of other people having these issues.
I do know it has something to do with file size. I have not had similar issues with smaller files, but when doing large imports these problems start occurring regularly. I need to be able to automate this process as it will be a recurring interim data feed until our vendor / IT are able to implement a stable production feed.
Any help would be welcome.
Could this be related to the issues in this post?
The issue in another is about out of memory, did you run into the same exception? From your description, it seems there’s no error/exception, it just hangs. When you run automation with verbose log, were you able to see at which step it hangs? If you can paste the verbose log here, we can investigate it for you.
We do have some user reported that certain Excel will take extra long time to process ( TSS-1140), and we are still investigating. In general, for large data, it takes less resource to process csv file than Excel, you might want to convert your Excel files to csv format to speed up the import.
Thanks!
No I am not getting any error messages. I tested importing all CSV in the folder and it freezes up the same as with excel files. It typically freezes somewhere in the middle of reading/ inserting rows for the first or second file.
The interesting part is if I do a separate import for each file using automation then it runs without issue. If I do an import based upon the folder it hangs. Why would they be that different? Is there some kind of memory handling issue in folder imports?
I have received out of memory errors on my machine, but not while doing this automation. I have a 32 bit computer so its only able to address approximately 3 gigs of RAM. OS and other apps eat 2.4 gigs when my machine is idle. I am starting to suspect the issue is memory related. Can anyone help support/refute this? If it is my problem then I will need to justify an upgrade.
Can you open a support case and give us access to the files you are using. Sometimes the issue is data driven. If you work with support person we can gather the info/files needed to reproduce and fix the issue.