Local Storage - Remove Duplicates

What is the easiest way to remove duplicates from a local storage table?

You can remove duplicates from a table using the Transformation and Cleanse or Data Browser windows.

To use these, right click on the table and select either Transformation and Cleanse or Data Browser. Make a Find Duplicates rule on the column(s) you want (Select the tab in Transform and Cleanse, or right click on the grid and select the option in the menu), and export the results to a csv file or a new table in local storage.

As for the difference between Transformation and Cleanse and Data Browser, both can use the same rules, though making rules in the former is a bit easier since the interface for making rules is always displayed. For the Data Browser, however, you can see all the data in the table.

You can remove duplicates from a table using the Transformation and Cleanse or Data Browser windows.

To use these, right click on the table and select either Transformation and Cleanse or Data Browser. Make a Find Duplicates rule on the column(s) you want (Select the tab in Transform and Cleanse, or right click on the grid and select the option in the menu), and export the results to a csv file or a new table in local storage.

As for the difference between Transformation and Cleanse and Data Browser, both can use the same rules, though making rules in the former is a bit easier since the interface for making rules is always displayed. For the Data Browser, however, you can see all the data in the table.

I’ve tried the Transform and Cleanse, however, my TDA freezes up and never returns data. This could be because of the size of the table though. I can try it again but I think I will have to stick with a manual process.

Sorry this did not work out, but do you mind if I ask a few questions? I do want to see this feature working for you.

  1. Can you tell me the steps you took leading up to this error? Which type of export were you using?

  2. What is the size of the table, approximately?

  3. Can you provide a sample set of data? If not, then could you provide the DDL?

  4. What server was this on?

  5. How long do you wait until TDA freezes?

Thank you for your time in trying this feature.

  1. Right Clicked local table and selected “transform and cleanse”
  2. The table has 2,5 million records, maybe 30 columns
  3. Just a local storage table within TDA 3.0.1.1734
  4. Freezes immediately, waited 5-10 minutes before forcing the program closed.

So it froze as soon as you entered the Transform and Cleanse window?

No tab appeared, only the hour glass cursor. When I use another smaller table it populated within a minute or two.

We cannot reproduce the issue, so far: We were able to send a table with 150 large columns to Transformation and Cleanse, and that you have 2.5 million rows does not matter to that window, as it only retrieves the first 1000. Juding by this, it may be a problem with how the table structure interacts with Transformation and Cleanse. Could you please send me the DDL of the table in question? You can send it to ryan.johnson@quest.com .

Second question: When you say local storage, do you mean the Local Storage that comes with TDA, or another local database?