DataSelf ETL+ Release History (Download Links)

Warning: Test the beta version carefully before using it for production. Click here for the upgrade steps.

Download Links:
- ETL+ 64 bits: DataSelf ETL Plus - Beta 2022.04.1302.exe
- ETL+ 32 bits: DataSelf ETL Plus 32bits - Beta 2022.04.1302.exe
- DataSelf Agent: DataSelf Agent.exe
Released: 04/13/2022 (beta release #7)

New Features

1. ETL Page: Renaming tables on the center panel renames the physical data warehouse tables.

2. ETL Page: Duplicating tables on the center panel duplicates the physical data warehouse tables. This is helpful when users want to perform tests on a table without affecting the official one.

3. ETL Page: Deleting tables on the center panel deletes the physical data warehouse tables. User is prompted to confirm the physical deletion.

4. ETL Page: Deleting ETL+ Sources deletes its mapped tables on the center panel and now also the physical data warehouse tables. User is prompted to confirm the physical table deletion.

5. Job: “Refresh Batch” renamed to “Job” - this feature can run scripts that are not “refreshes”.

6. DataSelf Agent: Greatly anticipated ETL+ auxiliary application to execute scheduled and remote tasks. Long-term vision: a user runs ETL+ on her laptop at a hotel and requests a data refresh from her on-premises ETL+ install, and orders a margarita (drink orders not included in ETL+ lol). Their on-prem DS Agent receives and executes the request on the on-prem ETL+. And then it notifies the user when the refresh is complete. Magic happens while the margarita buzz kicks in!

7. Agent: Introduces the concept of ETL+ Production installs where the locally deployed Agent runs scheduled jobs and remote tasks. For the time being, an ETL+ install will have to either use WTS or the Agent for running scheduled jobs.

8. Agent: ETL+ Production installs must have direct access to source systems and have the Agent locally deployed and running with a local Windows Admin user.

9. Agent: ETL+ will capture in the metadata the Window Device name to associate the ETL+ Entity and Job IDs to run by the Agent.

10. Agent: When assigning a computer to run the Agent for the first time, ETL+ assigns the username and password saved on the Job → Properties → Agent credentials. This user must have local Windows Admin Rights.

11. Agent: When assigning a computer to run the Agent for the first time, ETL+ creates an Agent shortcut in the Windows folder: C:\Users\<user_name>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup.

12. Agent: The shortcut from the item above will automatically run the Agent with the correct Windows user credentials every time Windows starts/reboots.

13. Agent: Runs as a Windows service (in the background) and checks the cloud metadata every 10 secs for instructions to execute.

14. Agent: Checks the cloud metadata if there are Jobs to run now. The rule is: (Jobs associated to run by this Agent) & Enabled & (Next Run Time <= now).

15. Agent: Checks the cloud metadata if there are on-demand tasks to execute. Currently, it can execute two tasks remotely: run a Job now, and kill the execution of a Job initiated on-demand.

16. Job: Added Agent column on the job list to show which app runs the schedule of a job. Options: WTS (Windows Task Scheduler) and DataSelf Agent.

17. Job: Agent column showing WTS: this is the default approach. The job runs via Windows Task Scheduler. ETL+ currently can only create and overwrite WTS tasks via the Schedule button.

18. Job - Icons: Renamed Run Job -> Run Job via ETL+.

19. Job - Icons: Added Run Job via Agent. This sends a message to the Agent to run the Job on demand.

20. Job - Icons: Added ... (More Actions).

21. Job - Icons: Added End Job via Agent icon under the ... icon.

22. Job - Icons: Added Enable icon under the ... icon.

23. Job - Icons: Added Disable icon under the ... icon.

24. Job - Icons: Moved Rename Job icon under the ... icon.

25. Job - Icons: Moved Delete Job icon under the ... icon.

26. Job - Right-click: Same changes as Job - Icons above.

27. Job: Agent column showing Agent Enabled: This means the Agent will run that job on its schedule.

28. Job: Agent column showing Agent Disabled: This means the Agent will not run that job on its schedule.

29. Job: Agent column showing Agent Running: This happens after a user triggers an on-demand Run Job via Agent. Currently, this Running status won’t know when the Job is completed. Users can monitor the Job running status via the Log or by the refresh email.

30. Job: Agent column showing Agent Stopping: This happens after a user triggers an on-demand End Job via Agent. Currently, this Stopping status won’t know when the Job ends. Users can monitor the Job running status via the Log.

31. Job - Pop up: Shows Job name

32. Job - Pop up: Shows Agent to run the Job. Computer Name if DataSelf Agent will run the Job.

33. Job - Pop up: Shows Job Schedule.

34. Job - Pop up: Shows Load Type.

35. Job - Pop up: Shows Last Run Time.

36. Job - Pop up: Shows Next Run Time.

37. Job: Removed Windows Task Scheduler (WTS) section.

38. Job: Removed WTS Settings.

39. Job: Added Properties icon which shows a section for Schedule, Load Type, and Agent.

40. Job - Schedule: Added radio buttons to set which engine runs schedules: WTS or DataSelf Agent.

41. Job - Schedule: When radio button = WTS, the feature works like in prior ETL+ releases.

42. Job - Schedule: When radio button = DataSelf Agent, the Agent will be responsible for running the Job’s schedule.

43. Job - Agent: Press the Apply button to associate a Job to run on this computer.

44. Design: When changing the Data Type of a column, ETL+ posts the SQL expressions automatically applied. Users can edit the expression.

45. Design: After a SQL expression is posted to a column, if a user changes the column’s Data Type again, ETL+ will prompt if it should update the SQL expression.

46. Design: Loading of MS SQL Timestamp columns automatically converted to char(23).

47. Load Engine: Depending on the source system, loading of Date and DateTime columns automatically converts invalid values to NULL.

48. Load Engine: Delta filter now becomes a physical index in the data warehouse.

49. Load Engine: PK columns used in delta loads now become physical indexes in the data warehouse.

50. Load Engine: Adjustments when recalculating MS SQL indexes for the filter column used in tables with Replace. Resulting in significant performance improvement when processing deltas in large tables.

51. Load Engine: Adjustments when recalculating MS SQL indexes for the filter and PK columns used in tables with Upsert. Resulting in significant performance improvement when processing deltas in large tables.

52. Load Engine: New ETL+ command line parameter -e EntityID.

53. Load Engine: New ETL+ command line parameter -j JobID.

54. Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, it’ll show the message “Another ETL+ is loading this table. Waiting for table release.”

55. Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, every 15 secs it’ll show the message “Checking table release…”

56. Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, but the other entity isn’t active for over 90 seconds, the new entity takes over the table load.

57. Load Engine: Checking if Log tables exist only during the opening of an entity (instead of during the load of every table).

58. Load Engine: Added MS Access as a new source type.

59. Load Engine - MS Access: Enter file path and password if required.

60. Load Engine: Support and adjustments to BigQuery ODBC.

61. Load Engine: Shows when the load started (metadata date/time).

62. User Interface: Runs a Job on its own window (runs via command line).

63. User Interface: Opens Agent page on its own window (runs via command line).

64. Command Line: added syntax: -e EntityID -j JobID -ui Page <action>.

65. Command Line: parameter -e followed by EntityID. Ex.: -e abccorp

66. Command Line: parameter -j followed by JobID. Ex.: -j 1

67. Command Line: parameter -ui followed by load -j JobID. Ex.: this runs JobID on a new Load session. This automatically closes once the job completes the refresh.

68. Command Line: parameter -ui followed by log. Ex.: this opens the Log page on a new ETL+ session. This session stays open until the user closes it. Click the Refresh icon to view the latest log rows.

69. Command Line: parameter -ui followed by agent. Ex.: this opens the Agent page on a new ETL+ session.

70. Security: New passwords require at least 12 complex characters (up from 8).

71. Security: New passwords now accept # @ as part of special characters.

72. Security: When ETL+ can’t connect to its metadata: "Unable to connect to cloud metadata via https and SSL SQL. ETL+ needs to whitelist your public IP, or your firewall or antimalware is preventing ETL+ connection to the cloud. Start ETL+ IP Whitelisting?"

73. Security: ETL Page - ETL SQL Statement: Clicking Confirm validates zData Warehouse schemas.

74. ETL page, title now shows the connected “Data Warehouse: <SqlServerName>.<Database>”

75. ETL Page, center panel: Renamed Data Warehouse to ETL Objects. More descriptive of what the panel is about. Objects currently include source headings (to group sources' tables together), SQL schemas, tables, and table columns.

76. ETL Page, center panel: zData Warehouse heading renamed to zData Warehouse (Reimports). More descriptive of the tables under this header.

77. ETL page, left panel renamed to “ETL Source”.

78. ETL page, left pane shortcut: select a source → (Ctrl + i) opens its Properties.

79. ETL page, left pane shortcut: double-click a table to add/map it to the load process.

80. ETL page, left pane shortcut: (F5) to refresh a source.

81. ETL page: Job icon updated.

82. ETL page: Settings icon updated.

83. ETL Page: Find feature to quickly highlight matches on sources, data warehouse tables, and SQL Statements.

84. ETL Page: Click the Find icon to expose/collapse the search box

85. ETL page, center pane shortcut: select a table → (Ctl + Del) key to delete.

86. ETL page, center pane shortcut: select a table → (2) key to duplicate.

87. ETL page, center pane shortcut: select a table → (F2) key to rename.

88. ETL page, center pane shortcut: select a table → (Ctrl + D) key to open Design.

89. ETL page, center pane shortcut: select a table → (Ctrl + C) key to load its Columns.

90. ETL page, center pane shortcut: (Ctrl + O) loads columns and expands its list.

91. ETL page, center pane shortcut: select a table → (Ctrl + P) key to preview.

92. ETL page, center pane shortcut: select a table → (Ctrl + L) key to Load Now.

93. ETL page, center pane: Added ... icon to the panel header for additional non-popular actions.

94. ETL page, center pane: select a table → ... → Show Columns.

95. ETL page, center pane: select a table → ... → Load All Columns.

96. ETL page, center pane: select a table → ... → Expand/Collapse All source groups.

97. ETL page, center pane: select a table → ... → Download to CSV.

98. ETL page shortcut: click (F1) key to open Help.

99. ETL page shortcut: click (Ctrl + F) to go to the Find box.

100. ETL page shortcut: click (Ctrl + G) to open the Log page.

101. ETL page shortcut: click (Ctrl + J) to open the Job page.

102. ETL page shortcut: click (Ctrl + S) to Save.

103. ETL page shortcut: click (Ctrl + O) inside of ETL SQL Statement to Confirm.

104. ETL Page: Right-click a data warehouse table → Download to CSV downloads the data in CSV format into ETL+ folder/DS_Downloads/<TableName>.csv.

105. Refresh Email Notification: Removed email attachments. No longer necessary with cloud logging.

106. Log Page: Added Rows Retrieved parameter. Enter a new value and press Refresh icon.

107. Log Page: (F5) shortcut to refresh Log page.

108. Log Page: Titles and radio buttons changed to ETL Log and Table Stats.

109. Log Page: Debug Level dropdown is available on this page now.

110. Log Page: Debug Level dropdown shows a summary of what the log level does.

111. Log Page: Debug Off now removes the log file on the local HD. For saving disk space and avoiding requiring ETL+ to Run as Admin when running from Windows protected folders

112. Log Page: Debug Off no longer counts and records the total number of rows before and after a table load. Performance improvement.

113. Log Page: New Debug Level 0 adds the log file on the local HD plus counts and records the total number of rows before and after a table load.

114. Log Page: Debug Level 1 adds Level 1 log plus Level 0 features.

115. Log Page: Debug Level 2 adds Level 2 log plus Level 0 features.

116. Log Page: The Debug dropdown description is contextual to the page’s radio buttons.

117. Log Page: Table Stats layout rearranged and some labels renamed for easier viewing.

118. Log Page: Default sorting of Log pages is the Index column.

119. Log Page: Row counting uses SQL metadata (instead of count(*)). Faster performance.

120. Logging: Checking if Log tables exist only during the opening of an entity (instead of during a load of every table).

121. Restore ETL SQL Statement As Of: New dropdown list on Settings allows users to retrieve prior ETL SQL Server Statements. Useful to recover deleted and changed statements from the past.

122. Login: Forgot Password? button. It’ll email the password when initiated from whitelisted IPs with a valid username (email address).

123. Job: New script Download to CSV to run on a schedule. Select a Job → + on the Steps panel -> Download to CSV → select a table -> Save.

124. Job: Find feature to quickly highlight matches on the Steps panel.

125. Job: Click the Find icon to expose/collapse the find box.

126. Job: Standardized user interaction as main ETL page so right-clicking on panel objects pops up the available features. Ex.: right-click a job → Run Job.

127. Job, left panel: Standardized user interaction as main ETL page so right-clicking on panel objects pops up the available features. Ex.: right-click a job → Run Job.

128. Job, left panel now has icons on the panel header (instead of inside of the panel).

129. Job, left panel now has two sets of icons, the left is active when selecting individual jobs, the right is independent of individual jobs

130. Job, Step panel: now shows the Job <ID> on the header.

131. Job, Step panel: Added Move to Top icon/feature.

132. Job, Step panel: Added Move to Bottom icon/feature.

133. Job, Steps pane shortcut: (Ctrl + F) to go to the Find box.

134. Job, Steps pane shortcut: select a script → (Ctrl + i) to open its properties.

135. Job, Steps pane shortcut: select a script → (Ctrl + Del) key to delete.

136. Job, Steps pane shortcut: select a row → (Ctrl + D) key to pushing it down one row.

137. Job, Steps pane shortcut: select a row → press and hold the (Ctrl + D) key to push it down.

138. Job, Steps pane shortcut: select a row → (Ctrl + U) key to pushing it up one row.

139. Job, Steps pane shortcut: select a row → press and hold the (Ctrl + U) key to push it up.

140. Job, Steps pane shortcut: (Ctrl + T) to create a new Tableau Extract Refresh script.

141. ETL+ Plain Deployment: New framework for an on-premises data warehouse that easily loads the zDateUntreated table from a cloud Excel file.

142. Metadata: Redesigned metadata for improved security, reporting, and indexing.

Fixed Issues

1. Tables with delta refresh no longer log the misleading message: “Error : Column name '<PK_column1,PK_column2,PK_column3etc>' does not exist in the target table or view.”

2. Regular users: When login in to ETL+, the list of available entities used to show deleted ones as well.

3. When manually loading a table, it will retry 3 times in case of failure (instead of 12 times).

4. Tables with delta refresh no longer run into the error: ‘Invalid column name ‘0_UnionID', but ETL+ still processes the delta refresh correctly.

5. When super users switched to a new entity within the same ETL+ session could create zEntity and zLog tables in the new entity’s data warehouse with the schema from the prior entity.

6. When super users switched to a new entity within the same ETL+ session could send refresh emails with the prior entity’s refresh email subject.

7. Deleting all Jobs no longer requires going to the ETL main page and back.

8. Deleting an entity and recreating it with the same ID would add a new row to zEntity table.

Known Issues

1. Clicking the Log icon after certain tasks (such as running a Job) might present an "Unhandled exception" error. Click Quit and re-open. It mostly only happens once.

2. Regular users: When login in to ETL+, in the first attempt, ETL+ will only open the first entity even when one uses another entity on the dropdown list.

3. Upon adding a new table from a NetSuite ODBC, the SQL syntax will have an extra ] at the end of the table name. To fix it: remove ] and save.

4. Many SQL Statements for MongoDB ODBC will only load using SELECT *.

5. Some Acumatica OData v4 tables might show the error “The contents of table TableName are not available for this user“ and fail to import. This is an Acumatica security issue and we’ve requested how to fix it. While we wait for the Acumatica fix, use OData v3 to import such tables.

6. Adding tables from Sage Intacct might not work.

7. Opening an entity from an older metadata version with the new one using a regular user (non Admin) will not update the metadata version in the cloud. Workaround: open it with an Admin user.

8. Job: If a table fails to refresh, the ETL+ might unexpectedly load several tables multiple times.

9. Sometimes ETL+ will mistakenly flag that the metadata has been changed and you need to exit and re-enter the entity. Just exit and re-enter.

10. Changing the data warehouse of an entity won’t show on the bottom left corner until the entity is closed and re-opened.

11. Regular users: Mousing over Sources doesn’t show the pop up with details.

12. The icon color and type for the column formats on the center panel aren’t 100% consistent yet.

13. Opening an entity with tables mapped to the zData Warehouse with an older ETL+ version will push such tables to other source databases on the center panel. DataSelf can fix that manually.

14. Importing large XML content into varchar(max) might run into error: server stack limit has been reached.

15. A few installs are hitting the error “There is insufficient system memory in resource pool 'internal' to run this query.” once in a while. It happened to only one table in a job, and a different on another job. Load All. First report started in v2021.09.

16. Sometimes ETL+ is closing unexpectedly after a data load.

17. Sometimes ETL+ runs into an “Unhandled exception” error. Close and re-open ETL+.

18. Locking the ETL+ editing when someone else is editing it doesn’t work in some situations.

19. ETL+ still tries to send email even when set to Cloud notification, or No Notification.

20. ETL+ delta extraction from NetSuite ODBC might not work when using dates for filtering.

21. When opening an entity created in prior ETL+ versions, the data warehouse panel will show the source header blank. To fix this, open the entity in the prior ETL+ version, make any modification and save it. Now open the entity in the latest ETL+ and the source header appears.

Download files:
64 bits: DataSelf ETL Plus.exe
32 bits: DataSelf ETL Plus 32bits.exe

Released: 02/05/2022

New Features

1. ETL Log: Log icon now retrieves cloud log. Users can retrieve the log from anywhere.

2. ETL Log: By default, the Log shows the latest logs at the top of the page (sorted <DateTime> Desc).

3. ETL Log: Sort the Log page by clicking the Log column headers.

4. ETL Log: Click Sequence header to sort by <RefreshRunID> Desc & <DateTime> Asc.

5. ETL Log: Refresh icon re-loads the latest log entries.

6. ETL Log: Logged dates now show the time zone (currently, PST only).

7. ETL Log: Option to view Stats or Summary logs.

8. ETL Log: Stats show row stats, processing time, and refresh criteria by loaded table.

9. ETL Log: Summary shows loaded tables, rows, error messages, refresh batch headers (admin user: machine username).

10. ETL Log: Creates and populates the Log Summary in the data warehouse table <dw schema>.<zRefreshLog_Summary>.

11. ETL Log: Creates and populates the Log Stats in the data warehouse table <dw schema>.<zRefreshLog_Stats>.

12. ETL Log: Log columns available to Super User only: PK_Entity, MachineUser.

13. Table labeling: Improved labeling to uniquely identify tables (MS SQL standard):

    1. <Source Name> - <SQL Schema>.<Table Name>

    2. Ex.: ExcelFile - dbo.BudgetTable

14. Table labeling: In multi-tenant data warehouses (such as in Advanced Analytics), the schema doesn’t show on the center panel since all loaded tables belong to the same schema (<EntityID>).

15. Table labeling: Refresh Batch Steps don’t show schemas to simplify the table list. Schemas can be seen on a pop up by mousing over tables.

16. Source - zData Warehouse: The data warehouse automatically shows as a data source for additional data preparations and transformations.

17. Source - zData Warehouse: “zData ” source is listed at the bottom of the panels.

18. Source - zData Warehouse: Uses a green source icon to tell it apart from regular sources.

19. Source - zData Warehouse: This source provides access to all data warehouse tables and views for single-tenant data warehouses.

20. Source - zData Warehouse: This source provides access to data warehouse tables and views where <SQL schema> = <EntityID> or <SQL schema> = <dbo> in multi-tenant data warehouses (DataSelf Analytics Advanced Edition).

21. Source - zData Warehouse: Source’s Properties disabled.

22. Source - zData Warehouse: Source 'sDelete disabled.

23. Source - zData Warehouse: Use Settings -> Data Warehouse for its configuration.

24. Source Properties: Renamed Dw Schema to Schema. Sets Alias as the source tables' schema.

25. Data Warehouse Panel - zData Warehouse: When importing zData Warehouse tables, they’ll be grouped under a green grouping icon.

26. Data Warehouse Panel: Renaming a table only applies to its name, not the schema.

27. Data Warehouse Panel: Tables imported from the data warehouse keep the same SQL schema as the data warehouse default schema.

28. Data Warehouse Panel - Columns: Double-click a data warehouse table to show:

    1. Column list

    2. Column formatting

    3. Primary key

29. Data Warehouse Panel - Columns: Click + / - by a table name to expand/collapse its column formatting.

30. Data Warehouse Panel - Columns: icon for regular columns.

31. Data Warehouse Panel - Columns: icon for transformed columns.

32. Data Warehouse Panel - Columns: icon for primary key columns.

33. Data Warehouse Panel - Columns: icon for primary key transformed columns.

34. Data Warehouse Panel - Design: added Design icon to access table Design page.

35. Data Warehouse Panel - Design: For many sources, the list of columns shows even when the SQL Statement involves transformations, table links, table unions and CTEs.

36. Design: New page for table design:

    1. Configure primary key columns

    2. Change column formats

    3. Apply column transformations

37. Design: Many sources don’t allow transformations during the extraction, or extraction transformations are limited and/or slow. The data is extracted and loaded into a temporary data warehouse table, and then transformed into to official table. These transformations happen in the data warehouse engine, therefore they are fast, efficient and provide a broad range of MS SQL transformation functions.

38. Design: Select a data warehouse table → Design to open the Design page.

39. Design: New page shows PK (primary key), Column Name, Dw Data Type, Transform SQL Expression.

40. Design: List shows gray and white banded lines.

41. Design: To change a Column’s format, use its Dw Data Type dropdown menu, or double-click the dropdown and type in the new format.

42. Design: Column MS SQL transformations available on Transform MS SQL Expression. Ex.: LEFT(ColumnName,5).

43. Design: Click the column header for Column Name or Source Data Type to sort the list.

44. Design: Press & hold & drag with the right button of a mouse on the border of a column to change its width/height.

45. Design: ETL+ will execute the Design changes when user clicks Confirm.

46. Design: If issues are found during Design changes execution, ETL+ will report which columns were successfully changed and which ones failed.

47. Design - PK: PK check boxes define a table’s primary key columns. Currently:

    1. PKs are necessary to process Upsert delta refreshes.

    2. No validation if the PK columns are the correct primary key.

    3. No creation of the physical PKs in the data warehouse.

    4. If physical PKs are needed, please create them directly in SQL after the first ETL+ import.

48. Design - PK: When importing a new MS SQL or Odata table that has PK in the source, ETL+ will automatically recognize and pre-configure its PK.

49. Design - PK: Check the PK check boxes to change/define the table’s primary key columns.

50. Design - PK: For text-based sources like OData, varchar(max) columns are automatically changed to varchar(50). We recommend reviewing the varchar size.

51. Design - PK: ETL+ logs the following when MS SQL can’t create/recalculate a PK: "Primary Key failure. PK requires rows with unique values and no NULLs. Review the PK fields.". This means the selected PK columns are not the table’s correct primary key - review/fix the PK configuration.

52. Design - PK: When upgrading an ETL+ entity, Primary Index columns automatically become Primary Key columns.

53. Column Formats: On the data warehous panel (center panel), double-click or right-click on a table name -> Show Columns to enable the expansion of this table’s column names and formats.

54. Column Formats: Click +/- by a column name to expand/collapse its column names and formats.

55. Column Formats: To reload column names and formats, Double-click again or right-click on a table name -> Show Columns again.

56. ETL Page - Load Type: Load Type section now only uses vertical real estate needed.

57. ETL Page - Load Type: Replace All has been renamed to Load All.

58. ETL Page - Load Type: Replace with Control has been renamed to Replace.

59. ETL Page - Load Type: Update has been renamed to Upsert.

60. ETL Page - Load Type: Hovering over the Load Type options shows its function.

61. ETL Page - Load Type: Hover over Load All: 1st: Load all. Ongoing: Reload all.

62. ETL Page - Load Type: Hover over Replace: 1st: Load all. Ongoing: Replace filtered rows.

63. ETL Page - Load Type: Hover over Upsert: 1st: Load all. Ongoing: Update modified, insert new rows.

64. ETL Page - Load Type: Hover over Append: 1st: Load all. Ongoing: Append all.

65. ETL Page - Load Type: Selecting Replace or Upsert expands its settings section.

66. ETL Page - Replace: Filter column now is a drop down.

67. ETL Page - Replace: The filter expression is automatically set to Max(FilterColumn).

68. ETL Page - Replace: Users can customize the filter variable. For instance:

    1. Constant: 1000

    2. Date constant: ‘02/15/2021’ (use quotes around the MM/DD/YYYY date format)

    3. SQL Expressions: for instance, getdate()-30

69. ETL Page - Upsert: Renamed Primary Index to Primary Key.

70. ETL Page - Upsert: Primary Key lists the PK columns (from the Design page).

71. ETL Page - Upsert: Primary Key lists the PK columns in the same order they appear in the SQL Statement.

72. ETL Page - Upsert: To edit the PK columns, click the Edit icon by the Primary Key or the Design icon on the table.

73. ETL Page - Upsert: Use the dropdown to select the Column with Last Modified Value.

74. ETL Page - Upsert: The Update filter is automatically set to Max(LastModifiedValueColumn).

75. ETL Page - SQL Statement: Support to MS SQL Server CTE (Common Table Expressions).

76. ETL Page - SQL Statement: Support to SQL statement comments. Syntax /* comment text */. This can only be used after an initial SELECT statement.

77. ETL Version Control: When opening an entity with a different ETL version than the last one used for the entity, ETL will notify and prompt what to do next.

78. ETL Version Control: When using a newer ETL+ version than the last one used on an entity, users will be prompted to upgrade or cancel.

79. ETL Version Control: When using an older ETL+ version than the last one used on an entity, users will be prompted to download the older version (by going to our download page) or cancel.

80. Load Engine: Ability to apply transformations after the load (previously, only available during extraction when the source SQL syntax allowed).

81. Load Engine: Source loading adds a 0_UnionID column with content = table schema. This is helpful for multi-company table linking. This new column will be further refined when more multi-company features become available.

82. Load Engine: Settings 0_Entity now creates 0_<EntityID> column (instead of 0_<schema>).

83. Load Engine: If the source table already has a 0_<EntityID> column, load engine will import this column (instead of creating it).

84. Load Engine: If the source table already has a [0_UnionID] column, load engine will import this column (instead of creating it).

85. Load Engine: ETL+ retries 3 times upon running into a load error in the UI, and then proceeds. In a multi table refresh, use the Cancel button to stop the refresh (Cancel applies before the next table load).

86. Load Engine - zEntity: Creates a <dw schema>.zEntity table with the following columns: PK_Entity, EntityID, EntityName, CYE_Value, DT_LatestLoadStart, FYrStart_Int2, PnLType, ReportingDate, 0_UnionID, 0_<EntityID>.

87. Load Engine - zEntity: If <dw schema>.zEntity already exists, ETL+ doesn’t overwrite it.

88. Load Engine - zEntity Columns: PK_Entity, EntityID and EntityName defining the entity.

89. Load Engine - zEntity.CYE_Value: It’s populated by Settings page -> CYE Value. This is used by many Financials templates to set up a description or a GL number to Current Year Earnings.

90. Load Engine - zEntity.DT_LatestLoadStart: At the start of a new load (individual or group of tables), ETL+ writes the current date & time on this column. Use case: to populate report DataAsOf fields.

91. Load Engine - zEntity.FYrStart_Int2: It’s populated by Settings page -> FYear Start: Mmm. A number between 0 and 12that defines the 1st calendar month of the first fiscal year (assumes period starts on the 1st of the month). Ex.: 2 for Feb 1st. 0 means a custom fiscal year. Use case: this field controls part of the logic on DataSelf’s 5_Date and 6_DatePeriod date tables.

92. Load Engine - zEntity.PnLType: It’s populated by Settings page -> P&L: Type. This column controls how DataSelf P&reports work: S → Standard P&L or E → EBIDTA type.

93. Load Engine - zEntity.ReportingDate: For customers that want to customize the actual reporting date. For instance, show this fiscal period metrics based on the end of last month’s date. It’s populated by Settings page -> Reporting Date. Options: 0 -> Today. 1 -> Yesterday, 2 -> Last month's end.

94. Load Engine - Load All: The load happens on a temporary table to allow users to query the official table with minimum interruption. Once the load completes, the temporary table becomes the official one. The temporary table is named Schema.TableName_TEMP.

95. Load Engine - Load All: The transition between temporary to official table happens very quickly when there are no transformations performed (delete official, rename temporary to official, create <0_EntityID> when applicable).

96. Load Engine - Load All: The transition between temporary to official table happens quickly when there are transformations performed (delete official, insert into official from temp table applying transformations).

97. Load Engine - Replace and Upsert: When user changes the mappings of a delta refresh table, and the table already has data, user is promped with:

    1. Column mappings modified. The delta refresh needs one "Load All" data load to sync up OR manual adjustments to the physical table.
       Force one "Load All" in the next load (manual or scheduled)?
       Yes/No/Cancel

98. Load Engine - Replace and Upsert: When user clicks Yes to the prompt above, ETL+ will force a Load All in the next table load (manual or scheduled), and then resumes the delta refresh.

99. Refresh Batch: Rename icon added for Refresh Batch Names.

100. Refresh Batch: Removed Run Refresh Batch icon from header (cleaner UI).

101. Refresh Batch: Removed Delete Refresh Batch icon from header (cleaner UI).

102. Refresh Batch - Steps: Select a step and Click & Hold the Up or Down icon to quickly move it.

103. Refresh Batch - Steps: Hover over Up and Down icons to show quick help pop up.

104. Refresh Batch - WTS: Repeat Task checkbox unchecked by default.

105. Refresh Batch - Steps: Pop up when hovering steps has been cleaned up and re-organized the layout to make it clearer what the step is about.

106. Data prep: If the source data brings in a Date and or DateTime value that is incorrect for SQL (common in text based data sources like Excel or OData, and Providex), apply transformations to fix the issue. Examples:

     1. Invalid date values (ex.: Feb 30): Convert the field to a character field.

     2. A date/time column has more characters than accepted by SQL. Remove the extra characters: CAST(LEFT(DateColumn),19) as DATETIME).

107. OData Extraction - Load All: ETL+ process the data extraction sorted based on the PK (usually available) or a key available from the OData metadata.

108. OData Extraction - Replace and Upsert: ETL+ process the data extraction sorted based on the delta refresh filter (ex.: InvoiceDate) plus the PK (usually available) or a key available from the OData metadata.

109. OData Extraction - Replace and Upsert: When starting a new load and there’s no dw.TableName_Temp2, ETL+ checks the delta refresh filter value from dw.TableName, applies the filter value into the source data extraction process, and loads the data into TableName_Temp. If the end of the extraction is reached before 100 pages, ETL+ deletes TableName, and renames TableName_Temp to TableName_Temp.

110. OData Extraction - Replace and Upsert: Starting an extraction like listed in the prior feature above, but if 100 pages are loaded without the end of the extraction, ETL+ moves/append TableName_Temp onto TableName_Temp2, deletes TableName_Temp, logs off Acumatica, logs in Acumatica again, and resumes the load into TableName_Temp. This process repeats until the end of the extraction, then TableName_Temp is appended onto TableName_Temp2, TableName is deleted, and TableName_Temp2 is renamed into TableName.

111. OData Extraction - Replace and Upsert: When starting a new load and there’s a TableName_Temp2 on the data warehouse, ETL+ starts the delta load from TableName_Temp2 (instead of from TableName) and processes the load as per the prior feature.

112. OData Extraction - Upsert: Be sure to configure the order of PK columns in the SQL statement to guarantee an ascending extraction. Failing to do this might cause delta data extraction inconsistencies. We highy recommend adding a date (like CreateDate) as the first column of a PK controlling an Upsert.

113. OData Extraction - Upsert: When executing OData extractions, the order of the PK columns in the OData extraction URL following the order of the columns the SQL statement (instead of alphabetical).

114. ODBC Configuration: Added an explanation about adding credentials to an ODBC driver.

115. ODBC Configuration: Added an explanation about potential reason for missing ODBCs is because of mismatch of ETL+ and ODBC 32/64-bit builds.

116. ODBC Extraction: Support to Pervasive ODBC.

117. Installation: For every entity, usually only one ETL+ installation requires setting up the connection to data sources. Therefore, we removed the installation step to connect every ETL+ to its data sources.

118. Installation: We removed the Windows Task Schedule step from the installation. Cleaner install.

119. All Users: On the user credentials screen, the Login button is preselected. Once credentials are entered, press Enter to login or click the Login button.

120. All Users - ETL+ Version Control: ETL+ flags its metadata with the ETL+ version used to maintain it.

121. All Users - ETL+ Version Control: When a user opens an entity with a different version than what’s on the metadata, ETL+ will prompt the user if he/she wants to proceed with the different version or Cancel.

122. All Users - ETL+ Version Control: If a user wants to open an entity with a newer or older ETL+ version than what’s in the metadata, the ETL+ will proceed and update the metadata ETL+ version information.

123. All Users - ETL+ Version Control: If a user doesn’t want to open an entity with a newer or older ETL+ version than what’s in the metadata, the ETL+ will prompt if the user wants to open ETL+ Release History webpage to download other versions.

124. Admin User: ETL+ admin user page now shows the ETL+ metadata version.

125. All Users: The new ETL+ installation flow no longer has a WTS step. This was confusing for many users.

126. Client Info: ETL+ license is now part of the metadata.

127. All Users: When opening an entity, the center panel cursor goes to the top of the page.

128. All Users: Added Send button on Client Info page to re-send product registration email.

129. All Users: Added Send button on Client Info page to re-send product registration email that now copies sales@ and support@ dataself emails.

130. Admin User: metadata.EntityAttribute new table to host customer and tech support information.

131. Admin User: Entities page has a search box - enter characters to find and select matching EntityIDs and Names.

132. Admin User: When landing on the Entities page, the cursor goes to the search box.

133. Admin User: When leaving the ETL page back to the Entities page, the prior Entity will pre-selected in the new Search box.

134. Admin User: When a single entity shows on the Entities page, just click Enter to Edit it (or click Edit with mouse).

135. Admin User: Tableau Script Deployment removed from Refresh Batch page (it was confusing).

136. Admin User- Duplicate Entity: Upon duplication of an Entity, ETL+ finds & replaces <OriginalEntityID> with <NewEntityID> in all SQL Statements. This is needed for many out of the box data warehouse table data manipulations. Ex.: for properly processing zEntity and zCalendarTable tables.

137. Admin User: ETL+ Customer Portal with easy access to consolidated metadata information.

138. Admin User: ETL+ Customer Portal can only be accessed from behind DataSelf’s firewall.

139. Admin User: Deployment of Advanced, Pro and Enterprise share the same foundation (ETL+ templates, Tableau tds, Tableau main and QA twb workbooks).

140. Admin User: Deployment of Pro and Enteprise replaces EntityID in SQL Statements with dbo (which is the default schema for such deployments).

141. Admin User: For Pro and Enterprise deployments, ETL+ will automatically create a SYSTEM_Date table in the data warehouse. Required for certain internal analytics processes.

142. Admin User: For Pro and Enterprise deployments, ETL+ will automatically create the zDateUntreated table for date manipulations.

143. Admin User: For Pro and Enterprise deployment, ETL+ will automatically create auxiliary tables for source database specific data manipulations (such as document types and GL Financials data sets).

144. Admin User: ETL+ License field controls the cloud template deployment script.

145. Admin User: Client Info page allows the configuration of the ETL+ License type: DataSelf ETL+, VAR, DIY, Advanced, Pro, Enterprise.

146. Admin User: On the Client Info page, checking the Tableau Deploy checkbox + License = VAR + clicking Save will deploy a DataSelf Advanced Analytics NFR system on https://bi1.dataself.com.

147. Admin User: On the Client Info page, checking the Tableau Deploy checkbox + License = Advanced + clicking Save will deploy a DataSelf Advanced Analytics system on https://dataselfbi.com.

148. Admin User: On the Client Info page, checking the Tableau Deploy checkbox + License = Pro + clicking Save will deploy a DataSelf Pro Analytics system on https://dataselfbi.com.

149. Admin User: On the Client Info page, checking the Tableau Deploy checkbox + License = Enterprise + clicking Save will deploy a DataSelf Enterprise Analytics system on https://dataselfbi.com.

150. Admin User: Analytics Settings is now part of the ETL+ metadata. It contains information such as Analytics type (Tableau, soon Power BI as well) and other settings to automate maintenance tasks such as the deployment of their templates and extract refresh.

151. Admin User: New Stored Procedure to automate the maintenance of analytics tasks such as the deployment of Analytics templates and extract refresh.

152. Admin User: The Analytics Server URL from Analytics Settings shows at the bottom of the Tableau Extract Refresh page.

153. Upgraded Microsoft Code Signing with RSA4096 encryption.

154. Compatible with Windows 11.

Fixed Issues

1. Product registration email missed the token information. Fixed.

2. Adding a source table to the data warehouse was disabling the center panel’s icons for that table. The user had to click another table and go back to the new table to enable the icons. Fixed.

3. WTS was automatically pre-checking the Repeat task every feature. Fixed.

4. WTS was not setting Run whether user is logged on or not. This is actually a security issue, be sure to enter a local Windows admin username and password in the ETL+ WTS page to create the WTS task with the setting above.

5. Upon installing ETL+ on a new computer, if it doesn’t start, right-click ETL+ → Run as administrator. This is a required security feature to kick off the IP whitelisting process.

6. Mapping of Acumatica OData columns with spaces in their names in GIs will come without spaces in OData. Ex.: “Geo Customer” in GI comes as “GeoCustomer” in OData.

Known Issues

1. Tables with delta refresh will present an error: ‘Invalid column name ‘0_UnionID', but ETL+ still processes the delta refresh correctly. To prevent the error message from happening, manually create a new column 0_UnionID varchar(100) on tables with that error message.

2. Clicking the Log icon after certain tasks (such as running a Refresh Batch) might present a "Unhandled exception" error. Click Quit and re-open. It mostly only happens once.

3. Upon adding a new table from a NetSuite ODBC, the SQL syntax will have an extra ] at the end of the table name. To fix it: remove ] and save.

4. Some Acumatica OData v4 tables might show error “The contents of table TableName are not available for this user“ and fail to import. This is an Acumatica security issue and we’ve requested how to fix it. While we wait for the Acumatica fix, use OData v3 to import such tables.

5. When manually loading a table, it will retry 12 times in case of failure.

6. Adding tables from Sage Intacct might not work.

7. Deleting all Refresh Batches requires going to ETL main page and back to get a new one created.

8. Opening an entity from an older metadata version with the new one using a regular user (non Admin) will not update the metadata version in the cloud. Work around: open it with an Admin user.

9. Refresh Batch: If a table fails to refresh, the ETL+ might unexpectedly load several tables multiple times.

10. Sometimes ETL+ will mistakenly flag that the metadata has been changed and you need to exit and re-enter the entity. Just exit and re-enter.

11. Changing the data warehouse of an entity won’t show on the bottom left corner until the entity is closed and re-opened.

12. Regular users: Mousing over Sources doesn’t show the pop up with details.

13. Regular users: When login to ETL+, in the first attempt, ETL+ will only open the first entity even when one uses another entity on the dropdown list.

14. The icon color and type for the column formats on the center panel isn’t 100% consistent yet.

15. Opening an entity with tables mapped to the zData Warehouse with an older ETL+ version will push such tables to other source databased on the center panel. DataSelf can fix that manually.

16. Importing large XML content into varchar(max) might run into error: server stack limit has been reached.

17. Sometimes ETL+ is closing unexpectedly after a data load.

18. Sometimes ETL+ runs into an “Unhandled exception” error. Close and re-open ETL+.

19. Locking the ETL+ editing when someone else is editing it doesn’t work in some situations.

20. ETL+ still tries to send email even when set to Cloud notification, or No Notification.

21. ETL+ delta extraction from NetSuite ODBC might not work when using dates for filtering.

22. When opening an entity created in prior ETL+ versions, the data warehouse panel will show the source header blank. To fix this, open the entity in the prior ETL+ version, make any modification and save it. Now open the entity in the latest ETL+ and the source header appears.

Download file

64 bits: https://dataself.com/etl_plus/DataSelf ETL Plus 64bits - 2021.09.1301.exe
32 bits: https://dataself.com/etl_plus/DataSelf ETL Plus 32bits - 2021.09.1301.exe

Released: 09/13/2021

New Features

1. ETL UI: Added '?' help icon on main pages.

2. ETL UI: User Guide available at DataSelf ETL+ User Guide .

3. ETL UI: Many buttons have been replaced by icons.

4. ETL UI: Hover over icons to view their functions.

5. ETL UI: Hover over data structures such as data sources and tables to view metadata.

6. ETL UI: Contextual icons - they are grayed out when they don’t apply to selected objects (ex.: selecting a table on the Source panel disables Delete Data Source).

7. ETL page: Right clicking on the left two panels pops up the icon functions. The pop ups show the icons and their function names.

8. ETL page: Clean up of the ETL+ main page (some functions have been moved to other pages).

9. ETL page: ETL SQL Statement’s title shows the selected TableName (or Source.TableName when multiple sources are mapped).

10. ETL page: ‘Log File’ icon to open local log file in Notepad.

11. ETL page: ‘Refresh Schedule’ button changed/renamed to ‘Refresh Batch’ icon.

12. ETL page: ‘Refresh Batch Now’ function moved to the 'Refresh Batch' page.

13. Multiple data sources: When opening ETL+, the data source(s) will not list its table names by default. This enables a fast opening of ETL+ since it doesn’t have to wait to list source system table names dynamically. Click a data source and the ‘Refresh’ icon to list its table names - this process can take several seconds if the source system is slow to render its table name list.

14. Multiple data sources: Imports data from multiple data sources within the same ETL+ entity.

15. Multiple data sources: Source panel shows all data sources created.

16. Multiple data sources: Data sources are represented by a cylinder icon, and their table name list can be expanded or collapsed with the +/- icon by the data source icon. Tables are represented by a square window icon.

17. Multiple data sources: Select a table from a data source and click the ‘Add Table’ icon/function to map it to the ETL process.

18. Multiple data sources: Click the ‘Add Data Source’ icon/function, select a data source type (CSV, MS Excel, MS SQL Server, OData, ODBC), configure the connection information, click Connect (or Save).

19. Multiple data sources: Configuration of data sources now includes an editable Alias (DSN Alias).

20. Multiple data sources: Added ‘Save’ button (saves and closes without trying to connect to source).

21. Multiple data sources: When creating a new DSN, the Alias is auto populated as following:

    1. MS SQL: Database name.

    2. ODBC: ODBC name.

    3. Excel: File name.

    4. CSV: Windows Folder name.

    5. OData: Last word of the OData URL.

22. Multiple data sources: Configuration of data sources now includes a ‘Dw Schema’ checkbox. When checked, tables from this data source will be stored in the data warehouse using a SQL schema = DSN Alias. When unchecked, tables will be stored in the default data warehouse SQL schema.

23. Multiple data sources: In the DSN configuration page, use the ‘Change Data Source’ icon to replace it to another type (ex.: from OData to MS SQL).

24. Multiple data sources: The DSN pages have a Save button (helpful when doing remote DSN maintenance without access to the DSN).

25. Multiple data sources: Select a data source and click the ‘Delete Data Source’ icon/function to delete.

26. Multiple data sources: Select a data source and click the ‘Refresh’ icon/function to reconnect to the data source and list its tables.

27. Multiple data sources: Select a data source and click the ‘Properties’ icon/function to configure its settings.

28. Multiple data sources: Mousing over the left panel’s data source icons shows its metadata.

29. Multiple data sources: Mousing over the left panel’s tables from a data source shows its metadata.

30. Multiple data sources: The center panel organizes the data warehouse tables by data source alias.

31. Multiple data sources: The center panel data source alias icon is a square with 3 objects in it. You can expand/collapse the list of data warehouse tables with the +/- icon by this icon.

32. Multiple data sources: Mousing over the center panel’s data source alias icons shows its metadata.

33. Multiple data sources: Mousing over the center panel’s tables shows its metadata.

34. Multiple data sources: When a data source has its table names listed on the left panel, and the user clicks that same table on the center panel, the cursor will select the table on the source system list.

35. Multiple data sources: Clicking on a center panel’s data source alias icon and the ‘Load Now’ icon will load all of its mapped tables to the data warehouse.

36. Multiple data sources: Ability to import data from the data warehouse for further data prep.

37. Multiple data sources: When adding a table from a source, the cursor selects it on the center panel.

38. Load UI: The load UI page is now integrated in the ETL+ application (not a detached layer).

39. Load UI: Cancel button stops the load process when the table currently loading ends.

40. Load UI: Cancel button stops the load process between OData extraction blocks and tables.

41. Load UI: Shows how many rows have been loaded from the table being extracted now.

42. Load UI: Shows how many rows per table have been loaded: (X rows <action>).

43. Load UI: Shows the type of load: the <action> variable in (X rows <action>):

    1. Replace All: loaded

    2. Replace with Control: replaced

    3. Update: updated

    4. Append: appended

44. Load UI: Refreshing a long list of steps scrolls up showing the latest steps at the bottom of the list.

45. Load UI: Shows at the bottom of the table list when waiting (sleep mode) versus extracting data.

46. Load UI: Shows when refreshing each Tableau Extract.

47. Load UI: New background and font colors provide better contrast.

48. Load UI: Shows Source.TableName when multiple sources are available, otherwise only TableName.

49. Load UI: Creates MS SQL indexes for the filter column used in tables with Replace.

50. Load UI: Creates MS SQL indexes for the filter and PK columns used in tables with Upsert.

51. Cloud Watchdog: Monitors ETL+ scheduled refresh cloud logging and sends email notifications.

52. Cloud Watchdog: Notifies “Missed execution?” when a scheduled Refresh Batch hasn’t started 120 seconds after the scheduled time.

53. Cloud Watchdog: Notifies “Taking too long?” when a scheduled Refresh Batch takes longer than (10 mins + 1.5 x Last Refresh Time) to complete. This might be an indication that the refresh will not complete.

54. Cloud Watchdog: Always notifies when ETL+ “Notification Email” is set to “Cloud - Always Notify”.

55. Cloud Watchdog: Notifies on error when ETL+ “Notification Email” is set to “Cloud - Notify on Error”.

56. Settings: Creation of the Settings page and added its icon on ETL+ main page.

57. Settings: ETL version posted.

58. Settings: Page provides access to ‘Client Info’, notification about ‘3 secs between error retries', ‘Debug’ drop down, ‘0_EntityID Column’, ‘Data Warehouse’ configuration, ‘About’, 'Notification Email’.

59. Settings: ‘Refresh Batch Notification Email’ includes dropdown for:

    1. Always Notify: ETL+ always sends notification email.

    2. Notify on Error: ETL+ only sends email on error.

    3. Cloud - Always Notify: Cloud Watchdog always sends notification email.

    4. Cloud - Notify on Error: Cloud Watchdog only sends email on error.

    5. No Notification: No notifications from ETL+ nor Watch Watch Dog.

60. Settings: “Create Now” button added to “New Data Warehouse” page.

61. Refresh Batch: Renaming ‘Refresh Schedule’ to ‘Refresh Batch’.

62. Refresh Batch: Re-layout of the page.

63. Refresh Batch: Bottom left section shows WTS summary configuration.

64. Refresh Batch: ‘WTS Settings’ button provides access to WTS settings and Apply button.

65. Refresh Batch: To run a Batch either click the ‘Refresh Now' icon on a batch name, or select the Batch and click 'Refresh Now’ icon on the panel header.

66. Refresh Batch: Click the 'Delete' icon on a batch name to delete it.

67. Refresh Batch: ‘Add’ and ‘Create Shortcut’ became icons on the batch name list.

68. Refresh Batch: List of Refresh Batch Steps shows twice as much vertical real estate.

69. Refresh Batch: Clearer description of ‘All Data Warehouse Tables (include new ones)’ checkbox.

70. Refresh Batch: Click a refresh step and ‘Move Up’ & ‘Move Down’ icons to re-order the refresh steps.

71. Refresh Batch: New step types for ‘Tableau Extract Refresh', ‘Cloud Script’, 'Command Line’.

72. Refresh Batch: Click ‘+' icon ('Add Script’), and select a script type to create.

73. Refresh Batch: Select ‘Tableau Extract Refresh’ to configure an extract to be refreshed. These extracts must be hosted at https://dataselfbi.com and pull data from DataSelf's cloud data warehouses.

74. Refresh Batch: ‘Tableau Extract Refresh’ script requires the extract name.

75. Refresh Batch: ‘Tableau Extract Refresh’ default values for other fields:

    1. Project = <EntityID>

    2. Parent Project = NULL

    3. Tableau Site = <EntityID>

76. Refresh Batch: Select ‘Cloud Script’ to provide its name. Consult DataSelf to discuss your cloud script options (such as running other cloud ETL+ extractions, DataSelf Automatic Distribution, and Python custom applications).

77. Refresh Batch: Select ‘Command Line’ to provide its name and parameters. This will run the configured command line locally. Example of use cases: running locally Tableau extract refreshes, or on-premises DataSelf Automatic Distribution.

78. Refresh Batch: Select a script step, and click icons ‘Delete Script’, ‘Configure Script', or 'Run Script’ as needed.

79. Refresh Batch: Removed ‘Refresh Type' since this can now be done directly in 'Refresh Batch Steps’.

80. Refresh Batch: Moved ‘Debug’ drop down to Settings page.

81. Refresh Batch: Removed ‘Enabled’ from batch list (not functional yet).

82. Refresh Batch: Other UI buttons/icons become disabled when editing scripts.

83. Refresh Batch: Icons on the Steps panel differentiate objects between tables and scripts.

84. Refresh Batch: Mousing over icons provide their functions and name and/or metadata.

85. Refresh Batch: Mousing over a table pops up its metadata (DSN and refresh settings).

86. Refresh Batch: Removed ‘On Demand Data Refresh’ UI with delay start - no longer needed.

87. Refresh Batch: Shows ETL+ version on the top right.

88. Refresh Batch: With multiple data sources, Batch Steps show DataSourceName.TableName.

89. Refresh Batch: Added ‘Tableau Template Deployment’ icon - available to admin users only.

90. Refresh Batch: ‘Log File’ icon to open local log file in Notepad.

91. Engine: Notifies when ETL+ doesn’t find the required .NET installed.

92. Engine: When an extraction errors out, retry 3 times (3 seconds apart).

93. Engine: When manually running extractions, connection errors prompt for Retry, Cancel or Skip.

94. OData Extraction: For OData tables without built-in indexes, ETL+ will only extract data in blocks when indexes have been configured in ETL+. Without indexes, such extractions will be continuous (not in blocks).

95. OData Extraction: Character fields are loaded into varchar(max) in the data warehouse and are limited to the first 250 characters (next release will give users control of target column format).

96. OData Extraction: When running an RWC or Update extraction from large datasets, ETL+ logs to Acumatica, loads in sets of up to 100 extraction blocks into a temp table, loads the temp table to the target table, logs off, and then repeats this process for the next set of extraction blocks. This helps Acumatica rendering speed, and provides intermediary starting points in case of issues in the middle of the extraction process.

97. OData Extraction: When processing delta refreshes, the page size will automatically expand when the delta filter cannot get over the next page of data. The expanded page size only applies to the table being loaded. Message: The delta extraction filter requires an Extraction Pagination expansion from <current> to <new> rows.

98. OData Extraction: Ability to insert texts that include SQL special characters (such as ' \* *\).

99. OData Extraction: Debug Log file includes OData URLs executed when Debug is set to Level 2.

100. Acumatica OData v3 delta refreshes: Acumatica does NOT accept GI formula fields (ex.: =[GLHistoryByPeriod.FinPeriodID]) as filter for RWC and Update.

101. Acumatica OData v4 delta refreshes: Acumatica requires that filters for RWC and Update must not have NULLs and ETL+ extraction blocks must be bigger than the max number of records for each block of records with the same filter value.

102. OData v4 Extraction: Changed extraction process for large tables when OData v4 refused to render data.

103. OData v4 Extraction: When working with RWC or Update and using date filters, ETL+ filtering adapts to the date format of the source Acumatica.

104. OData v4 Extraction: If a table doesn’t have a native OData index, ETL+ errors out and logs “Extraction time out. Table without native OData index. Please create one via Replace with Control or Update.”

105. ODBC Extraction: Sage Intacct ODBC support.

106. ODBC Extraction: Quickbooks Online ODBC support.

107. OBDC Extraction: MongoDB ODBC support (non-relational database).

108. OBDC Extraction: Using native MondoDB ' (single quote) or ANSI [] (brackets) around reserved words and nested collections.

109. ODBC Extraction: Support to views from source system (previously, only tables).

110. Logging: ETL+ logs description and time for new functions/scripts (Tableau Extract Refresh, etc).

111. Logging: When launching ETL+ via command line, add the 3rd parameter “1” and ETL+ will log locally “Start ETL+” even before loading the cloud metadata.

112. Logging: Posts Source.TableName when multiple sources are available, otherwise only TableName.

113. Logging: Posts the number of rows loaded.

114. Logging: Posts the type of load: the <action> variable in (X rows <action>):

     1. Replace All: loaded

     2. Replace with Control: replaced

     3. Update: updated

     4. Append: appended

115. Logging: For large OData tables with RWC or Update, logs when 100 extraction blocks are loaded.

116. Logging: Posts 32 or 64bit ETL+ version on log refresh headers.

117. Client Information: ETL version posted.

118. Client Information: The following fields were moved to more relevant pages: Data source, Settings, 0_EntityID Column, Notification Email.

119. Client Information: Creation of a Dropdown for DataSelf License type (DataSelf ETL+, VAR, DIY, Advanced, Pro, Enterprise). Only available for super admin.

120. Client Information: Addition of checkbox to deploy out-of-the-box Tableau Templates for Advanced Analytics.

121. User Logging Page: ETL version posted.

122. Error handling: Notifies and logs incorrect Column names used in 'Replace with Control' or 'Update'.

123. Error Handling: Improved message description when ETL+ can’t connect to its data warehouse.

124. Error Handling: Added message describing that MS SQL doesn’t accept insert in timestamp columns.

125. Error Handling: For certain extraction errors, retry the extraction 3 times.

126. New Deployments: Removed step to load data now (it caused confusion).

127. New Deployments: ‘Deploy Tableau Script’ runs on dw3 cloud server the following command line: “C:\DataSelf\<SourceSystem>\Deploy.Bat <EntityID> <DwServer> <DwDbName>”

128. Super admin: ‘Client Info’ and ‘Refresh Batch’ pages allow the deployment of out-of-the-box ‘Tableau Templates’ (site, tdsx, twb) for Advanced Analytics and QA templates for Pro and Enterprise.

129. Super admin: Entity creation populates EntityID in Refresh Batch Email Notification’s Subject.

130. Super admin: Entity creation populates user email in Refresh Batch Email Notification’s Email.

Resolved Issues

1. Counter of Load UI page fixed (it was always counting 1 extra item).

2. Handling of OData v4 having more than one table with the same name.

3. Creation of EntityID now requires a unique ID.

4. Errors out in case of an attempt to create a duplicated data warehouse Schema.TableName.

5. Tagging of the NextRefreshTime metadata field wasn’t populating correctly.

6. More than 4 index fields in Update was looping in some OData extractions.

7. OData XML extraction of non-standard characters fixed.

Known Issues

1. Importing large XML content into varchar(max) might run into error: server stack limit has been reached.

2. Deleting all Refresh Batches requires going to ETL main page and back to get a new one created.

3. Changing the data warehouse of an entity won’t show on the bottom left corner until the entity is closed and re-opened.

4. Sometimes ETL+ is closing unexpectedly after a data load.

5. Sometimes ETL+ runs into an “Unhandled exception” error. Close and re-open ETL+.

6. Locking the ETL+ editing when someone else is editing it doesn’t work in some situations.

7. ETL+ still tries to send email even when set to Cloud notification, or No Notification.

8. ETL+ delta extraction from NetSuite ODBC might not work when using dates for filtering.

9. When opening an entity created in prior ETL+ versions, the data warehouse panel will show the source header blank. To fix this, open the entity in the prior ETL+ version, make any modification and save it. Now open the entity in the latest ETL+ and the source header appears.

10. Importing tables from the data warehouse works correctly but it might incorrectly log an error message.

Release date: 03/31/2021

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 64bits - 2021.03.3103.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 32bits - 2021.03.3103.exe

New Features

1. Integrated IP Whitelisting. If ETL+ can’t connect to DataSelf cloud servers, it whitelists the new IP once the user enters the correct product registration information.

2. Friendlier installation. ETL+ is now an executable-only application. One just needs to download and run it. No installer, no changes to the Windows environment, no need to Run as Admin (unless in some systems when creating Windows Task Schedule tasks).

3. ETL+ automatically sorts columns alphabetically when adding new source system tables.

4. Enable SELECT * from MS SQL and OData data sources. Preview only works with MS SQL sources.

5. OData extraction ignores duplicated column names (avoiding OData → SQL extraction errors).

6. Refresh Schedule page: “All Tables” checkbox to include new tables added to the ETL process.

7. Refresh Schedule page: “Tableau Subset” checkbox runs a dedicated Tableau refresh batch for the selected Refresh Schedule. Example: the default Refresh Schedule refreshes all Tableau extracts overnight, and a 2nd one refreshes a subset of Tableau extracts hourly. The creation of Tableau refresh batches continues to be a manual process done by DataSelf.

8. Refresh Schedule page: “Create Shortcut” checkbox to create ETL+ auto refresh shortcuts.

9. Refresh Schedule page set to 800x600 resolution.

10. Refresh Schedule page lists Schedule ID (3rd parameter when running ETL+ via command line).

11. Windows Task Scheduler: “Repeat tasks every” X minutes (ex.: run every 5 mins starting at 8am).

12. Windows Task Scheduler: “For a duration of” X hours (ex.: run every 5 mins starting at 8am for 12 hours). To configure indefinite tasks, set duration to 24 hours.

13. Windows Task Scheduler: Enable configuration of “Recurs every” X days (ex.: once a week).

14. Windows Task Scheduler: Schedules tasks into the future (to prevent users from mistakenly configuring tasks in the past).

15. Windows Task Scheduler: UI re-layout for better clarity and more real estate for configurations.

16. Speed: “Update” delta refresh creates data warehouse indexes for the filter and index columns.

17. Speed: “Replace with Control” delta refresh creates a data warehouse index for the filter column.

18. SQL Integrity: “Update” delta refresh turns varchar(max) filter and index columns into char(50).

19. SQL Integrity: “Replace with Control” delta refresh turns a varchar(max) filter column into char(50).

20. When running ETL+ for the 1st time, the DataSelf cloud servers connection only happens after the user has accepted the EULA.

21. If ETL+ can’t download the EULA: it’ll show the EULA page blank (probably an internet connection, firewall or antivirus issue).

22. When ETL+ can’t connect to DataSelf cloud servers: shows error about IP whitelisting and/or firewall and antivirus adjustments. And then tries to whitelist IP (see feature 1 above).

23. Super admin: Settings page to select, modify or new data warehouse and its SQL schema. Blank schema = EntityID. Enter the new name to change the schema. Ex.: dbo.

24. Super admin: Data warehouse configuration page has a Multi-Tenant checkbox. Check it for Advanced Analytics data warehouses, and uncheck it for Enterprise Analytics.

25. Super admin: Client Info page: “0_EntityID Column” checkbox to create this column.

26. Super admin: ETL+ logging in the cloud.

27. Super admin: ETL+ cloud auto refresh log includes EntityID, TableName, Begin Date/Time, End Date/Time, Rows Before, Rows Deleted, Rows Updated, Rows Kept, Rows Inserted, Rows After, Refresh Criteria, Result Type, ETL+ User, Refresh Schedule ID, Refresh Schedule Run ID, Windows Computer Name, Windows Username.

28. Super admin: “Entity” box converts all characters to lower case.

29. Super admin: Creation of new OData entities sets the OData extraction blocks to 1000 records and 100ms pause (instead of 0 recs and 0ms pause which causes issues).

30. Super admin: Wider Entity ID column on the Entities page.

Resolved Issues

1. Super admin: Issues when creating new entities.

2. The 32-bit version wasn’t saving ODBC credentials.

Known Issues

1. When configuring Acumatica OData delta refreshes, in the case that OData doesn’t render the indexes of a table (Acumatica bug?), you might get an “invalid type for an index” or “invalid for use as a key column” error message. To fix it, keep the delta configuration as is, change the “Load Type” to “Replace All”, do a table refresh, and switch back to delta refresh.

2. Failing to entering correct credentials 5 times in a row closes ETL+ but allows immediate 5 new re-attempts. The correct process is to force a 5 minute wait after 5 failed attempts.

Release date: 02/28/2021

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 64bits - 2021.02.2803.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 32bits - 2021.02.2803.exe

New Features

1. Move Windows User variables to the Windows ProgramData area.

2. Simplifies running ETL+ and its auto refresh processes regardless of Windows users.

3. Collaboration: When a user is editing an ETL+ entity, other users can only log, view or copy its settings. To release editing to other users, Save the entity or close ETL+.

   Other users trying to edit it will receive the message: “username is currently editing this entity. You can only view it. To re-load the modified entity, exit and re-enter ETL.”

4. Collaboration: When a user tries to edit ETL+ but the metadata has changed by another user, ETL+ gives the message: ETL+ metadata has changed. In order to edit your ETL+, please exit and re-enter to re-load your metadata.

5. Allowing users to open more than one ETL+ at the same time.

6. When running Refresh Schedules in parallel, log each of them on their own block (easier to read log file).

7. Table Load Types in Refresh Schedules: Refresh Schedules can run the default table load type (Refresh All, Replace with Control, Update, Append), or force a "Replace All". This helpful when using delta refreshes on a frequent schedule, and then forcing a replace all on a less frequent schedule to capture old records that have been deleted or modified.

8. ETL+ lets the same Refresh Schedule run in parallel, but might issue the error message: [DS ERROR : 0902] ETL+ cloud log conflict. Two or more ETL+ are running the same Refresh Schedule at the same time. The data refresh might be ok.

9. ETL+ allows the refreshing of the same data warehouse table in parallel. Users need to understand the implications of using this feature since it can lead to data consistency issues. Ex.: it’s Ok to append different data sets in parallel, but a full refresh in parallel is likely not desirable.

10. When editing ETL+, add an * to the Save button to indicate changes haven’t been saved yet. Clicking Save removes the * flag.

11. Support to OData v4. First release. Test it before using it for production.

12. Resized ETL+ UI to fit small resolutions down to 800x600.

13. ETL+ UI can be switched between default size, full screen mode and minimized.

14. New redesigned ETL+ metadata for better auto documentation, faster performance and reporting.

15. Upgrade from prior ETL+ versions requires the re-creation of entities, users and statements.

16. When refreshing tables on ETL+ UI, the refresh progress indicator shows the Refresh Schedule name at the top, the table counter and % progress inside of the circle, and the table being imported at the bottom.

17. Show ETL version and 32/64bits on the main page.

18. Ability to define MS SQL Server data warehouse’s database schema. The default schema = ETL+ EntityID. ETL+ admin users can set the schema to any other value (such as dbo). Multi-tenant data warehouses' ID=1 will always have schema = ETL+ EntityID.

19. RWC and Update will do the first refresh in full refresh mode.

20. Removed dozens of locally saved variables that are no longer used (they were used before metadata moved to the cloud).

21. "Leave without saving?" when a user tries to close ETL+ without saving changes.

22. For users with access to multiple entities, ETL+ remembers the last entity opened.

23. Show ETL version and 32/64bits on the top right of main pages.

24. Strong encryption of ProgramData info.

25. Log changes to the source system’s connection string.

26. Left and right trim to all user input fields (avoids spaces in usernames and passwords).

27. ETL+ captures who saved an ETL SQL Statement version. Currently, only available in the ETL+ metadata.

28. More informative error message when ETL+ cannot connect to its metadata or data warehouse.

29. If two or more ETL try to refresh the same table at the same time, if the table has RWC or Update, it might give the error message: [DS ERROR : 0910] Two or more ETL+ tried to refresh the XYZ data warehouse table at the same time. This can cause data consistency issues. Please run a full refresh for this table again.

30. ETL+ version number set to YYYY.MM.DDNN (NN is sequential starting on 01).

31. ETL+ EULA and About pages inform the latest ETL+ version and URL for download (in case users are installing an older version accidentally).

Resolved Issues

1. New features number 1 and 2 resolve several auto refresh friction points.

2. Fixed data auto refresh not running from some servers via command line.

3. Fixed a testing feature that prevented ETL+ from connecting to the cloud from some servers.

4. Fixed the auto email of new product registrations.

5. Fixed ETL+ Excel and CSV data sources to include headers on row 1.

6. Fixed Update and Replace with Control refresh from ODBC sources.

7. Fixed error message showing data warehouse structure names.

Release date: 01/13/2021

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 64bits - 2021.1.001.1301.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 32bits - 2021.1.001.1301.exe

New Features

1. Multiple refresh schedules.

2. Add Debug Mode dropdown to Batch page. Levels: Debug Off, Debug L1, Debug L2. The debug applies to each DSBI site, and all of its refreshes and users.

3. Allow ETL+ to run multiple refreshes of the same Entity in parallel.

4. Make UI 99 to be sorted by EntityID (not Entity Name).

5. Add the dw name to the bottom left corner of the ETL+ UI.

6. Improve UI titles and descriptions for clarity and self documentation.

Resolved Issues

1. Test "Communications" and "User" tables with all fields from Sage CRM source.

2. Scheduler isn't remembering the saved schedule settings.

3. ETL+ was hanging in WTS. Issue with the feature that puts notifications on the system tray.

4. When trying to upload 2 tables now, on the dw table list: clicking on a table, pressing and holding Crtl and clicking on a 2nd table crashes the ETL.

5. Log when refresh runs via shortcut shows [DS ERROR: 9997] - Thread was being aborted.

Release date: 12/14/2020

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf_ETL_Plus_64bits_2020_4_012_1402.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf_ETL_Plus_32bits_2020_4_012_1402.exe

New Features

1. Create button on the Source Tables column for "Include".

2. When creating the ETL+ password, re-label "Password" to "New Password".

3. Selecting DBs in the MS SQL credentials page: Refresh Button to reload the list of DBs.

4. Delete Data Warehouse Table now has a "Confirm Deletion?" prompt.

5. Password cannot have # as the only special character. Message: Password cannot have # as the only special character.

6. For CSV Data Sources: "The first row has a Column names" should be checked and grayed out and non-editable.

7. For CSV Data Sources: "The first row has a Column names" Delimiter should have comma ',' and grayedout and non-editable.

8. Replace with Control: Order By field box removed. It automatically creates the index based on the Control field.

9. Debug Mode creates a log file: Log entityID_debug.txt.

10. Support for NetSuite ODBC.

11. Download CSV and Excel files with URLs with strings after the file name.

12. Ability to use subqueries in SQL Statements.

13. 0_EntityID to all tables.

14. Replace with Control delta refresh.

15. Update delta refresh.

16. Download CSV and Excel files from URLs into DS_SourceFiles folder where the ETL+ is, and then extract data to data warehouse.

Resolved Issues

1. Refresh and Upload buttons fail with THROW Error.

2. The ETL is not accepting edits in Client Information and Refresh Schedule. DS ERROR 04000 - Transaction Failed.

3. When saving a data warehouse table, ETL shows: "Do you want to cancel all changes made in this section?".

4. In the Microsoft Excel data source, "the first row has a Column names" should be checked.

5. It takes too long to enable other functions after saving a SQL Statement.

6. Fixed OData extract: DSID: nfr_wac_acumatica1, Dw Table: 'TST jn - Prod Order Detail - In Process Released', Upload fails.

7. It wasn’t running the Tbl refresh at the end of the auto refresh. 

8. The data refresh email should only be sent when running the full refresh. 

9. Mapping of Odata fields failed.

10. When adding New Table, the cursor will allow to change this new table name. However, it can only save the new name if all other table names listed are above "New Target Table".

11. The ODBC Data Source box had two Connect buttons.

12. Using Preview in OData on some tables causing closing, freezing or error message.

Release date: 10/27/2020, First Release

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf_ETL_Plus_64bits_2020_4_010_2701.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf_ETL_Plus_32bits_2020_4_010_2701.exe

Released Features

1. Minimum system requirements: 1 CPU, 2GB of RAM, 5GB of free diskspace, Windows 10, Windows Server 2012 R2 or newer, 32 and 64bit.

2. Installation process: EULA, user credentials and token, IP whitelisting, connection to data source, scheduling the data auto refresh via Windows Task Scheduler (WTS), refreshing data now, and optionally maintaining ETL+ metadata.

3. Microsoft certified application.

4. Creation of Windows Task Scheduler (WTS) task to run the data auto refresh, including schedule and user to run WTS service.

5. Cloud-based ETL+ metadata allows remote maintenance.

6. Cloud uptime of 99.9%.

7. Data source built-in connectors:

   1. ODBC (32 and 64 bits)

   2. MS SQL Server

   3. Odata version 3

   4. Excel (will prompt to install MS Access driver if required)

   5. CSV (will prompt to install MS Access driver if required)

8. Refresh of tables and subsequent Tableau data sources via encoded batch files.

9. Auto data refresh email notification.

10. Data load type: Replace all.

11. Create evolution bar for the data extraction when running via ETL+ UI.

12. Add refresh evolution process in system tray when running as a service.

13. Stores data and metadata in SSL secured multi-tenant AWS MS SQL Server data warehouse.

14. Upload data to AWS cloud servers via SSL secured IP-to-IP connection.

15. Locally encrypt user credentials to ETL+ and on-promises systems.

16. Credentials using user token, and requiring user to create an user with their email and complex password (minimum of 8 characters with lower/upper case, numbers and special characters).

17. If a users fails to enter correct credentials 5 times in a row, block access for 5 minutes.

18. ETL+ UI panes: source tables, data warehouse tables, SQL Statements.

19. About page to show EULA and product release.

20. Data warehouse table functions: new, duplicate, rename, delete, preview, upload now.

21. Refresh Now with options to Refresh Full, DW only, Tableau only.

22. Refresh Now with counter of 9 seconds (configurable), and buttons for Cancel, Pause, Run Now.

23. ETL SQL Statement with Edit Mode and Confirm button.

24. Locally saved ETL+ log file with refresh steps.

25. Email refresh process log to maintenance users.

26. Creates shortcut on Desktop to run ETL+ auto refresh processes.