DataSelf ETL+ Release History

Download file

64 bits: https://dataself.com/etl_plus/DataSelf ETL Plus - Beta 2021.07.2203.exe

New Features

1. UI: Many buttons have been replace by icons.

2. UI: Mousing over icons shows tooltips with the icon function name and/or metadata.

3. UI: Contextual icons - they are grayed out when they don’t apply to selected objects (ex.: selecting a table on the Source panel disables Delete Data Source).

4. UI: Right clicking on the left two panels pops up the icon functions. The pop ups show the icons and their function names.

5. UI: Clean up of the ETL+ main page (some functions have been moved to other pages).

6. Multiple data sources: When opening ETL+, the data source(s) will not list its table names by default. This enables a fast opening of ETL+ since it doesn’t have to wait to list source system table names dynamically. Click a data source and the ‘Refresh’ icon to list its table names - this process can take several seconds if the source system is slow to render its table name list.

7. Multiple data sources: Imports data from multiple data sources within the same ETL+ entity.

8. Multiple data sources: Source panel shows all data sources created.

9. Multiple data sources: Data sources are represented by a cylinder icon, and their table name list can be expanded or collapsed with the +/- icon by the data source icon. Tables are represented by a square window icon.

10. Multiple data sources: Select a table from a data source and click the ‘Add Table’ icon/function to map it to the ETL process.

11. Multiple data sources: Click the ‘Add Data Source’ icon/function, select a data source type (CSV, MS Excel, MS SQL Server, OData, ODBC), configure the connection information, click Connect (or Save if available).

12. Multiple data sources: Configuration of data sources now includes an Alias (DSN Alias).

13. Multiple data sources: Configuration of data sources now includes a ‘Dw Schema’ checkbox. When checked, tables from this data source will be stored in the data warehouse using a SQL schema = DSN Alias. When unchecked, tables will be stored in the default data warehouse SQL schema.

14. Multiple data sources: In the DSN configuration page, use the ‘Change Data Source’ icon to replace it to another type (ex.: from OData to MS SQL).

15. Multiple data sources: The DSN pages have a Save button (helpful when doing remote DSN maintenance without access to the DSN).

16. Multiple data sources: Select a data source and click the ‘Delete Data Source’ icon/function to delete.

17. Multiple data sources: Select a data source and click the ‘Refresh’ icon/function to reconnect to the data source and list its tables.

18. Multiple data sources: Select a data source and click the ‘Properties’ icon/function to configure its settings.

19. Multiple data sources: Mousing over the left panel’s data source icons shows its metadata.

20. Multiple data sources: Mousing over the left panel’s tables from a data source shows its metadata.

21. Multiple data sources: The center panel organizes the data warehouse tables by data source alias.

22. Multiple data sources: The center panel data source alias icon is a square with 3 objects in it. You can expand/collapse the list of data warehouse tables with the +/- icon by this icon.

23. Multiple data sources: Mousing over the center panel’s data source alias icons shows its metadata.

24. Multiple data sources: Mousing over the center panel’s tables shows its metadata.

25. Multiple data sources: When a data source has its table names listed on the left panel, and the user clicks that same table on the center panel, the cursor will select the table on the source system list.

26. Multiple data sources: Clicking on a center panel’s data source alias icon and the ‘Load Now’ icon will load all of its mapped tables to the data warehouse.

27. Multiple data sources: ‘Refresh Schedule’ button changed/renamed to ‘Refresh Batch’ icon.

28. Multiple data sources: ‘Refresh Batch Now’ has been moved to the 'Refresh Batch' page.

29. Multiple data sources: Ability to import data from the data warehouse for further data prep.

30. Load UI: The load UI page is now integrated in the ETL+ application (not a detached layer).

31. Load UI: The load UI page now has the Cancel button. In a multi-table load process, clicking Cancel stops the load process when the table currently loading ends.

32. Load UI: For OData, it shows how many records have been extracted.

33. Cloud Watchdog: Monitors ETL+ scheduled refresh logging in the cloud and notifies administrators:

    1. On error

    2. When a scheduled refresh doesn’t start

    3. When a scheduled refresh takes longer than expected

34. Cloud Watchdog: Emails notifications when ETL+ Notification Email = Always Cloud (for cases when ETL+ has no access to its SMTP server - deployments with tight firewall rules).

35. Settings: Creation of the Settings page and added its icon on ETL+ main page.

36. Settings: Page provides access to ‘Client Info’, notification about ‘3 secs between error retries', ‘Debug’ drop down, ‘0_EntityID Column’, ‘Data Warehouse’ configuration, ‘About’, 'Notification Email’.

37. Settings: Notification Email includes dropdown for:

    1. Default: ETL+ always sends notification email)

    2. On Error: ETL+ only sends email on error

    3. Always Cloud: Cloud Watchdog always sends notification

38. Refresh Batch: Renaming Refresh Schedule to Refresh Batch, re-layout and clean up.

39. Refresh Batch: Bottom left section shows WTS summary configuration.

40. Refresh Batch: ‘WTS Settings’ button provides means to configure and apply WTS settings.

41. Refresh Batch: Click a 'Refresh Now' icon on the batch list to run it.

42. Refresh Batch: Add, Delete, Create Shortcut, and Enable became icons on the batch list.

43. Refresh Batch: List of Refresh Batch Steps shows twice as much vertical real estate (lists twice as many objects).

44. Refresh Batch: Refresh Batch Steps includes 3 new functions/scripts: ‘Tableau Extract Refresh', ‘Cloud Script’, 'Command Line’.

45. Refresh Batch: Clearer description of ‘All Data Warehouse Tables (include new ones)’ checkbox.

46. Refresh Batch: Click a step and ‘Move Up’ and/or ‘Move Down’ icons to re-order refresh sequence.

47. Refresh Batch: Click ‘+' icon ('Add Script’), and select a script type to create.

48. Refresh Batch: Select ‘Tableau Extract Refresh’ to configure an extract to be refreshed. These extracts must be hosted at https://dataselfbi.com and pull data from DataSelf's cloud data warehouses.

49. Refresh Batch: ‘Tableau Extract Refresh’ script needs the extract name. If the extract is not in the default Project (named ETL+'s Entity ID), enter the ‘Project’ name. If the extract is in a sub project, enter the 'Parent Project' name.

50. Refresh Batch: Select ‘Cloud Script’ to provide its name. Consult DataSelf to discuss your cloud script options (such as running other cloud ETL+ extractions, DataSelf Automatic Distribution, and Python custom applications).

51. Refresh Batch: Select ‘Command Line’ to provide its name and parameters. This will run the configured command line locally. Example of use cases: running locally Tableau extract refreshes, or on-premises DataSelf Automatic Distribution.

52. Refresh Batch: Select a script step, and click icons ‘Delete Script’, ‘Configure Script', or 'Run Script’ as needed.

53. Refresh Batch: Other UI buttons/icons become disabled when editing scripts.

54. Refresh Batch: Icons on the Steps panel differentiate objects between tables and scripts.

55. Refresh Batch: Mousing over icons provide its function, name and/or metadata.

56. Refresh Batch: ‘Run Refresh Batch’ has a simplified UI with a counter starting at 20 secs.

57. Logging: ETL+ logs description and time for new functions/scripts (Tableau Extract Refresh, etc).

58. Logging: When launching ETL+, the very first log is “ETL+ launched”.

59. Engine: Notifies when ETL+ doesn’t find the required .NET installed (.NET v7 or newer).

60. OData Extraction: When RWC or Update is configured, ETL+ will load in sets of up to 100 extraction blocks, execute the load of that set to the target table, and then run the next set of extraction blocks. This is helpful when extracting data from large tables. It helps Acumatica rendering speed, and provides intermediary starting points in case of failures or extraction quitting in the middle of the process.

61. OBDC Extraction: Support to MongoDB ODBC. Use native MondoDB ' (single quote) or ANSI [] (brackets) around reserved words and nested collections.

62. Improved error message description when ETL+ can’t connect to its data warehouse.

63. Added error message describing that MS SQL doesn’t accept insert in timestamp columns.

64. Super admin can now trigger the deployment of the out-of-the-box Tableau Site and Extract templates.

Resolved Issues

1. Counter of Load UI page fixed (it was always counting 1 extra item).

2. Handling of OData v4 having more than one table with the same name.

3. Creation of EntityID now requires a unique ID.

Known Issues

1. Changing the data warehouse of an entity won’t show on the bottom left corner until the entity is closed and re-opened.

Release date: 03/31/2021

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 32 bits.exe

New Features

1. Integrated IP Whitelisting. If ETL+ can’t connect to DataSelf cloud servers, it whitelists the new IP once the user enters the correct product registration information.

2. Friendlier installation. ETL+ is now an executable-only application. One just needs to download and run it. No installer, no changes to the Windows environment, no need to Run as Admin (unless in some systems when creating Windows Task Schedule tasks).

3. ETL+ automatically sorts columns alphabetically when adding new source system tables.

4. Enable SELECT * from MS SQL and OData data sources. Preview only works with MS SQL sources.

5. OData extraction ignores duplicated column names (avoiding OData → SQL extraction errors).

6. Refresh Schedule page: “All Tables” checkbox to include new tables added to the ETL process.

7. Refresh Schedule page: “Tableau Subset” checkbox runs a dedicated Tableau refresh batch for the selected Refresh Schedule. Example: the default Refresh Schedule refreshes all Tableau extracts overnight, and a 2nd one refreshes a subset of Tableau extracts hourly. The creation of Tableau refresh batches continues to be a manual process done by DataSelf.

8. Refresh Schedule page: “Create Shortcut” checkbox to create ETL+ auto refresh shortcuts.

9. Refresh Schedule page set to 800x600 resolution.

10. Refresh Schedule page lists Schedule ID (3rd parameter when running ETL+ via command line).

11. Windows Task Scheduler: “Repeat tasks every” X minutes (ex.: run every 5 mins starting at 8am).

12. Windows Task Scheduler: “For a duration of” X hours (ex.: run every 5 mins starting at 8am for 12 hours). To configure indefinite tasks, set duration to 24 hours.

13. Windows Task Scheduler: Enable configuration of “Recurs every” X days (ex.: once a week).

14. Windows Task Scheduler: Schedules tasks into the future (to prevent users from mistakenly configuring tasks in the past).

15. Windows Task Scheduler: UI re-layout for better clarity and more real estate for configurations.

16. Speed: “Update” delta refresh creates data warehouse indexes for the filter and index columns.

17. Speed: “Replace with Control” delta refresh creates a data warehouse index for the filter column.

18. SQL Integrity: “Update” delta refresh turns varchar(max) filter and index columns into char(50).

19. SQL Integrity: “Replace with Control” delta refresh turns a varchar(max) filter column into char(50).

20. When running ETL+ for the 1st time, the DataSelf cloud servers connection only happens after the user has accepted the EULA.

21. If ETL+ can’t download the EULA: it’ll show the EULA page blank (probably an internet connection, firewall or antivirus issue).

22. When ETL+ can’t connect to DataSelf cloud servers: shows error about IP whitelisting and/or firewall and antivirus adjustments. And then tries to whitelist IP (see feature 1 above).

23. Super admin: Settings page to select, modify or new data warehouse and its SQL schema. Blank schema = EntityID. Enter the new name to change the schema. Ex.: dbo.

24. Super admin: Data warehouse configuration page has a Multi-Tenant checkbox. Check it for Advanced Analytics data warehouses, and uncheck it for Enterprise Analytics.

25. Super admin: Client Info page: “0_EntityID Column” checkbox to create this column.

26. Super admin: ETL+ logging in the cloud.

27. Super admin: ETL+ cloud auto refresh log includes EntityID, TableName, Begin Date/Time, End Date/Time, Rows Before, Rows Deleted, Rows Updated, Rows Kept, Rows Inserted, Rows After, Refresh Criteria, Result Type, ETL+ User, Refresh Schedule ID, Refresh Schedule Run ID, Windows Computer Name, Windows Username.

28. Super admin: “Entity” box converts all characters to lower case.

29. Super admin: Creation of new OData entities sets the OData extraction blocks to 1000 records and 100ms pause (instead of 0 recs and 0ms pause which causes issues).

30. Super admin: Wider Entity ID column on the Entities page.

Resolved Issues

1. Super admin: Issues when creating new entities.

2. The 32-bit version wasn’t saving ODBC credentials.

Known Issues

1. When configuring Acumatica OData delta refreshes, in the case that OData doesn’t render the indexes of a table (Acumatica bug?), you might get an “invalid type for an index” error message. To fix it, keep the delta configuration as is, change the “Load Type” to “Replace All”, do a table refresh, and switch back to delta refresh.

2. Failing to entering correct credentials 5 times in a row closes ETL+ but allows immediate 5 new re-attempts. The correct process is to force a 5 minute wait after 5 failed attempts.

Release date: 02/28/2021

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 64 bits - 2021.02.2803.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 32 bits - 2021.02.2803.exe

New Features

1. Move Windows User variables to the Windows ProgramData area.

2. Simplifies running ETL+ and its auto refresh processes regardless of Windows users.

3. Collaboration: When a user is editing an ETL+ entity, other users can only log, view or copy its settings. To release editing to other users, Save the entity or close ETL+.

   Other users trying to edit it will receive the message: “username is currently editing this entity. You can only view it. To re-load the modified entity, exit and re-enter ETL.”

4. Collaboration: When a user tries to edit ETL+ but the metadata has changed by another user, ETL+ gives the message: ETL+ metadata has changed. In order to edit your ETL+, please exit and re-enter to re-load your metadata.

5. Allowing users to open more than one ETL+ at the same time.

6. When running Refresh Schedules in parallel, log each of them on their own block (easier to read log file).

7. Table Load Types in Refresh Schedules: Refresh Schedules can run the default table load type (Refresh All, Replace with Control, Update, Append), or force a "Replace All". This helpful when using delta refreshes on a frequent schedule, and then forcing a replace all on a less frequent schedule to capture old records that have been deleted or modified.

8. ETL+ lets the same Refresh Schedule run in parallel, but might issue the error message: [DS ERROR : 0902] ETL+ cloud log conflict. Two or more ETL+ are running the same Refresh Schedule at the same time. The data refresh might be ok.

9. ETL+ allows the refreshing of the same data warehouse table in parallel. Users need to understand the implications of using this feature since it can lead to data consistency issues. Ex.: it’s Ok to append different data sets in parallel, but a full refresh in parallel is likely not desirable.

10. When editing ETL+, add an * to the Save button to indicate changes haven’t been saved yet. Clicking Save removes the * flag.

11. Support to OData v4. First release. Test it before using it for production.

12. Resized ETL+ UI to fit small resolutions down to 800x600.

13. ETL+ UI can be switched between default size, full screen mode and minimized.

14. New redesigned ETL+ metadata for better auto documentation, faster performance and reporting.

15. Upgrade from prior ETL+ versions requires the re-creation of entities, users and statements.

16. When refreshing tables on ETL+ UI, the refresh progress indicator shows the Refresh Schedule name at the top, the table counter and % progress inside of the circle, and the table being imported at the bottom.

17. Show ETL version and 32/64bits on the main page.

18. Ability to define MS SQL Server data warehouse’s database schema. The default schema = ETL+ EntityID. ETL+ admin users can set the schema to any other value (such as dbo). Multi-tenant data warehouses' ID=1 will always have schema = ETL+ EntityID.

19. RWC and Update will do the first refresh in full refresh mode.

20. Removed dozens of locally saved variables that are no longer used (they were used before metadata moved to the cloud).

21. "Leave without saving?" when a user tries to close ETL+ without saving changes.

22. For users with access to multiple entities, ETL+ remembers the last entity opened.

23. Show ETL version and 32/64bits on the top right of main pages.

24. Strong encryption of ProgramData info.

25. Log changes to the source system’s connection string.

26. Left and right trim to all user input fields (avoids spaces in usernames and passwords).

27. ETL+ captures who saved an ETL SQL Statement version. Currently, only available in the ETL+ metadata.

28. More informative error message when ETL+ cannot connect to its metadata or data warehouse.

29. If two or more ETL try to refresh the same table at the same time, if the table has RWC or Update, it might give the error message: [DS ERROR : 0910] Two or more ETL+ tried to refresh the XYZ data warehouse table at the same time. This can cause data consistency issues. Please run a full refresh for this table again.

30. ETL+ version number set to YYYY.MM.DDNN (NN is sequential starting on 01).

31. ETL+ EULA and About pages inform the latest ETL+ version and URL for download (in case users are installing an older version accidentally).

Resolved Issues

1. New features number 1 and 2 resolve several auto refresh friction points.

2. Fixed data auto refresh not running from some servers via command line.

3. Fixed a testing feature that prevented ETL+ from connecting to the cloud from some servers.

4. Fixed the auto email of new product registrations.

5. Fixed ETL+ Excel and CSV data sources to include headers on row 1.

6. Fixed Update and Replace with Control refresh from ODBC sources.

7. Fixed error message showing data warehouse structure names.

Release date: 01/13/2021

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 64 bits 2021.1.001.1301.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf ETL Plus 32 bits 2021.1.001.1301.exe

New Features

1. Multiple refresh schedules.

2. Add Debug Mode dropdown to Batch page. Levels: Debug Off, Debug L1, Debug L2. The debug applies to each DSBI site, and all of its refreshes and users.

3. Allow ETL+ to run multiple refreshes of the same Entity in parallel.

4. Make UI 99 to be sorted by EntityID (not Entity Name).

5. Add the dw name to the bottom left corner of the ETL+ UI.

6. Improve UI titles and descriptions for clarity and self documentation.

Resolved Issues

1. Test "Communications" and "User" tables with all fields from Sage CRM source.

2. Scheduler isn't remembering the saved schedule settings.

3. ETL+ was hanging in WTS. Issue with the feature that puts notifications on the system tray.

4. When trying to upload 2 tables now, on the dw table list: clicking on a table, pressing and holding Crtl and clicking on a 2nd table crashes the ETL.

5. Log when refresh runs via shortcut shows [DS ERROR: 9997] - Thread was being aborted.

Release date: 12/14/2020

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf_ETL_Plus_64bits_2020_4_012_1402.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf_ETL_Plus_32bits_2020_4_012_1402.exe

New Features

1. Create button on the Source Tables column for "Include".

2. When creating the ETL+ password, re-label "Password" to "New Password".

3. Selecting DBs in the MS SQL credentials page: Refresh Button to reload the list of DBs.

4. Delete Data Warehouse Table now has a "Confirm Deletion?" prompt.

5. Password cannot have # as the only special character. Message: Password cannot have # as the only special character.

6. For CSV Data Sources: "The first row has a Column names" should be checked and grayed out and non-editable.

7. For CSV Data Sources: "The first row has a Column names" Delimiter should have comma ',' and grayedout and non-editable.

8. Replace with Control: Order By field box removed. It automatically creates the index based on the Control field.

9. Debug Mode creates a log file: Log entityID_debug.txt.

10. Support for NetSuite ODBC.

11. Download CSV and Excel files with URLs with strings after the file name.

12. Ability to use subqueries in SQL Statements.

13. 0_EntityID to all tables.

14. Replace with Control delta refresh.

15. Update delta refresh.

16. Download CSV and Excel files from URLs into DS_SourceFiles folder where the ETL+ is, and then extract data to data warehouse.

Resolved Issues

1. Refresh and Upload buttons fail with THROW Error.

2. The ETL is not accepting edits in Client Information and Refresh Schedule. DS ERROR 04000 - Transaction Failed.

3. When saving a data warehouse table, ETL shows: "Do you want to cancel all changes made in this section?".

4. In the Microsoft Excel data source, "the first row has a Column names" should be checked.

5. It takes too long to enable other functions after saving a SQL Statement.

6. Fixed OData extract: DSID: nfr_wac_acumatica1, Dw Table: 'TST jn - Prod Order Detail - In Process Released', Upload fails.

7. It wasn’t running the Tbl refresh at the end of the auto refresh. 

8. The data refresh email should only be sent when running the full refresh. 

9. Mapping of Odata fields failed.

10. When adding New Table, the cursor will allow to change this new table name. However, it can only save the new name if all other table names listed are above "New Target Table".

11. The ODBC Data Source box had two Connect buttons.

12. Using Preview in OData on some tables causing closing, freezing or error message.

Release date: 10/27/2020, First Release

Download Files

64 bits: https://www.dataself.com/etl_plus/DataSelf_ETL_Plus_64bits_2020_4_010_2701.exe

32 bits: https://www.dataself.com/etl_plus/DataSelf_ETL_Plus_32bits_2020_4_010_2701.exe

Released Features

1. Minimum system requirements: 1 CPU, 2GB of RAM, 5GB of free diskspace, Windows 10, Windows Server 2012 R2 or newer, 32 and 64bit.

2. Installation process: EULA, user credentials and token, IP whitelisting, connection to data source, scheduling the data auto refresh via Windows Task Scheduler (WTS), refreshing data now, and optionally maintaining ETL+ metadata.

3. Microsoft certified application.

4. Creation of Windows Task Scheduler (WTS) task to run the data auto refresh, including schedule and user to run WTS service.

5. Cloud-based ETL+ metadata allows remote maintenance.

6. Cloud uptime of 99.9%.

7. Data source built-in connectors:

   1. ODBC (32 and 64 bits)

   2. MS SQL Server

   3. Odata version 3

   4. Excel (will prompt to install MS Access driver if required)

   5. CSV (will prompt to install MS Access driver if required)

8. Refresh of tables and subsequent Tableau data sources via encoded batch files.

9. Auto data refresh email notification.

10. Data load type: Replace all.

11. Create evolution bar for the data extraction when running via ETL+ UI.

12. Add refresh evolution process in system tray when running as a service.

13. Stores data and metadata in SSL secured multi-tenant AWS MS SQL Server data warehouse.

14. Upload data to AWS cloud servers via SSL secured IP-to-IP connection.

15. Locally encrypt user credentials to ETL+ and on-promises systems.

16. Credentials using user token, and requiring user to create an user with their email and complex password (minimum of 8 characters with lower/upper case, numbers and special characters).

17. If a users fails to enter correct credentials 5 times in a row, block access for 5 minutes.

18. ETL+ UI panes: source tables, data warehouse tables, SQL Statements.

19. About page to show EULA and product release.

20. Data warehouse table functions: new, duplicate, rename, delete, preview, upload now.

21. Refresh Now with options to Refresh Full, DW only, Tableau only.

22. Refresh Now with counter of 9 seconds (configurable), and buttons for Cancel, Pause, Run Now.

23. ETL SQL Statement with Edit Mode and Confirm button.

24. Locally saved ETL+ log file with refresh steps.

25. Email refresh process log to maintenance users.

26. Creates shortcut on Desktop to run ETL+ auto refresh processes.