ETL UI: Added '?' help icon on main pages.
ETL UI: User Guide available at https://dataself.atlassian.net/wiki/spaces/DS/pages/1569980439 .
ETL UI: Many buttons have been replaced by icons.
ETL UI: Hover over icons to view their functions.
ETL UI: Hover over data structures such as data sources and tables to view metadata.
ETL UI: Contextual icons - they are grayed out when they don’t apply to selected objects (ex.: selecting a table on the Source panel disables Delete Data Source).
ETL page: Right clicking on the left two panels pops up the icon functions. The pop ups show the icons and their function names.
ETL page: Clean up of the ETL+ main page (some functions have been moved to other pages).
ETL page: ETL SQL Statement’s title shows the selected TableName (or Source.TableName when multiple sources are mapped).
ETL page: ‘Log File’ icon to open local log file in Notepad.
ETL page: ‘Refresh Schedule’ button changed/renamed to ‘Refresh Batch’ icon.
ETL page: ‘Refresh Batch Now’ function moved to the 'Refresh Batch' page.
Multiple data sources: When opening ETL+, the data source(s) will not list its table names by default. This enables a fast opening of ETL+ since it doesn’t have to wait to list source system table names dynamically. Click a data source and the ‘Refresh’ icon to list its table names - this process can take several seconds if the source system is slow to render its table name list.
Multiple data sources: Imports data from multiple data sources within the same ETL+ entity.
Multiple data sources: Source panel shows all data sources created.
Multiple data sources: Data sources are represented by a cylinder icon, and their table name list can be expanded or collapsed with the +/- icon by the data source icon. Tables are represented by a square window icon.
Multiple data sources: Select a table from a data source and click the ‘Add Table’ icon/function to map it to the ETL process.
Multiple data sources: Click the ‘Add Data Source’ icon/function, select a data source type (CSV, MS Excel, MS SQL Server, OData, ODBC), configure the connection information, click Connect (or Save).
Multiple data sources: Configuration of data sources now includes an editable Alias (DSN Alias).
Multiple data sources: Added ‘Save’ button (saves and closes without trying to connect to source).
Multiple data sources: When creating a new DSN, the Alias is auto populated as following:
MS SQL: Database name.
ODBC: ODBC name.
Excel: File name.
CSV: Windows Folder name.
OData: Last word of the OData URL.
Multiple data sources: Configuration of data sources now includes a ‘Dw Schema’ checkbox. When checked, tables from this data source will be stored in the data warehouse using a SQL schema = DSN Alias. When unchecked, tables will be stored in the default data warehouse SQL schema.
Multiple data sources: In the DSN configuration page, use the ‘Change Data Source’ icon to replace it to another type (ex.: from OData to MS SQL).
Multiple data sources: The DSN pages have a Save button (helpful when doing remote DSN maintenance without access to the DSN).
Multiple data sources: Select a data source and click the ‘Delete Data Source’ icon/function to delete.
Multiple data sources: Select a data source and click the ‘Refresh’ icon/function to reconnect to the data source and list its tables.
Multiple data sources: Select a data source and click the ‘Properties’ icon/function to configure its settings.
Multiple data sources: Mousing over the left panel’s data source icons shows its metadata.
Multiple data sources: Mousing over the left panel’s tables from a data source shows its metadata.
Multiple data sources: The center panel organizes the data warehouse tables by data source alias.
Multiple data sources: The center panel data source alias icon is a square with 3 objects in it. You can expand/collapse the list of data warehouse tables with the +/- icon by this icon.
Multiple data sources: Mousing over the center panel’s data source alias icons shows its metadata.
Multiple data sources: Mousing over the center panel’s tables shows its metadata.
Multiple data sources: When a data source has its table names listed on the left panel, and the user clicks that same table on the center panel, the cursor will select the table on the source system list.
Multiple data sources: Clicking on a center panel’s data source alias icon and the ‘Load Now’ icon will load all of its mapped tables to the data warehouse.
Multiple data sources: Ability to import data from the data warehouse for further data prep.
Multiple data sources: When adding a table from a source, the cursor selects it on the center panel.
Load UI: The load UI page is now integrated in the ETL+ application (not a detached layer).
Load UI: Cancel button stops the load process when the table currently loading ends.
Load UI: Cancel button stops the load process between OData extraction blocks and tables.
Load UI: Shows how many rows have been loaded from the table being extracted now.
Load UI: Shows how many rows per table have been loaded: (X rows <action>).
Load UI: Shows the type of load: the <action> variable in (X rows <action>):
Replace All: loaded
Replace with Control: replaced
Load UI: Refreshing a long list of steps scrolls up showing the latest steps at the bottom of the list.
Load UI: Shows at the bottom of the table list when waiting (sleep mode) versus extracting data.
Load UI: Shows when refreshing each Tableau Extract.
Load UI: New background and font colors provide better contrast.
Load UI: Shows Source.TableName when multiple sources are available, otherwise only TableName.
Cloud Watchdog: Monitors ETL+ scheduled refresh cloud logging and sends email notifications.
Cloud Watchdog: Notifies “Missed execution?” when a scheduled Refresh Batch hasn’t started 120 seconds after the scheduled time.
Cloud Watchdog: Notifies “Taking too long?” when a scheduled Refresh Batch takes longer than (10 mins + 1.5 x Last Refresh Time) to complete. This might be an indication that the refresh will not complete.
Cloud Watchdog: Always notifies when ETL+ “Notification Email” is set to “Cloud - Always Notify”.
Cloud Watchdog: Notifies on error when ETL+ “Notification Email” is set to “Cloud - Notify on Error”.
Settings: Creation of the Settings page and added its icon on ETL+ main page.
Settings: ETL version posted.
Settings: Page provides access to ‘Client Info’, notification about ‘3 secs between error retries', ‘Debug’ drop down, ‘0_EntityID Column’, ‘Data Warehouse’ configuration, ‘About’, 'Notification Email’.
Settings: ‘Refresh Batch Notification Email’ includes dropdown for:
Always Notify: ETL+ always sends notification email.
Notify on Error: ETL+ only sends email on error.
Cloud - Always Notify: Cloud Watchdog always sends notification email.
Cloud - Notify on Error: Cloud Watchdog only sends email on error.
No Notification: No notifications from ETL+ nor Watch Watch Dog.
Settings: “Create Now” button added to “New Data Warehouse” page.
Refresh Batch: Renaming ‘Refresh Schedule’ to ‘Refresh Batch’.
Refresh Batch: Re-layout of the page.
Refresh Batch: Bottom left section shows WTS summary configuration.
Refresh Batch: ‘WTS Settings’ button provides access to WTS settings and Apply button.
Refresh Batch: To run a Batch either click the ‘Refresh Now' icon on a batch name, or select the Batch and click 'Refresh Now’ icon on the panel header.
Refresh Batch: Click the 'Delete' icon on a batch name to delete it.
Refresh Batch: ‘Add’ and ‘Create Shortcut’ became icons on the batch name list.
Refresh Batch: List of Refresh Batch Steps shows twice as much vertical real estate.
Refresh Batch: Clearer description of ‘All Data Warehouse Tables (include new ones)’ checkbox.
Refresh Batch: Click a refresh step and ‘Move Up’ & ‘Move Down’ icons to re-order the refresh steps.
Refresh Batch: New step types for ‘Tableau Extract Refresh', ‘Cloud Script’, 'Command Line’.
Refresh Batch: Click ‘+' icon ('Add Script’), and select a script type to create.
Refresh Batch: Select ‘Tableau Extract Refresh’ to configure an extract to be refreshed. These extracts must be hosted at https://dataselfbi.com and pull data from DataSelf's cloud data warehouses.
Refresh Batch: ‘Tableau Extract Refresh’ script requires the extract name.
Refresh Batch: ‘Tableau Extract Refresh’ default values for other fields:
Project = <EntityID>
Parent Project = NULL
Tableau Site = <EntityID>
Refresh Batch: Select ‘Cloud Script’ to provide its name. Consult DataSelf to discuss your cloud script options (such as running other cloud ETL+ extractions, DataSelf Automatic Distribution, and Python custom applications).
Refresh Batch: Select ‘Command Line’ to provide its name and parameters. This will run the configured command line locally. Example of use cases: running locally Tableau extract refreshes, or on-premises DataSelf Automatic Distribution.
Refresh Batch: Select a script step, and click icons ‘Delete Script’, ‘Configure Script', or 'Run Script’ as needed.
Refresh Batch: Removed ‘Refresh Type' since this can now be done directly in 'Refresh Batch Steps’.
Refresh Batch: Moved ‘Debug’ drop down to Settings page.
Refresh Batch: Removed ‘Enabled’ from batch list (not functional yet).
Refresh Batch: Other UI buttons/icons become disabled when editing scripts.
Refresh Batch: Icons on the Steps panel differentiate objects between tables and scripts.
Refresh Batch: Mousing over icons provide their functions and name and/or metadata.
Refresh Batch: Mousing over a table pops up its metadata (DSN and refresh settings).
Refresh Batch: Removed ‘On Demand Data Refresh’ UI with delay start - no longer needed.
Refresh Batch: Shows ETL+ version on the top right.
Refresh Batch: With multiple data sources, Batch Steps show DataSourceName.TableName.
Refresh Batch: Added ‘Tableau Template Deployment’ icon - available to admin users only.
Refresh Batch: ‘Log File’ icon to open local log file in Notepad.
Engine: Notifies when ETL+ doesn’t find the required .NET installed.
Engine: When an extraction errors out, retry 3 times (3 seconds apart).
Engine: When manually running extractions, connection errors prompt for Retry, Cancel or Skip.
OData Extraction: For OData tables without built-in indexes, ETL+ will only extract data in blocks when indexes have been configured in ETL+. Without indexes, such extractions will be continuous (not in blocks).
OData Extraction: Character fields are loaded into varchar(max) in the data warehouse and are limited to the first 250 characters (next release will give users control of target column format).
OData Extraction: When running an RWC or Update extraction from large datasets, ETL+ logs to Acumatica, loads in sets of up to 100 extraction blocks into a temp table, loads the temp table to the target table, logs off, and then repeats this process for the next set of extraction blocks. This helps Acumatica rendering speed, and provides intermediary starting points in case of issues in the middle of the extraction process.
OData Extraction: When processing delta refreshes, the page size will automatically expand when the delta filter cannot get over the next page of data. The expanded page size only applies to the table being loaded. Message: The delta extraction filter requires an Extraction Pagination expansion from <current> to <new> rows.
OData Extraction: Ability to insert texts that include SQL special characters (such as ' \* *\).
OData Extraction: Debug Log file includes OData URLs executed when Debug is set to Level 2.
Acumatica OData v3 delta refreshes: Acumatica does NOT accept GI formula fields (ex.: =[GLHistoryByPeriod.FinPeriodID]) as filter for RWC and Update.
Acumatica OData v4 delta refreshes: Acumatica requires that filters for RWC and Update must not have NULLs and ETL+ extraction blocks must be bigger than the max number of records for each block of records with the same filter value.
OData v4 Extraction: Changed extraction process for large tables when OData v4 refused to render data.
OData v4 Extraction: When working with RWC or Update and using date filters, ETL+ filtering adapts to the date format of the source Acumatica.
OData v4 Extraction: If a table doesn’t have a native OData index, ETL+ errors out and logs “Extraction time out. Table without native OData index. Please create one via Replace with Control or Update.”
ODBC Extraction: Sage Intacct ODBC support.
ODBC Extraction: Quickbooks Online ODBC support.
OBDC Extraction: MongoDB ODBC support (non-relational database).
OBDC Extraction: Using native MondoDB ' (single quote) or ANSI  (brackets) around reserved words and nested collections.
ODBC Extraction: Support to views from source system (previously, only tables).
Logging: ETL+ logs description and time for new functions/scripts (Tableau Extract Refresh, etc).
Logging: When launching ETL+ via command line, add the 3rd parameter “1” and ETL+ will log locally “Start ETL+” even before loading the cloud metadata.
Logging: Posts Source.TableName when multiple sources are available, otherwise only TableName.
Logging: Posts the number of rows loaded.
Logging: Posts the type of load: the <action> variable in (X rows <action>):
Replace All: loaded
Replace with Control: replaced
Logging: For large OData tables with RWC or Update, logs when 100 extraction blocks are loaded.
Logging: Posts 32 or 64bit ETL+ version on log refresh headers.
Client Information: ETL version posted.
Client Information: The following fields were moved to more relevant pages: Data source, Settings, 0_EntityID Column, Notification Email.
Client Information: Creation of a Dropdown for DataSelf License type (DataSelf ETL+, VAR, DIY, Advanced, Pro, Enterprise). Only available for super admin.
Client Information: Addition of checkbox to deploy out-of-the-box Tableau Templates for Advanced Analytics.
User Logging Page: ETL version posted.
Error handling: Notifies and logs incorrect Column names used in 'Replace with Control' or 'Update'.
Error Handling: Improved message description when ETL+ can’t connect to its data warehouse.
Error Handling: Added message describing that MS SQL doesn’t accept insert in timestamp columns.
Error Handling: For certain extraction errors, retry the extraction 3 times.
New Deployments: Removed step to load data now (it caused confusion).
New Deployments: ‘Deploy Tableau Script’ runs on dw3 cloud server the following command line: “C:\DataSelf\<SourceSystem>\Deploy.Bat <EntityID> <DwServer> <DwDbName>”
Super admin: ‘Client Info’ and ‘Refresh Batch’ pages allow the deployment of out-of-the-box ‘Tableau Templates’ (site, tdsx, twb) for Advanced Analytics and QA templates for Pro and Enterprise.
Super admin: Entity creation populates EntityID in Refresh Batch Email Notification’s Subject.
Super admin: Entity creation populates user email in Refresh Batch Email Notification’s Email.