ETL Log: Log icon now retrieves cloud log. Users can retrieve the log from anywhere.
ETL Log: By default, the Log shows the latest logs at the top of the page (sorted <DateTime> Desc).
ETL Log: Sort the Log page by clicking the Log column headers.
ETL Log: Click Sequence
header to sort by <RefreshRunID> Desc
& <DateTime> Asc
.
ETL Log: Refresh
icon re-loads the latest log entries.
ETL Log: Logged dates now show the time zone (currently, PST only).
ETL Log: Option to view Stats or Summary logs.
ETL Log: Stats
show row stats, processing time, and refresh criteria by loaded table.
ETL Log: Summary
shows loaded tables, rows, error messages, refresh batch headers (admin user: machine username).
ETL Log: Creates and populates the Log Summary in the data warehouse table <dw schema>.<zRefreshLog_Summary>.
ETL Log: Creates and populates the Log Stats in the data warehouse table <dw schema>.<zRefreshLog_Stats>.
ETL Log: Log columns available to Super User only: PK_Entity, MachineUser.
Table labeling: Improved labeling to uniquely identify tables (MS SQL standard):
<Source Name> - <SQL Schema>.<Table Name>
Ex.: ExcelFile - dbo.BudgetTable
Table labeling: In multi-tenant data warehouses (such as in Advanced Analytics), the schema doesn’t show on the center panel since all loaded tables belong to the same schema (<EntityID>).
Table labeling: Refresh Batch Steps don’t show schemas to simplify the table list. Schemas can be seen on a pop up by mousing over tables.
Source - zData Warehouse: The data warehouse automatically shows as a data source for additional data preparations and transformations.
Source - zData Warehouse: “zData ” source is listed at the bottom of the panels.
Source - zData Warehouse: Uses a green source icon to tell it apart from regular sources.
Source - zData Warehouse: This source provides access to all data warehouse tables and views for single-tenant data warehouses.
Source - zData Warehouse: This source provides access to data warehouse tables and views where <SQL schema> = <EntityID> or <SQL schema> = <dbo> in multi-tenant data warehouses (DataSelf Analytics Advanced Edition).
Source - zData Warehouse: Source’s Properties
disabled.
Source - zData Warehouse: Source 'sDelete
disabled.
Source - zData Warehouse: Use Settings
-> Data Warehouse
for its configuration.
Source Properties: Renamed Dw Schema
to Schema
. Sets Alias as the source tables' schema.
Data Warehouse Panel - zData Warehouse: When importing zData Warehouse tables, they’ll be grouped under a green grouping icon.
Data Warehouse Panel: Renaming a table only applies to its name, not the schema.
Data Warehouse Panel: Tables imported from the data warehouse keep the same SQL schema as the data warehouse default schema.
Data Warehouse Panel - Columns: Double-click a data warehouse table to show:
Column list
Column formatting
Primary key
Data Warehouse Panel - Columns: Click +
/ -
by a table name to expand/collapse its column formatting.
Data Warehouse Panel - Columns: icon for regular columns.
Data Warehouse Panel - Columns: icon for transformed columns.
Data Warehouse Panel - Columns: icon for primary key columns.
Data Warehouse Panel - Columns: icon for primary key transformed columns.
Data Warehouse Panel - Design: added Design icon to access table Design page.
Data Warehouse Panel - Design: For many sources, the list of columns shows even when the SQL Statement involves transformations, table links, table unions and CTEs.
Design: New page for table design:
Configure primary key columns
Change column formats
Apply column transformations
Design: Many sources don’t allow transformations during the extraction, or extraction transformations are limited and/or slow. The data is extracted and loaded into a temporary data warehouse table, and then transformed into to official table. These transformations happen in the data warehouse engine, therefore they are fast, efficient and provide a broad range of MS SQL transformation functions.
Design: Select a data warehouse table → Design
to open the Design page.
Design: New page shows PK
(primary key), Column Name
, Dw Data Type
, Transform SQL Expression
.
Design: List shows gray and white banded lines.
Design: To change a Column’s format, use its Dw Data Type
dropdown menu, or double-click the dropdown and type in the new format.
Design: Column MS SQL transformations available on Transform MS SQL Expression
. Ex.: LEFT(ColumnName,5).
Design: Click the column header for Column Name
or Source Data Type
to sort the list.
Design: Press & hold & drag with the right button of a mouse on the border of a column to change its width/height.
Design: ETL+ will execute the Design changes when user clicks Confirm
.
Design: If issues are found during Design changes execution, ETL+ will report which columns were successfully changed and which ones failed.
Design - PK: PK check boxes define a table’s primary key columns. Currently:
PKs are necessary to process Upsert delta refreshes.
No validation if the PK columns are the correct primary key.
No creation of the physical PKs in the data warehouse.
If physical PKs are needed, please create them directly in SQL after the first ETL+ import.
Design - PK: When importing a new MS SQL or Odata table that has PK in the source, ETL+ will automatically recognize and pre-configure its PK.
Design - PK: Check the PK check boxes to change/define the table’s primary key columns.
Design - PK: For text-based sources like OData, varchar(max) columns are automatically changed to varchar(50). We recommend reviewing the varchar size.
Design - PK: ETL+ logs the following when MS SQL can’t create/recalculate a PK: "Primary Key failure. PK requires rows with unique values and no NULLs. Review the PK fields.". This means the selected PK columns are not the table’s correct primary key - review/fix the PK configuration.
Design - PK: When upgrading an ETL+ entity, Primary Index
columns automatically become Primary Key
columns.
Column Formats: On the data warehous panel (center panel), double-click
or right-click on a table name -> Show Columns
to enable the expansion of this table’s column names and formats.
Column Formats: Click +
/-
by a column name to expand/collapse its column names and formats.
Column Formats: To reload column names and formats, Double-click
again or right-click on a table name -> Show Columns
again.
ETL Page - Load Type: Load Type section now only uses vertical real estate needed.
ETL Page - Load Type: Replace All
has been renamed to Load All
.
ETL Page - Load Type: Replace with Control
has been renamed to Replace
.
ETL Page - Load Type: Update
has been renamed to Upsert
.
ETL Page - Load Type: Hovering over the Load Type options shows its function.
ETL Page - Load Type: Hover over Load All
: 1st: Load all. Ongoing: Reload all.
ETL Page - Load Type: Hover over Replace
: 1st: Load all. Ongoing: Replace filtered rows.
ETL Page - Load Type: Hover over Upsert
: 1st: Load all. Ongoing: Update modified, insert new rows.
ETL Page - Load Type: Hover over Append
: 1st: Load all. Ongoing: Append all.
ETL Page - Load Type: Selecting Replace or Upsert expands its settings section.
ETL Page - Replace: Filter column now is a drop down.
ETL Page - Replace: The filter expression is automatically set to Max(FilterColumn).
ETL Page - Replace: Users can customize the filter variable. For instance:
Constant: 1000
Date constant: ‘02/15/2021’ (use quotes around the MM/DD/YYYY date format)
SQL Expressions: for instance, getdate()-30
ETL Page - Upsert: Renamed Primary Index
to Primary Key
.
ETL Page - Upsert: Primary Key
lists the PK columns (from the Design
page).
ETL Page - Upsert: Primary Key
lists the PK columns in the same order they appear in the SQL Statement.
ETL Page - Upsert: To edit the PK columns, click the Edit
icon by the Primary Key or the Design
icon on the table.
ETL Page - Upsert: Use the dropdown to select the Column with Last Modified Value.
ETL Page - Upsert: The Update filter is automatically set to Max(LastModifiedValueColumn).
ETL Page - SQL Statement: Support to MS SQL Server CTE (Common Table Expressions).
ETL Page - SQL Statement: Support to SQL statement comments. Syntax /* comment text */. This can only be used after an initial SELECT statement.
ETL Version Control: When opening an entity with a different ETL version than the last one used for the entity, ETL will notify and prompt what to do next.
ETL Version Control: When using a newer ETL+ version than the last one used on an entity, users will be prompted to upgrade or cancel.
ETL Version Control: When using an older ETL+ version than the last one used on an entity, users will be prompted to download the older version (by going to our download page) or cancel.
Load Engine: Ability to apply transformations after the load (previously, only available during extraction when the source SQL syntax allowed).
Load Engine: Source loading adds a 0_UnionID column with content = table schema. This is helpful for multi-company table linking. This new column will be further refined when more multi-company features become available.
Load Engine: Settings 0_Entity
now creates 0_<EntityID> column (instead of 0_<schema>).
Load Engine: If the source table already has a 0_<EntityID> column, load engine will import this column (instead of creating it).
Load Engine: If the source table already has a [0_UnionID] column, load engine will import this column (instead of creating it).
Load Engine: ETL+ retries 3 times upon running into a load error in the UI, and then proceeds. In a multi table refresh, use the Cancel
button to stop the refresh (Cancel applies before the next table load).
Load Engine - zEntity: Creates a <dw schema>.zEntity table with the following columns: PK_Entity, EntityID, EntityName, CYE_Value, DT_LatestLoadStart, FYrStart_Int2, PnLType, ReportingDate, 0_UnionID, 0_<EntityID>.
Load Engine - zEntity: If <dw schema>.zEntity already exists, ETL+ doesn’t overwrite it.
Load Engine - zEntity Columns: PK_Entity, EntityID and EntityName defining the entity.
Load Engine - zEntity.CYE_Value: It’s populated by Settings
page -> CYE Value
. This is used by many Financials templates to set up a description or a GL number to Current Year Earnings.
Load Engine - zEntity.DT_LatestLoadStart: At the start of a new load (individual or group of tables), ETL+ writes the current date & time on this column. Use case: to populate report DataAsOf fields.
Load Engine - zEntity.FYrStart_Int2: It’s populated by Settings
page -> FYear Start: Mmm
. A number between 0 and 12that defines the 1st calendar month of the first fiscal year (assumes period starts on the 1st of the month). Ex.: 2 for Feb 1st. 0 means a custom fiscal year. Use case: this field controls part of the logic on DataSelf’s 5_Date and 6_DatePeriod date tables.
Load Engine - zEntity.PnLType: It’s populated by Settings
page -> P&L: Type
. This column controls how DataSelf P&reports work: S → Standard P&L or E → EBIDTA type.
Load Engine - zEntity.ReportingDate: For customers that want to customize the actual reporting date. For instance, show this fiscal period metrics based on the end of last month’s date. It’s populated by Settings
page -> Reporting Date
. Options: 0 -> Today. 1 -> Yesterday, 2 -> Last month's end
.
Load Engine - Load All: The load happens on a temporary table to allow users to query the official table with minimum interruption. Once the load completes, the temporary table becomes the official one. The temporary table is named Schema.TableName_TEMP.
Load Engine - Load All: The transition between temporary to official table happens very quickly when there are no transformations performed (delete official, rename temporary to official, create <0_EntityID> when applicable).
Load Engine - Load All: The transition between temporary to official table happens quickly when there are transformations performed (delete official, insert into official from temp table applying transformations).
Load Engine - Replace and Upsert: When user changes the mappings of a delta refresh table, and the table already has data, user is promped with:
Column mappings modified. The delta refresh needs one "Load All" data load to sync up OR manual adjustments to the physical table.
Force one "Load All" in the next load (manual or scheduled)?
Yes/No/Cancel
Load Engine - Replace and Upsert: When user clicks Yes
to the prompt above, ETL+ will force a Load All
in the next table load (manual or scheduled), and then resumes the delta refresh.
Refresh Batch: Rename
icon added for Refresh Batch Names.
Refresh Batch: Removed Run Refresh Batch
icon from header (cleaner UI).
Refresh Batch: Removed Delete Refresh Batch
icon from header (cleaner UI).
Refresh Batch - Steps: Select a step and Click & Hold the Up or Down icon to quickly move it.
Refresh Batch - Steps: Hover over Up and Down icons to show quick help pop up.
Refresh Batch - WTS: Repeat Task checkbox unchecked by default.
Refresh Batch - Steps: Pop up when hovering steps has been cleaned up and re-organized the layout to make it clearer what the step is about.
Data prep: If the source data brings in a Date and or DateTime value that is incorrect for SQL (common in text based data sources like Excel or OData, and Providex), apply transformations to fix the issue. Examples:
Invalid date values (ex.: Feb 30): Convert the field to a character field.
A date/time column has more characters than accepted by SQL. Remove the extra characters: CAST(LEFT(DateColumn),19) as DATETIME).
OData Extraction - Load All: ETL+ process the data extraction sorted based on the PK (usually available) or a key available from the OData metadata.
OData Extraction - Replace and Upsert: ETL+ process the data extraction sorted based on the delta refresh filter (ex.: InvoiceDate) plus the PK (usually available) or a key available from the OData metadata.
OData Extraction - Replace and Upsert: When starting a new load and there’s no dw.TableName_Temp2, ETL+ checks the delta refresh filter value from dw.TableName, applies the filter value into the source data extraction process, and loads the data into TableName_Temp. If the end of the extraction is reached before 100 pages, ETL+ deletes TableName, and renames TableName_Temp to TableName_Temp.
OData Extraction - Replace and Upsert: Starting an extraction like listed in the prior feature above, but if 100 pages are loaded without the end of the extraction, ETL+ moves/append TableName_Temp onto TableName_Temp2, deletes TableName_Temp, logs off Acumatica, logs in Acumatica again, and resumes the load into TableName_Temp. This process repeats until the end of the extraction, then TableName_Temp is appended onto TableName_Temp2, TableName is deleted, and TableName_Temp2 is renamed into TableName.
OData Extraction - Replace and Upsert: When starting a new load and there’s a TableName_Temp2 on the data warehouse, ETL+ starts the delta load from TableName_Temp2 (instead of from TableName) and processes the load as per the prior feature.
OData Extraction - Upsert: Be sure to configure the order of PK columns in the SQL statement to guarantee an ascending extraction. Failing to do this might cause delta data extraction inconsistencies. We highy recommend adding a date (like CreateDate) as the first column of a PK controlling an Upsert.
OData Extraction - Upsert: When executing OData extractions, the order of the PK columns in the OData extraction URL following the order of the columns the SQL statement (instead of alphabetical).
ODBC Configuration: Added an explanation about adding credentials to an ODBC driver.
ODBC Configuration: Added an explanation about potential reason for missing ODBCs is because of mismatch of ETL+ and ODBC 32/64-bit builds.
ODBC Extraction: Support to Pervasive ODBC.
Installation: For every entity, usually only one ETL+ installation requires setting up the connection to data sources. Therefore, we removed the installation step to connect every ETL+ to its data sources.
Installation: We removed the Windows Task Schedule step from the installation. Cleaner install.
All Users: On the user credentials screen, the Login button is preselected. Once credentials are entered, press Enter
to login or click the Login
button.
All Users - ETL+ Version Control: ETL+ flags its metadata with the ETL+ version used to maintain it.
All Users - ETL+ Version Control: When a user opens an entity with a different version than what’s on the metadata, ETL+ will prompt the user if he/she wants to proceed with the different version or Cancel
.
All Users - ETL+ Version Control: If a user wants to open an entity with a newer or older ETL+ version than what’s in the metadata, the ETL+ will proceed and update the metadata ETL+ version information.
All Users - ETL+ Version Control: If a user doesn’t want to open an entity with a newer or older ETL+ version than what’s in the metadata, the ETL+ will prompt if the user wants to open ETL+ Release History webpage to download other versions.
Admin User: ETL+ admin user page now shows the ETL+ metadata version.
All Users: The new ETL+ installation flow no longer has a WTS step. This was confusing for many users.
Client Info: ETL+ license is now part of the metadata.
All Users: When opening an entity, the center panel cursor goes to the top of the page.
All Users: Added Send
button on Client Info
page to re-send product registration email.
All Users: Added Send
button on Client Info
page to re-send product registration email that now copies sales@ and support@ dataself emails.
Admin User: metadata.EntityAttribute new table to host customer and tech support information.
Admin User: Entities page has a search box - enter characters to find and select matching EntityIDs and Names.
Admin User: When landing on the Entities page, the cursor goes to the search box.
Admin User: When leaving the ETL page back to the Entities page, the prior Entity will pre-selected in the new Search box.
Admin User: When a single entity shows on the Entities page, just click Enter
to Edit it (or click Edit
with mouse).
Admin User: Tableau Script Deployment removed from Refresh Batch page (it was confusing).
Admin User- Duplicate Entity: Upon duplication of an Entity, ETL+ finds & replaces <OriginalEntityID> with <NewEntityID> in all SQL Statements. This is needed for many out of the box data warehouse table data manipulations. Ex.: for properly processing zEntity and zCalendarTable tables.
Admin User: ETL+ Customer Portal with easy access to consolidated metadata information.
Admin User: ETL+ Customer Portal can only be accessed from behind DataSelf’s firewall.
Admin User: Deployment of Advanced, Pro and Enterprise share the same foundation (ETL+ templates, Tableau tds, Tableau main and QA twb workbooks).
Admin User: Deployment of Pro and Enteprise replaces EntityID in SQL Statements with dbo (which is the default schema for such deployments).
Admin User: For Pro and Enterprise deployments, ETL+ will automatically create a SYSTEM_Date
table in the data warehouse. Required for certain internal analytics processes.
Admin User: For Pro and Enterprise deployments, ETL+ will automatically create the zDateUntreated
table for date manipulations.
Admin User: For Pro and Enterprise deployment, ETL+ will automatically create auxiliary tables for source database specific data manipulations (such as document types and GL Financials data sets).
Admin User: ETL+ License field controls the cloud template deployment script.
Admin User: Client Info
page allows the configuration of the ETL+ License
type: DataSelf ETL+, VAR, DIY, Advanced, Pro, Enterprise.
Admin User: On the Client Info
page, checking the Tableau Deploy
checkbox + License
= VAR
+ clicking Save
will deploy a DataSelf Advanced Analytics NFR system on https://bi1.dataself.com.
Admin User: On the Client Info
page, checking the Tableau Deploy
checkbox + License
= Advanced
+ clicking Save
will deploy a DataSelf Advanced Analytics system on https://dataselfbi.com.
Admin User: On the Client Info
page, checking the Tableau Deploy
checkbox + License
= Pro
+ clicking Save
will deploy a DataSelf Pro Analytics system on https://dataselfbi.com.
Admin User: On the Client Info
page, checking the Tableau Deploy
checkbox + License
= Enterprise
+ clicking Save
will deploy a DataSelf Enterprise Analytics system on https://dataselfbi.com.
Admin User: Analytics Settings
is now part of the ETL+ metadata. It contains information such as Analytics type (Tableau, soon Power BI as well) and other settings to automate maintenance tasks such as the deployment of their templates and extract refresh.
Admin User: New Stored Procedure to automate the maintenance of analytics tasks such as the deployment of Analytics templates and extract refresh.
Admin User: The Analytics Server URL from Analytics Settings
shows at the bottom of the Tableau Extract Refresh page.
Upgraded Microsoft Code Signing with RSA4096 encryption.
Compatible with Windows 11.