Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Expand
titleDownload, Release Notes, Release Date

Important: Beta version for testing purposes only! It hasn’t been extensevely tested yet.

Download files:
64 bits: soon
32 bits: soon

Released: soon

New Features - Work in Progress

  1. Data Warehouse Column Formatting: Double-click a data warehouse table to enable column formatting and primary key display mode (a + is enabled by the tables).

  2. Data Warehouse Column Formatting: Click + / - by a table name to expand/collapse its column formatting.

  3. Data Warehouse Column Formatting: (blue star) icon for regular columns.

  4. Data Warehouse Column Formatting: (blue star) icon for transformed columns.

  5. Data Warehouse Column Formatting: (blue star) icon for primary key columns.

  6. Data Warehouse Column Formatting: (blue star) icon for primary key transformed columns.

  7. Transform: New page to apply transformations after the load process. Reason: Many sources don’t allow transformations during the extraction or when extraction transformations are limited and/or slow. The data is extracted and loaded into a temporary data warehouse table, and then transformed into to official table. These transformations happen in the data warehouse engine, therefore they are fast, efficient and provide a broad range of MS SQL transformation functions.

  8. Transform: Select a data warehouse table → Edit Columns to open the Transform page.

  9. Transform: New page shows PK (primary key), Column Name, Source Data Type, Dw Data Type, Transform MS SQL Expression.

  10. Transform - PK: Check the PK check boxes to define the table’s primary key columns. Currently, ETL+ doesn’t validate if the PK columns are the correct primary key.

  11. Transform - PK: ETL+ creates the SQL PK columns set to Not Null.

  12. Transform - PK: For text-based sources (like OData), when checking the PK checkbox for a varchar(max) column, ETL+ automatically changes its to varchar(50) and pops up: "PK cannot use varchar(Max). Format changed to varchar(50). We recommend reviewing the varchar size." -> Ok.

  13. Transform - PK: ETL+ logs the following when MS SQL can’t create/recalculate a PK: "Primary Key failure. PK requires rows with unique values and no NULLs. Review the PK fields.". This means the selected PK columns are not the table’s correct primary key - review/fix the PK configuration.

  14. Transform - PK: When adding new MS SQL and OData tables with PKs, ETL+ automatically configures the PK columns. User can change the PK configuration afterwards.

  15. Transform - PK: From text-based sources (like OData), PK columns with varchar(max) format are automatically converted to varchar(50). User can change the PK configuration afterwards.

  16. Transform - PK: When upgrading an ETL+ entity, Primary Index columns automatically become Primary Key columns. Oct version?

  17. Transform: To view the source’s Column format, click the Refresh icon on top of Source Data Type. Depending on your source, this can take time (even minutes from OData v4).

  18. Transform: To change a Column’s format, use its Dw Data Type dropdown menu, or double-click the dropdown and type in the new format.

  19. Transform: Column MS SQL transformations available on Trasnform Transform MS SQL Expression. Ex.: LEFT(ColumnName,5).

  20. Transform: Click the column header for Column Name or Source Data Type to sort the list.

  21. Source Properties: Renamed Dw Schema to Schema. This checkbox sets Alias as the schema for tables imported from this source.

  22. Settings - PK: Ability to create data warehouse physical PKs. Reason: PKs can be taxing for data load and storage (CPU, RAM and disk space), but might dramatically improve query performance.

  23. Settings - PK: New “Create PK” dropdown with options:

    1. Update Only: Default option. Only creates physical data warehouse PKs for tables with Update delta feature.

    2. Always: Creates physical data warehouse PKs for all tables with PKs.

    3. Never: Doesn’t create physical data warehouse PKs.

  24. ETL Page - Update: Renamed Primary Index to Primary Key.

  25. ETL Page - Update: Primary Key lists the PK columns (from the Transform page).

  26. ETL Page - Update: To edit the PK columns, click the Edit icon by the Primary Key or the Edit Columns icon on the table.

  27. ETL Page - Replace with Control: Now with more control options:

    1. >= Max value from the data warehouse

    2. >= Constant

    3. >= SQL Expression. Ex. 1: getdate()-90. Ex. 2: max(ColumnName)-1000)

  28. Load Engine - Replace All: The load happens on a temporary table to allow users to query the official table with minimum interruption. Once the load completes, the temporary table becomes the official one. The temporary table is named Schema.TableName_TEMP.

  29. Load Engine - Replace All: The transition between temporary to official table happens in a fraction of a second quickly when there are no transformations performed (delete official -> , rename temporary to official, create <0_EntityID> when applicable).

  30. Load Engine - Replace All: The transition between temporary to official table happens quickly when there are transformations performed (delete official -> bulk , insert into official from temp table applying transformations). Avg cloud bulk insert speed: 200MB/s.

  31. Load Engine: Ability to apply transformations after the load (previously, only available during extraction when the source SQL syntax allowed).

  32. Data Warehouse Tables: All tables now show schema , (including dbo). <Schema>.<TableName>.

  33. Data Warehouse Tables: Renaming a table only applies to its name, not the schema (schemas might be changed in Source’s Properties and/or Data Warehouse Settings).

  34. Data Warehouse Source: The data warehouse automatically shows as a data source for additional data preparations and transformations.

  35. Data Warehouse Source: This source provides access to all data warehouse tables and views for single-tenant data warehouses (DataSelf Analytics Pro and Enterprise Editions).

  36. Data Warehouse Source: This source provides access to data warehouse tables and views where <SQL schema> = <EntityID> or <SQL schema> = <dbo> in multi-tenant data warehouses (DataSelf Analytics Advanced Edition).

  37. Data Warehouse - Source: Source’s Properties disabled.

  38. Data Warehouse - Source: Source 'sDelete disabled.

  39. Data Warehouse - Source: Use Settings -> Data Warehouse for configuration.

  40. Data Warehouse - Source: Source credentials are sync’ed up its data warehouse configuration.

  41. Data Warehouse - Center Panel: Tables imported from the data warehouse are automatically renamed to <TableName_DW>. Users can then rename them as needed.

  42. Data Warehouse - Center Panel: Tables imported from the data warehouse keep the same SQL schema as from the original source. Ex.: Abc table imported from ERP source using alias as schema will be stored in <Schema>.<TableName>=<ERP>.<Abc>. Upon re-importing it from the data warehouse, this table will be automatically set to <ERP>.<Abc_DW>.

  43. Data Warehouse - Center Panel: When importing a data warehouse table into itself (usually, a mistake), ETL+ will log: "Warning: Data warehouse source table imported into itself. This is usually a mistake. Fix it by renaming the table.". Oct version?

  44. Sources: Addition to native extraction from cloud Microsoft Dynamics CRM.

  45. Source Properties: Checkbox for 0_UnionID. For this version, this field will be populated with “AAA”.

  46. Remove Log Level 1 from local disk. Retrieve live log from the cloud instead. Oct version?

  47. Refresh Batch - WTS: Repeat Task checkbox unchecked by default.

Resolved Issues

  1. fdfsafdaaaa

Known Issues

  1. Importing large XML content into varchar(max) might run into error: server stack limit has been reached.

  2. Deleting all Refresh Batches requires going to ETL main page and back to get a new one created.

  3. Changing the data warehouse of an entity won’t show on the bottom left corner until the entity is closed and re-opened.

  4. Sometimes ETL+ is closing unexpectedly after a data load.

  5. Sometimes ETL+ runs into an “Unhandled exception” error. Close and re-open ETL+.

  6. Locking the ETL+ editing when someone else is editing it doesn’t work in some situations.

  7. ETL+ still tries to send email even when set to Cloud notification, or No Notification.

  8. ETL+ delta extraction from NetSuite ODBC might not work when using dates for filtering.

  9. When opening an entity created in prior ETL+ versions, the data warehouse panel will show the source header blank. To fix this, open the entity in the prior ETL+ version, make any modification and save it. Now open the entity in the latest ETL+ and the source header appears.

  10. Importing tables from the data warehouse works correctly but it might incorrectly log an error message.

...