Versions Compared

Old Version 104

changes.mady.by.user Joni Girardi

Saved on

compared with

New Version 105

changes.mady.by.user Lori Wright

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Expand
titleDownload, Release Notes, Release Date

Important: Beta version for testing purposes only! It hasn’t been extensevely tested yet.

Download files:
64 bits: soon
32 bits: soon

Released: soon

New Features - Work in Progress

  1. Data Warehouse Column Formatting: Double-click a data warehouse table to enable column formatting and primary key display mode (a + is enabled by the tables).

  2. Data Warehouse Column Formatting: Click + / - by a table name to expand/collapse its column formatting.

  3. Data Warehouse Column Formatting: (blue star) icon for regular columns.

  4. Data Warehouse Column Formatting: (blue star) icon for transformed columns.

  5. Data Warehouse Column Formatting: (blue star) icon for primary key columns.

  6. Data Warehouse Column Formatting: (blue star) icon for primary key transformed columns.

  7. TransformDesign: New page to apply transformations after the load process. Reasonfor column design: configure primary key columns, change column formats, apply column transformations.

  8. Design - Column Transformations: Many sources don’t allow transformations during the extraction, or when extraction transformations are limited and/or slow. The data is extracted and loaded into a temporary data warehouse table, and then transformed into to official table. These transformations happen in the data warehouse engine, therefore they are fast, efficient and provide a broad range of MS SQL transformation functions.

  9. TransformDesign: Select a data warehouse table → Edit Columns Design to open the Transform Design page.

  10. TransformDesign: New page shows PK (primary key), Column Name, Source Data Type, Dw Data Type, Transform MS SQL Expression.

  11. Transform Design - PK: Check the PK check boxes to define the a table’s primary key columns. Currently, ETL+ doesn’t validate :

    Transform

    1. PKs are necessary to process Update delta refreshes.

    2. No validation if the PK columns are the correct primary key.

  12. Transform - PK: ETL+ creates the SQL PK columns set to Not Null.

  13. Transform - PK: For text-based sources (like OData), when checking the PK checkbox for a varchar(max) column, ETL+ automatically changes its to varchar(50) and pops up: "PK cannot use varchar(Max). Format changed to varchar(50). We recommend reviewing the varchar size." -> Ok.

    1. No creation of the physical PKs in the data warehouse.

    2. If physical PKs are needed, please create them directly in SQL after the first ETL+ import.

  15. Design - PK: When importing a new MS SQL or Odata table that has PK in the source, ETL+ will automatically recognize and pre-configure its PK.

  16. Design - PK: Check the PK check boxes to change/define the table’s primary key columns.

  17. Design - PK: ETL+ creates the SQL PK columns set to Not Null.

  18. Design - PK: For text-based sources like OData, varchar(max) columns are automatically changed to varchar(50). We recommend reviewing the varchar size.

  19. Design - PK: ETL+ logs the following when MS SQL can’t create/recalculate a PK: "Primary Key failure. PK requires rows with unique values and no NULLs. Review the PK fields.". This means the selected PK columns are not the table’s correct primary key - review/fix the PK configuration.

  20. Transform Design - PK: When adding new MS SQL and OData tables with PKs, ETL+ automatically configures the PK columns. User can change the PK configuration afterwards.

  21. Transform - PK: From text-based sources (like OData), PK columns with varchar(max) format are automatically converted to varchar(50). User can change the PK configuration afterwards.

  22. Transform - PK: When upgrading an ETL+ entity, Primary Index columns automatically become Primary Key columns. Oct version?

  23. Transformupgrading an ETL+ entity, Primary Index columns automatically become Primary Key columns. Oct version?

  24. Design : To view the source’s Column format, click the Refresh icon on top of Source Data Type. Depending on your source, this can take time (even minutes from OData v4).

  25. TransformDesign: To change a Column’s format, use its Dw Data Type dropdown menu, or double-click the dropdown and type in the new format.

  26. TransformDesign: Column MS SQL transformations available on Transform MS SQL Expression. Ex.: LEFT(ColumnName,5).

  27. TransformDesign: Click the column header for Column Name or Source Data Type to sort the list.

  28. Source Properties: Renamed Dw Schema to Schema. This checkbox sets Alias as the schema for tables imported from this source.

  29. Settings - PK: Ability to create data warehouse physical PKs. Reason: PKs can be taxing for data load and storage (CPU, RAM and disk space), but might dramatically improve query performance.

  30. Settings - PK: New “Create PK” dropdown with options:

    1. Update Only: Default option. Only creates physical data warehouse PKs for tables with Update delta feature.

    2. Always: Creates physical data warehouse PKs for all tables with PKs.

    3. Never: Doesn’t create physical data warehouse PKs. for tables with Update delta feature.

    4. Always: Creates physical data warehouse PKs for all tables with PKs.

    5. Never: Doesn’t create physical data warehouse PKs.

  31. ETL Page - Load Type: Load Type section is now colapsed.

  32. ETL Page - Load Type: Selecting Replace or Update will expand its settings section.

  33. ETL Page - Load Type: Hovering over the Load Type options shows its function.

  34. ETL Page - Load Type: Replace All has been renamed to Load.

  35. ETL Page - Load Type: Replace with Control has been renamed to Replace.

  36. ETL Page - Load Type: Hover over Load: Full load. Delete & load.

  37. ETL Page - Load Type: Hover over Replace: Delta Load. 1st Load. Ongoing: Replace filtered rows.

  38. ETL Page - Load Type: Hover over Update: Delta Load. 1st Load. Ongoing: Add new, update modified rows.

  39. ETL Page - Load Type: Hover over Append: Full load. 1st load. Ongoing: Append.

  40. ETL Page - Replace: Filter column now is a drop down.

  41. ETL Page - Replace: The filter expression is automatically set to Max(FilterColumn).

  42. ETL Page - Replace: Users can customize the filter SQL pressions. For instance:

    1. Integer constant: 1000

    2. Date constant: ‘02/15/2021’ (use quotes around the MM/DD/YYYY date format)

    3. SQL Expressions. Ex.: getdate()-30

  43. ETL Page - Update: Renamed Primary Index to Primary Key.

  44. ETL Page - Update: Primary Key lists the PK columns (from the Transform Design page).

  45. ETL Page - Update: To edit the PK columns, click the Edit icon by the Primary Key or the Edit Columns Design icon on the tablethe table.

  46. ETL Page - Update: Use the dropdown to select the Column with Last Modified Value.

  47. ETL Page - Replace with Control: Now with more control options:

  48. >= Max value from the data warehouse

  49. >= Constant

    50. >= SQL Expression. Ex. 1: getdate()-90. Ex. 2: max(ColumnName)-1000)

    Update: The Update filter is automatically set to Max(LastModifiedValueColumn).

  50. Load Engine - Replace AllLoad: The load happens on a temporary table to allow users to query the official table with minimum interruption. Once the load completes, the temporary table becomes the official one. The temporary table is named Schema.TableName_TEMP.

  51. Load Engine - Replace AllLoad: The transition between temporary to official table happens quickly when there are no transformations performed (delete official, rename temporary to official, create <0_EntityID> when applicable).

  52. Load Engine - Replace AllLoad: The transition between temporary to official table happens quickly when there are transformations performed (delete official, insert into official from temp table applying transformations).

  53. Load Engine: Ability to apply transformations after the load (previously, only available during extraction when the source SQL syntax allowed).

  54. Data Warehouse Tables: All tables now show schema (including dbo). <Schema>.<TableName>.

  55. Data Warehouse Tables: Renaming a table only applies to its name, not the schema (schemas might be changed in Source’s Properties and/or Data Warehouse Settings).

  56. Data Warehouse Source: The data warehouse automatically shows as a data source for additional data preparations and transformations.

  57. Data Warehouse Source: This source provides access to all data warehouse tables and views for single-tenant data warehouses (DataSelf Analytics Pro and Enterprise Editions).

  58. Data Warehouse Source: This source provides access to data warehouse tables and views where <SQL schema> = <EntityID> or <SQL schema> = <dbo> in multi-tenant data warehouses (DataSelf Analytics Advanced Edition).

  59. Data Warehouse - Source: Source’s Properties disabled.

  60. Data Warehouse - Source: Source 'sDelete disabled.

  61. Data Warehouse - Source: Use Settings -> Data Warehouse for configuration.

  62. Data Warehouse - Source: Source credentials are sync’ed up its data warehouse configuration.

  63. Data Warehouse - Center Panel: Tables imported from the data warehouse are automatically renamed to <TableName_DW>. Users can then rename them as needed.

  64. Data Warehouse - Center Panel: Tables imported from the data warehouse keep the same SQL schema as from the original source. Ex.: Abc table imported from ERP source using alias as schema will be stored in <Schema>.<TableName>=<ERP>.<Abc>. Upon re-importing it from the data warehouse, this table will be automatically set to <ERP>.<Abc_DW>.

  65. Data Warehouse - Center Panel: When importing a data warehouse table into itself (usually, a mistake), ETL+ will log: "Warning: Data warehouse source table imported into itself. This is usually a mistake. Fix it by renaming the table.". Oct version?

  66. Sources: Addition to native extraction from cloud Microsoft Dynamics CRM. Alpha version.

  67. Source Properties: Checkbox for 0_UnionID. For this version, this field will be populated with “AAA”“001”.

  68. Remove Log Level 1 from local disk. Retrieve live log from the cloud instead. Oct version?ETL Log icon now retrieves live cloud log.

  69. Refresh Batch - WTS: Repeat Task checkbox unchecked by default.

Resolved Issues

  1. aaaa

Known Issues

  1. Importing large XML content into varchar(max) might run into error: server stack limit has been reached.

  2. Deleting all Refresh Batches requires going to ETL main page and back to get a new one created.

  3. Changing the data warehouse of an entity won’t show on the bottom left corner until the entity is closed and re-opened.

  4. Sometimes ETL+ is closing unexpectedly after a data load.

  5. Sometimes ETL+ runs into an “Unhandled exception” error. Close and re-open ETL+.

  6. Locking the ETL+ editing when someone else is editing it doesn’t work in some situations.

  7. ETL+ still tries to send email even when set to Cloud notification, or No Notification.

  8. ETL+ delta extraction from NetSuite ODBC might not work when using dates for filtering.

  9. When opening an entity created in prior ETL+ versions, the data warehouse panel will show the source header blank. To fix this, open the entity in the prior ETL+ version, make any modification and save it. Now open the entity in the latest ETL+ and the source header appears.

  10. Importing tables from the data warehouse works correctly but it might incorrectly log an error message.

...