ETL page: title shows the connected “Data Warehouse: [<SqlServerName>].[<Database>]”
ETL page: Settings icon updated.
ETL Page - Find: Added Find feature to quickly highlight matches on sources, data warehouse tables, and SQL Statements.
ETL Page - Find: Click the Find icon to expose/collapse the search box.
ETL Page - Find: Find is case insensitive for source, table, and column names, and is case sensitive for ETL SQL Statements (to be improved in future releases).
ETL Page: click (Ctrl + F) to open the Find box.
ETL Page: click (Ctrl + G) to open the Log page.
ETL Page: click (F1) to go to Help.
ETL page, left panel renamed to “ETL Source”.
ETL Source - MS Access: Added MS Access as a new source type.
ETL Source - MS Access: Enter file path and password if required.
ETL Source - MS Access: Replace delta refresh available.
ETL Source - MS Access: Upsert delta refresh available.
ETL Source - PostgreSQL: Added PostgreSQL as a new source type.
ETL Source - PostgreSQL: Enter server address, port, credentials, and database name.
ETL Source - PostgreSQL: Replace delta refresh available.
ETL Source - PostgreSQL: Upsert delta refresh available.
ETL Source - BigQuery ODBC: Support and adjustments to BigQuery ODBC.
ETL Source - Google Sheet (beta): Support to extract data from Google Sheets. 64bit beta version.
ETL Source - Google Sheet (beta): Enter Spreadsheet ID, Target (optional). More instructions to be provided.
ETL Source - Google Sheet (beta): Date fields are automatically converted to varchar(max) to prevent failures with invalid dates in MS SQL. Users can enter SQL Expressions on the Design page to convert invalid dates to valid values.
ETL Source - MySQL (beta): Support to extract data from MySQL. 64bit beta version.
ETL Source - MySQL (beta): Enter server name, port, credentials, and database name.
ETL Source - Oracle (beta): Support to extract data from Oracle. 64bit beta version.
ETL Source - Oracle (beta): Enter Default or TNS, host name, port, credentials, and service name.
ETL Source: Deleting ETL+ Sources deletes its mapped tables on the center panel and now also the physical data warehouse tables. User is prompted to confirm the physical table deletion.
ETL Source: select a source → (Ctrl + i) opens its Properties.
ETL Source: double-click a table to add/map it to the load process.
ETL Source: (F5) to refresh a source.
ETL Page, center panel: Renamed to ETL Objects. More descriptive of what the panel is about. Objects currently include source headings (to group sources' tables together), SQL schemas, tables, and table columns.
ETL Objects: Renaming tables on the center panel renames the physical data warehouse tables.
ETL Objects: Duplicating tables on the center panel duplicates the physical data warehouse tables. This is helpful when users want to perform tests on a table without affecting the official one.
ETL Objects: Deleting tables on the center panel deletes the physical data warehouse tables. User is prompted to confirm the physical deletion.
ETL Objects: zData Warehouse heading renamed to zData Warehouse (Reimports).
ETL Objects: Job icon updated.
ETL Objects: select a table → (Ctrl + Del) key to delete.
ETL Objects: select a table → (Ctrl + 2) key to duplicate.
ETL Objects: select a table → (F2) key to rename.
ETL Objects: select a table → (Ctrl + D) key to open Design.
ETL Objects: select a table → (Ctrl + O) key to re-load and show Columns and data types.
ETL Objects: select a table → (Ctrl + P) key to preview.
ETL Objects: select a table → (Ctrl + L) key to Load Now.
ETL Objects: click (Ctrl + J) to open the Job page.
ETL Objects: Added ... icon to the panel header for additional non-popular actions.
ETL Objects: select a table → ... → Show Columns.
ETL Objects: select a table → ... → Download to CSV.
ETL Objects: ... → Load All Columns.
ETL Objects: ... → Expand All / Collapse All for expanding/collapsing the source headers.
ETL Objects - CSV: Added Export to CSV tables that have been loaded to the data warehouse.
ETL Objects - CSV: Right-click an ETL Object table → More Actions → Download to CSV downloads data in CSV format into the folder: ETL+ folder/DS_Downloads/<EntityID>.<TableName>.csv (ex.: abcinc.TName1.csv).
ETL page: click (Ctrl + S) to Save.
ETL page: click (Ctrl + O) inside of ETL SQL Statement to Confirm.
Job page: “Refresh Batch” renamed to “Job” - this feature can run scripts that are not “refreshes”.
Design page: When changing the Data Type of a column, ETL+ posts the SQL expressions automatically applied. Users can edit the expression.
Design page: After a SQL expression is posted to a column, if a user changes the column’s Data Type again, ETL+ will prompt if it should update the SQL expression.
Design page: Loading of MS SQL Timestamp columns automatically converted to char(23).
DataSelf Agent: Greatly anticipated ETL+ auxiliary application to execute scheduled and remote tasks. Long-term vision: a user runs ETL+ on her laptop at a hotel and requests a data refresh from her on-premises ETL+ install, and orders a margarita (drink orders not included in ETL+ lol). Their on-prem DS Agent receives and executes the request on the on-prem ETL+. And then it notifies the user when the refresh is complete. Magic happens while the margarita buzz kicks in!
Agent: A locally deployed Agent can run scheduled jobs.
Agent: A locally deployed Agent can run on-demand jobs.
Agent: Because of the Agent framework, users can now stop a running job no matter if it’s running via ETL+ UI or via an Agent.
Agent: An ETL+ Job can be associated with one Agent or to WTS.
Agent: ETL+ Production installs must have direct access to source systems and have the Agent locally deployed and running with a local Windows Admin user.
Agent: ETL+ will capture in the metadata the Windows Device name to associate the ETL+ Entity and Job IDs to run by the Agent.
Agent: When assigning a computer to run the Agent for the first time, ETL+ assigns the username and password saved on the Job → Properties → Agent credentials. This user must have local Windows Admin Rights.
Agent: Runs as a Windows service (in the background) and checks the cloud metadata every 10 secs for instructions to execute.
Agent: Checks the cloud metadata if there are Jobs to run now. The rule is: (Jobs associated to run by this Agent) & Enabled & (Next Run Time <= now).
Agent: Checks the cloud metadata if there are on-demand tasks to execute. Currently, it can execute two tasks: run a Job now, and kill the execution of a running Job.
Agent: DateTime values related to the Agent are fixed in PST. Next release will add flexibility to this setting.
Agent Auto Start: When assigning a computer to run the Agent for the first time, ETL+ creates a WTS task to automatically run the Agent.
Agent Auto Start: On a given computer, only one Agent runs at a time. Attempts to run a second session in parallel will only log the attempt.
Agent Auto Start: The WTS that runs the Agent is scheduled to run every hour. This can be manually changed to run on Windows Startup, and change Setting to stop the app after 3 days.
Agent Protocol: The communication between the Agent and the cloud is via a single metadata table.
Agent Protocol: The Agent and ETL+ use the metadata table to monitor actions such as terminating a running Job.
Agent Scheduling: Once per day schedules.
Agent Scheduling: Repeat task every X days.
Agent Scheduling: Repeat task every X minutes within Y hours.
Agent Scheduling: Future schedules (ex.: run it next year).
Agent app: Right-click on the Agent on Windows Explorer → Properties → Details, see the Agent version on File version.
Agent app - Agent Log.txt: Creates Agent Log.txt log file on the same directory where the Agent is. This file contains high-level Agent debugging and monitoring logs.
Agent app - Agent Log.txt: Logs the Agent version upon its start.
Agent app - Agent Log.txt: Logs failed attempts to run the Agent (it’s usually ok since only one Agent can run on each computer).
Agent app - Agent Log.txt: Logs the ETL+ command line running a Job.
Agent app - Agent Log.txt: Logs the end of a Job run.
Job: Added Status column on the job list to show which app runs the Job. Options: WTS (Windows Task Scheduler) to DataSelf Agent.
Job: Status column = WTS Enabled: The job is assigned to run via Windows Task Scheduler like in prior ETL+ versions.
Job: Status column = WTS Disabled: The job is assigned to run via Windows Task Scheduler like in prior ETL+ versions. However, ETL+ will not run the Job because it’s disabled even if/when WTS tries to run it.
Job: Status = Agent Offline: The Job is assigned to an Agent that is either not running, or is unable to access DataSelf cloud.
Job: Status = Agent Ready: The Job is assigned to an Agent and is ready to run scheduled or on-demand Jobs.
Job: Status = Agent Disabled: The Job is assigned to an Agent but it won’t run the Job.
Job: Status = Agent Running: The Job is running.
Job: Status = Agent Stopping: The Job is/was running, but the user just clicked End Job in ETL+.
Job: Added Load column showing Default or Forced Load All.
Job: Added Start at column showing a summary of the schedule description.
Job: Moved Run Job icon to the left panel header.
Job: Added Job Properties icon on the left panel header.
Job: Added ... (More Actions) icon.
Job: Added ... → Run Job via Agent icon. This sends a message to the Agent to run the Job on demand.
Job: Added ... → End Job icon.
Job: Moved Rename Job icon under the ... icon.
Job: Moved Delete Job icon under the ... icon.
Job: Moved Create Win Shortcut icon under the ... icon.
Job - Right-click: Same changes as the icons above.
Job - Pop up: Shows Job name
Job - Pop up: Shows Agent to run the Job. Computer Name if DataSelf Agent will run the Job.
Job - Pop up: Shows Job Schedule.
Job - Pop up: Shows Load Type.
Job - Pop up: Shows Last Run Time.
Job - Pop up: Shows Next Run Time.
Job: Removed Windows Task Scheduler (WTS) section.
Job: Removed WTS Settings.
Job: Clicking the Job Properties icon shows a section for Schedule, Load Type, and App.
Job: Settings icon updated.
Job - Find: Added Find feature to quickly highlight matches on Jobs and Job Steps.
Job: click (Ctrl + F) to open the Find box.
Job: click (Ctrl + G) to open the Log page.
Job: click (F1) to go to Help.
Job: Standardized user interaction as main ETL page so right-clicking on panel objects pops up the available features. Ex.: right-click a job → Run Job.
Job: Added Refresh icon that updates Job status.
Job: Added Expand icon to show additional scheduling columns.
Job: Expand additional column: Last Run.
Job: Expand additional column: Next Run.
Job: Expand additional column: Server.
Job: The Refresh feature enters in a 10-sec auto-refresh mode when the columns are expanded.
Job Steps: now shows the Job <ID> on the header.
Job Steps: Added Move to Top icon/feature.
Job Steps: Added Move to Bottom icon/feature.
Job Steps: select a script → (Ctrl + i) to open its properties.
Job Steps: select a script → (Ctrl + Del) key to delete.
Job Steps: select a row → (Ctrl + D) key to push it down one row.
Job Steps: select a row → press and hold the (Ctrl + D) key to push it down.
Job Steps: select a row → (Ctrl + U) key to push it up one row.
Job Steps: select a row → press and hold the (Ctrl + U) key to push it up.
Job Steps: (Ctrl + T) to create a new Tableau Extract Refresh script.
Job Steps: New script Download to CSV to run on a schedule. Select a Job → + on the Steps panel → Download to CSV → select a table -> Save.
Job Steps: Right-click on a Job Step shows a popup with the icons/actions above.
Job Properties - Schedule: Layout re-arranged.
Job Properties - Schedule: Added Enabled checkbox. Unchecking it will prevent the Job from running via the command line (typically, this is done by the Agent and WTS).
Job Properties - Schedule: The Start time zone shows PST when the Agent is selected.
Job Properties - Schedule: The Start doesn’t show a time zone when WTS is selected. This assumes it’ll be run based on the time zone of the WTS running the Job.
Job Start at column shows PST when the Agent is selected. It doesn’t show a time zone when WTS is selected (this assumes it’ll be run based on the time zone of the WTS running the Job).
Job Last Run column shows PST when the Agent is selected. It doesn’t show a time zone when WTS is selected (this assumes it’ll be run based on the time zone of the WTS running the Job).
Job Next Run column shows PST when the Agent is selected. It doesn’t show a time zone when WTS is selected (this assumes it’ll be run based on the time zone of the WTS running the Job).
Job App: New section to manage the application running Jobs.
Job App: In a new installation, the App dropdown has the following options:
Deploy the Agent on this device (<DeviceName>)
WTS.
Job App: To deploy the Agent on a device, select Deploy the Agent on this device (<DeviceName>) on the dropdown, enter credentials with local Windows Admin rights on the right panel, check Assign the Agent on this device for all Jobs if applicable, press Apply. If the Agent hasn’t been downloaded yet to the ETL+ folder, this will download the Agent to the ETL+ folder, create a WTS task to run the Agent automatically during reboots, and run the Agent as a service.
Job App: Repeat the step above to re-deploy the Agent - in case the Agent is running as expected.
Job App: To upgrade the Agent, delete DataSelf Agent.exe from the ETL+ folder and deploy the Agent again.
Job App: Once an Agent is assigned to a Job, its App dropdown shows Agent on <DeviceName> or Agent on <DeviceName> (this device). This informs the Agent is assigned to a Job and the Windows Computer name running that Agent.
Job App: If there are multiple Agents available to run a Job, if required, use the App dropdown to assign a different Agent to a Job. For instance, the production server will be down for a few days, and the user assigns a Job to a backup server to run the Job during that period.
Job App: ETL+ and Job refresh will work the same way as in prior versions when the App dropdown shows WTS.
Job App: When the App = WTS, the right panels shows the credentials for the local Windows Admin user and the Apply button (similar to prior versions).
Job App: When the App = WTS, users can disable the scheduled refresh of a Job using the Schedule Enable checkbox. This doesn’t affect the WTS activities per se, but it’ll prevent ETL+ from actively running such disabled Jobs.
Load Engine: Depending on the source system, loading of Date and DateTime columns automatically converts invalid values to NULL.
Load Engine: The delta filter column now creates a physical index in the data warehouse.
Load Engine: PK columns used in delta loads now create physical indexes in the data warehouse.
Load Engine: Adjustments when recalculating MS SQL indexes for the filter column used in tables with Replace. Resulting in significant performance improvement when processing deltas in large tables.
Load Engine: Adjustments when recalculating MS SQL indexes for the filter and PK columns used in tables with Upsert. Resulting in significant performance improvement when processing deltas in large tables.
Command Line: parameter -e <EntityID>. Ex.: -e abccorp
Command Line: parameter -j <JobID>. Ex.: -j 1
Command Line: New command syntax: -e <EntityID> -j <JobID> -ui <Page> <action>.
Command Line: parameter -e <EntityID> -ui load -j <JobID>. Ex.: this runs JobID on a new Load session. This automatically closes once the job completes the refresh.
Command Line: parameter -e <EntityID> -ui log. Ex.: this opens the Log page on a new ETL+ session. This session stays open until the user closes it. Click the Refresh icon to view the latest log rows.
Command Line: parameter -e <EntityID> -ui agent. Ex.: opens the Agent page from an ETL+ session.
Command Line: ETL+ uses the latest locally saved encrypted credentials when running command lines.
Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, it’ll show the message “Another ETL+ is loading this table. Waiting for table release.”
Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, every 15 secs it’ll show the message “Checking table release…”
Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, but the other entity isn’t active for over 90 seconds, the new entity takes over the table load.
Load Engine: The mechanism to check for “Another ETL+ is loading this table. Waiting for table release.” checks for which Statement.RefreshRunID is loading the statement and when was the last date/time the loading was active on Statement.LastRefreshBg.
Load Engine: Checking if Log tables exist only during the opening of an entity (instead of during the load of every table).
Load Engine: Shows when the load started (metadata date/time).
Job Loading Page: Runs on its own window (runs via command line).
Agent Monitor Page: Opens Agent page on its own window (runs via command line).
Security: New passwords require at least 12 complex characters (up from 8).
Security: New passwords now accept # @ as part of special characters.
Security: When ETL+ can’t connect to its metadata: "Unable to connect to cloud metadata via https and SSL SQL. ETL+ needs to whitelist your public IP, or your firewall or antimalware is preventing ETL+ connection to the cloud. Start ETL+ IP Whitelisting?"
Security: ETL Page - ETL SQL Statement: For multi-tenant data warehouses, clicking Confirm validates that only dbo and entityID schemas can be read from zData Warehouse.
Log Page: Runs on its own window (runs via command line). This allows users to keep it open for monitoring the results of load tests.
Log Page: It will remain open even if the user closes the main ETL+ session. This can be useful for using the Log page for monitoring ETL+ loads exclusively.
Log Page: Click Close to close it.
Log Page: Added Rows Retrieved parameter. Enter a new value and press Refresh icon.
Log Page: (F5) shortcut to refresh Log page.
Log Page: Titles and radio buttons changed to ETL Log and Table Stats.
Log Page: Table Stats layout rearranged and some labels renamed for easier viewing.
Log Page: Default sorting of Log pages is the Index column.
Log Page: User Admin users have access to an additional column that shows the (unmasked) Source Error message returned by source system.
Log - Table Stats: Table row counting uses SQL metadata (instead of count(*)). Faster performance.
Logging: Checking if Log tables exist only during the opening of an entity (instead of during a load of every table).
Settings - Refresh Email Notification: Removed email attachments. No longer necessary with cloud logging.
Settings: Debug Level dropdown shows a summary of what the log level does.
Settings: Debug Off now removes the log file on the local HD. For saving disk space and avoiding requiring ETL+ to Run as Admin when running from Windows-protected folders
Settings: Debug Off no longer counts and records the number of rows before and after a table load. Performance improvement.
Settings: New Debug Level 0 adds the log file on the local HD plus counts and records the total number of rows before and after a table load.
Settings: Debug Level 1 adds Level 1 log plus Level 0 features.
Settings: Debug Level 2 adds Level 2 log plus Level 0 features.
Settings - Restore ETL SQL Statement As Of: New dropdown list on Settings allows users to retrieve prior ETL SQL Server Statements. Useful to recover deleted and changed statements from the past.
Login: Forgot Password? button. It’ll email the password when initiated from whitelisted IPs with a valid username (email address).
ETL+ Plain Deployment: New framework for an on-premises data warehouse that easily loads the zDateUntreated table from a cloud Excel file.
Cloud Scripts: Increased to up to 7 parallel processes.
Metadata: Redesigned metadata for improved security, reporting, and indexing. This makes prior ETL+ versions incompatible and separated with the new version.
Metadata AWS platform: Doubled the CPU capacity.
Metadata AWS platform: Doubled the RAM capacity.
Login within the same month release: Starting on this version, users will be able to upgrade and downgrade ETL+ seamlessly within the same ETL+ month release.
Login within the same month release: Removed prompt about opening an entity with a different metadata version within the same ETL+ month release.
Login with a newer month’s release: Opening an entity with a newer ETL+ version than the last opened release is seamless.
Login with an older month’s release: ETL+ will suggest the user to upgrade ETL+ and opens the ETL+ Upgrade page.
Creation of .ink files in \ProgramData\DataSelf\ETLplus removed.
zEntity: Process to guarantee that there’s no duplication of EntityID values.
Admin user: Remembers the last admin credentials in DataSelf protected computers.
Admin user: Renamed “Tableau Deploy” to “Cloud Script” on the Client Information page.
Admin user: Added Unlock Editing button to release an entity locked by a user.
Support to Windows Server 2022.
UI Pages: Data warehouse was removed from the bottom left corner since it’s prominently listed on the ETL main page.