ETL page, left panel renamed to “ETL Source”.
ETL Page, center panel: Renamed Data Warehouse to ETL Objects. More descriptive of what the panel is about. Objects currently include source headings (to group sources' tables together), SQL schemas, tables, and table columns.
ETL Page: Renaming tables on the center panel renames the physical data warehouse tables.
ETL Page: Duplicating tables on the center panel duplicates the physical data warehouse tables. This is helpful when users want to perform tests on a table without affecting the official one.
ETL Page: Deleting tables on the center panel deletes the physical data warehouse tables. User is prompted to confirm the physical deletion.
ETL Page: Deleting ETL+ Sources deletes its mapped tables on the center panel and now also the physical data warehouse tables. User is prompted to confirm the physical table deletion.
ETL page, title now shows the connected “Data Warehouse: [<SqlServerName>].[<Database>]”
ETL Page, ETL Objects: zData Warehouse heading renamed to zData Warehouse (Reimports). More descriptive of the tables under this header.
ETL page, left pane shortcut: select a source → (Ctrl + i) opens its Properties.
ETL page, left pane shortcut: double-click a table to add/map it to the load process.
ETL page, left pane shortcut: (F5) to refresh a source.
ETL page: Job icon updated.
ETL page: Settings icon updated.
ETL Page: Find feature to quickly highlight matches on sources, data warehouse tables, and SQL Statements.
ETL Page: Click the Find icon to expose/collapse the search box
ETL page, center pane shortcut: select a table → (Ctrl + Del) key to delete.
ETL page, center pane shortcut: select a table → (Ctrl + 2) key to duplicate.
ETL page, center pane shortcut: select a table → (F2) key to rename.
ETL page, center pane shortcut: select a table → (Ctrl + D) key to open Design.
ETL page, center pane shortcut: select a table → (Ctrl + O) key to load its Columns.
ETL page, center pane shortcut: select a table → (Ctrl + P) key to preview.
ETL page, center pane shortcut: select a table → (Ctrl + L) key to Load Now.
ETL page, center pane: Added ... icon to the panel header for additional non-popular actions.
ETL page, center pane: select a table → ... → Show Columns.
ETL page, center pane: select a table → ... → Load All Columns.
ETL page, center pane: select a table → ... → Expand/Collapse All source groups.
ETL page, center pane: select a table → ... → Download to CSV.
ETL page shortcut: click (F1) key to open Help.
ETL page shortcut: click (Ctrl + F) to go to the Find box.
ETL page shortcut: click (Ctrl + G) to open the Log page.
ETL page shortcut: click (Ctrl + J) to open the Job page.
ETL page shortcut: click (Ctrl + S) to Save.
ETL page shortcut: click (Ctrl + O) inside of ETL SQL Statement to Confirm.
ETL Page: Right-click a data warehouse table → Download to CSV downloads the data in CSV format into ETL+ folder/DS_Downloads/<TableName>.csv.
Job: “Refresh Batch” renamed to “Job” - this feature can run scripts that are not “refreshes”.
DataSelf Agent: Greatly anticipated ETL+ auxiliary application to execute scheduled and remote tasks. Long-term vision: a user runs ETL+ on her laptop at a hotel and requests a data refresh from her on-premises ETL+ install, and orders a margarita (drink orders not included in ETL+ lol). Their on-prem DS Agent receives and executes the request on the on-prem ETL+. And then it notifies the user when the refresh is complete. Magic happens while the margarita buzz kicks in!
Agent: Introduces the concept of ETL+ Production installs where the locally deployed Agent runs scheduled jobs and remote tasks.
Agent: An ETL+ Job can be associated to one Agent to run it on schedule or on-demand, or via WTS.
Agent: ETL+ Production installs must have direct access to source systems and have the Agent locally deployed and running with a local Windows Admin user.
Agent: ETL+ will capture in the metadata the Windows Device name to associate the ETL+ Entity and Job IDs to run by the Agent.
Agent: When assigning a computer to run the Agent for the first time, ETL+ assigns the username and password saved on the Job → Properties → Agent credentials. This user must have local Windows Admin Rights.
Agent: When assigning a computer to run the Agent for the first time, ETL+ creates an WTS task to automatically run the Agent when restarting the computer.
Agent: Runs as a Windows service (in the background) and checks the cloud metadata every 10 secs for instructions to execute.
Agent: Checks the cloud metadata if there are Jobs to run now. The rule is: (Jobs associated to run by this Agent) & Enabled & (Next Run Time <= now).
Agent: Checks the cloud metadata if there are on-demand tasks to execute. Currently, it can execute two tasks remotely: run a Job now, and kill the execution of a running Job.
Agent: DateTime values related to the Agent are fixed in PST. Next release will add flexibility to this setting.
Agent Protocol: The communication between the Agent and the cloud is via a single metadata table.
Agent Protocol: The Agent and ETL+ use the metadata table to monitor actions such as terminate a running Job.
Job: Added Status column on the job list to show which app runs the schedule of a job. Options: WTS (Windows Task Scheduler) and DataSelf Agent.
Job: Status column showing WTS: The job runs via Windows Task Scheduler. ETL+ currently can only create and overwrite WTS tasks via the Schedule button.
Job: Status column showing the Agent status: Ready, Disable, Running, Stopping.
Job - Icons: Added Run Job via Agent. This sends a message to the Agent to run the Job on demand.
Job - Icons: Added ... (More Actions).
Job - Icons: Added End Job via Agent icon under the ... icon.
Job - Icons: Added Enable icon under the ... icon.
Job - Icons: Added Disable icon under the ... icon.
Job - Icons: Moved Rename Job icon under the ... icon.
Job - Icons: Moved Delete Job icon under the ... icon.
Job - Right-click: Same changes as Job - Icons above.
Job: Agent column showing Agent Ready: This means the Agent will run that job on its schedule.
Job: Agent column showing Agent Disabled: This means the Agent will not run that job on its schedule.
Job: Agent column showing Agent Running: This happens after a user triggers an on-demand Run Job via Agent. Currently, this Running status won’t know when the Job is completed. Users can monitor the Job running status via the Log or by the refresh email.
Job: Agent column showing Agent Stopping: This happens after a user triggers an on-demand End Job via Agent. Currently, this Stopping status won’t know when the Job ends. Users can monitor the Job running status via the Log.
Job - Pop up: Shows Job name
Job - Pop up: Shows Agent to run the Job. Computer Name if DataSelf Agent will run the Job.
Job - Pop up: Shows Job Schedule.
Job - Pop up: Shows Load Type.
Job - Pop up: Shows Last Run Time.
Job - Pop up: Shows Next Run Time.
Job: Removed Windows Task Scheduler (WTS) section.
Job: Removed WTS Settings.
Job: Added Properties icon which shows a section for Schedule, Load Type, and Agent.
Job - Schedule: Added radio buttons to set which engine runs schedules: WTS or DataSelf Agent.
Job - Schedule: When radio button = WTS, the feature works like in prior ETL+ releases.
Job - Schedule: When radio button = DataSelf Agent, the Agent will be responsible for running the Job’s schedule.
Job - Agent: Press the Apply button to associate a Job to run on this computer.
Job: Added icon to open the Log page.
Design: When changing the Data Type of a column, ETL+ posts the SQL expressions automatically applied. Users can edit the expression.
Design: After a SQL expression is posted to a column, if a user changes the column’s Data Type again, ETL+ will prompt if it should update the SQL expression.
Design: Loading of MS SQL Timestamp columns automatically converted to char(23).
Load Engine: Depending on the source system, loading of Date and DateTime columns automatically converts invalid values to NULL.
Load Engine: The delta filter column now creates a physical index in the data warehouse.
Load Engine: PK columns used in delta loads now create physical indexes in the data warehouse.
Load Engine: Adjustments when recalculating MS SQL indexes for the filter column used in tables with Replace. Resulting in significant performance improvement when processing deltas in large tables.
Load Engine: Adjustments when recalculating MS SQL indexes for the filter and PK columns used in tables with Upsert. Resulting in significant performance improvement when processing deltas in large tables.
Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, it’ll show the message “Another ETL+ is loading this table. Waiting for table release.”
Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, every 15 secs it’ll show the message “Checking table release…”
Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, but the other entity isn’t active for over 90 seconds, the new entity takes over the table load.
Load Engine: Checking if Log tables exist only during the opening of an entity (instead of during the load of every table).
Load Engine: Added MS Access as a new source type.
Load Engine - MS Access: Enter file path and password if required.
Load Engine: Added PostgreSQL as a new source type.
Load Engine - PostgreSQL: Enter server address, port, database, username and email.
Load Engine - PostgreSQL: Delta refreshes based on date/time are operational.
Load Engine: Support and adjustments to BigQuery ODBC.
Load Engine: Shows when the load started (metadata date/time).
Command Line: parameter -e <EntityID>. Ex.: -e abccorp
Command Line: parameter -j <JobID>. Ex.: -j 1
Command Line: New command syntax: -e <EntityID> -j <JobID> -ui <Page> <action>.
Command Line: parameter -e <EntityID> -ui load -j <JobID>. Ex.: this runs JobID on a new Load session. This automatically closes once the job completes the refresh.
Command Line: parameter -e <EntityID> -ui log. Ex.: this opens the Log page on a new ETL+ session. This session stays open until the user closes it. Click the Refresh icon to view the latest log rows.
Command Line: parameter -e <EntityID> -ui agent. Ex.: opens the Agent page from an ETL+ session.
Command Line: Engine checks encrypted locally saved credentials when running command lines.
User Interface: Runs a Job on its own window (runs via command line).
User Interface: Opens Agent page on its own window (runs via command line).
Security: New passwords require at least 12 complex characters (up from 8).
Security: New passwords now accept # @ as part of special characters.
Security: When ETL+ can’t connect to its metadata: "Unable to connect to cloud metadata via https and SSL SQL. ETL+ needs to whitelist your public IP, or your firewall or antimalware is preventing ETL+ connection to the cloud. Start ETL+ IP Whitelisting?"
Security: ETL Page - ETL SQL Statement: Clicking Confirm validates zData Warehouse schemas.
Refresh Email Notification: Removed email attachments. No longer necessary with cloud logging.
Log Page: Added Rows Retrieved parameter. Enter a new value and press Refresh icon.
Log Page: (F5) shortcut to refresh Log page.
Log Page: Titles and radio buttons changed to ETL Log and Table Stats.
Log Page: Table Stats layout rearranged and some labels renamed for easier viewing.
Log Page: Default sorting of Log pages is the Index column.
Log Page: Row counting uses SQL metadata (instead of count(*)). Faster performance.
Logging: Checking if Log tables exist only during the opening of an entity (instead of during a load of every table).
Settings: Debug Level dropdown shows a summary of what the log level does.
Settings: Debug Off now removes the log file on the local HD. For saving disk space and avoiding requiring ETL+ to Run as Admin when running from Windows protected folders
Settings: Debug Off no longer counts and records the total number of rows before and after a table load. Performance improvement.
Settings: New Debug Level 0 adds the log file on the local HD plus counts and records the total number of rows before and after a table load.
Settings: Debug Level 1 adds Level 1 log plus Level 0 features.
Settings: Debug Level 2 adds Level 2 log plus Level 0 features.
Restore ETL SQL Statement As Of: New dropdown list on Settings allows users to retrieve prior ETL SQL Server Statements. Useful to recover deleted and changed statements from the past.
Login: Forgot Password? button. It’ll email the password when initiated from whitelisted IPs with a valid username (email address).
Job: New script Download to CSV to run on a schedule. Select a Job → + on the Steps panel -> Download to CSV → select a table -> Save.
Job: Find feature to quickly highlight matches on the Steps panel.
Job: Click the Find icon to expose/collapse the find box.
Job: Standardized user interaction as main ETL page so right-clicking on panel objects pops up the available features. Ex.: right-click a job → Run Job.
Job, left panel: Standardized user interaction as main ETL page so right-clicking on panel objects pops up the available features. Ex.: right-click a job → Run Job.
Job, left panel now has icons on the panel header (instead of inside of the panel).
Job, left panel now has two sets of icons, the left is active when selecting individual jobs, the right is independent of individual jobs
Job, Step panel: now shows the Job <ID> on the header.
Job, Step panel: Added Move to Top icon/feature.
Job, Step panel: Added Move to Bottom icon/feature.
Job, Steps pane shortcut: (Ctrl + F) to go to the Find box.
Job, Steps pane shortcut: select a script → (Ctrl + i) to open its properties.
Job, Steps pane shortcut: select a script → (Ctrl + Del) key to delete.
Job, Steps pane shortcut: select a row → (Ctrl + D) key to pushing it down one row.
Job, Steps pane shortcut: select a row → press and hold the (Ctrl + D) key to push it down.
Job, Steps pane shortcut: select a row → (Ctrl + U) key to pushing it up one row.
Job, Steps pane shortcut: select a row → press and hold the (Ctrl + U) key to push it up.
Job, Steps pane shortcut: (Ctrl + T) to create a new Tableau Extract Refresh script.
ETL+ Plain Deployment: New framework for an on-premises data warehouse that easily loads the zDateUntreated table from a cloud Excel file.
Cloud Scripts: Increased to up to 7 parallel processes.
Metadata: Redesigned metadata for improved security, reporting, and indexing.
Admin user: Remembers the last credentials in DataSelf protected computers.