ETL+ Job Page (formerly Refresh Batch) [v2022.02]
- Cortlandt _
1. Refresh Batch
A Refresh Batch allows the grouping of data warehouse tables and scripts. When executing a batch, only those tables and scripts are executed in the selected order.
Example: Create a batch to load data to sales tables and then refresh the _Sales_Invoice Tableau data source. Create another batch to do the same only for GL tables and Tableau data sources.
1. Refresh Batch Name panel: Lists the available refresh batches.
2. ID: Refresh Batch ID. Use this ID when running the batch from a Window command line or shortcut.
3. Run Refresh Batch: Click the icon to execute a Refresh Batch. Clicking the icon by the ID runs that particular batch. Clicking the icon on the panel header executes the selected Refresh Batch.
4. Refresh Batch Name: Shows the batch name. Double-click on a name to edit it. By default:
- The batch name is: DataSelf <EntityID> #<BatchID>
- When creating WTS tasks via ETL+, the task is named with this name.
- When creating Windows shortcuts via ETL+, the shortcut is named with this name.
5. Rename: Select a Refresh Batch Name and click the icon to rename.
6. Add Refresh Batch: Click the icon to create a new Refresh Batch.
7. Create Windows Shortcuts: Click the icon to create Windows Shortcuts from which you can execute Refresh Batches by running them on Windows Explorer. Clicking the icon creates shortcuts to all batches on the same directory where ETL+ is installed.
Learn more at: ETL+ Command Line Interface [v2021.09]
8. Delete Refresh Batch: Click its icon to delete a Refresh Batch.
31. Load Type: Controls the data warehouse table load.
- Default: Uses the Load Type configuration on the ETL+ main page.
- Forced Load All: Forces this batch’s data warehouse tables to process Load All load. Use case example: for tables with delta load, force a load all in a less frequent schedule to capture deleted records.
Learn more at https://dataself.atlassian.net/wiki/spaces/DS/pages/1571225705 .
9. Refresh Batch Steps
Defines the steps for each refresh batch, including the maintenance of scripts for Tableau, command line and cloud scripts.
9. Refresh Batch Steps panel: This panel lists the objects that belong to the selected Refresh Batch on the left panel, and their execution order. In other words, running a Refresh Batch on the left panel will only execute the objects with checked checkboxes on the listed order.
10. Object Checkboxes: Use the checkboxes to define which objects belong to the selected refresh batch on the left panel.
11. Object Name: Lists the names of each ETL+ object.
12. Move Up/Down: Allows the reordering of a Refresh Batch Step execution order. Select a step → Click & hold the Up or Down icon to quickly reorder the line. Use case: move to the top the tables and Tableau extract to finish before other steps.
13. Add Script: Click the icon to create a new script, select one of the options below and follow the prompts.
Tableau Extract Refresh: Runs Tableau Extract Refreshes when the client’s Tableau and data warehouse are hosted by DataSelf. Enter the case-sensitive parameters below:
Name of an existing Tableau Extract available.
Project (optional): Leave blank if data sources in the Tableau project = ETL+ EntityID.
Parent Project (optional): Leave blank if data sources are NOT stored in a Tableau sub project.
Tableau Site (optional): Leave blank if your Tableau site = ETL+ EntityID.
This version assumes the Tableau Extracts are at:
Production: https://dataselfbi.com
NFR licenses: https://bi1.dataself.com
Tableau Extract Refresh for Tableau Online, Private-cloud or On-premise:
Use ETL+ Command Line script (next item below) approach to run a Windows batch to refresh your extract in sync with table loads.
Visit the following page for Tableau syntax to be used in your batch files:
https://help.tableau.com/current/pro/desktop/en-us/extracting_TDE.htm
Example: tableau refreshextract -s https://tlb.com -u doe -p "abc" --datasource "TdsName"
Command Line: Runs a command line script on the computer where ETL+ is running. Enter the following parameters:
Alias name.
The script to run as a Windows Command Line (such as C:\MyFolder\MyScript.bat).
Arguments (optional): Arguments to the command line.
Start at (optional): Folder to run the script (usually not needed).
Cloud Script: Runs a DataSelf Cloud Script (typically for running Tableau, Python and other data extractions or Automatic Distribution scripts). Contact DataSelf to discuss.
Enter the following parameters: Script name.
When triggered, this runs the following on the dw3 cloud server: “C:\DataSelf\CloudScript\<EntityID>_<ScriptName>.bat”.
14. Delete Script: Select a script and click this icon to delete it.
15. Run Script: Select a script and click this icon to execute it.
16. Configure Script: Select a script and click this icon to configure it.
17. All Data Warehouse Tables (including new ones): Selecting this checkbox will automatically add new data warehouse tables to the selected Refresh Batch.
18. Tables: Data warehouse tables show with that icon.
19. Tableau Extract Refresh Script: Tableau Extract Refresh shows with that icon.
20: Command Line Script: Command Line Scripts show with that icon.
21. Cloud Script: Cloud Scrip show with that icon.
22. Windows Task Scheduler (WTS)
Important: This section is only applicable to ETL+ installations running official data extractions.
Creates/overwrites tasks in WTS to execute Refresh Batches on schedule.
22. Windows Task Schedule panel.
23. Shows the latest configured WTS schedule for the selected Refresh Batch. Currently, there’s no indication in ETL+ if the WTS task has been created and/or is enabled.
24. WTS Settings: Click the button to configure and apply WTS settings for the selected refresh batch.
- See image below for details.
- Be sure to enter a local Windows Admin user in the Run as credentials.
- Clicking the Apply button will create/overwrite the WTS task named as the Refresh Batch Name.
- In some systems, you might need to run ETL+ as Windows Administrator to apply these tasks properly.
- This page works the same way as WTS settings. You might want to adjust the task settings directly in WTS. If you do so, do not press Apply on this page since it’ll overwrite the WTS task with the same name.
Other
25. Help: Click the ? icon to view the help page.
26. Settings: Click the icon for ETL+ settings. Click here for details.
27. ETL+ Build:
Version numbering: vYYYY.MM.DDBB. Ex.: v2021.08.2703 means the 3rd build on 08/27/2021.
ETL+ is available in 32 and 64 bits. See xxx for details.
28. Bottom left corner:
User: your ETL+ username currently logged to ETL+.
Entity: the Entity ID , Data Warehouse name connected to this ETL+ session. Ex.: abc,aa_dw3 = entity abc, data warehouse name aa_dw3.
29. Back: Go back to ETL+ main page.
30. Save: ‘*' shows on this button when this ETL+ is being edited. Click Save to save the changes. Or click Back and Yes to 'Leave without saving?' prompt. Clicking icons # 13, 14 and 19 automatically save changes.
Access
Access this ETL+ page (window) from https://dataself.atlassian.net/wiki/spaces/DS/pages/1570013215