ETL+ Agent (DataSelf Agent)

Owned by Cortlandt _
Jun 24, 2023
3 min read

DataSelf Agent Vision

DataSelf Agent and DataSelf ETL+ is a well-integrated suite of data management tools that work across computers and networks. The tools allow easy, consistent, secured, centralized, and fast ways to manage and troubleshoot extractions from an organization's data silos (on-premises, private and public cloud). The Agent runs as a service on Windows machines and monitors requests from local and remote authorized ETL+ users to kick off ETL+ tasks such as running scheduled Jobs, previewing and/or loading data from individual tables, mapping new tables and columns from source systems, etc.

DataSelf Agent in Action

The Agent’s remote administration capability dramatically reduces configuration and maintenance tasks such as changes to data mapping, ad-hoc data refreshes, and schedule changes. With the Agent, ETL+ users can trigger these functions for all of their cloud and on-premises data sources without logging into other computers such as servers with on-premises or private-cloud data sources. The Agent is a reliable, remotely managed, secure, and integrated replacement for Windows Task Scheduler (WTS) to run ETL+ jobs, such as nightly data refreshes. Clients can decide to leverage the Agent or WTS.

The Agent uses the same outbound cloud functionality as ETL+. The Agent frequently reads from its cloud metadata if any other ETL+ installation has requested either an immediate local table load or a change to the regular jobs schedule. Thus, a user running ETL+ on a remote location can securely request scheduling changes or ad hoc table load tasks to be completed by their on-premise ETL+ installations.

Use Case Example

Mary’s boss emails her asking for a refresh in a KPI now. At a conference hotel, she runs ETL+ on her laptop (availability on a web browser coming soon), and requests a data refresh from her on-premises ETL+ install that will update the desired KPI. Behind the scenes, her ETL+ writes the data refresh request on ETL+ cloud metadata instantly. On her company’s on-premises server, ETL+ Agent reads the cloud metadata every few seconds to see if there are requests queued. Upon reading Mary’s request, the Agent executes it on the on-prem ETL+. Once the request execution completes, the on-premises ETL+ notifies the cloud that Mary’s request has been finished (including a detailed cloud log). Mary sees that the request has finished on her ETL+ and notifies her boss that the KPI has been refreshed.

Product Name

Name: DataSelf Agent 
Nickname: DS Agent, or Agent
Application name (file name): DataSelf Agent.exe

Agent Roadmap

ETL+ v2022.08: Agent initial release

  • Agent assignment for all or individual scheduled jobs.

  • Allowing users to enable/disable scheduled jobs from anywhere.

  • The Agent runs scheduled jobs by monitoring the cloud metadata every 10 secs.

  • An Agent install can manage and run multiple Jobs. Every Job can only be managed and run by a unique Agent install.

  • To set up a Job to run on a different computer, the user has to go to that computer and deploy the Agent (instructions here TBD).

  • The Agent is automatically initiated by a Windows Task Scheduler task and must run with a Windows Local Admin user (configured in ETL+).

  • The Agent introduces the concept of ETL+ Production installs where the locally deployed Agent runs scheduled jobs and remote tasks. ETL+ Production installs must have direct access to source systems and have the Agent locally deployed and running with a local Windows Admin user.

  • The Agent captures the Window Device name in the metadata to associate the ETL+ Entity and Job IDs to run by the Agent.

  • The Agent becomes the default ETL+ scheduling tool (WTS is optional).

  • The Agent date/times are in PST, as shown on the ETL+ Agent UI.

  • Users can remotely run a Job on demand.

  • Users can remotely terminate a Job currently running.

  • The Agent updates the cloud metadata with Jobs' running/idle status.

  • The Agent creates a local log file in the same folder where the Agent is.

 

v2022.08