Administer > Utility Tools > ETL Toolkit

ETL Toolkit

You use IDE to develop your content. After the IDE generates all content artifacts, you can use the ITBA application to install the Content Pack, activate the data source, and run the ETL for data loading. But the IDE only supports very basic ETL transformation use cases, so if you want to do additional customization, you must change the Vertica script, and test it yourself. To do so, use the ETL Toolkit .

After the data is loaded into the Target database, you verify that the data is correct. If the data is not correct, check the logs. For details, see Logs and the LogTool Once you have checked the issue in the logs, open the ETL toolkit to make the corrections and check them. You can also clean the data of the last ETL run, using the Clean ETL Data function in the ETL Toolkit, and rerun the ETL.

Recommended: Use the ETL Toolkit only in the Development environment.

ClosedUse Case

  1. The user completes the content development in the IDE. For details, see
  2. The user deploys the content artifacts generated by the IDE.
  3. To test the ETL flow, the user copies the entire folder to the ITBA content folder, and installs the content pack using the Content Pack manager. For details, see Data Management - Install Content Pack
  4. To test the new entity, the user opens the ETL debug tool, runs each flow step by step, until all the steps are running successfully. For details, see below.

After checking that the data is correct, the unit testing is complete.

ClosedEnable the ETL Toolkit

Note It is recommended to check the logs to find out what the probable issues are before using the ETL Tookit. For details, see Logs and the LogTool.

Recommended: 

  • Make sure the ETL Toolkit is disabled when not in use.
  • Use the ETL Toolkit only in the Development environment.

To install the tool, do the following on the ITBA server:

  1. Go to https://HPLN.hpe.com/group/it-business-analytics, click Resources > Tools, and download the relevant version of the dw-etl-tool.ZIP file.
  2. Unzip the ETL ToolKit File and upload the file to $HPBA_HOME/apps.

  3. Run $HPBA_HOME/bin/ enable-dw-etl-tool.sh.

  4. Enter the ETL Toolkit using the glassfish user and password.

ClosedAccess the ETL Toolkit

Open http://<hostname>:10002/dw-etl-tool/services.do
where Username is the glassfish user name and the password is the glassfish password.

ClosedDisable the ETL Toolkit

Recommended: Make sure the ETL Toolkit is disabled when not in use.

To disable the ETL Toolkit, execute the following script:

  1. Go to cd $HPBA_HOME/bin on the ITBA server.
  2. Run the following command: ./disable-dw-etl-tool.sh
  3. Enter the ETL Toolkit application using the glassfish user and password.

ClosedExecute a specific extractor or ETL job step

  1. Make sure you have enabled the ETL Toolkit in the ITBA server.

  2. Login to the ETL Toolkit using the glassfish user and password.

  3. Select the relevant data source instance in the Data Source Instance Name.

    The lower part of the left pane displays the list of relevant Extractors and ETL jobs.

  4. Expand the relevant job to view the detailed steps of the job.

    • For extractor jobs, the following tree opens: <main_extractor_job> > <entity_extractor_job>.
    • For ETL jobs, the following tree opens: <main_ELT_job> > <entity_ETL_job><entity_SQL_Statement_section>
  5. Select the relevant step and click Execute to run the selected step. Click Confirm.

    The lower part of the page displays the running status of the action.

ClosedAbort the running of a specific extractor or ETL job step

  1. Make sure you have enabled the ETL Toolkit in the ITBA server.

  2. Login to the ETL Toolkit using the glassfish user and password.

  3. Select the relevant data source instance in the Data Source Instance Name.

    The lower part of the left pane displays the list of relevant Extractor and ETL jobs.

  4. Expand the relevant job to view the detailed steps of the job.

    • For extractor jobs, the following tree opens: <main_extractor_job> > <entity_extractor_job>.
    • For ETL jobs, the following tree opens: <main_ELT_job> > <entity_ETL_job><entity_SQL_Statement_section>
  5. Select the relevant step (that is currently running) and click Abort.

    Click Confirm.

    The lower part of the page displays the running status of the action.

ClosedView the script details

  1. Make sure you have enabled the ETL Toolkit in the ITBA server.

  2. Login to the ETL Toolkit using the glassfish user and password.

  3. Select the relevant data source instance in the Data Source Instance Name.

    The lower part of the left pane displays the list of relevant Extractor and ETL jobs.

  4. Expand the relevant job to view the detailed steps of the job.

  5. Double-click the relevant <entity_SQL_Statement_section> step to view the details of the script.

    Note This is only available for <entity_SQL_Statement_section> steps.

    You can click the to transfer the script to the Edit pane where you can edit the script.

    You can then click Execute to execute the corrected script to check it.

    Note Make sure you make the same change to the script once you have tested it, as the scripts in the ETL Toolkit and in the ITBA application are not synchronized.

    The lower part of the page displays the running status of the action.

ClosedClean the ETL data

  1. Make sure you have enabled the ETL Toolkit in the ITBA server.

  2. Login to the ETL Toolkit using the glassfish user and password.

  3. Select the relevant data source instance in the Data Source Instance Name.

    The left pane displays the list of relevant Extractor and ETL jobs.

  4. Expand ETL job and select the relevant ETL job.

  5. Click Clean ETL Data to clean all the data in the Target database. Click Confirm.

    The lower part of the page displays the running status of the action.

Tip  

Associated tables data are deleted after running Clean ETL Data. For detailed information, see the $HPBA_HOME/glassfish/glassfish/domains/BTOA/logs/dwhETLToolkit.log.

ClosedUninstall a Content Pack

Tip It is NOT recommended to take this action Uninstall Content Pack in the production environment.

To uninstall a Content Pack:

  1. Make sure you have enabled the ETL Toolkit in the ITBA server.

  2. Login to the ETL Toolkit using the glassfish user and password.

  3. Click Uninstall Content Pack.

  4. Select the Content Pack and click Confirm.

  5. Check the result.

Tip  

Associated tables are droped after running Uninstall Content Pack. For detailed Information, see the $HPBA_HOME/glassfish/glassfish/domains/BTOA/logs/dwhETLToolkit.log

ClosedETL Toolkit Page

User interface elements are described below (when relevant, unlabeled elements are shown in angle brackets):

UI Element

Description

Data Source Instance Name Select the relevant data source instance name. The list of instances corresponds to the data source instances you activated in the ADMIN > Data Management >  Connect Data Source. For details, see Data Management - Connect the Data Sources.
Execute

Runs the selected step of the extractor or ETL job.

Click Confirm.

Abort

Aborts the currently running step of the extractor or ETL job.

Click Confirm.

<View Script>

Displays the details of the script corresponding to the selected step.

Note This is only available for <entity_SQL_Statement_section> steps.

You can click the to transfer the script to the Edit pane where you can edit the script.

You can then click Execute to execute the corrected script to check it.

Note Make sure you make the same change to the script once you have tested it, as the scripts in the ETL Toolkit and in the ITBA application are not synchronized.

Clean ETL Data

Cleans all the data of the relevant ETL job from the Target database.

If you select All from the instance drop-list, all ETL data of instances in target tables is deleted. If you select one specific instance, only the data related to the instance is deleted.

Click Confirm.

Uninstall Content Pack Uninstalls the selected Content Pack from the ITBA application and from the ETL Toolkit.
<Left pane>

The list of relevant Extractor and ETL jobs of the selected data source instance.

<Right pane>

The right pane allows you to edit a script after you select it and click the .

<Lower pane> The running status of the action.