Develop > Develop Content > Extend Content > IDE > IDE User Interface Overview > Engineer Stream Designer

Diagram Editor - Engineer Stream Designer Tasks

The Engineer Stream Designer enables you to define the integrations and how they are transformed and connected. This view defines the mapping relations for the entity from the source to the target. You can create new integrations based on graphical diagrams, using the Engineer Stream Designer in the IDE tool.

Note Make sure you have selected the IDE perspective before beginning to work on your designer diagram.

ClosedTo access:

In the IDE main page, select File > New > Other > DWH IDE Wizards > Engineer Stream Designer.

The Engineer Stream Designer enables you to extend and add content from the Integration and Source Content Packs into the target model. Using a new or existing Target entity, you can add new or existing content from the Integration and Source entities. You then perform transformation and Source Integration mapping for new and pre-loaded entities. The Source, Integration and Target entities are mapped using the Transformation Mapping connector, which creates a new mapping entity. All further mapping is done to the new entity in order to complete the Target model.

Note All Source entities must be connected to an Integration entity.

Entities in the Engineer Stream Designer are mapped as follows:

  • Source Entity: Mapping to the correct source, meaning the version the data is extracted from.

  • Integration Entity: Contains the mapping of all fields.

  • Target Entity: Several consolidation entities can be consolidated into one entity.

It is recommended to map from the Source to the Target.

ClosedSource Extractors

OOTB set of existing supported extractors are:

  • Generic Extractors

    • DB Extractor: Supports MSSQL, Oracle and MySQL databases.

    • File Extractor: Supports general CSV file.

  • Source Specific Extractor

    The supported extractors for the OOTB Content Packs are as follows:

    Content Pack Extractor Type
    PPM(Oracle) DB
    SM(MSSQL, Oracle, DB2) DB
    ALM REST API
    CSA REST API
    AWS SDK
    AWSCW SDK
    AM DB
    SA DB
    VPV "REST API"
    AZURE "REST API"

    Connecting to the integration:

    ETL Gen templates in the back-end generate integration templates which utilize Data Collection Service (DCS). Load the Source metadata to the IDE and then map it to the correct consolidation entity in the relevant fields. To create a custom DCS extractor, see the ..

ClosedImportant Information

Note Creating new Target entities with Associated Dimensions can only be done in the Architect Target Designer.

ClosedRecommended

When you generate ETL artifacts with IDE, if you just put the source and integration entities in the engineer diagram, but do not map any columns with target attributes (no associated dimensions) from source to target (Left to Right), the IDE does not generate the data model correctly, and the ConsolidationReferences in the Extraction model is incomplete.

It is recommended to map at least one column to the target attributes for every integration entity.

ClosedCreate a new Engineer Stream Designer in the IDE

  1. In the IDE main page, select File > New > Other > DWH IDE Wizards > Engineer Stream Designer and click Next.

  2. Enter or select the parent directory and enter the file name for the Engineer Stream Designer diagram - make sure you do not change the suffix of the file name - and click Next.

  3. Select the Target, Integration, and Source Content Pack directories and click Finish.

ClosedDefine a new Target entity

  1. In the Engineer Stream Designer, in the palette area, select New Target Entity and click on the canvas.

  2. Name the entity.

  3. Add relevant columns to the entity.

  4. Click Save.

ClosedLoad an existing Target entity

  1. In the Engineer Stream Designer, in the palette area, select Load Target Entity and click on the canvas. The Resource Selector dialog box opens.

  2. Select an entity. The entity is displayed with all of the previously added columns and dimensions.

  3. Click Save.

ClosedDefine a new integration entity

  1. In the Stream Designer, in the palette area, select New Integration Entity in the Integration section and click on the canvas. An integration entity is displayed.

  2. Enter a name. The name can be up to 44 characters. An Integration entity is created.

  3. Click Integration Column to add a column to the entity.

  4. Perform mapping if required.

  5. Click Save.

ClosedLoad an existing integration entity

  1. In the Stream Designer, in the palette area, select Load Integration Entity in the Integration section and click on the canvas. The Resource Selector dialog box opens.

  2. Select an entity. The entity is displayed with all of the previously added mappings.

ClosedDefine a new source entity

  1. In the Engineer Stream Designer, in the palette area, select New Source Entity in the Source section and click on the canvas. A Source entity is displayed.

  2. Click Source-Integration Relation and drag the display arrow from the Source entity to an Integration entity. All columns from the Integration entity are copied to the Source entity with default values.

  3. Click Save.

    Note The Source-Integration Relation tool only enables you to link Source entities to Integration entities of the same product (content pack). For example, a Source entity from cp-sm-9.4-dbdict-mssql can only be linked to Integration entities from cp-sm and not cp-ppm.

ClosedLoad an existing source entity

  1. In the Engineer Stream Designer, in the palette area, select Load Source Entity in the Source section and click on the canvas. The Resource Selector dialog box opens.

  2. Select an entity. The entity is displayed with its mapped integration entity.

ClosedAdd a column

  1. In the Engineer Stream Designer, in the palette area, select Target Column or Integration Column.

  2. Click in the Column area of your entity and enter the Column ID in the text box.

    The column is added to the entity.

  3. Click Save.

Note Source data is replicated from the integration entity. If you add or delete columns from the Integration, however, it is not automatically done in the Source.

To add: Once you have added the column, delete the mapping link and re-map from the Source to Integration.

To delete: Right-click the deleted column, delete the mapping link and re-map from the Source to Integration.

Alternatively, you can use the Quick Fix mechanism on the error that appears in the Problems tab. For details, see Manage Validations.

ClosedMap from the Integration to Target

Once you have added columns to the Integration entity you can perform SSI mapping.

  1. In the Engineer Stream Designer, in the palette area, click Column Mapping in the Relations section to map the Integration to the Target entity.

  2. Drag the arrow from the Integration column to the Target column.

  3. After you have performed the first SSI mapping, a transformation SSI entity is created to which you can perform further mapping. Any integration entity from the same CP that should be mapped to this Target, must be mapped to the transformation entity.

  4. Click Save.

    Note If an Integration column is already mapped to a Target column, you cannot perform further mapping from the same Content Pack.

    Two Integration entity columns from the same Content Pack cannot be mapped to the same Target entity column.

ClosedEngineer Stream Designer View

ClosedPalette Area

UI Element

 Description
Target Entity

  • New Target Entity. Select to display a new Target entity.

  • Load Target Entity. Opens the Resource Selector dialog box. Select the relevant entity for your model.

  • Target Column. Adds columns to entities. Column properties, for example column type, can be edited in the Properties tab.

    Each added attribute impacts the relevant entity generations.
Integration Entity

Contains the following Integration entity options:

  • New Integration Entity. Select to display a new Integration entity.

  • Load Integration Entity. Opens the Resource Selector dialog box. Select the relevant entity for your model.

  • Integration Column. Adds columns to entities. Column properties, for example column type, can be edited in the Properties tab.

    Each added attribute impacts the relevant entity generations.
Source Entity

Contains the following Source entity options:

  • New Source Entity. Select to display a new Source entity.

  • Load Source Entity. Opens the Resource Selector dialog box.Select the relevant entity for your model.

  • Source to Integration Relation. The Source must be mapped to the Integration in order to map to the Target. A green arrow is displayed between the entities.
Transformation

Click to map the displayed entities.

  • Column Mapping. Mapping between the Integration and Target columns. First time mapping is done from the Integration column to the Target column. A black arrow is displayed between the entities.

    First-time mapping creates a new entity for Transformation. Subsequent mappings are done to the new Transformation entity.
ClosedProperties tab for Integration

To view the Properties tab, right-click on the specific entity and click Show Properties View.

Note Properties data is replicated in the Integration and Source columns. If you add or delete columns from the Integration, however, it is not automatically done in the Source.

To add: Once you have added the column in the properties tab, delete the mapping link and re-map from the Source to Integration.

To delete: Right-click the deleted column, delete the mapping link and re-map from the Source to Integration.

UI Element

 Description
General tab

Displays the General entity attributes.
Columns tab

Displays column information:

  • ID. The ID of the column. It is taken from the mapped integration column and is read-only.

  • Business name. The complete business name.

  • Description. Enter a description of the entity.

  • Data Type. Varchar, Int 16, Int 32, Float, Localized Date Time

  • Length. Only for Varchar data type. The length of a Varchar column must be more than 0 and less than 4000.

  • Null Allowed. Select if the field can have the value NULL. If you specify a Business Key, at least one of the columns must not Null Allowed.

  • Business Key. Primary key of the Integration field order. Select a number for the order of columns for placement. When no number has been assigned, you can only select 1. Subsequently you can assign any number to a column. If you change a number, all other assigned columns are automatically shifted down a spot in the order.

    Example If you change the Business Key of a column to 1, the one that was specified previously as 1 becomes 2.

  • Last Modified. The last date that the source has been changed. Select the button if you want a query from a certain date and not all possible data.
ClosedProperties tab for Source

To view the Properties tab, right-click on the specific entity and click Show Properties View.

UI Element

 Description
General tab

General attributes:

  • ID. The entity ID.

  • Content Pack. The Source Content Pack information.

Data Source SQL Clauses:

  • Group by. The section in the SQL Query for the aggregation.

  • Table Name/Join Condition. Default is Integration entity name. If you change the table name, you must also change it in the columns. Enter the Join Condition for an SQL query with join.

  • Where. Where to extract for the SQL query for filtering data from the source.
Columns tab

Displays column information. The Source entity receives all of the Integration entity properties.

  • ID. The ID of the column. It is taken from the mapped integration column and is read-only.

  • Type. Select Fixed, Direct, or Rule. If you select Fixed, you must enter a Fixed Value. if you select Rule, you must enter the Calculation Rule.

  • Not Applicable.

  • Table Name. Based on table name entered in the General tab. Relevant only if Direct type is selected.

  • Column Name. The field name. Must be what appears in the Source entity.

  • Fixed Value. The fixed value of the column. Relevant only if Fixed type is selected.

  • Calculation Rule. The calculation rule of the column. Relevant only if Rule type is selected. The IDE does not support non UTF-8 characters in the Calculation Rule field.

For details on the Properties for Target entities, see Properties tab for Target entity and specific fields

For details on the Properties tab for Associated Dimensions, see Properties tab for Associated Dimensions.

Send Help Center feedback