4. Implement the Extractor

This extractor extends the BaseExtractor and requires at least the following implementation:

public boolean checkConnection() throws ExtractorException
protected abstract void doExtract(String batchId, Map<String, String> lastModifiedDateMap, List<DcsEntity> entities) throws ExtractorException,
            InterruptedException; 

The batchId is a unique number indicates a distinguished data extraction job.

The lastModifiedDateMap contains all the last modified date for all the entities, which is critical for delta data extraction.

The entities contains all the valid entities for the current extraction. Entities that appears in the extraction model but not in the source model are ignored and are not included in the entities parameter.

For the initial load, the CFM (Content Flow Manager) transforms the initial load period to a specified date value as the last modified date.

For example, if the current time is ‘2015-06-01 00:00:00’ and the initial load period is 6 months, the calculated last modified date for all entities is ‘2015-01-01 00:00:00’. The date time is formatted with the time zone in the ITBA server. Each entity will have a separate entry in the lastModifiedDateMap.

For a delta load, CFM fetches the last modified date from the TSNP tables according to the column that is marked as lastmodifieddate=’Y’ in the extraction model.

Activation & Registration

• Test the connection.

• Verify that the relevant source is available and that the Data Warehouse can connect to it.

• The connection details of the specific source such as product version, data source type, connection properties must be send to the extractor, initialized, and must invoke the checkConnection method. Once the check connection is successful, its details are registered into the DCS framework and saved for later data extraction. Otherwise, an exception is thrown to the upper layer.

  The check operation uses these details in order to validate that the source is accessible from the Data Warehouse.

Data Source Extraction

• Extract the relevant data from the source.

• Use the Source Model:

  • Describes the required data and data structure of the source.

  • Defines the structure of the result output text file.

    The doExtract(…) should connect to the source and read the relevant data defined in the SourceModel. For each extractor, there may be more than one entity for extraction. All the entities can be extracted in sequence order in the extractor that you defined. Another option is that you create an entity level extractor and let the entity extractor run in parallel. The CP level extractor will manage all the entity level extractor tasks.

    Every entity extraction should use the createFlatFileWriter(…) that is implemented in the BaseExtractor to create the FileFileWriter and write the result. Generally, FileFIleWriter has two methods: writeHeader(String[] headers) and writeLine(String[] values), which are used to write the header values and the values for each row data. The headers and values are escaped so that the flat file can be imported into the Vertica DB.

    SourceModel stores information about the data structure of the source. It defines the name of the entity and the fields related to this entity. All the entities contains both the extraction model and the source model, and they are loaded by the BaseExtractor before the data extraction. Each entity is related to a DcsEntity object, which contains corresponding source model and extraction model. You can get the field details through the extraction model and source model reference.

    

    In order to get the fields and their order, you can use getColumns() of the DMExtractionModel (defined and initialized in the BaseExtractor).

• Initial load and delta load

  • The initial load means that all the data is loaded according the range defined by the initial load period. This process takes a lot of time depending on the length of the period you want to extract. A one time initial load is executed after setting up a new data source connection.

  • After the initial load, the following data extraction is taken as a delta load. Generally, it extracts the data that changed since the last extraction time except for delete cases.

• Delete case

  • If an entity is marked as delete (isdelete="Y") in the extraction model, it means that the columns that are defined as business key (businesskey="Y") need to be fully extracted from the data source, whatever it is initial load or delta load. You should ignore the last modified date value for that entity or use the value in the initial load.

• Extraction status: The Extractor manages and stores the status to DB (PostgreSQL).

  To store the operation status info to DB:

  status.setStatus(StatusEnum.COMPLETED.getValue());
  flushStatus();
  

  It is very important that the extractor developer save the status of all the entities because the BaseExtractor includes a logic that checks all the statuses of the entities. An extractor may have a lot of entities that can be extracted in parallel. If and only if all the entities are successfully extracted, the whole extractor can be successful. If any of the entity fails to extract data, the whole extractor fails.