Deployment Configuration

This section describes the configuration of the Azure Data Factory module. It is recommended to follow the configuration for the automated Terraform Setup. For the manual configuration of the build-pipeline see the Azure DevOps section below.

Terraform Setup

In the config file of the Terraform Setup you find the configuration for the Azure Data Factory module under the cloe_modules block:

azure_data_factory:
    deploy: true
    parameters:
        <env>:
            serviceConnectionName:  # (1)
            subscriptionId:         # (2)
            resourceGroupName:      # (3)

1. Name of the used service connection.
2. ID of the used azure subscription.
3. Name of the used ressource group.

Azure DevOps

Open your build-pipeline definition and change the following values:

Info

The stage deployCLOEARMTemplate is only part of the auto arm example (see Azure Data Factory).

1. Stage BuildCLOEArtifact:

   • Configure the highlighted parameters:

       template: /templates/generator/cloe.exec.cli.yml@templates # (1)
       parameters: # (2)
         cliArgs: |
           gen-datafactory \
             $(Build.Repository.LocalPath)/PATH_TO_MODEL \
             $(Build.Repository.LocalPath)/io/ \
             $(Build.Repository.LocalPath)/io/
         outputPath: '$(Build.Repository.LocalPath)/io'
         artifactPublishName: cloedatafactory
   

   1. If you work in a customers DevOps you need to change the path to the CLOE Pipeline Templates Repository by changing the template path.
   2. If you do not use the recommended folder structure, you can change the path to the CLOE Model by changing the cliArgs. Just change the highlighted line to your CLOE model path.

2. Now do the pipeline setup and run the pipeline once. After the first execution a "parameters_file.json" file is created. Put this file in your CLOE model repository. After that switch back to your build-pipeline definition and configure the following stage.

3. Stage deployCLOEARMTemplate:

   • Configure the highlighted parameters:

       environment: '' # (1)
       artifactName: 'cloedatafactory'
       serviceConnectionName: '' # (2)
       templateFileName: '**/factory_template.json'
       templateParametersFileName: '**/PATH_TO_PARAMETER_FILE' # (3)
       subscriptionId: '' # (4)
       resourceGroupName: '' # (5)
   

   1. Specify the DevOps environment to be used e.g. "DEV".
   2. Name of the used service connection.
   3. Replace PATH_TO_PARAMETER_FILE with the path to your model (e.g. 'parameters_file.json'). Path is relative to repository root ($(Build.Repository.LocalPath)).
   4. ID of the used azure subscription.
   5. Name of the used ressource group.

   Info

   If you need help for the following parameters, ask the data platform team: serviceConnectionName, subscriptionId, resourceGroupName.

Configure ADF parameter file

1. If you are planning on using git integration in your datafactory, complete the following steps:

   • Add the following parameter to the parameter file located in your CLOE model repository and configure the highlighted values:

       "FactoryProperties": {
           "value": {
               "repoConfiguration": {
                   "collaborationBranch": "", # (1)
                   "accountName": "", # (2)
                   "projectName": "", # (3)
                   "type": "FactoryVSTSConfiguration",
                   "repositoryName": "", # (4)
                   "rootFolder": "" # (5)
               }
           }
       }
   

   1. Name of the branch used to publish changes to datafactory.
   2. Azure Devops organization name or GitHub account name.
   3. Name of the git project.
   4. Name of the git repository used for datafactory development.
   5. Path to the folder that will contain the datafactory templates.