Metadata source

In the configuration file there must always be information on the source of the metadata. This metadata will be fetched and transformed into the standardized YAML models.

You must always provide the type parameter to tell the program which kind of metadata source to expect.

Snowflake table

Currently, the only supported metadata source is a Snowflake table or view that stores the metadata for other tables. To fetch the metadata provide the following information in the configuration file. The following is an example when retrieving SAP metadata from SAP's DD03L table that has been loaded to Snowflake.

To fetch the metadata the framework needs to connect against Snowflake. The connection parameters are not stored in the configuration file, but are provided via environment variables.

metadata:
  type: "snowflake_table"
  schema: "CORE"
  table: "TBL_DD03L"
  columns:
    table_name: "TABNAME"
    column_name: "FIELDNAME"
    key:
      name: "KEYFLAG"
      truth_value: "X"
    datatype: "DATATYPE"
    length: "LENG"
    precision: "LENG"
    scale: "DECIMALS"
    date_precision: "LENG"
  order_by: "POSITION"
  dtype_mapping:
    source: "sap"
    target: "snowflake"

Mandatory parameters

parameter	description
schema	database schema of the source table
table	name of table storing the metadata information

Additional parameters

parameter	description
columns	required metadata information map the column names expected by the framework to the actual column names in the source table
order_by	column that stores e.g. ordinal information
dtype_mapping	if you need to map the datatypes between the source and target system provide the mapping specification here