How To: Configure Logging¶
Loggers can be configured via the NessySettings class, that uses
pydantic-settings to load
settings from environment variables.
You will get meaningful warnings if the environment variables are not set the
way they are expected. The NessySettings class will provide default values for
all settings, so that the framework can be used without setting any environment
variables.
Warning
To provide an easy switch for framework users, whether to send logs to a
specific target or not, the NessySettings will always overrule the
specific logger settings. This means e.g., that if the NessySettings are
set to not log to a specific target, the logger will not add the handler for
this target.
This means, e.g., that if the environment variable nessy_target_log_analytics
is set to False, no Log Analytics handler will be attached to the logger.
The LoggerMixin is responsible
for setting up the logger and adding the handlers to it. It differentiates
between two types of loggers, the console logger and the tabular logger. While
console loggers only log to the console, tabular loggers log to a table in the
Unity Catalog or the Azure Log Analytics.
Environment Variables¶
The following section defines the environment variables that can be set to configure the loggers.
General¶
| Attribute | Type | Default Value | Description |
|---|---|---|---|
| nessy_log_level_console | int | logging.INFO | The log level for the console logger. |
| nessy_log_level_tabular | int | logging.INFO | The log level for the tabular logger. |
| nessy_log_format_console | str | %(asctime)s - %(message)s | The format of the console logger. |
Azure Log Analytics¶
The Azure Log Analytics handler will log to Azure Log Analytics, if
nessy_target_log_analytics is set to True.
| Attribute | Type | Default Value | Description |
|---|---|---|---|
| nessy_target_log_analytics | bool | False | Whether to log to Azure Log Analytics. |
| nessy_log_analytics_workspace_id | str | None | None | The workspace ID for Azure Log Analytics. |
| nessy_log_analytics_shared_key | str | None | None | The shared key for Azure Log Analytics. |
Unity Catalog¶
The Unity Catalog handler will log to a table in the Unity Catalog, if
nessy_target_unity_catalog_table is set to True.
Warning
The Handler will automatically use the credentials of the user that is running the code. This means that the user needs to have the necessary permissions to write to the catalog table.
| Attribute | Type | Default Value | Description |
|---|---|---|---|
| nessy_target_unity_catalog_table | bool | False | Whether to log to the Unity Catalog Table. |
| uc_workspace_url | str | None | None | Determines which Databricks workspace to use for the REST API. |
| uc_warehouse_id | str | None | None | Determines which SQL Warehouse to use for INSERT Statements to the Logging Table. |
| nessy_uc_catalog_name | str | None | None | The catalog name for the Unity Catalog Table. |
| nessy_uc_schema_name | str | None | None | The schema name for the Unity Catalog Table. |