Superset
Important Capabilities
Capability | Status | Notes |
---|---|---|
Detect Deleted Entities | ✅ | Optionally enabled via stateful_ingestion |
Table-Level Lineage | ✅ | Supported by default |
This plugin extracts the following:
- Charts, dashboards, and associated metadata
See documentation for superset's /security/login at https://superset.apache.org/docs/rest-api for more details on superset's login api.
CLI based Ingestion
Install the Plugin
pip install 'acryl-datahub[superset]'
Starter Recipe
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source:
type: superset
config:
# Coordinates
connect_uri: http://localhost:8088
# Credentials
username: user
password: pass
provider: ldap
sink:
# sink configs
Config Details
- Options
- Schema
Note that a .
is used to denote nested fields in the YAML recipe.
Field | Description |
---|---|
connect_uri string | Superset host URL. Default: http://localhost:8088 |
database_alias map(str,string) | |
display_uri string | optional URL to use in links (if connect_uri is only for ingestion) |
options object | Default: {} |
password string | Superset password. |
platform_instance string | The instance of the platform that all assets produced by this recipe belong to |
provider string | Superset provider. Default: db |
username string | Superset username. |
env string | The environment that all assets produced by this connector belong to Default: PROD |
domain map(str,AllowDenyPattern) | A class to store allow deny regexes |
domain. key .allowarray | |
domain. key .denyarray | |
domain. key .ignoreCaseboolean | Whether to ignore case sensitivity during pattern matching. Default: True |
stateful_ingestion StatefulStaleMetadataRemovalConfig | Superset Stateful Ingestion Config. |
stateful_ingestion.enabled boolean | The type of the ingestion state provider registered with datahub. Default: False |
stateful_ingestion.remove_stale_metadata boolean | Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. Default: True |
The JSONSchema for this configuration is inlined below.
{
"title": "SupersetConfig",
"description": "Base configuration class for stateful ingestion for source configs to inherit from.",
"type": "object",
"properties": {
"platform_instance": {
"title": "Platform Instance",
"description": "The instance of the platform that all assets produced by this recipe belong to",
"type": "string"
},
"env": {
"title": "Env",
"description": "The environment that all assets produced by this connector belong to",
"default": "PROD",
"type": "string"
},
"stateful_ingestion": {
"title": "Stateful Ingestion",
"description": "Superset Stateful Ingestion Config.",
"allOf": [
{
"$ref": "#/definitions/StatefulStaleMetadataRemovalConfig"
}
]
},
"connect_uri": {
"title": "Connect Uri",
"description": "Superset host URL.",
"default": "http://localhost:8088",
"type": "string"
},
"display_uri": {
"title": "Display Uri",
"description": "optional URL to use in links (if `connect_uri` is only for ingestion)",
"type": "string"
},
"domain": {
"title": "Domain",
"description": "regex patterns for tables to filter to assign domain_key. ",
"default": {},
"type": "object",
"additionalProperties": {
"$ref": "#/definitions/AllowDenyPattern"
}
},
"username": {
"title": "Username",
"description": "Superset username.",
"type": "string"
},
"password": {
"title": "Password",
"description": "Superset password.",
"type": "string"
},
"provider": {
"title": "Provider",
"description": "Superset provider.",
"default": "db",
"type": "string"
},
"options": {
"title": "Options",
"default": {},
"type": "object"
},
"database_alias": {
"title": "Database Alias",
"description": "Can be used to change mapping for database names in superset to what you have in datahub",
"default": {},
"type": "object",
"additionalProperties": {
"type": "string"
}
}
},
"additionalProperties": false,
"definitions": {
"DynamicTypedStateProviderConfig": {
"title": "DynamicTypedStateProviderConfig",
"type": "object",
"properties": {
"type": {
"title": "Type",
"description": "The type of the state provider to use. For DataHub use `datahub`",
"type": "string"
},
"config": {
"title": "Config",
"description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).",
"default": {},
"type": "object"
}
},
"required": [
"type"
],
"additionalProperties": false
},
"StatefulStaleMetadataRemovalConfig": {
"title": "StatefulStaleMetadataRemovalConfig",
"description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
"type": "object",
"properties": {
"enabled": {
"title": "Enabled",
"description": "The type of the ingestion state provider registered with datahub.",
"default": false,
"type": "boolean"
},
"remove_stale_metadata": {
"title": "Remove Stale Metadata",
"description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
"default": true,
"type": "boolean"
}
},
"additionalProperties": false
},
"AllowDenyPattern": {
"title": "AllowDenyPattern",
"description": "A class to store allow deny regexes",
"type": "object",
"properties": {
"allow": {
"title": "Allow",
"description": "List of regex patterns to include in ingestion",
"default": [
".*"
],
"type": "array",
"items": {
"type": "string"
}
},
"deny": {
"title": "Deny",
"description": "List of regex patterns to exclude from ingestion.",
"default": [],
"type": "array",
"items": {
"type": "string"
}
},
"ignoreCase": {
"title": "Ignorecase",
"description": "Whether to ignore case sensitivity during pattern matching.",
"default": true,
"type": "boolean"
}
},
"additionalProperties": false
}
}
}
If you were using database_alias
in one of your other ingestions to rename your databases to something else based on business needs you can rename them in superset also
source:
type: superset
config:
# Coordinates
connect_uri: http://localhost:8088
# Credentials
username: user
password: pass
provider: ldap
database_alias:
example_name_1: business_name_1
example_name_2: business_name_2
sink:
# sink configs
Code Coordinates
- Class Name:
datahub.ingestion.source.superset.SupersetSource
- Browse on GitHub
Questions
If you've got any questions on configuring ingestion for Superset, feel free to ping us on our Slack.