dbt_airflow_factory.k8s package

Submodules

dbt_airflow_factory.k8s.k8s_operator module

Factories creating Airflow Operators running DBT tasks.

class KubernetesPodOperatorBuilder(dbt_execution_env_parameters: dbt_airflow_factory.dbt_parameters.DbtExecutionEnvironmentParameters, kubernetes_execution_parameters: dbt_airflow_factory.k8s.k8s_parameters.KubernetesExecutionParameters)[source]

Bases: dbt_airflow_factory.operator.DbtRunOperatorBuilder

Builder of Kubernetes Pod Operator running a single DBT task.

Parameters
create(name: str, command: str, model: Optional[str] = None, additional_dbt_args: Optional[List[str]] = None) → airflow.models.baseoperator.BaseOperator[source]

Create Airflow Operator running a single DBT task.

Parameters

  • name (str) – task name.

  • command (str) – DBT command to run.

  • model (Optional[str]) – models to include.

  • additional_dbt_args (Optional[List[str]]) – Additional arguments to pass to dbt.

Returns

Airflow Operator running a single DBT task.

Return type

BaseOperator

dbt_execution_env_parameters: dbt_airflow_factory.dbt_parameters.DbtExecutionEnvironmentParameters

POD representing DBT operator config file.

kubernetes_execution_parameters: dbt_airflow_factory.k8s.k8s_parameters.KubernetesExecutionParameters

POD representing Kubernetes operator config file.

dbt_airflow_factory.k8s.k8s_parameters module

POD representing Kubernetes operator config file.

class KubernetesExecutionParameters(image: str, namespace: str = 'default', image_pull_policy: Optional[str] = None, node_selectors: Optional[dict] = None, tolerations: Optional[list] = None, labels: Optional[dict] = None, limit: Optional[dict] = None, requests: Optional[dict] = None, annotations: Optional[dict] = None, envs: Optional[Dict[str, str]] = None, secrets: Optional[List[airflow.kubernetes.pre_7_4_0_compatibility.secret.Secret]] = None, is_delete_operator_pod: bool = True, config_file: Optional[str] = None, execution_script: str = 'dbt --no-write-json', in_cluster: Optional[bool] = None, cluster_context: Optional[str] = None, startup_timeout_seconds: int = 120, **kwargs: Any)[source]

Bases: object

POD representing Kubernetes operator config file.

Parameters

  • image (str) – tag of Docker image you wish to launch.

  • namespace (str) – the namespace to run within Kubernetes.

  • image_pull_policy (str) – Specify a policy to cache or always pull an image.

  • node_selectors (dict) – A dict containing a group of scheduling rules.

  • tolerations (list) – A list of Kubernetes tolerations.

  • labels (dict) – labels to apply to the Pod. (templated)

  • limit (dict) – A dict containing resources limits.

  • requests (dict) – A dict containing resources requests.

  • annotations (dict) – non-identifying metadata you can attach to the Pod.

  • envs (Optional[Dict[str, str]]) – Environment variables initialized in the container. (templated)

  • secrets (List[Secret]) – Kubernetes secrets to inject in the container. They can be exposed as environment vars or files in a volume.

  • is_delete_operator_pod (bool) – What to do when the pod reaches its final state, or the execution is interrupted. If True: delete the pod.

  • config_file (Optional[str]) – The path to the Kubernetes config file. (templated) If not specified, default value is ~/.kube/config

  • execution_script (str) – Script that will be executed inside pod.

  • in_cluster (bool) – Run kubernetes client with in_cluster configuration.

  • cluster_context (str) – Context that points to kubernetes cluster. Ignored when in_cluster is True. If None, current-context is used.

property env_vars

Return dict containing environment variables to set in Kubernetes.

For Airflow 1, the function returns a dictionary.

Beginning with Airflow 2, KubernetesPodOperator expects a list of V1EnvVar instances instead. Hence, for Airflow 2, the function returns a List[k8s.V1EnvVar].

Returns

Dictionary or list containing environment variables.

property resources

Return dict containing resources requests and limits.

In the Airflow 1, it was expected to be a real dictionary with request_memory, request_cpu, limit_memory, and limit_cpu as keys. So for Airflow 1, the function returns such a dictionary.

Beginning with Airflow 2, KubernetesPodOperator expects V1ResourceRequirements class instead. Hence, for Airflow 2, the function returns instance of this class.

Returns

Dictionary containing resources requests and limits.

property secrets: Optional[List[airflow.kubernetes.pre_7_4_0_compatibility.secret.Secret]]

Return list containing secrets to be set in Kubernetes. :return List containing kubernetes Secrets

dbt_airflow_factory.k8s.k8s_parameters_loader module

POD representing Kubernetes operator config file.

class KubernetesExecutionParametersLoader[source]

Bases: object

static create_config(dag_path: str, env: str, execution_env_config_file_name: str) → dbt_airflow_factory.k8s.k8s_parameters.KubernetesExecutionParameters[source]