Skip to content
This repository has been archived by the owner on Jul 4, 2023. It is now read-only.

Latest commit

 

History

History

apache-airflow

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 

Apache Airflow

https://airflow.apache.org/

Airflow is a platform created by the community to programmatically author, schedule and monitor workflows

Airflow

Provision

In order to provision apache airflow you need bastetools, docker, minikube as dependencies.

vagrant up --provision-with basetools,docker,minikube,postgresql,dbt,apache-airflow

Web UI Access

To access the web UI visit http://localhost:18889. Default login is:

Username: admin
Password: admin

Further Info

Airflow is deployed on Minikube (Kubernetes) using Helm, and additional values are supplied in the values.yaml file.

Example DAGs are supplied in the dags folder and they are mounted into the airflow scheduler pod, see the details in the values.yaml file

Airflow Information

In the dags folder you will find 2 dags

  • example-dag.py
  • test-ssh.py

The example-dag.py runs dbt commands by using the SSHOperator and ssh'ing into Hashiqube. The test-ssh.py just ssh into hashiqube to test the connection

Airflow DAGs

Airflow

Airflow Connections

Airflow

Airflow DAG run

Airflow

Airflow Task Instance

Airflow

Airflow Task Instance Result

Airflow

Links and further reading