Airflow on Kubernetes: Dynamic Workflows Simplified - Daniel Imberman, Bloomberg & Barni Seetharaman

Поделиться
HTML-код
  • Опубликовано: 3 авг 2024
  • Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
    Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
    Airflow on Kubernetes: Dynamic Workflows Simplified - Daniel Imberman, Bloomberg & Barni Seetharaman, Google
    Apache Airflow is an open source workflow orchestration engine that allows users to write Directed Acyclic Graph (DAG)-based workflows using a simple Python library. Airflow offers a wide range of native operators for services ranging from Spark and HBase to Google Cloud Platform (GCP) and Amazon Web Services (AWS). Until recently, the Airflow user experience has been hindered by the need to launch and maintain statically-sized Celery-based Airflow clusters. These clusters were both expensive (over and under-utilization) and complex (multiple points of failure). To address these issues, we developed and published a native Kubernetes Operator and Kubernetes Executor for Apache Airflow. These products allow one-step Airflow deployments, dynamic allocation of Airflow worker pods, full power over run-time environments, and per-task resource management.
    To learn more: sched.co/GrUO
  • КиноКино

Комментарии • 4

  • @NVME57
    @NVME57 3 года назад +3

    Wow. Just wow.
    I'm going to implement this in my company.
    Thank you.

  • @automata8973
    @automata8973 3 года назад +2

    I think argo workflow project fits better !

  • @user-uw7st6vn1z
    @user-uw7st6vn1z 2 года назад

    cloud native will take everything away from everyone....

  • @hubstrangers3450
    @hubstrangers3450 4 года назад

    In simple terms is it just use the API, rest will be taken care.....sounds like returning to menu driven environment, either future generations of what's under the hood, dosen't possess the skills to develop something "Organically", drawback of cloud accounts, in addition to being a modern day taxation, also, prefer to take away the skills, knowledge away from future generations.....