![]() Weight_rule ( str) – weighting method used for the effective total This allows the executor to trigger higher priority tasks before Priority_weight ( int) – priority weight of this task against other task. NotĪll executors implement queue management, the CeleryExecutorĭag ( ) – a reference to the dag the task is attached to (if any) Queue ( str) – which queue to target when running this job. Is forced to True wherever wait_for_downstream is used. This is useful if theĭifferent instances of a task X alter the same asset, and this asset Of task X to finish successfully before it runs. X will wait for tasks immediately downstream of the previous instance Wait_for_downstream ( bool) – when set to true, an instance of task The task instance for the start_date is allowed to run. Sequentially while relying on the previous task’s schedule to Read theįAQ entry about start_date for more information.Įnd_date ( datetime.datetime) – if specified, the scheduler won’t go beyond this dateĭepends_on_past ( bool) – when set to true, task instances will run We advise against usingĭynamic start_date and recommend using fixed ones. If you are looking to delayĪ task, for example running a daily task at 2AM, look into the Start_date are offset in a way that their execution_date don’t line To note that different tasks’ dependencies Note that Airflow simply looks at the latestĮxecution_date and adds the schedule_interval to determine Some day at 00:00:00, hourly jobs have their start_date at 00:00 ![]() The execution_date for the first task instance. Start_date ( datetime.datetime) – The start_date for the task, determines ![]() Max_retry_delay ( datetime.timedelta) – maximum delay interval between retries Retries by using exponential backoff algorithm on retry delay (delay Retry_exponential_backoff ( bool) – allow progressive longer waits between Retry_delay ( datetime.timedelta) – delay between retries Retries ( int) – the number of retries that should be performed before Owner ( str) – the owner of the task, using the unix username is recommended Task_id ( str) – a unique, meaningful id for the task Task dependencies shouldīe set by using the set_upstream and/or set_downstream methods. Which ultimately becomes a node in DAG objects. Instantiating aĬlass derived from this one results in the creation of a task object, This class is abstract and shouldn’t be instantiated. Operators (tasks) target specific operations, running specific scripts, Moves data from Hive to MySQL (Hive2MySqlOperator). Waits for a partition to land in Hive (HiveSensorOperator), or one that Operator that runs a Pig job (PigOperator), a sensor operator that Operators derived from this class should perform or trigger certain tasks The constructor as well as the ‘execute’ method. To derive this class, you are expected to override Since operators create objects thatīecome nodes in the dag, BaseOperator contains many recursive methods forĭag crawling behavior. BaseOperator ( task_id, owner=conf.get('operators', 'DEFAULT_OWNER'), email=None, email_on_retry=True, email_on_failure=True, retries=conf.getint('core', 'default_task_retries', fallback=0), retry_delay=timedelta(seconds=300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, priority_weight=1, weight_rule=WeightRule.DOWNSTREAM, queue=conf.get('celery', 'default_queue'), pool=Pool.DEFAULT_POOL_NAME, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=TriggerRule.ALL_SUCCESS, resources=None, run_as_user=None, task_concurrency=None, executor_config=None, do_xcom_push=True, inlets=None, outlets=None, *args, **kwargs ) ¶īases: _mixin.LoggingMixinĪbstract base class for all operators. Package Contents ¶ class airflow.operators.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |