How to Orchestrate AWS Glue Jobs with Airflow: Step-by-Step

Your data lands in S3 every day, from Airbyte, Fivetran or a small Python script on Lambda. You wrote a Spark job in AWS Glue that turns it into reports. Now someone has to run that job every morning, and it should not be you.

This guide is for data engineers and technical founders who want AWS Glue jobs to run on a schedule, wait for fresh data, retry on failure and alert when something breaks. I'll show the exact Airflow DAG I use, updated for Airflow 3, Amazon MWAA and Glue 5.x.

Why orchestrate AWS Glue jobs with Airflow?#

Glue is good at one thing: running Spark (or Python shell) jobs without managing a cluster. It is not great at deciding when to run them, what should happen before and after, and who gets pinged when a run fails.

That is where Airflow fits. I like it for Glue pipelines for four reasons:

• Scheduling and dependencies in code. The DAG says "wait for today's data, then run the Glue job, then refresh the dashboard". It lives in Git and gets reviewed like any other code.
• Retries, timeouts and alerts are built in, and the same for every task.
• One orchestrator for everything. Next month you will want to add a dbt run, a Lambda call or a second Glue job. You add a task, not a new tool.
• Run history and backfills. You can see every run, rerun one day, or backfill a month with one command. If your data stack is still a patchwork of cron jobs and manual clicks, orchestration is usually the first thing we fix when building data foundations for a team.

What does the architecture look like?#

Here is the setup this guide builds. Raw data lands in S3, Airflow waits for it, triggers a Glue job, and the Glue job writes a curated, partitioned table that Athena or your warehouse can query.

Airflow never touches the data itself. It only tells Glue what to run, with which parameters, and waits for the result. That separation keeps your Airflow workers small and cheap.

What do you need before you start?#

• Airflow 3.x (self-hosted, Astronomer, or Amazon MWAA, which now supports Airflow 3) or Airflow 2.x if you have not migrated yet. I note the differences below.
• The Amazon provider: apache-airflow-providers-amazon. It is preinstalled on MWAA. Locally:

pip install "apache-airflow-providers-amazon"

• An S3 bucket for Glue scripts and temporary files.
• An IAM role for Glue (the role the job runs as), with read access to the raw data and write access to the curated zone.
• AWS permissions for Airflow to start and monitor Glue jobs (details in the security section).

Step 1: Write a Glue job that takes parameters#

The most common mistake I see is a Glue script that reads "everything in the bucket" on every run. It gets slower and more expensive every day, and a rerun can double-count data.

Instead, make the script process one date, passed in by Airflow. Here is a Glue 5.x PySpark job that builds a daily sales report from raw payment data:

import sys

from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from pyspark.sql import functions as F

args = getResolvedOptions(
    sys.argv, ["JOB_NAME", "run_date", "source_path", "target_path"]
)

sc = SparkContext.getOrCreate()
glue_context = GlueContext(sc)
spark = glue_context.spark_session
job = Job(glue_context)
job.init(args["JOB_NAME"], args)

# Only overwrite the partitions we write, not the whole table
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

charges = spark.read.parquet(f"{args['source_path']}dt={args['run_date']}/")

daily_sales = (
    charges.filter(F.col("status") == "succeeded")
    .groupBy("currency", "product_id")
    .agg(
        F.count("*").alias("orders"),
        (F.sum("amount") / 100).alias("revenue"),
    )
    .withColumn("report_date", F.lit(args["run_date"]).cast("date"))
)

(
    daily_sales.write.mode("overwrite")
    .partitionBy("report_date")
    .parquet(args["target_path"])
)

job.commit()

Two details matter here:

• Dynamic partition overwrite means rerunning 2026-03-14 replaces only that day's partition. Your pipeline becomes idempotent, so retries and backfills are safe.
• Parameters are resolved at runtime. getResolvedOptions reads the values Airflow passes: --run_date, --source_path and --target_path. If you want ACID tables, time travel or schema evolution, write to Apache Iceberg instead of plain Parquet (Glue supports it natively with the --datalake-formats job argument). I cover the table format choice in building a lakehouse on AWS with Glue.

Step 2: Build the Airflow DAG#

The original version of this DAG had two task groups: one to upload the Glue script to S3, one to run it. I kept that structure and added what a production pipeline needs: a sensor that waits for fresh data, runtime parameters, deferrable execution and sane defaults.

from datetime import timedelta

from pendulum import datetime
from airflow.sdk import dag, task_group
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from airflow.providers.amazon.aws.transfers.local_to_s3 import (
    LocalFilesystemToS3Operator,
)

GLUE_BUCKET = "acme-glue-assets"
RAW_BUCKET = "acme-raw"
CURATED_PATH = "s3://acme-curated/reports/daily_sales/"
JOB_NAME = "daily_sales_report"
GLUE_ROLE = "glue-daily-sales-report"
SCRIPT_LOCAL_PATH = "/usr/local/airflow/include/glue/daily_sales_report.py"
SCRIPT_S3_KEY = f"scripts/{JOB_NAME}.py"

GLUE_JOB_CONFIG = {
    "GlueVersion": "5.1",  # or "5.0" / "4.0" depending on your region
    "WorkerType": "G.1X",
    "NumberOfWorkers": 2,
    "Timeout": 60,  # minutes, fail fast instead of burning DPUs
    "DefaultArguments": {
        "--enable-metrics": "true",
        "--job-language": "python",
    },
}


@dag(
    start_date=datetime(2026, 1, 1, tz="UTC"),
    schedule="0 6 * * *",
    catchup=False,
    max_active_runs=1,
    tags=["glue", "reporting"],
    default_args={
        "aws_conn_id": "aws_default",
        "retries": 2,
        "retry_delay": timedelta(minutes=5),
    },
)
def glue_daily_sales_report():
    wait_for_raw_data = S3KeySensor(
        task_id="wait_for_raw_data",
        bucket_name=RAW_BUCKET,
        bucket_key="stripe/charges/dt={{ ds }}/*.parquet",
        wildcard_match=True,
        deferrable=True,
        timeout=3 * 60 * 60,  # give up after 3 hours
    )

    @task_group(group_id="glue_scripts_to_s3")
    def upload_scripts():
        LocalFilesystemToS3Operator(
            task_id="upload_glue_script",
            filename=SCRIPT_LOCAL_PATH,
            dest_bucket=GLUE_BUCKET,
            dest_key=SCRIPT_S3_KEY,
            replace=True,
        )

    @task_group(group_id="run_glue_jobs")
    def run_jobs():
        GlueJobOperator(
            task_id="run_daily_sales_report",
            job_name=JOB_NAME,
            script_location=f"s3://{GLUE_BUCKET}/{SCRIPT_S3_KEY}",
            s3_bucket=GLUE_BUCKET,
            iam_role_name=GLUE_ROLE,
            create_job_kwargs=GLUE_JOB_CONFIG,
            update_config=True,
            script_args={
                "--run_date": "{{ ds }}",
                "--source_path": f"s3://{RAW_BUCKET}/stripe/charges/",
                "--target_path": CURATED_PATH,
            },
            wait_for_completion=True,
            deferrable=True,
            stop_job_run_on_kill=True,
        )

    wait_for_raw_data >> upload_scripts() >> run_jobs()


glue_daily_sales_report()

What each piece does#

• S3KeySensor waits until today's files exist. With deferrable=True it hands the waiting to the Airflow triggerer, so it does not hold a worker slot for three hours.
• LocalFilesystemToS3Operator copies the Glue script from your Airflow project to S3, where Glue expects it. replace=True overwrites the previous version.
• GlueJobOperator creates the Glue job if it does not exist, runs it and waits for the result. With update_config=True it also pushes changes to create_job_kwargs (workers, Glue version, timeout) to the existing job.
• script_args is templated, so {{ ds }} becomes the run's date (for example 2026-03-14) and lands in your script as --run_date.
• deferrable=True on the Glue operator frees the worker while Spark runs. On a 40-minute job, that is 40 minutes of worker capacity back.
• max_active_runs=1 stops two runs of the same DAG from writing to the same table at the same time during a backfill.

Airflow 2.x vs Airflow 3: what changes in this DAG?#

My original version used schedule_interval="0 20 * * *". That breaks on Airflow 3, so switch to schedule even if you are still on 2.x. It makes the upgrade one less thing to worry about.

Running several Glue jobs from one DAG#

Task groups start paying off when you have more than one job. A simple loop keeps the DAG readable:

GLUE_JOBS = {
    "daily_sales_report": "s3://acme-curated/reports/daily_sales/",
    "daily_refunds_report": "s3://acme-curated/reports/daily_refunds/",
}

@task_group(group_id="run_glue_jobs")
def run_jobs():
    for name, target in GLUE_JOBS.items():
        GlueJobOperator(
            task_id=f"run_{name}",
            job_name=name,
            script_location=f"s3://{GLUE_BUCKET}/scripts/{name}.py",
            s3_bucket=GLUE_BUCKET,
            iam_role_name=GLUE_ROLE,
            create_job_kwargs=GLUE_JOB_CONFIG,
            update_config=True,
            script_args={"--run_date": "{{ ds }}", "--target_path": target},
            deferrable=True,
        )

These jobs run in parallel inside the group. If one depends on another, chain them with >> as usual.

Step 3: Move script deployment out of the DAG#

Uploading the script from the DAG on every run is fine for a prototype. In production I prefer to deploy Glue scripts from CI, for three reasons:

• The script in S3 always matches what was reviewed and merged in Git.
• Airflow does not need write access to your scripts bucket.
• A broken script fails in CI, not at 6 a.m. in production. Here is a minimal GitHub Actions workflow that syncs Glue scripts to S3 on every merge to main, using OIDC so there are no AWS keys stored in GitHub:

name: deploy-glue-scripts

on:
  push:
    branches: [main]
    paths: ["glue/**"]

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - uses: aws-actions/configure-aws-credentials@v5
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-glue-deploy
          aws-region: eu-west-1

      - name: Sync Glue scripts to S3
        run: aws s3 sync glue/ s3://acme-glue-assets/scripts/ --exclude "*" --include "*.py"

Once this is in place, delete the glue_scripts_to_s3 task group and keep only the sensor and the Glue task. If you want the full picture of testing and deploying data code, I wrote a separate guide on CI/CD for data engineering.

How should Airflow authenticate to AWS?#

Do not paste an access key and secret into an Airflow connection. Use roles:

• On MWAA, leave the aws_default connection without credentials. The operators then use the MWAA execution role.
• On EKS or ECS, use IRSA, EKS Pod Identity or the ECS task role.
• Locally (Astro CLI or the MWAA local runner), mount your AWS profile or SSO session and point the connection at it. The role Airflow runs as needs a short list of permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetJob",
        "glue:CreateJob",
        "glue:UpdateJob",
        "glue:StartJobRun",
        "glue:GetJobRun",
        "glue:BatchStopJobRun"
      ],
      "Resource": "arn:aws:glue:eu-west-1:123456789012:job/*"
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::123456789012:role/glue-daily-sales-report"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetObject"],
      "Resource": ["arn:aws:s3:::acme-raw", "arn:aws:s3:::acme-raw/*"]
    }
  ]
}

iam:PassRole is the one people forget. Without it, CreateJob fails because Airflow is not allowed to hand the Glue role to the job. Add logs:GetLogEvents on the Glue log groups if you want the operator to stream job logs into the Airflow UI with verbose=True.

Airflow vs Glue Workflows vs Step Functions: which should you use?#

Airflow is not the only way to schedule Glue. Here is how I choose:

My rule: if the pipeline has one job, keep it simple with a Glue trigger or EventBridge. The moment you have three or more steps, or steps outside AWS Glue, Airflow pays for itself.

Production checklist for Glue jobs in Airflow#

• Every Glue job takes a date (or batch id) and processes only that slice.
• Writes are idempotent (dynamic partition overwrite, MERGE on Iceberg, or Glue job bookmarks).
• Timeout is set on the Glue job, and timeout on sensors.
• retries and retry_delay are set in default_args.
• max_active_runs=1 on DAGs that write to the same table.
• Failures send an alert (an on_failure_callback to Slack or email).
• Non-urgent jobs use the Glue Flex execution class ("ExecutionClass": "FLEX") to cut cost.
• No AWS access keys stored in Airflow or GitHub.
• A basic data check runs after the job (row counts, totals vs source). That last point is the one teams skip, and it is the one that hurts. In one audit for a game studio, we found €454K of Stripe revenue missing from their reporting. The pipeline ran green every day. It just was not checking that the numbers matched.

FAQ#

Do I need Amazon MWAA to run AWS Glue jobs from Airflow?#

No. The GlueJobOperator works on any Airflow deployment with the Amazon provider installed: self-hosted, Docker, Kubernetes or Astronomer. MWAA is simply the lowest-effort option on AWS because the provider and IAM integration are already set up.

How do I pass parameters from Airflow to an AWS Glue job?#

Use the script_args argument of GlueJobOperator, with keys prefixed by --, for example {"--run_date": "{{ ds }}"}. It is templated, so you can use Airflow macros. In the Glue script, read the values with getResolvedOptions(sys.argv, ["run_date"]).

Should I use deferrable mode for Glue jobs?#

Yes, in most cases. Without it, an Airflow worker sits idle polling Glue for the entire run. With deferrable=True, the triggerer handles the waiting and the worker slot is freed for other tasks. It requires a running triggerer, which MWAA and Astronomer provide.

Can Airflow trigger a Glue job when new files land in S3?#

Yes. The simplest approach is a deferrable S3KeySensor at the start of a scheduled DAG, as shown above. For true event-driven runs on Airflow 3, you can route S3 event notifications to SQS and use an asset watcher to trigger the DAG as soon as files arrive.

What is the difference between GlueJobOperator and GlueJobSensor?#

GlueJobOperator starts a run and, by default, waits for it to finish. GlueJobSensor only waits for a run that was started elsewhere. Use the sensor when you set wait_for_completion=False on the operator, or when another system starts the job.

Key takeaways#

• Airflow decides when and in what order things run. Glue does the heavy Spark work. Keep those roles separate.
• Parameterize every Glue job by date and make writes idempotent, so retries and backfills are safe.
• Use deferrable=True, timeouts and max_active_runs=1 to keep the pipeline cheap and predictable.
• Deploy scripts from CI and authenticate with IAM roles, not keys.
• Add a data check after the job. A green pipeline is not the same as correct numbers. Want your Glue, Airflow and S3 pipelines set up properly, or cleaned up so they stop breaking at 6 a.m.? We design and run data integration pipelines that your team owns. Book a call with Dublo.