AWS Lambda CI/CD with GitHub Actions: A Data Engineer's Guide

Most data engineers ship their first AWS Lambda function the same way: copy the code, open the console, paste, click Deploy. It works until the second change, when nobody knows which version is live or who edited it.

This guide is for data engineers who want a boring, repeatable deployment instead. We will take a small Python extraction job, package it as a container image, and deploy it to Lambda with a GitHub Actions pipeline that runs on every push to main.

What is AWS Lambda, in one paragraph?#

AWS Lambda is a compute service that runs your code without you provisioning or managing servers. You supply code in a supported runtime (or a container image), you define what triggers it, and AWS handles the rest.

You pay only for the compute time you consume, billed per millisecond. If your function runs for 3 minutes a day, you pay for 3 minutes. When it is not running, you pay nothing.

When should a data engineer use AWS Lambda?#

Lambda is a great fit for small, event-driven or scheduled jobs. In my work, it shows up in three patterns again and again:

• Extracting data from an API on a schedule and landing it in S3.
• Processing a file as soon as it is uploaded to an S3 bucket.
• Consuming small batches from Kinesis, SQS or DynamoDB Streams. The hard limit to remember is the timeout: a Lambda function can run for 15 minutes at most. There are other limits worth knowing before you commit, all listed in the AWS Lambda quotas documentation.

If your workload lives in the Glue column, I covered that setup in how to orchestrate AWS Glue jobs with Airflow.

The example: a Python job that extracts API data to S3#

Say we have a Python script on our machine that calls an API and stores the result as a JSON file in S3. We want it on Lambda, triggered once a day.

Here is the repository layout we will end up with:

lambda-api-extract/
├── .github/
│   └── workflows/
│       └── deploy.yml
├── src/
│   └── app.py
├── tests/
│   └── test_app.py
├── Dockerfile
└── requirements.txt

And the handler itself:

# src/app.py
import json
import os
from datetime import datetime, timezone

import boto3
import requests

s3 = boto3.client("s3")

API_URL = os.environ["API_URL"]
BUCKET = os.environ["BUCKET_NAME"]
PREFIX = os.environ.get("S3_PREFIX", "raw/api")


def handler(event, context):
    response = requests.get(API_URL, timeout=30)
    response.raise_for_status()
    payload = response.json()

    run_ts = datetime.now(timezone.utc)
    key = f"{PREFIX}/dt={run_ts:%Y-%m-%d}/extract_{run_ts:%H%M%S}.json"

    s3.put_object(
        Bucket=BUCKET,
        Key=key,
        Body=json.dumps(payload).encode("utf-8"),
        ContentType="application/json",
    )

    records = len(payload) if isinstance(payload, list) else 1
    return {"status": "ok", "s3_key": key, "records": records}

Pretty basic. Writing to a date-partitioned prefix (dt=YYYY-MM-DD) means Athena, Glue or DuckDB can read the raw files later without extra work.

Why not just paste the code in the Lambda console?#

You could. Open the AWS console, create a function, paste the code, done.

Now you want to change it. You go back to the console. How do you track what changed, who changed it, and how to undo it? You don't.

We are software engineers. Code that runs in production lives in version control and gets deployed by a pipeline. Here we will use Git and GitHub Actions. It takes about an hour to set up and saves you from every "who touched the Lambda?" conversation afterwards. I wrote more about the general principle in CI/CD for data engineering.

Zip file or container image: how should you package a Lambda?#

Lambda accepts two package types: a .zip archive or a container image stored in Amazon ECR. I default to container images for data jobs.

The main reason is portability. The same image runs on Lambda today and on ECS Fargate tomorrow if the job outgrows the 15 minute limit. The second reason is size: data libraries like pandas, pyarrow or DuckDB blow past the zip limits quickly.

Who says ECR, says Docker. Who says Docker, says Dockerfile:

# Dockerfile
FROM public.ecr.aws/lambda/python:3.13

COPY requirements.txt ${LAMBDA_TASK_ROOT}/
RUN pip install --no-cache-dir -r ${LAMBDA_TASK_ROOT}/requirements.txt

COPY src/ ${LAMBDA_TASK_ROOT}/

CMD ["app.handler"]

# requirements.txt
requests>=2.32,<3

The AWS base image already ships boto3 and the Lambda Runtime Interface Client, so you only install what your code adds. CMD ["app.handler"] tells Lambda to call the handler function in app.py.

What does the CI/CD pipeline look like?#

The pipeline has one job: every push to main should test the code, build a new image, push it to ECR and point the Lambda function at it.

My rule for CI/CD: think CLI first#

Here is my tip: when you hear "CI/CD", think "CLI".

If you deploy something on AWS, start with the AWS CLI. For Databricks, the Databricks CLI. For GCP, the gcloud CLI. Anything you click in a console can be done from the CLI, and anything you can do from the CLI can run in a pipeline.

The AWS CLI reference is huge, but a Ctrl+F for "lambda" gets you what you need in seconds.

Creating a Lambda function from the CLI#

Creating a function from a container image is one command, create-function, with fairly obvious parameters: the function name, the package type, the image URI, the execution role and the timeout.

aws lambda create-function \
  --function-name api-extract \
  --package-type Image \
  --code ImageUri=123456789012.dkr.ecr.eu-west-3.amazonaws.com/api-extract:abc1234 \
  --role arn:aws:iam::123456789012:role/api-extract-lambda-execution \
  --timeout 300 \
  --memory-size 512

Updating a Lambda function with a new image#

Updating is the same logic with update-function-code: the function name and the new image URI.

aws lambda update-function-code \
  --function-name api-extract \
  --image-uri 123456789012.dkr.ecr.eu-west-3.amazonaws.com/api-extract:def5678

That is 80% of the pipeline. The remaining 20% is authentication, building the image and wiring it into GitHub Actions.

How do you connect GitHub Actions to AWS without access keys?#

The original version of this setup stored an AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as GitHub secrets. Please don't do that anymore. Long-lived keys leak, never expire and are painful to rotate.

Use OpenID Connect (OIDC) instead. GitHub issues a short-lived token for each workflow run, and AWS exchanges it for temporary credentials on a role you control. You do this setup once per AWS account.

Step 1: register GitHub as an identity provider#

aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com

Step 2: create a deploy role that only your repo can assume#

The trust policy restricts the role to your repository and the main branch:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:your-org/lambda-api-extract:ref:refs/heads/main"
        }
      }
    }
  ]
}

Then give the role only what the pipeline needs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ecr:GetAuthorizationToken",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:InitiateLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:CompleteLayerUpload",
        "ecr:PutImage"
      ],
      "Resource": "arn:aws:ecr:eu-west-3:123456789012:repository/api-extract"
    },
    {
      "Effect": "Allow",
      "Action": [
        "lambda:GetFunction",
        "lambda:CreateFunction",
        "lambda:UpdateFunctionCode",
        "lambda:GetFunctionConfiguration"
      ],
      "Resource": "arn:aws:lambda:eu-west-3:123456789012:function:api-extract"
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::123456789012:role/api-extract-lambda-execution"
    }
  ]
}

Keep two roles separate in your head. The deploy role is what GitHub Actions assumes to push images and update the function. The execution role is what the Lambda function runs as at runtime (here: write to the S3 bucket and to CloudWatch Logs). Never merge them.

Step 3: create the ECR repository#

aws ecr create-repository \
  --repository-name api-extract \
  --image-tag-mutability IMMUTABLE \
  --image-scanning-configuration scanOnPush=true

Immutable tags mean an image tagged with a Git SHA can never be overwritten. That is what makes rollbacks trustworthy.

The complete GitHub Actions workflow#

Add the deploy role ARN and the execution role ARN as repository secrets (AWS_DEPLOY_ROLE_ARN, LAMBDA_EXECUTION_ROLE_ARN). They are not sensitive like keys, but it keeps account IDs out of the code.

# .github/workflows/deploy.yml
name: deploy-api-extract-lambda

on:
  push:
    branches: [main]
  pull_request:
  workflow_dispatch:

permissions:
  id-token: write   # required for OIDC
  contents: read

env:
  AWS_REGION: eu-west-3
  ECR_REPOSITORY: api-extract
  FUNCTION_NAME: api-extract

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: false

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - uses: actions/setup-python@v6
        with:
          python-version: "3.13"

      - name: Install dependencies
        run: pip install -r requirements.txt boto3 pytest ruff

      - name: Lint
        run: ruff check src tests

      - name: Unit tests
        env:
          PYTHONPATH: src
        run: pytest -q

  deploy:
    needs: test
    if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v5

      - name: Configure AWS credentials (OIDC)
        uses: aws-actions/configure-aws-credentials@v5
        with:
          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Log in to Amazon ECR
        id: ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push image
        id: image
        env:
          REGISTRY: ${{ steps.ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          IMAGE_URI="$REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"
          docker buildx build \
            --platform linux/amd64 \
            --provenance=false \
            -t "$IMAGE_URI" \
            --push .
          echo "uri=$IMAGE_URI" >> "$GITHUB_OUTPUT"

      - name: Create or update Lambda function
        env:
          IMAGE_URI: ${{ steps.image.outputs.uri }}
          EXECUTION_ROLE_ARN: ${{ secrets.LAMBDA_EXECUTION_ROLE_ARN }}
        run: |
          if aws lambda get-function --function-name "$FUNCTION_NAME" > /dev/null 2>&1; then
            echo "Updating $FUNCTION_NAME"
            aws lambda update-function-code \
              --function-name "$FUNCTION_NAME" \
              --image-uri "$IMAGE_URI" \
              --publish
            aws lambda wait function-updated-v2 --function-name "$FUNCTION_NAME"
          else
            echo "Creating $FUNCTION_NAME"
            aws lambda create-function \
              --function-name "$FUNCTION_NAME" \
              --package-type Image \
              --code ImageUri="$IMAGE_URI" \
              --role "$EXECUTION_ROLE_ARN" \
              --architectures x86_64 \
              --timeout 300 \
              --memory-size 512 \
              --environment "Variables={API_URL=https://api.example.com/v1/orders,BUCKET_NAME=my-raw-bucket}"
            aws lambda wait function-active-v2 --function-name "$FUNCTION_NAME"
          fi

A few details that matter more than they look:

• permissions: id-token: write is what allows the workflow to request an OIDC token. Without it, the AWS credentials step fails.
• --provenance=false stops Buildx from attaching an attestation manifest. Lambda rejects those images with a "media type not supported" error, which is a classic first-deploy headache.
• --platform linux/amd64 matches the x86_64 architecture of the function. If you switch to Graviton (arm64, usually cheaper), change both together.
• aws lambda wait blocks until Lambda has finished applying the change, so the job fails loudly if the update fails instead of reporting a green run.
• environment: production lets you add a required reviewer in GitHub before anything hits AWS, if your team wants a manual gate.
• Pull requests run the test job only. Nothing deploys until the code is merged. Push to main and the first run creates the function. Every run after that updates it.

Adding tests to the pipeline#

The original version of this article skipped tests to keep things simple. A pipeline that deploys untested code just ships bugs faster, so here is the minimum I put in every repo:

# tests/test_app.py
import os

os.environ.setdefault("API_URL", "https://api.example.com/v1/orders")
os.environ.setdefault("BUCKET_NAME", "test-bucket")
os.environ.setdefault("AWS_DEFAULT_REGION", "eu-west-3")

from unittest.mock import MagicMock, patch

import app


def test_handler_writes_json_to_s3():
    fake_response = MagicMock()
    fake_response.json.return_value = [{"id": 1}, {"id": 2}]
    fake_response.raise_for_status.return_value = None

    with patch.object(app.requests, "get", return_value=fake_response), \
         patch.object(app, "s3") as fake_s3:
        result = app.handler({}, None)

    assert result["records"] == 2
    assert result["s3_key"].startswith("raw/api/dt=")
    fake_s3.put_object.assert_called_once()

It mocks the API and S3, so it runs in seconds with no AWS access. For integration tests, you can run the container locally with the built-in Runtime Interface Emulator:

docker build --platform linux/amd64 -t api-extract:local .
docker run --rm -p 9000:8080 \
  -e API_URL=https://api.example.com/v1/orders \
  -e BUCKET_NAME=my-dev-bucket \
  api-extract:local

# in another terminal
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

How do you schedule the Lambda to run daily?#

Use EventBridge Scheduler. It supports time zones (useful when the business says "every morning at 6" and means local time) and has built-in retries.

aws scheduler create-schedule \
  --name api-extract-daily \
  --schedule-expression "cron(0 6 * * ? *)" \
  --schedule-expression-timezone "Europe/Paris" \
  --flexible-time-window Mode=OFF \
  --target '{"Arn":"arn:aws:lambda:eu-west-3:123456789012:function:api-extract","RoleArn":"arn:aws:iam::123456789012:role/scheduler-invoke-api-extract"}'

The schedule's role only needs lambda:InvokeFunction on this function. This is a one-time setup, so I keep it out of the deploy pipeline.

How do you roll back a bad Lambda deployment?#

Because every image is tagged with an immutable Git SHA, a rollback is pointing the function at the previous image:

aws lambda update-function-code \
  --function-name api-extract \
  --image-uri 123456789012.dkr.ecr.eu-west-3.amazonaws.com/api-extract:<previous-sha>

Even better, revert the commit in Git and let the pipeline redeploy. That way Git stays the single source of truth for what is running.

When should you move from the CLI to infrastructure as code?#

The CLI approach is perfect for one function with a stable configuration. Once you manage several functions, schedules, roles and buckets, the if exists, update, else create pattern stops scaling, because it only deploys code, not configuration changes like memory or environment variables.

My usual path: start with the CLI pipeline above, move the function, roles, schedule and ECR repo into Terraform once there are three or more of them, and keep GitHub Actions to build images and run terraform apply. The "think CLI first" rule still holds, since Terraform and SAM are CLIs too.

Checklist before your first deploy#

• Code, Dockerfile and workflow live in one Git repository
• GitHub OIDC provider registered in the AWS account
• Deploy role trusted only by your repo and main branch
• Separate execution role with least-privilege S3 and CloudWatch Logs access
• ECR repository with immutable tags and scan on push
• Image built for the same architecture as the function, with provenance disabled
• Tests run before the deploy job
• EventBridge Scheduler (or an S3 event) set up to trigger the function
• A CloudWatch alarm on the function's Errors metric, so failures don't stay silent

FAQ#

Should I deploy AWS Lambda as a zip file or a container image?#

For data jobs, use a container image. It handles heavy libraries like pandas, pyarrow or DuckDB (up to 10 GB versus 250 MB unzipped for a zip), it can be tested locally with Docker, and the same image can run on ECS Fargate if the job outgrows Lambda's 15 minute limit. Zip archives are fine for tiny functions with almost no dependencies.

How do I deploy to AWS from GitHub Actions without storing access keys?#

Use OpenID Connect. Register token.actions.githubusercontent.com as an identity provider in IAM, create a role whose trust policy only accepts your repository and branch, and give the workflow id-token: write permission. The aws-actions/configure-aws-credentials action then exchanges the GitHub token for short-lived AWS credentials on each run.

Why does Lambda say my container image's media type is not supported?#

Recent versions of Docker Buildx attach provenance attestations to images by default, which creates a manifest format Lambda does not accept. Build with --provenance=false (and set the platform explicitly, for example linux/amd64) and push again.

Is AWS Lambda a good choice for ETL pipelines?#

For small, frequent jobs, yes: API extracts, file-triggered processing, light transformations and loading into S3 or a warehouse. For anything that can run longer than 15 minutes or needs distributed processing on large datasets, use AWS Glue, EMR Serverless or ECS on Fargate instead, and orchestrate everything with a scheduler like Airflow.

How do I run unit tests for a Lambda function in CI?#

Keep the handler a plain Python function, mock external calls (the API with unittest.mock, S3 with mocks or the moto library), and run pytest in a job that must pass before the deploy job starts. Add an integration test that runs the container locally with the Runtime Interface Emulator if you want extra confidence.

Key takeaways#

• Treat a Lambda function like any other production code: Git, tests, a pipeline.
• Container images on ECR give you room for real dependencies and an easy exit to Fargate.
• OIDC plus a narrowly scoped deploy role replaces long-lived AWS keys.
• Two CLI commands (create-function, update-function-code) cover the actual deploy.
• When you hear CI/CD, think CLI first. Move to Terraform or SAM when the number of resources grows. Small Lambda extracts like this one are usually the first brick of a bigger pipeline that feeds a warehouse, dbt models and dashboards. If you want that whole layer built properly, see how we approach data integration and data foundations.

Want your pipelines deployed this way without spending your team's time on it? Book a call with Dublo and we will set up tested, version-controlled deployments for your data jobs.