Running Binaries in AWS Lambda

One of the major perks of Lambda functions is their simple approach to helping us automate infrastructure tasks. We just provide the code and it does it’s thing. In this article, I’ll go through such a scenario, but with the added caveat of executing a binary from directly within the Lambda.

The outcome of this will be a database backup from a MySQL RDS instance. Now, if you’ve ever done MySQL backups before, you’ll be fully aware of mysqldump, and this is exactly what we’ll be running in our Lambda.

Let’s get started (using Python by the way!)

Firstly, we need to do some fetching of the database connection details. We’ll simply store the database host as an environment variable, along with the name of the database we want to backup. For the credentials, we’ll fetch these from AWS Parameter store.

db_host     = os.getenv('DB_HOST')
db_name     = os.getenv('DB_NAME')
file_name   = os.getenv("FILE_NAME"
ssm_client  = boto3.client('ssm')

def get_ssm_parameter(param_path):
    return ssm_client.get_parameter(
        Name=f"/{param_path}", WithDecryption=True
    )['Parameter']['Value']

The above function takes the path of the parameter, and returns the value of it (you’ll obviously want some error handling here for paths that may not exist).

Now we can move onto our lambda_handler function:

def lambda_handler(event, context):
    os.environ['PATH'] = os.environ['PATH'] = ':' + os.environ['LAMBDA_TASK_ROOT']

This line is super important, hence why I’ve left it on it’s own. This is the key to allowing us to run executables within Lambda. The following blog post has some good detail around this: https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/

Next, we can grab our parameters from Parameter store:

db_username = get_ssm_parameter("/path/to/database/username")
db_password = get_ssm_parameter("/path/to/database/password")

Okay, before we move any further, we need to create a shell script. The shell script is where we’ll run mysqldump.

#!/bin/sh

DB_HOST=$1
DB_NAME=$2
DB_USER=$3
DB_PASS=$4
FILE_NAME=$5

export MYSQL_PWD="${DB_PASS}"

/tmp/mysqldump --host ${DB_HOST} --user ${DB_USER} --max_allowed_packet=1G \
    --lock-tables=false --routines ${DB_NAME} | gzip -c > /tmp/${FILE_NAME}

We have a few variables defined here for all our basic connection details, but there’s also FILE_NAME. Since this is a database backup that we will likely want to run periodically, it makes sense to have the date appended, so we keep this dynamic and pass it through to mysqldump.
The database password is exported as an environment variable MYSQL_PWD which will be automatically picked up by mysqldump.

Finally, the command itself. You’ll notice that we run /tmp/mysqldump. In Lambda, the /tmp/ directory is where AWS provides us with 512MB of ephemeral storage, but more importantly, executables can only be ran from within this directory, so we’ll need to run the shell script and mysqldump binary from in here. We also store our final backup file here after zipping it up.

Now, back to our lambda_handler function. Before invoking the shell script, we first need to copy our script and binary to the /tmp/ directory and ensure they have permissions to execute:

shutil.copyfile('./backup.sh', '/tmp/backup.sh')
os.chmod('/tmp/backup.sh', 0o755)

shutil.copyfile('./mysqldump', '/tmp/mysqldump')
os.chmod('/tmp/mysqldump', 0o755)

Finally, we can execute the shell script via subprocess like so:

connect_str = f"/tmp/backup.sh {db_host} {db_name} {db_username} {db_password} {file_name}"
subprocess.run(connect_str, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, check=True)

Now for a bit of extra detail which may come in handy if you find yourself troubleshooting issues. AWS Lambda runs on Amazon Linux in the backend, and the packages (and package versions) that this OS has installed may differ depending on the OS version being used. The beauty of serverless means we cannot control this, but it may cause issues with binaries you want to run. Unfortunately I can’t remember if it was mysqldump or gzip that I encountered issues with OS binary dependencies, but I had to bundle the binaries for libssl.so.1.1 and libcrypto.so.1.1 along with my function. Of course, I also had to include the binary for gzip itself. Something to keep in mind!

Thanks for reading, and good luck! 🙂