Home   About Me   Blog  

How to use any programming language on AWS lambda.(01 October 2018)

Intro
AWS lambda is a service that lets you run code without provisioning or managing servers.
At the time of writing this blogpost, AWS lambda natively supports only the following programming languages/runtimes; Node.js, Java, Python, .NET Core and Go
Sometimes you may have a usecase for AWS lambda(serverless) but the programming language that you are using is not natively supported by AWS lambda.
If that is the case, worry not; you can still use your favorite programming language with AWS lambda even if AWS lambda does not natively support that language.
In this article, we explore how to do exactly that.

What up now
Let's say that our favorite programming language is Nim and we want to run our program written with Nim on AWS lambda.
The way to do that(at least the one we will explore in this blog) is by creating a shim using a programming language that is already supported on AWS lambda.
The shim will accept lambda requests, it will then pass those requests to our Nim program and finally it will accept/listen for responses from our Nim program and marshal them into a form that AWS lambda can understand.

We will use python to create the shim(but you can use any of the other AWS lambda supported languages.)
The workflow is;

  1. The python program/shim gets a request from AWS lambda.
  2. it serializes that request into json.
  3. it writes that json into stdin
  4. The Nim program reads from stdin
  5. it unmarshals what it has read from stdin and acts on it.
  6. it creates a json marshaled response
  7. it writes that json response to stdout
  8. The python program/shim reads that response from stdout
  9. it unmarshals what it read(the response)
  10. it sends the response back to AWS lambda.
So the python program and the Nim program communicate using JSON over stdin/stdout.
NB: You do not have to use json over stdin/stdout if you do not want; you could use rpc, unix sockets etc.

We have talked about python talking to the Nim program, but how will python talk to the Nim program if we can't run Nim on AWS lambda?
The answer lies with the fact that AWS supports running arbitrary executables
So, as long as you are using a programming language that can compile binaries(that are either statically linked or built for the matching version of Amazon Linux) then you are good to go.

The Shim
Let's create our shim python program, lambda.py;

# lambda.py
import os
import sys
import json
import shutil
import traceback
import subprocess

os.environ["PATH"] = (
    os.environ["PATH"] + ":" + os.environ.get("LAMBDA_TASK_ROOT", "LAMBDA_TASK_ROOT")
)

def handle(event, context):
    try:
        proc = subprocess.Popen(
            ["./main"],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            universal_newlines=True
        )

        # write to binary program
        write_data = json.dumps({"event": event}) + "\n"
        proc.stdin.write(write_data)
        proc.stdin.flush()

        # read from binary program
        line = proc.stdout.readline()

        event = json.loads(line)

        proc.stdin.close()
        proc.stdout.close()

        proc.terminate()
        proc.wait(timeout=1.2)
    except Exception as e:
        exc_type, exc_value, exc_traceback = sys.exc_info()
        traceback.print_exception(exc_type, exc_value, exc_traceback, limit=2, file=sys.stdout)
        return {"error": repr(e)}
    return event
                
The os.environ["PATH"] part is necessary for you to be able to reference(via relative path) any binaries that you upload.
So, the python shim;
The part where you need to be careful in writing the shim is: These precautions are necessary to avoid deadlocks [1] [2]

The Nim program
Our actual Nim program won't do much; NB: the previous caveats about eliminating deadlocks also apply to the Nim program(or whichever programming language you are using.)
The nim program, main.nim;

# main.nim
import json
import times
import system

# read from stdin
var request: string = readLine(stdin)
let jsonReq = parseJson(request)

# for this example, we only use event; in real life you may also want to use context as well.
let event = jsonReq["event"]

var response =  %*
    {
    "EchoEvent": event,
    "Message": "hello fom Nim version: " & system.NimVersion,
    "CurrentTime":  format(times.now(), "d MMMM yyyy HH:mm")

    }

# write to stdout
echo response
                
You can execute the Nim program locally on your machine:

echo '{"event": "myLambdaEventName", "context": "myLambdaContext"}' | nim compile --run main.nim

Hint: operation successful (34568 lines compiled; 0.703 sec total; 59.941MiB peakmem; Debug Build) [SuccessX]

{"EchoEvent":"myLambdaEventName","Message":"hello fom Nim version: 0.19.0","CurrentTime":"1 October 2018 13:55"}
                


Tie it up together
We have created the python shim and the Nim program, to tie it all together we need to: To create the Nim executable, run;

nim c -d:release main.nim
                
This creates an executable in the current directory, the executable is called main
The command, nim c -d:release main.nim should be ran from a Linux 64bit computer(since that is the architecture of the AWS lamda machines.)
If your programming language supports cross compilation, you can cross-compile to linux 64bit(without having to change machines). Nim supports cross compilation but - this was my first Nim program ever - I was not able to cross-compile so I had to compile using a 64bit machine running ubuntu.

Next we create a zipfile

zip mylambda.zip main lambda.py
                
This creates a zip file called mylambda.zip in the current directory.

Run in AWS lambda
Finally, let's open up the AWS lambda console and:

1. create an AWS lambda function where we chose Python 3.6 as the runtime:

create lambda function

2. upload our zip file:

upload lambda zip file
make sure the runtime is Python 3.6 and Handler is lambda.handle ie our python file is called lambda.py and the function inside it is def handle(event, context)

3. click save and you'll be taken to a page that shows the unzipped file contents:

unzipped lambda contents

4. create a AWS lamda test:

create lambda test

5. execute/run that lambda test:

execute lambda test


As you can see, the lambda function was executed/invoked and it ran calling our Nim binary which returned results accordingly.

Conclusion
We have been able to use our language of choice(Nim) on AWS lambda even though lambda doesn't support Nim as a programming language.
Shims are a powerful way of connecting two seemingly incompatible systems.
If you are interested in running your programs on AWS lambda without having to write your own shim, TJ Holowaychuk has you covered with his up project which I had written about in an earlier blog post
This all begs the question; instead of AWS rolling out access to lambda one language at a time, why cant they create a common interface/shim that all languages can target?

All the code in this blogpost can be found at: https://github.com/komuw/komu.engineer/tree/master/blogs/05


Update(06 March 2021);
On 29th November 2018(~60 days after I wrote this blogpost), AWS announced Lambda support for any programming language.
And no, I don't think this blogpost had anything to do with it.