How to use any programming language on AWS lambda.(01 October 2018)

Intro
AWS lambda is a service that lets you run code without provisioning or managing servers.
At the time of writing this blogpost, AWS lambda natively supports only the following programming languages/runtimes; Node.js, Java, Python, .NET Core and Go
Sometimes you may have a usecase for AWS lambda(serverless) but the programming language that you are using is not natively supported by AWS lambda.
If that is the case, worry not; you can still use your favorite programming language with AWS lambda even if AWS lambda does not natively support that language.
In this article, we explore how to do exactly that.

What up now
Let's say that our favorite programming language is Nim and we want to run our program written with Nim on AWS lambda.
The way to do that(at least the one we will explore in this blog) is by creating a shim using a programming language that is already supported on AWS lambda.
The shim will accept lambda requests, it will then pass those requests to our Nim program and finally it will accept/listen for responses from our Nim program and marshal them into a form that AWS lambda can understand.

We will use python to create the shim(but you can use any of the other AWS lambda supported languages.)
The workflow is;

1. The python program/shim gets a request from AWS lambda.
2. it serializes that request into json.
3. it writes that json into stdin
4. The Nim program reads from stdin
5. it unmarshals what it has read from stdin and acts on it.
6. it creates a json marshaled response
7. it writes that json response to stdout
8. The python program/shim reads that response from stdout
9. it unmarshals what it read(the response)
10. it sends the response back to AWS lambda.

So the python program and the Nim program communicate using JSON over stdin/stdout.
NB: You do not have to use json over stdin/stdout if you do not want; you could use rpc, unix sockets etc.

We have talked about python talking to the Nim program, but how will python talk to the Nim program if we can't run Nim on AWS lambda?
The answer lies with the fact that AWS supports running arbitrary executables
So, as long as you are using a programming language that can compile binaries(that are either statically linked or built for the matching version of Amazon Linux) then you are good to go.

The Shim
Let's create our shim python program, lambda.py;

# lambda.py
import os
import sys
import json
import shutil
import traceback
import subprocess

os.environ["PATH"] = (
    os.environ["PATH"] + ":" + os.environ.get("LAMBDA_TASK_ROOT", "LAMBDA_TASK_ROOT")
)

def handle(event, context):
    try:
        proc = subprocess.Popen(
            ["./main"],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            universal_newlines=True
        )

        # write to binary program
        write_data = json.dumps({"event": event}) + "\n"
        proc.stdin.write(write_data)
        proc.stdin.flush()

        # read from binary program
        line = proc.stdout.readline()

        event = json.loads(line)

        proc.stdin.close()
        proc.stdout.close()

        proc.terminate()
        proc.wait(timeout=1.2)
    except Exception as e:
        exc_type, exc_value, exc_traceback = sys.exc_info()
        traceback.print_exception(exc_type, exc_value, exc_traceback, limit=2, file=sys.stdout)
        return {"error": repr(e)}
    return event
                

The os.environ["PATH"] part is necessary for you to be able to reference(via relative path) any binaries that you upload.
So, the python shim;

• receives request from AWS lambda(lambda calls the handle function for every invocation)
• runs the binary Nim program(./main) via python's subprocess lib, converts the lamda request to json and it then writes that request to the Nim program's stdin
• it reads from the Nim program's stdout using readline and converts what it has read(the response) from json
• it returns that response to AWS lambda

The part where you need to be careful in writing the shim is:

• you need to make sure that you terminate whatever you are writing with a newline, +"\n"
• you should actually flush what you have written, proc.stdin.flush()
• you should use readline() to read from Nim program's stdout as opposed to read()

These precautions are necessary to avoid deadlocks [1] [2]

The Nim program
Our actual Nim program won't do much;

• it will just read from stdin
• create a response that has the event it got from the shim echoed back, the current time and a message containing Nim's version
• it will convert that response to json and write it to stdout.

NB: the previous caveats about eliminating deadlocks also apply to the Nim program(or whichever programming language you are using.)
The nim program, main.nim;

# main.nim
import json
import times
import system

# read from stdin
var request: string = readLine(stdin)
let jsonReq = parseJson(request)

# for this example, we only use event; in real life you may also want to use context as well.
let event = jsonReq["event"]

var response =  %*
    {
    "EchoEvent": event,
    "Message": "hello fom Nim version: " & system.NimVersion,
    "CurrentTime":  format(times.now(), "d MMMM yyyy HH:mm")

    }

# write to stdout
echo response
                

You can execute the Nim program locally on your machine:

echo '{"event": "myLambdaEventName", "context": "myLambdaContext"}' | nim compile --run main.nim

Hint: operation successful (34568 lines compiled; 0.703 sec total; 59.941MiB peakmem; Debug Build) [SuccessX]

{"EchoEvent":"myLambdaEventName","Message":"hello fom Nim version: 0.19.0","CurrentTime":"1 October 2018 13:55"}
                

Tie it up together
We have created the python shim and the Nim program, to tie it all together we need to:

• compile the Nim program into an executable(statically linked, ideally)
• create a zip file containing the Nim executable and the python program
• upload that zip file to AWS lambda

To create the Nim executable, run;

nim c -d:release main.nim
                

This creates an executable in the current directory, the executable is called main
The command, nim c -d:release main.nim should be ran from a Linux 64bit computer(since that is the architecture of the AWS lamda machines.)
If your programming language supports cross compilation, you can cross-compile to linux 64bit(without having to change machines). Nim supports cross compilation but - this was my first Nim program ever - I was not able to cross-compile so I had to compile using a 64bit machine running ubuntu.

Next we create a zipfile

zip mylambda.zip main lambda.py
                

This creates a zip file called mylambda.zip in the current directory.

Run in AWS lambda
Finally, let's open up the AWS lambda console and:

1. create an AWS lambda function where we chose Python 3.6 as the runtime:



2. upload our zip file:


make sure the runtime is Python 3.6 and Handler is lambda.handle ie our python file is called lambda.py and the function inside it is def handle(event, context)

3. click save and you'll be taken to a page that shows the unzipped file contents:



4. create a AWS lamda test:



5. execute/run that lambda test:




As you can see, the lambda function was executed/invoked and it ran calling our Nim binary which returned results accordingly.

Conclusion
We have been able to use our language of choice(Nim) on AWS lambda even though lambda doesn't support Nim as a programming language.
Shims are a powerful way of connecting two seemingly incompatible systems.
If you are interested in running your programs on AWS lambda without having to write your own shim, TJ Holowaychuk has you covered with his up project which I had written about in an earlier blog post
This all begs the question; instead of AWS rolling out access to lambda one language at a time, why cant they create a common interface/shim that all languages can target?

All the code in this blogpost can be found at: https://github.com/komuw/komu.engineer/tree/master/blogs/05


Update(06 March 2021);
On 29th November 2018(~60 days after I wrote this blogpost), AWS announced Lambda support for any programming language.
And no, I don't think this blogpost had anything to do with it.