For work lately I've been studying Amazon Lambda, a service they provide under the AWS umbrella. The basic theory is that you can use it to serve requests, but not pay anything for the compute time that it isn't serving anyone. It also ties nicely in with Step Functions, another Amazon product that lets you describe workflows and have your data obey them.
I was tasked with building a couple Lambda functions, and also with exploring the Apex tool, meant for administering and developing Lambda functions. My manager had already set up a repository with a "hello world" example written in JS. Because JS was the first language that Lambda supported, that seemed like a good target to continue with. But I quickly got frustrated with JS because most libraries, including the AWS library and the libraries I could find for parsing Zip files, were callback based, even though in 2017 most new code is written with async/await or at the very least Promises. I messed around with promisifying those libraries but it seemed too fiddly and I started to resent the developer experience I found myself in. I noticed that Apex supported Rust and I thought I would try that instead.
Suffice it to say that I think there are still quite a few rough edges in using Rust on Apex. In particular, lots of stuff is underdocumented. I finally got a proof-of-concept working and I thought I would write down some of the things I had to struggle with to try to help the next person doing this.
In order to build a Rust application that runs on Lambda, you have to target musl, which is an alternate libc. I'm not sure if this is because the Lambda machines only have musl installed and you have to match it, or if it's just that libc isn't installed and the Rust executable has to be statically linked. Either way, this was pretty easy with rustup: just do rustup target add x86_64-unknown-linux-musl.
AWS doesn't natively support Rust, but Apex has a "shim" that lets you execute Rust through a NodeJS parent process, which is supported. Unfortunately, AWS recently disabled support for NodeJS 0.10, and the version of Apex I installed hadn't been updated yet. The fix was on master but it wasn't released yet. I spent some time struggling to figure out how to use Go to build and run a local clone of a thing, but eventually I stumbled on using go get [Apex Github repo URL].
Although the Apex Rust example doesn't include serde as an explicit dependency, you need it to be available if you are using serde_derive. Otherwise you will get the error can't find crate for _serde.
I needed access to AWS services, so I tried to use the rusoto library. This requires the Rust OpenSSL library, which needs headers from OpenSSL, which needs to be build against Musl. I got into a lot of mischief here:
Of course, Musl isn't packaged for Fedora, so you have to download it and build from source. Nothing too bad here, it seemed like I had all the headers I needed. I used a --prefix of /home/ethan/.local/musl. (Note that the configure script doesn't recognize ~.) I didn't want it to go into ~/.local because it will copy include files and libraries over whatever else was already there. Then, per the musl wiki, you can compile other projects (like OpenSSL!) using musl-gcc. ~/.local/musl/bin wasn't on my PATH so I did CC="/home/ethan/.local/musl/bin/musl-gcc".
The OpenSSL "target" that I used was linux-x86_64.
OpenSSL wouldn't build without linux, asm, and asm-generic in include, and Musl doesn't provide them. I ended up just symlinking in the system ones and that seemed to work OK.
OpenSSL 1.1 series seemed to rely on setcontext, getcontext, and makecontext, which Musl doesn't provide (see this thread for more information). There might be a configure flag you can pass to make it not require those functions, but I didn't want to spend any time on it. I was able to get the LTS release of 1.0 to build successfully, and for this exercise that was enough (but obviously this would be something worth fixing in a real project).
Of course, you need to provide a --prefix to OpenSSL to get it to find the Musl installation.
I think I had the wrong compiler flags -- I tried both -static (because that's what it said in the wiki and I didn't know any better), but that gave me a linker error that said I should recompile with -fPIC. But then I did that too and the final product wouldn't run on Lambda (but see below). Eventually I just got rid of both and it seems to have worked out fine?
The Rust OpenSSL library will invoke pkg-config to get the compiler flags it needs, and pkg-config will refuse to cross-compile unless you set an PKG_CONFIG_ALLOW_CROSS environment variable. I don't really know why this exists, since I just set it even though I was clearly out of my depth.
If you aren't careful when you build the Rust OpenSSL libraries, they can link against the system OpenSSL library instead of the Musl-based one you wanted. I think because the system library is linked dynamically against libc, the final Rust executable you get will be dynamically linked, and won't run on Lambda. (Normally, the executable I got from apex deploy was statically linked -- again, I'm not sure if this is a strict requirement or just a default or what.) Then, when you deploy your artifact, it won't start up and the logs won't tell you why. (I'm guessing the dynamic linker is failing to find something but the exec() call is returning a success to the NodeJS shim.)
In order to make sure the Rust OpenSSL library picks up your changes to environment, you can cargo clean (i.e. blow away the target directory) before doing a rebuild. I had been trying to rm target/foo/deps/openssl* and I think that didn't pick up the changes.
The OpenSSL library, when built, hard-codes a path to where it expects root certificates to be. That path can be different on your target system. (On Lambda, it's /etc/pki/tls.) You can specify it by passing --openssldir to the OpenSSL configure script.
But then, when you do make install, OpenSSL won't install anything because it will fail to write to /etc/pki/tls. This is true even when there's a --prefix argument, and it's even true when you do make install_sw. You can work around it by providing an INSTALL_PREFIX to make install as documented in the INSTALL file. Then it will "install" to INSTALL_PREFIX + PREFIX, which is a little annoying.
What I finally ended up with was:
env CC="/home/ethan/.local/musl/bin/musl-gcc" ./Configure --prefix=/home/ethan/.local/musl --openssldir=/etc/pki/tls linux-x86_64 make make INSTALL_PREFIX=/tmp/openssl-build install # Then, in the project directory: env PATH=/home/ethan/.local/musl/bin:$PATH PKG_CONFIG_ALLOW_CROSS=1 OPENSSL_DIR=/tmp/openssl-build/home/ethan/.local/musl apex deploy my-function
Your Rust program can't print to stdout because of https://github.com/apex/apex/issues/625. All output has to be through stderr.
I also hit https://github.com/rusoto/rusoto/pull/569 which hasn't made it into an official release at the time of this writing. Fortunately, Cargo lets you specify a git repo as the "version" of a dependency, so I just built against git master.
I also hit https://github.com/mvdnes/zip-rs/issues/14 because I wanted to look for one file, and if it wasn't there, look for another file, and I couldn't get the lifetimes to work out right. I ended up just duplicating some code to create two ZipArchives. Yuck.
I really enjoy working in Rust, even when it's difficult, because it feels like a "good" difficulty where I'm learning and growing as a programmer. I probably enjoy it more than I should because I spent way longer on this than I should have to demonstrate how "bleeding edge" this stuff is. My hope is that by documenting this, other people will start to use it a bit more and file off the sharp edges until it starts to be viable for everyone.
Thanks dalias on #musl, @tj on the Apex Slack, and sfackler on #rust for their invaluable help while I muddled through all this!