A Real World PHP Lambda App Part 7: Load Testing with serverless-artillery

Submitted by nigel on Thursday 12th April 2018

Load testing is an important aspect of delivering an enterprise solution. Load testing can monitor the system's response times for each of the transactions during a set period of time. Load testing can also raise attention to any problems in the application software and fix these bottlenecks before they become more problematic. 

So we've established that load testing is necessary. Tools for load testing Lambda functions are scarce, but there is serverless-artillery, which is of course based on the artillery nodejs application. So this is our choice for load testing our voting app. 

Installing serverless-artillery
First step is to install globally the parent artillery app
$ npm install -g artillery
Now lets install serverless-artillery
$ npm install -g serverless-artillery
And let's see what we've got on our system after that
$ ls -als /usr/local/lib/node_modules/
total 48
4 drwxr-xr-x 12 nigel www-data 4096 Apr 12 11:11 .
4 drwxr-xr-x  6 root  root     4096 Apr 17  2017 ..
4 drwxrwxr-x  7 nigel www-data 4096 Apr 12 10:07 artillery
4 drwxrwxr-x  4 nigel www-data 4096 Apr 17  2017 casper-chai
4 drwxrwxr-x  9 nigel www-data 4096 Apr 17  2017 casperjs
4 drwxrwxr-x  6 nigel www-data 4096 Apr 30  2017 localtunnel
4 drwxrwxr-x  6 nigel www-data 4096 Apr 17  2017 mocha
4 drwxrwxr-x  4 nigel www-data 4096 Apr 17  2017 mocha-casperjs
4 drwxrwxr-x  4 nigel www-data 4096 Nov 26 19:30 n
4 drwxrwxr-x 11 nigel www-data 4096 Nov 26 19:26 npm
4 drwxrwxr-x  6 nigel www-data 4096 Dec 23 11:15 serverless
4 drwxrwxr-x  6 nigel www-data 4096 Apr 12 11:10 serverless-artillery
Your directory listing of the available packages will obviously be different - but it looks like we are good to go.
Configuring serverless-artillery
We now need a home for our serverless-artillery configuration. I opted (perhaps confusingly) to create a directory called serverless-artillery in the usual place for my projects - /var/www/html. Once that is created, we can run the configure command.
$ cd /var/www/html
$ mkdir serverless-artillery
$ cd serverless-artillery
$ slsart configure
Now if we look at the directory we can see the normal artefacts we would expect in a severless project.
$ ls -las
total 112
 4 drwxrwxr-x   3 nigel    www-data  4096 Apr 12 11:27 .
 4 drwxrwxr-x  17 www-data www-data  4096 Apr 12 11:16 ..
48 -rw-rw-r--   1 nigel    www-data 46843 Apr 12 11:27 handler.js
 4 drwxrwxr-x 152 nigel    www-data  4096 Apr 12 11:27 node_modules
 4 -rw-rw-r--   1 nigel    www-data   222 Apr 12 11:27 package.json
44 -rw-rw-r--   1 nigel    www-data 41915 Apr 12 11:27 package-lock.json
 4 -rw-rw-r--   1 nigel    www-data  1329 Apr 12 11:27 serverless.yml
The serverless.yml needs editing because the AWS region is completely missing so it will default to us-east-1 unless we change it.
$ head -20 serverless.yml | tail -6
provider: # Using Node JS v4.3 on AWS
  name: aws
  region: eu-west-1
  runtime: nodejs6.10
    - Effect: "Allow"
Above you can see I have now added a region of eu-west-1 which reflects the region I am operating out of.
Deploying the serverless-artillery Lambda Function
Deployed function
Something I hadn't appreciated when I first installed and configured serverless-artillery is that it is a standalone Lambda function in its own right. I assumed incorrectly it somehow was attached to a pre-existing Lambda function as a plugin. The documentation for serverless-artillery didn't make this clear so I added a GitHub issue and the problem will be addressed by the project's maintainer. So now we know it's a standalone function, it needs deploying. The deployment is a little different - see below. The screenshot above shows the function listed on the AWS console.
$ slsart deploy
	Deploying Lambda to AWS...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (5.45 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
Serverless: Stack update finished...
Service Information
service: serverless-artillery-Bk1YNhhsM
stage: dev
region: eu-west-1
stack: serverless-artillery-Bk1YNhhsM-dev
api keys:
  loadGenerator: serverless-artillery-Bk1YNhhsM-dev-loadGenerator
	Deploy complete.
Create New 'test' Stage
Test Lambda Functions
Now would be a really good time to create a new stage called test to keep my testing environment separate from my development stage. To enable us to take advantage of the ability to flip between different stages on the command line, we have to make a minor change to the serverless.yml file. Change the stage to the output from the cat / grep combination below.
$ cd /var/www/html/serverless-vote
$ cat serverless.yml | grep stage | grep dev
  stage: ${opt:stage, 'dev'}
This effectively means take the stage from the command line option stage but if that doesn't exist, use dev as the default. Now we can do a regular deploy with a switch for the stage. Don't forget we will also need to sync assets for our new test stage.
$ sls deploy --stage test --no-color
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (12.53 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
Serverless: Stack update finished...
Service Information
service: vote
stage: test
region: eu-west-1
stack: vote-test
api keys:
  GET - https://bnz9qsvtm1.execute-api.eu-west-1.amazonaws.com/test/
  POST - https://bnz9qsvtm1.execute-api.eu-west-1.amazonaws.com/test/
  POST - https://bnz9qsvtm1.execute-api.eu-west-1.amazonaws.com/test/thank_you
  vote_get: vote-test-vote_get
  vote_post: vote-test-vote_post
  thank_you: vote-test-thank_you
$ sls syncToS3 --stage test
Serverless: s3,sync,dist/,s3://vote-test-webapps3bucket-ncaevy1jcsry/
upload: dist/images/favicon.ico to s3://vote-test-webapps3bucket-ncaevy1jcsry/images/favicon.ico
upload: dist/css/default.css to s3://vote-test-webapps3bucket-ncaevy1jcsry/css/default.css
upload: dist/images/badzilla-logo32x32.png to s3://vote-test-webapps3bucket-ncaevy1jcsry/images/badzilla-logo32x32.png
Serverless: stderr undefined
Serverless: Successfully synced to the S3 bucket
It should now be possible to point a web browser at the url provided by AWS, and by navigating to the Lambda function list on the AWS console, the test stage functions can be seen (see screenshot above).
Auto Scale DynamoDB
DynamoDB Auto Scaling

We are going to load test the vote_post function, so we should ensure that the DynamoDB Auto Scaling feature is enabled, along with minimum and maximum read and write capacity parameters. Navigate to Services->DynamoDB->Tables->{vote_test}->Capacity and you can see in the screenshot above I've elected to put the minimum capacity for read and writes to by 10 per second. I am not intending to really stretch DynamoDB too far for my load testing since this is a hobby project and DynamoDB costs can escalate out of hand quickly without prudence. 

Create the Artillery Script
serverless-artillery uses Artillery scripts to run load tests. They are easy to create and are constructed in yml format. Below is the script for our vote_post function load test. We are going to simulate the submission of our POSTed vote form multiple time. Note that despite the test will be run from our serverless-artillery directory, I have created the script in the serverless-vote directory so it is included in our vote project's git repo.
$ pwd
$ mkdir artillery
$ cd artillery
$ cat vote_post.yml
  # this hostname will be used as a prefix for each URI in the flow unless a complete URI is specified
  target: "https://bnz9qsvtm1.execute-api.eu-west-1.amazonaws.com"
    - duration: 300
      arrivalRate: 1
      rampTo: 10
  - name: "Post form"
      - post:
          url: "/test"
            first_name: "Fred"
            last_name: "Blogs"
            optradio: "phpstorm"
Running the Load Test
Now the time has come to execute our test. We navigate back to the serverless-artillery directory and invoke the Lambda function with a path back to our yml script.
$ cd ../serverless-artillery/
$ slsart invoke -p ../serverless-vote/artillery/vote_post.yml 
	Invoking test Lambda
	Your function has been invoked. The load is scheduled to be completed in 300 seconds.
Ok so now our test will run for five minutes.
Analysing the Load Test Results
DynamoDB Overview
DynamoDB Detail
DynamoDB List Items
Lambda Metrics

Once the load test completes there are monitoring screens that can be examined to determine the results. By navigating to the Metrics tab on the AWS Console DynamoDB vote_test, we can see the topmost screenshot above. The red line on the graph indicates where I have set the minimum provisioning thresholds. If we look at the Write Capacity metric we can also see a small blue line of the number of writes per second during the test. We didn't quite achieve ten per second. It's also useful to look at the rightmost metric - Throttled Write Requests. Thankfully the count is zero and this is important. I have noticed empirically that with throttling, there comes a nasty side effect - the AWS SDK putItem can timeout thus causing an exception to be thrown in the model code. 

The second screenshot shows a zoomed view of the Write Capacity with five minute intervals. The most writes per second reached 8 diring our test. 

By navigating to the Items tab (screenshot 3) we can see that we have a whole swathe of records that have been created in the DynamoDB database - this is exactly what we would expect. 

Next we navigate to Services->Lambda->Functions->{vote_test_vote_post}->Monitoring and we can see three metrics - all of which are important. We achieved over 1500 invocations of the function in the five minutes (leftmost graphic). The Invocation Duration is interesting - we averaged a completion time of 67 milliseconds which is very fast. There was a maximum of 343 milliseconds - still very acceptable, and clearly juding where the maximum peak is in comparison to the average, there can't have been too many functions that were near the top end of that maximum. The rightmost metric - Invocation Errors - is crucial. If we got time outs caused by DynamoDB write throttling, we would see some activity on this metric. Thankfully there is nothing so we know all is good. 

We can also use the command line - for instance to get the number of records created during the run we can issue:
$ aws dynamodb scan --table-name=vote_test
{snipped output}
            "optradio": {
                "S": "phpstorm"
            "datetime": {
                "S": "2018-04-13T16:10:18"
            "surname": {
                "S": "Blogs"
            "id": {
                "S": "6949cf80-336f-4536-a86d-4be164861597"
            "forename": {
                "S": "Fred"
    "ScannedCount": 1596, 
    "ConsumedCapacity": null
Hopefully this has given you a good taster for using artillery and serverless-artillery. As you can see, it really is quite easy.