Overview

Single AMI deployments are a way to deploy the Kill Bill stack, that is both Kill Bill server and the administrative UI (KAUI), in the same EC2 instance. We provide a Single-Tier option that is suitable for testing and experiments, and a Multi-Tier option for production deployments. Both of these deployment options rely on the same underlying AMI, but for the Multi-Tier option there is a list of prerequisite — e.g external database — along with a startup script to be provided when launching the instance.

Single-Tier Deployments

In order to get started as quickly as possible we offer an AMI that also bundles the mysql database as part of the same image. This makes it very easy to get started, as this only requires the click of a button. It is perfect for test environments but we don’t recommend it for production deployments.

After launching the AMI you should have a setup that looks like the following:

single tier ami deployment

In this model, the AMI comes with a running instance of nginx that acts as a reverse proxy and spreads the incoming traffic to either Kill Server or KAUI based on the incoming port.

In order to get started, follow the instructions below:

  1. Start the installation process by going to AWS Marketplace: click Continue to Subscribe and accept Terms. You will need some AWS permissions aws-marketplace:* to click on the Subscribe button.

  2. After a few minutes, click Continue to Configuration and Continue to Launch

    • Make sure you enable the `auto-assign `public IP:

std enable public ip

  • Make sure you open the ingress port 443, and 8443 on your security group:

std security group

  • Make sure you specify an existing known key-pair (for SSH access)

std key pair

Upon success, once the instance is running, Kill Bill is running on port 8443 and Kaui on port 443.

You can log-in to Kaui by going to https://<INSTANCE_IP_>:443 (make sure your IP address can address this port, or add it to the security group as needed). Default credentials are: admin/password. The first few requests might be a bit slow as Kill Bill initializes itself.

Take a look at our Getting Started guide for an introduction to Kaui.

Multi-Tier Deployments

For production deployments, allocating a separate RDS database instance and allowing multiple nodes to handle traffic behind a load balancer is the recommended option. In this model, we rely on AWS ELB to correctly spread the traffic among the various nodes, and also to route traffic to either Kill Server or KAUI based on the incoming port. Note that such deployment option requires a bit more setup and so we also provide an (easier) way to deploy using CloudFormation templates. However, the benefits of this option is that it provides more control in the deployment.

multi tier ami deployment

The steps are the following:

Notes: We are waiting for AWS Marketplace team to approve our Multi-Tier AMI, and will update this doc with the link when it is ready.

[1] Setup of the RDS instance

You can chose any options that are mysql or postgresql compatible. Using Aurora (mysql) is probably a good choice. Make sure to correctly setup your VPC, and subnets.

[2] Start the instances

The steps are similar to the one described for the section Single-Tier Deployment above, but in addition, when starting the instance, you will need to specify the properties associated with your RDS instance. In order to make it easy, we have created a User data template script, which will need to be edited to specify the real values:

#!/bin/bash

DB_PROPS="/var/tmp/db.props.$$"
KB_PROPS="/var/tmp/kb.props.$$"

cat <<_EOF > $DB_PROPS
#
# EDIT FOLLOWING DB PROPERTIES BELOW AS NEEDED
#
DB_SERVER=db-instance-1.cah16olm8gkg.us-east-1.rds.amazonaws.com:3306
DB_USER=admin
DB_PASSWORD=5Xn21rhxr3Km4jC7N7ol
KILLBILL_DB_NAME=killbill
KAUI_DB_NAME=kaui
_EOF

cat <<_EOF > $KB_PROPS
#
# EDIT FOLLOWING KB PROPERTIES BELOW AS NEEDED
#
org.killbill.dontexist=foo
_EOF

su -l -c "cd /var/lib/tomcat/bin && /var/lib/tomcat/bin/updateProperties.sh $DB_PROPS $KB_PROPS" tomcat

Specifying such user data script in the AWS UI is shown below:

userdata

Make sure that the security group for the EC2 instance also includes the security group for the RDS instance to ensure your EC2 instances can connect to RDS.

[3] Install database schema

In order to install the database schema, you will need to ssh to one EC2 instance using your private key — make sure you have alowed incoming SSH traffic on port 22 in your security group.

# SSH as ubuntu user
> ssh -i  <LOCATION_KEY>/<KEY_NAME>.pem [email protected]<EC2_NAME>
# Move to tomcat user
> sudo su - tomcat

Once logged in, and reusing the credentials, DB url examples from our user data script, create the 2 databases killbill and kaui:

> mysql -h db-instance-1.cah16olm8gkg.us-east-1.rds.amazonaws.com -u admin -p5Xn21rhxr3Km4jC7N7ol
mysql> create database killbill;
mysql> create database kaui;
^D

Then, you can install the schemas:

In order to install it:

# install killbill schema -- assumes it was saved as killbill.ddl
> mysql -h db-instance-1.cah16olm8gkg.us-east-1.rds.amazonaws.com -u admin -p5Xn21rhxr3Km4jC7N7ol killbill < killbill.ddl
# install kaui schema -- assumes it was saved as kaui.ddl
> mysql -h db-instance-1.cah16olm8gkg.us-east-1.rds.amazonaws.com -u admin -p5Xn21rhxr3Km4jC7N7ol kaui < kaui.ddl

[4] Restart service

> sudo service killbill stop
# Optionally clean existing logs
> rm /var/lib/tomcat/logs/*
> sudo service killbill start

[5] Add the ELB in front of the EC2 instances

Configuration

SSL Certificates

We have configured nginx to listen to port 443 (and forward traffic to KAUI) and so by default accessing the service from a web browser will show a Not Secure site. In order to make the site secure, you will need to add a valid certificate. The easiest option to add the certificate is to rely on Let’s Encrypt, a Certificate Authority (CA) that provides an easy way to obtain and install free TLS/SSL certificates.

The steps to add the certifcate are fairly simple and rely on a tool called cerbot.

[1]. Verify cerbot is installed or install it

Our latest image should have cerbot by default, but if not this can be added using the following:

Note: Check our TroubleShooting section to see how to login to the EC2 machine

# As root sudo su -
sudo su -
# Install cerbot if not already present
add-apt-repository -y ppa:certbot/certbot
apt install -y python-certbot-nginx

[2]. Stop all services

cerbot will need to have access to port 80 and 443

# Stop kaui
> service kaui stop
# Stop killbill
> service killbill stop
# Stop nginx
> service killbill nginx

Make sure your AWS security groups allow incoming traffic on port 80 and port 443 for all IPs

[3]. Setup your domain’s CNAME Record to point to the public DNS of your EC2 instance.

Create a CNAME to map your public DNS — e.g https://ec2-18-234-168-57.compute-1.amazonaws.com — to a legit CNAME, otherwise cerbot will fail with the following error: `Error creating new order

Cannot issue for "ec2-18-234-168-57.compute-1.amazonaws.com": The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy`

The configuration of your CNAME needs to happen from the UI of your domain provider (Namecheap, Cloudflare, …​)

[4]. Modify the nginx killbill.conf server sections

The server_name by default specifies _. Instead you need to replace this with your CNAME:

server {
    listen 443;
    server_name <CNAME>;

...

[5]. Run certbot

> sudo certbot --nginx -d <CNAME>

If successful you will see a message like:

...
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/deployment.killbill.io/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/deployment.killbill.io/privkey.pem
   Your cert will expire on 2020-05-11. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

[6]. Restart all services

# Start nginx
> service killbill nginx
# Start killbill
> service killbill stop
# Start kaui
> service kaui stop

Note: You can edit your security groups and remove the port 80, and also reduce the visibility for other ports by specifying a tighter range of incoming IPs.

The Let’s Encrypt certifcates are only valid 90 days and will therefore neeed to be renewed. certbot will create a cron entry under /etc/cron.d/certbot to make this process transparent.

Kill Bill

Kill Bill global properties are defined in /var/lib/killbill/config/killbill.properties. This is where you can change the MySQL credentials for instance. Take a look at our configuration guide for more details.

This is also where you can change the default Kill Bill admin credentials: specify org.killbill.security.shiroResourcePath=/var/lib/killbill/config/shiro.ini and create the shiro.ini file accordingly (see our RBAC guide for more details).

TroubleShooting Section

After launching the EC2 instance, the full stack should come up, with all services enabled and running, that is:

  • An nginx instance receiving traffic on port 443 and 8443

  • A instance of Kill Bill server listening on 127.0.0.1:8080 (and receiving external traffic through nginx on port 8443)

  • A instance of Kaui listening on 127.0.0.1:3000 (and receiving external traffic through nginx on port 443)

  • A local mysql server running on port 3306

In this section, we will provide some tips to verify the health of the system, and what to do when things are not working

SSH to EC2 Instances

From the EC2 dashboard, in the instance Description tab, you can copy the public DNS, called Public DNS (IPv4). Then, using the private key you specified when creating the instance:

# SSH as ubuntu user
> ssh -i  <LOCATION_KEY>/<KEY_NAME>.pem [email protected]<PUBLIC_DNS>
# Move to tomcat user
> sudo su - tomcat

Service Health

All services are started using System V init scripts, and so the status of the service can be retrieved:

  • Kill Bill server: service killbill status

  • KAUI server: service kaui status

  • Nginx server: service nginx status

Similarly one can start, stop the services using simalar command — e.g service kaui stop to stop KAUI.

In order to verify the health of the Kill Bill server instance, you can issue the following commands:

Log Files

Tomcat logs are under /var/lib/tomcat/logs/:

  • KAUI logs: /var/lib/tomcat/logs/kaui.out

  • Kill Bill server logs: /var/lib/tomcat/logs/catalina.out

Nginx logs can be found under /var/log/nginx/

  • Access logs: /var/log/nginx/access.log

  • Error logs: /var/log/nginx/error.log

System Diagnostics

In order to get some support, the first thing we would require is some information about your deployment. We have created a diagnostic command for that effect:

# Login as 'tomcat'
> sudo su - tomcat
#
# Assume a 'bob/lazar' tenant
# Assume some credentials 'admin/password'
#
> kpm  diagnostic \
  --killbill-credentials=admin password \
  --bundles-dir=/var/lib/killbill/bundles \
  --database-name=killbill \
  --database-credentials=root root \
  --killbill-api-credentials=bob lazar \
  --kaui-web-path=/var/lib/tomcat/webapps2 \
  --killbill-url=http://127.0.0.1:8080 \
  --database-host=127.0.0.1:3306

...
Diagnostic data exported under /tmp/killbill-diagnostics-20200213-23204-u93ah5/killbill-diagnostics-02-13-20.zip

Database

In order to access the database, one can use the following command mysql -u root -proot. There is one killbill and one kaui database created and used by the respective application

Nginx

The configuration files are located under /etc/nginx/ — e.g /etc/nginx/sites-enabled/killbill.conf

Upgrade steps

Note: you must switch to the tomcat user first in order to upgrade Kill Bill or Kaui (sudo su - tomcat).

The configuration file /var/lib/killbill/kpm.yml specifies the Kill Bill version (and its plugins) running on the instance. After updating this file with the new version(s), simply execute $KPM_INSTALL_KB_CMD, delete the cached directory /var/lib/tomcat/webapps/ROOT and restart the instance.

A similar process can be used for Kaui: update /var/lib/kaui/kpm.yml, run $KPM_INSTALL_KAUI_CMD, delete the cached directory /var/lib/tomcat/webapps2/ROOT and restart the instance.