There are at least three different ways to setup a working Kill Bill system on AWS. Two options are based on a single Amazon Machine Image (AMI) containing the complete Kill Bill stack, that is both the Kill Bill server and the administrative UI (KAUI). These options are single-tier and multi-tier. The single-tier option is designed to deploy a limited version of Kill Bill quickly for trial and experimentation. The multi-tier option is designed for production deployments; this requires more setup than other options, but provides the most control over the deployment.

A third alternative available for deploying a production system, discussed here, is CloudFormation. This deployment method leverages the capabilites of the AWS infrastructure to provide a robust production ready deployment with a single click. The entire CloudFormation configuration, or stack, is defined by a CloudFormation Template.

The features of the CloudFormation system include:

  • Both Kill Bill and KAUI instances can be scaled up or down using AWS autoscaling groups

  • AWS CloudWatch provides metrics to follow what is happening

  • The RDS database based on AWS Aurora comes automatically configured and ready for use

Overview

Running Kill Bill on AWS using our CloudFormation Template is the easiest and fastest way to get started with a production cluster. It is also the only method of installation that is certified by the core developers for a highly available, horizontally scalable and production-ready installation.

With the click of a button, the template will install and configure:

  • Kill Bill and Kaui on a custom AMI optimized for AWS workloads (integrated with CloudWatch, SQS, SES, X-Ray and more)

  • Auto Scaling Groups, to automatically scale up and down the number of EC2 instances as needed (such as when batches of invoices are generated)

  • A load balancer, integrated with our internal healthchecks to promptly take unhealthy instances out of rotation

  • An RDS Aurora Cluster with automatic failover

The following diagram shows the various AWS entities that will be created by CloudFormation:

cf stack

All resources for this system run within a single AWS Virtual Private Cloud (VPC), providing a dedicated block of IP addresses which must be located in a single region. The cloud is partitioned into availability zones, which are accessed by subnets. The resources must be distributed over at least two availability zones.

Multiple instances of KAUI and the Kill Bill server are deployed in the VPC, each running on its own Ubuntu Linux server. AWS autoscaling is used to increase and decrease the number of instances for each package as needed.

Access to these instances is managed by an AWS Elastic Load Balancer (ELB). The ELB routes each request to the correct package and distributes the requests across the available instances. The ELB accepts traffic using HTTP by default, but we recommend upgrading to HTTPS for higher security (see below). Using HTTP, the ELB will listen on port 80 for routing traffic to the Kill Bill instances, and on port 9090 for accessing KAUI.

The back end of the system is an Aurora database manager provided through the AWS Relational Database System (RDS). Aurora is a robust database system developed by AWS, compatible with MySQL and Postgres. There are separate databases maintained for Kill Bill and KAUI.

Finally, the complete CloudFormation system makes use of AWS scalable and reliable S3 storage technology, and incorporates AWS CloudWatch for monitoring and reports.

Installation

Login to AWS

To begin, log in to Amazon Web Services at https://aws.amazon.com. If you are new to AWS, you will be asked to create an account and provide billing information. You will need to sign in as a Root User. This should take you to the AWS Management Console, which provides links to all available services.

Check the upper right corner of your screen to be sure you are in the appropriate region. All resources you create will be placed in this region, and may not be accessible from other regions.

Setup the VPC and Subnets

All resources for your CloudFormation deployment must be placed within a single VPC. To prepare for CloudFormation deployment you will first need to setup and identify your VPC and subnets.

1. Setup your VPC

From the Services menu item at the top of the main AWS page, under Networking and Content Delivery, select VPC. This will open the VPC Dashboard. Then select Your VPCs from the left menu.

Normally you will see one VPC, which AWS provides by default. This VPC will automatically be used for all your resources. If you have more than one, you need to select the one you want to use and be sure to set its ID as a parameter for your CloudFormation configuration. If there is no VPC listed, you must create one. The only parameter you need to set is the ipv4 CIDR block, which designates a range of (private) IP addresses. A suggested value is 192.168.0.0/16.

2. Setup your Subnets

From the left menu of the VPC Dashboard select Subnets. This shows a list of your subnets. By default AWS creates one subnet for each Availability Zone in your region. You may create your own subnets, as long as you give each a CIDR block representing a unique subset of your VPC, and assign each to a specific availability zone.

Setup a Key Pair

The Kill Bill CloudFormation stack requires a key pair. The key pair provides the credentials you will need to login to your EC2 instances. If you already have a key pair, you are all set. Otherwise you will need to create one.

To create a key pair, from the EC2 console scroll down to Networks & Security / Key Pairs. Select Create Key Pair and follow the instructions. For details about key pairs, see the AWS documentation. Important: You must save the private key that will be generated in this step. If you lose this key, you will not be able to login to your instances.

Configure and Launch

The setup process starts at the AWS Marketplace using our official CloudFormation template. Go to this page, then click Continue to Subscribe.

cf subscribe

THe next page gives the AWS Terms and Conditions. You must accept these conditions if asked, then click Continue to Configuration. This will take you to the page titled Configure This Software:

cf configure

There is nothing to change on this page. Click Continue to Launch. THe next page is titled Launch this Software. There is nothing to do here either. Click Launch.

The next page is designated Step 1: Specify Template and titled Create stack:

cf step1

Once again everything is filled in for you. Click Next. This brings up the Stack Details page (your complete configuration is called a stack):

cf details

Now you have some work to do! This page requires that a number of configuration parameters be filled in. All of these are important, and many are critical.

First, you need to provide a name for your stack. Any name will do, as long as it meets the stated rules. Then you will need to carefully set a series of parameters:

  • DBClass: the database instance type to use for RDS. This normally should not be changed.

  • DBName: the database name for Kill Bill. This is preset to killbill. Do not change it.

  • DBUser: database admin username. The username you choose for the database administrator.

  • DBPassword: database admin password. The password you choose for the database administrator. This must include at least one letter, at least one digit, and no other character types.

  • EnableCloudWatchMetrics: whether to enable metrics in CloudWatch. This is strongly recommended for production.

  • EnvType: the purpose of this configuration: test, dev (development), or prod (production). There is no difference in the stack being created but this value will be sent to CloudWatch as a dimension.

  • HTTPLocation: the IP address range allowed to access the load balancer, in the form of a CIDR block. You can use 0.0.0.0/0 initially and adjust access later on.

  • InstanceType: the EC2 instance type to use for Kill Bill. This normally should not be changed.

  • KBAdminPassword: the password to be used for the default root user which has all permissions. By default this is set to password. Please change it! The requirements are the same as for DBPassword.

  • KauiDBName: database name for Kaui. This is preset to Kaui. Do not change it.

  • KauiServerCapacity: the initial number of Kaui instances in the Auto Scaling group. We recommend using the default value, 2.

  • KeyName: name of an existing EC2 KeyPair to enable SSH access to the instances. If you don’t have one, see the AWS documentation.

  • KillBillServerCapacity: the initial number of Kill Bill instances in the Auto Scaling group. Again we recommend the default value of 2.

  • RDSSubnets: the subnets to use for the RDS instance. Select two or more from your subnets, which must be in two or more availability zones.

  • SetEnvironmentVariables: a comma-delimited list of environment variables to set.

  • Subnets: the subnets to use for the KB and KAUi instances. Also two or more from your subnets in two or more availability zones. May or may not be the same as the RDS subnets.

  • VpcId: the VPC to use for the installation, which you identified earlier.

When all of these are set, click Next to go to Configure Stack Options. Now take a break. There is nothing to do here. Then click Next.

The final page gives you a chance to review. If everything seems OK, read and check any warnings at the bottom, then click Create Stack. you are off!

cf creating

If there are any errors, you will see a message and the Create will not begin. You will need to go back and fix the errors. Common errors may include using an invalid password form (which may give a misleading message), or not choosing subnets in at least two availability zones.

Otherwise, you will see that your stack is being created, and its status (shown in blue) will be CREATE_IN_PROGRESS. You may also check the Resources tab to see the many resources that are being created to make up the complete stack.

If the create succeeds, the status will eventually change to CREATE_COMPLETE (shown in green). This may take a fairly long time.

Update Configuration

To update the Kill Bill configuration parameters, select the CloudFormation stack and click on Update. The update stack consists of the following 4 steps:

  • Step 1 - Update stack: select Use current template and click Next

cf update config step1

  • Step 2 - Specify stack details: update any of the parameters and click Next

cf update config step2

  • Step 3 - Configure stack options: no changes are needed at this step

cf update config step3

  • Step 4 - Review: review all the changes made to the parameters, scroll down and check I acknowledge that AWS CloudFormation might create IAM resources and click on Submit.

cf update config step4b

CloudFormation stack will start updating all these configurations, and upon completion, its status will change to UPDATE_COMPLETE.

Enabling HTTPS

Your deployment is initially accepting external communications, such as KAUI interaction, using HTTP. For secure and private communication we recommend the use of HTTPS. For information about HTTPS and how to create a certificate see Using HTTPS. If you don’t already have a certificate we recommend creating one using the Amazon Certificate Manager (ACM).

To configure the load balancer to accept SSL connections, proceed as follows:

First, find the load balancer in the EC2 console:

find lb

Two HTTP listeners are configured by default, on ports 9090 (Kaui) and 80 (Kill Bill). You need to add two additional HTTPS listeners.

For example, to expose Kaui behind port 443, the configuration would look like this (note the Forward To section):

add lb listener

When requested, follow the instructions to import your certificate.

You will also need to allow HTTPS traffic in your security group:

lb security group

The load balancer is now configured to redirect SSL traffic on port 443 to Kaui. You can do the same for Kill Bill (using port 8443) and disable the HTTP rules in your Security Group.

CloudWatch

Our AMIs come pre-configured with CloudWatch integration to allow for better diagnostics. Below is a screenshot of the metrics that come for free:

cloudwatch

To support business metrics, we also support the use of additional plugins suchs as our free Analytics plugin. This provides a subscription billing management solution as feature-rich as popular SaaS platforms, but that you can control. Below is a screenshot of these business metrics:

analytics

For installation support along the way, reach out to [email protected].

Testing

If you are using HTTP, go to your browser and type http://DNS_NAME, where DNS_NAME is the DNS name for your load balancer as given on the Load Balancer dashboard. You should see:

cf success

If you are using HTTPS you should be able to login to KAUI from your browser using the URL https://kaui.DOMAIN, where DOMAIN is your domain that you have used for your certificate. If you are using HTTP you will need to use the URL http://DNS_NAME where DNS_NAME is the DNS name for the KAUI load balancer as given on the Load Balancer dashboard.

The KAUI login screen should appear. For an introduction to KAUI, see our Getting Started guide. The default credentials are: admin / password. The first few requests might be a bit slow as Kill Bill initializes itself.

Similarly, you should be able to login directly to the Kill Bill server using the URL https://kaui.domain:8443 or the URL http://DNS_NAME:8443.

Congratulations! Your CloudFormation installation is ready to go!

TroubleShooting

Stack Creation

There are several possible issues that could arise when starting the stack from CloudFormation (CFN). The goal of this section is to provide some tips debugging the issues, and knowledge about where to find various logs. Also keep in mind that starting the CFN stack will typically take in the order of 15-20 minutes, mostly because of the time it takes to setup the RDS cluster, so be patient…​

Let’s start with the CloudFormation console: The stack will have a status which will be CREATE_IN_PROGRESS for the duration of the initialization, and will then either transition to CREATE_COMPLETE or CREATE_FAILED. In case of CREATE_FAILED, try to locate from the Resources tab which resource failed to initialize properly.

Possible Issues

The main possible issues can be summarized as:

Issues with the template

If you think there is an issue with the CFN template itself, please report any issue to [email protected]

AWS Errors

Typical Issues are:

  • Insufficient IAM Permissions

  • Limit Exceeded

  • Security Group Does Not Exist in VPC

  • RDS Cluster failed to come up

  • …​

Make sure to check the discussion above to ensure you have setup a valid VPC with valid subnets. Fow AWS specific issues, please refer to the AWS troubleshooting documentation

Service Unavailable

We suggest to check the following:

  1. Are there any issues reported in the CFN logs?

  2. Is the database up and running and accessible from the Kill Bill/Kaui EC2 instances?

  3. Is the database schema correctly installed?

  4. Is the Kill Bill/Kaui server correctly started and listening on the correct ports?

  5. Are the Kill Bill/Kaui servers accessible from the LB, respectively on the correct ports?

  6. Are there any errors or stack traces in our logs?

Practical Tips

SSH to EC2 Instances

In order to answer these questions, you will first need to be able to SSH to the EC2 instances:

From the EC2 dashboard, you can locate the instances by filtering on a prefix in the stack’s name, in my case my-stack, and as indicated below you will see the instances for Kill Bill server and KAUI. In the example below we see one of each:

ec2 instances

You can select one instance and then from the description tag, you will have access to:

  1. Public DNS

  2. The security group

ec2 description

You will need to first click on the security group link to open the inbound port 22 required for SSH, as shown below:

security group

Then, you can issue the SSH command, by copying the Public DNS from the description tab:

# SSH as ubuntu user
> ssh -i  <LOCATION_KEY>/<KEY_NAME>.pem ubuntu@<PUBLIC_DNS>
# Move to tomcat user
> sudo su - tomcat

CloudFormation Logs

If there are any issue with CFN, it should be available from /var/log/cfn-init-cmd.log or /var/log/cfn-init.log

Other logs of interest may include:

/var/log/cfn-wire.log
/var/log/cloud-init-output.log
/var/log/awslogs.logs
/var/log/xray/xray.log

Kill Bill/KAUI Server Logs

The Kill Bill/KAUI server logs are located under /var/lib/tomcat/logs/, with the main 2 interesting logs being:

  • killbill.out: All Kill Bill server logs, configured as INFO by default

  • kaui.out: All KAUI server logs, configured as INFO by default

  • localhost_access_log…​: Access requests to the servers

The configuration of the logging (log rotation, log level, …​) can be found in /var/lib/killbill/config/logback.xml

If you update the logback.xml (or any configuration file) you will need to restart the service. In order to restart the service, you can run as root the following command:

# Restart  killbill server instance
> service killbill restart
# Restart  kaui server instance
> service kaui restart

Access to the Database

From any Kill Bill EC2 instance, it is possible to access the RDS database. On each node, there is a mysql client installed allowing database access. The database hostname can be obtained from the CFN Resources screen, or one can also extract this information from the killbill.properties file:

> grep 'org.killbill.dao' /var/lib/killbill/config/killbill.properties
org.killbill.dao.password=killbill
org.killbill.dao.url=jdbc:mysql:aurora://mystack-test-rdscluster-1qwiqitatcb9m.cluster-cah16olm8gkg.us-east-1.rds.amazonaws.com:3306/killbill
org.killbill.dao.user=killbill

Based on such info, the following command would allow you to get a mysql prompt:

> mysql -h mystack-test-rdscluster-1qwiqitatcb9m.cluster-cah16olm8gkg.us-east-1.rds.amazonaws.com -u killbill -pkillbill killbill
> show tables
> ...

Service Health

Since both Kill Bill/KAUI server listen on port 8080, you can check if the service is running by issuing the following command:

telnet 127.0.0.1 8080
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

For the Kill Bill server specifically some useful commands are:

# Healthcheck
> curl http://127.0.0.1:8080/1.0/healthcheck
# Check which Kill Bill & plugin versions
> curl -u admin:<KBAdminPassword> http://127.0.0.1:8080/1.0/kb/nodesInfo | jq
[
  {
    "nodeName": "ip-192-168-65-236.ec2.internal",
    "bootTime": "2020-02-02T21:26:44.000Z",
    "lastUpdatedDate": "2020-02-02T21:26:44.000Z",
    "kbVersion": "0.22.1",
    "apiVersion": "0.53.17",
    "pluginApiVersion": "0.26.3",
    "commonVersion": "0.23.7",
    "platformVersion": "0.39.12",
    "pluginsInfo": [
      {
        "bundleSymbolicName": "org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
        "pluginKey": null,
        "pluginName": "org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
        "version": null,
        "state": "RUNNING",
        "isSelectedForStart": true,
        "services": []
      },
      {
        "bundleSymbolicName": "org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
        "pluginKey": null,
        "pluginName": "org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
        "version": null,
        "state": "RUNNING",
        "isSelectedForStart": true,
        "services": []
      }
    ]
  }
]

Diagnostic Command

The diagnostic option of the kpm command creates an extensive report for a given tenant that may be useful for troubleshooting. To run this command:

# Login as 'tomcat'
> sudo su - tomcat
#
# Details about DB host can be extracted from '/var/lib/killbill/config/killbill.properties'
#
# Run the command with your access credentials:
#
> kpm  diagnostic \
  --killbill-credentials=ADMIN PASSWORD \
  --bundles-dir=/var/lib/killbill/bundles \
  --database-name=killbill \
  --database-credentials=DBUSER DBPASS \
  --killbill-api-credentials=KEY SECRET \
  --killbill-web-path=/var/lib/tomcat/webapps \
  --database-host=DBHOST

You will need to edit this command to include:

  1. Your KAUI username and password (ADMIN PASSWORD)

  2. Your database credentials (DBUSER DBPASS)

  3. The key and secret key for your tenant (KEY SECRET)

  4. Your database host (see '/var/lib/killbill/config/killbill.properties' )

The last line of the response should look like:

Diagnostic data exported under /tmp/killbill-diagnostics-20200212-26849-c0rrz3/killbill-diagnostics-02-12-20.zip

Note that there is also a --account-export=<account_id> flag to export the data associated with a specific Kill Bill account_id.

Upgrade Steps

Newer AMIs

The Kill Bill core team will provide new AMIs whenever necessary.

Because the CloudFormation from AWS Marketplace will always reflect the latest AMI ids, you can simply update the stack with the latest CloudFormation template and the instances in the AutoScaling groups will be updated automatically. We strongly recommend that you always test the upgrade in a test environment first.

We recommend that you rely on the CloudFormation ChangeSet functionality to get a sense of what would be updated if the change was submitted. For more information about the CloudFormation ChangeSet functionality see this documentation. Below is a summary of the steps:

[1]. Download the new CloudFormation template

Each AMI is defined by a CloudFormation template. To access the template for the latest AMI, go to the Marketplace page as described under Configure and Launch above. Check that the page lists the desired version, then scroll down to the Usage Information section. Expand the link View Cloudformation Template. Below the diagram that appears, click Download Cloudformation Template. Save the template file. This will be a long text (JSON) file with a name ending in template.

change set usage information

[2]. Create a new ChangeSet

Go to the CloudFormation dashboard and select you current stack. Then select Stack Details from the left menu. You should see the following page:

create change set

Select Create Change Set. On the page that appears, Select Replace Current Template, then select Upload a Template File. Finally, upload the file you downloaded in Step 1.

You will now revisit several pages that you saw when the stack was created. First, you will see the page Specify Stack Details. At this time there should be no changes required. Click Next.

The next page will be the Configure Stack Options. Again, no changes required.

THe last page is the Review page. If everything looks good, scroll to the bottom. You will see the following message, that you will need to acknowledge:

change set capabilities

Finally click Create Change Set. You can provide an optional description in the popup that appears, then select Create Change Set again. Your change set will be created. You will initially see the status CREATE_PENDING. Wait until the status message changes to CREATE_COMPLETE.

[3]. Apply the ChangeSet

It is important to remember that at this point your Kill Bill installation has not changed. Your change set is ready and waiting when you do want to use it. When that time comes, return to the cloudformation dashboard, select your stack and select the change sets tab. Select your change set, then click Execute.

change set execute

Your new resources will be created and any old ones no longer needed will be deleted. The status of the stack will show as UPDATE_IN_PROGRESS. For a short time the stack may be in an unusable state. When the status changes to UPDATE_COMPLETE, the stack has been fully updated to the new version.