There are three different ways to setup a working Kill Bill system on AWS, using software preconfigured by Kill Bill. Two options are based on a single Amazon Machine Image (AMI) containing the complete Kill Bill stack, that is both the Kill Bill server and the administrative UI (KAUI). These options are single tier and multitier. The single tier option is designed to deploy a limited version of Kill Bill quickly for trial and experimentation. For production deployments, we offer the multitier option. A third alternative available for deploying a production system is CloudFormation templates. The multitier option requires more setup than CloudFormation, but provides more control over the deployment.
This document describes the multitier option. This configuration uses two or more EC2 instances and a separate database provided by the AWS Relational Database Service (RDS). It also includes an AWS Elastic Load Balancer (ELB) to correctly spread the traffic among the various nodes, and to route traffic to either the Kill Bill Server or KAUI based on the incoming port.
The diagram below shows the principal components of the multitier system: The ELB, the RDS, and two or more EC2 instances. Each EC2 is an instance of Ubuntu Linux, running both Kill Bill and KAUI within a
tomcat server. Clients interact only with the ELB, which routes requests to the rest of the system.
Login to AWS
To begin, log in to Amazon Web Services at https://aws.amazon.com. If you are new to AWS, you will be asked to create an account and provide billing information. You will need to sign in as a Root User. This should take you to the AWS Management Console, which provides links to all available services.
Check the upper right corner of your screen to be sure you are in the appropriate region. All resources you create will be placed in this region, and may not be accessible from other regions.
In addition, AWS places all resources within a Virtual Private Cloud (VPC). A default VPC will be created and used automatically in the following steps. However, if you have access to other VPCs, you will need to ensure that all Kill Bill resources are deployed in the same one.
Setup the Database Manager
Once you are logged in, the first step is to setup the RDS instance. This process begins with the RDS dashboard, which should be available from the Services menu. When the dashboard appears, select Databases from the left menu, and click the red button at the top right that reads Create Database:
1. Set the Configuration
You will be taken to the Create Database page. The first choice you will have is between Standard Create, which allows you to set a full range of configuration parameters, or Easy Create, which sets most of these parameters to defaults. We recommend Easy Create in most cases.
The next section offers you a choice of several database types. Kill Bill can work with any database type that is
postgres compatible. For robust production use, Amazon Aurora is probably a good choice. Here we will illustrate the simpler steps setting up a MariaDB database.
The next choice determines the instance size. We suggest the Production option as this will provide the most robust configuration.
The last section asks you to:
Specify a name for your database
Give a username for the administrative account (we suggest that you do not use the default name)
Provide a password for the administrative acount (we suggest you let AWS generate one for you)
2. Create the DBM
When the password is setup and confirmed, click Create Database in the lower right corner. You will return to the main Databases screen, which should now look like this:
This display shows that your database is starting. After a few minutes, the status will change to Available (You may need to reload the page to see this). At this time you can click on the database name to get more information, including the full name of the instance.
On the page that appears you should see a panel named Connectivity and Security. The left side of this panel shows the full name of the endpoint, which you will need shortly, and the port number, which is normally 3306.
3. Setup the Security Rules
Lastly, on the Connectivity and Security panel, locate and click on the link for the default VPC security group. You will need to add an inbound security rule, because the database by default does not allow external access. In the panel for this group, click on Inbound Rules and select Edit Inbound Rules. Next click on Add rule. In the Type column select
MYSQL/Aurora. The port will be set to 3306 automatically. In the Source column, click on the search icon and select
0.0.0.0/0. Finally, click on Save Rules in the bottom right. Your database is ready to go.
Edit the Configuration Script
To set up the EC2 instances you will need to provide them with information needed to connect to the databases. We provide a brief configuration script to simplify this process. The template for this script is as follows:
#!/bin/bash DB_PROPS="/var/tmp/db.props.$$" KB_PROPS="/var/tmp/kb.props.$$" cat <<_EOF > $DB_PROPS # # EDIT THE FOLLOWING DB PROPERTIES AS NEEDED: # DB_SERVER=DB-INSTANCE-NAME:3306 DB_USER=ADMIN-NAME DB_PASSWORD=PASSWORD KILLBILL_DB_NAME=killbill KAUI_DB_NAME=kaui _EOF cat <<_EOF > $KB_PROPS # # EDIT THE FOLLOWING KB PROPERTIES AS NEEDED: # org.killbill.dontexist=foo _EOF su -l -c "cd /var/lib/tomcat/bin && /var/lib/tomcat/bin/updateProperties.sh $DB_PROPS $KB_PROPS" tomcat
First, you need to edit the database properties. DB_SERVER should be set to the full name of the DB instance, as given in the Connectivity and Security panel (see above). The port number 3306 is required. DB_USER and DB_PASSWORD should be set to the administrator credentials you have chosen for the RDS instance.
Second, you may optionally edit any Kill Bill properties that you need to change from the standard defaults. For more information see the Kill Bill Configuration Guide.
Launch EC2 Instances
The next step is to launch the number of EC2 instances you want, all based on the Kill Bill single AMI.
1. Subscribe to the AMI
To start the installation process, point your browser to the Kill Bill AMI at AWS Marketplace .
You should see the following image at the top of your screen:
Click Continue to Subscribe. The next page will give the AWS Terms and Conditions:
Accept the terms if asked. You will then see a new message confirming that you have subscribed. Next, click Continue to Configuration.
2. Configure the Instances
The next page will give several configuration options:
In most cases you should accept the defaults. Then click Continue to Launch.
The next page will give you several options for the launch method. We recommend that you choose Launch through EC2.
All other options will disappear. Click Launch.
The next page is titled Choose Instance Type. Your instance type is already chosen, so click Configure Instance Details at the bottom of the page.
The next page will provide you with a long list of options. The first option is Number of Instances. Set the number of instances you wish to launch. Each instance will have essentially the same configuration, including the same image, the same subnet and availability zone, and the same security group.
After setting the number of instances, scroll down to the bottom of the page. The last section is titled Advanced Settings. In this section you should set the configuration file you produced above. The setting panel should look like this:
Now click Review and Launch at the bottom of the page. The following page is headed Review Instance Launch. This provides a chance to review the details of your chosen instances. Then click Launch at the bottom of the page.
3. Setup a Key Pair
Next you will see a very important popup that asks you to choose or create a key pair.
The key pair provides the credentials you will need to login to your EC2 instances. For details about key pairs, see the AWS documentation. We recommend that you create a new key pair. All your instances can use the same one. Give the key pair a simple, easy to remember name such as
My-Key-Pair. Then click Download Key Pair. Important: You must save the private key that will be generated in this step. If you lose this key, you will not be able to login to your instances.
4. Launch your Instances
When the key pair is generated, click Launch Instances. You should see the screen below:
Your instances are finally launching! To follow what is happening on the EC2 Dashboard, scroll all the way down to the bottom, and click View Instance at the bottom right. This will take you to the Instances screen which is part of the EC2 Dashboard.
In a short time, the Instance State for each instance should indicate Running. You will need to scroll to the right to see all of the information available about your instances.
5. Setup Security Rules
You are almost set, but there is one more thing you need to do, and that is to scroll down in the menu on the left side to select Security Groups. You should see a list of two or more groups. Select the group whose name begins with
Kill Bill on AWS, then scroll to the bottom and select the tab for Inbound Rules. You should see:
These rules enable the ports that must be open to access KAUI and Kill Bill from a browser. However, for access through the ELB these ports will be different. In addition, to enable direct login to your instance using SSH, you need to add one more port.
Click on Edit Inbound Rules. then do the following:
For the rule that specifies Type: HTTPS, Port Range: 443, change the type to CUSTOM TCP and the Port Range to 3000.
For the rule that specifies Type: CUStOM TCP, Port Range: 8443, change the Port Range to 8080.
Finally, add a rule with the following elements: Type: SSH, Protocol: TCP, Port Range: 22, Source: 0.0.0.0/0.
Your Inbound Rules should now look like this:
6. Login to an Instance
Now that your instances are set up, you need to ensure that you can login to them for configuration and maintenance when needed. To login, use the secure shell command:
ssh -i PRIVATE_KEY.pem [email protected]_IP
Here PRIVATE_KEY is the pathname where you have stored the private key that was downloaded when you generated your key pair, and INSTANCE_IP is the IPV4 address for any one of your instances as described earlier. The private key will not work unless its access controls are set to readable by the owner only.
On Windows versions before Windows 10, you may need to download a program called PuTTY to enable
ssh. On Windows 10
ssh is available but may need to be activated through the Settings screen.
The first time you login, you will see a warning message asking if you want to add this host to your list of hosts. You should answer
You will now be able to explore your instance and perform various configuration and maintenance tasks. To exit from your login, type
Create the Databases
Kill Bill requires two databases, with the names
kaui. We provide predefined schemas for these databases.
To create the databases, you will need to login to one of your instances as described above. Once you are logged in, you can use the
mysql command to create the two databases
kaui. The credentials for this command are the same ones you set up for the database and copied to the configuration file. Note that the DB-INSTANCE-NAME should not include the port number.
The password will not be echoed when it is typed.
> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p Enter Password: mysql> create database killbill; mysql> create database kaui; mysql> exit
The next step is to install the schemas. These can be found at:
One easy way to do this is to return to your local computer (type
exit) and download the schemas, then use the
sftp command to upload them to your EC2 instance home directory with the commands:
sftp -i PRIVATE_KEY.pem [email protected]_IP put killbill.ddl put kaui.ddl exit
Once the files are successfully uploaded, login again to your instance using the
ssh command. You can now install the schemas:
> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p killbill < killbill.ddl Enter Password: > mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p kaui < kaui.ddl Enter Password:
To ensure that the databases are setup correctly, login to
mysql again, then try the SHOW TABLES command:
> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p Enter Password: use killbill show tables; use kaui show tables; exit
show tables command should display a list of table names for the database.
You can now login to KAUI from your browser using the URL http://INSTANCE_IP:3000, where INSTANCE_IP is the IPV4 address for your instance, given on your dashboard as Public IPV4 Address. This should display the KAUI login screen. For an introduction to KAUI, see our Getting Started guide. The default credentials are:
password. The first few requests might be a bit slow as Kill Bill initializes itself.
In addition, you can login to the Kill Bill server using the URL http://INSTANCE_IP:8080. This provides access to certain detailed reports that may be needed for maintenance, including metrics, event logs, and the Swagger API pages.
If these logins succeed, your EC2 instances and your RDS databases are setup properly.
Add the ELB
The last major task is to setup the Elastic Load Balancer in front of the EC2 instances.
1. Configure the ELB
To begin, from the EC2 dashboard scroll down the left-hand menu and select Load Balancing / Load Balancers. Then click the Create Load Balancer button at the upper left.
You will be given a choice of several load balancer types. The type we will use is Application Load Balancer.
Click on the Create button in the Application Load Balancer box. This will bring up the page titled Step 1: Configure Load Balancer:
On this page you need to do the following:
Assign a name to your load balancer
Select a protocol for the listener. We recommend using HTTPS as discussed below.
Set the listener port to 443 (if using HTTPS) or 80 (if using HTTP).
Click Add Listener to add a second listener using the same protocol and set its port to 8443.
Scroll to the bottom and select at least two availability zones. IMPORTANT: You must select all of the zones that your EC2 instances are contained in. Otherwise, the load balancer will be unable to connect to these instances.
2. Configure Security
Now choose Next: Configure Security Settings. You will now see a page titled Step 2: Configure Security Settings.
If you have selected the HTTPS protocol, you will be required to create or provide an X.509 SSL Certificate. If you already have a certificate you can identify it or upload it here. Otherwise we recommend you click on Request a New Certificate from ACM. This will enable you to create a certificate using the Amazon Certificate Manager. Follow the steps described for the ACM in Using HTTPS, then return to this page and proceed to the next step.
If you are using HTTP, you will see a warning message, which can be ignored. In either case, the Security Policy should not be changed.
Your next step is to choose Next: Configure Security Groups. This will take you to a page titled Step 3: Configure Security Groups. This page will show you the existing security groups and offer the choice to choose an existing group or create a new one. We advise you to create a new group. The new security group should have inbound rules enabling inputs for KAUI and Kill Bill as shown here:
These rules assume you are using HTTPS. If you are using HTTP, the port for the first rule should be 80.
3. Setup Target Instances
The next step is to identify the target instances for your load balancer, which are collected into a target group. Each listener will have a separate target group, but you can only setup one right now. The other group will be added later.
Your group will consist of all of the instances you have launched. First, create the group, give it a simple name, and set the port to 3000:
Now click on Next: Register Targets. The purpose of this step is to identify the target instances that will be part of your target group. Initially, all your instances will be listed in the bottom section. To register them, select them all and click Add to Registered. Then proceed to Next: Review.
4. Create the Load Balancer
Check all settings, then click Create. Your load balancer will be created. Close the final page to see the Load Balancer list. The initial status for your new ELB will be provisioning. After a few minutes this will change to active.
5. Setup the Second Listener
You are almost done. Your final step is to modify the second listener to use a different port number to access the Kill Bill server. From the left side menu select Target Groups. Click Create Target Group. In the page that appears, set the protocol to HTTP and set the port to 8080. Then click Next.
The next page is provided to register the targets for this group. Once again you will register all your instances as targets, but this works just a little differently than before. The list of available targets is at the top. Select all of them, then click Include as Pending Below to make these targets pending as members of the group.
Finally, click Create Target Group to create the group.
Now you will need to associate this group with your load balancer’s second listener. Return to the Load Balancer console, select your load balancer, and choose the Listeners tab in the bottom information panel. Then in the second entry, click View/Edit Rules.
Your object now is to change the rule for this listener to point to your new target group. To do this:
Click on the pencil icon at the top
Click on the pencil icon that appears to the left of the rule
Click on the pencil icon that appears under the heading THEN
Select your new target group in the dropdown list
At the top of the page, click UPDATE
Go back to the listener tab and confirm that the second listener points to the new target group. Your load balancer is now ready.
When your ELB is complete you can proceed to testing. If you are using HTTPS you should be able to login to KAUI from your browser using the URL https://kaui.DOMAIN, where DOMAIN is your domain that you have used for your certificate. If you are using HTTP you will need to use the URL http://DNS_NAME where DNS_NAME is the DNS name for the KAUI load balancer as given on the Load Balancer dashboard.
The KAUI login screen should appear. For an introduction to KAUI, see our Getting Started guide. The default credentials are:
password. The first few requests might be a bit slow as Kill Bill initializes itself.
Congratulations! Your multi-tier installation is ready to go!
Kill Bill defines a number of global properties and per-tenant properties that can be varied. These properties are explained in the configuration guide. Default values for these properties are built into the Kill Bill code; these values can be overridden by values defined in the file
/var/lib/killbill/config/killbill.properties. For example, this is where you can change the database URL and credentials.
This file also defines the location of the
shiro.ini file, which by default is in the same directory. This file defines the Kill Bill admin credentials, along with any other users and their roles. See the RBAC guide for details about this file.
If you make changes to these configuration files, remember to make the same changes to all your EC2 instances!
From time to time new versions of Kill Bill and KAUI may be released. This section explains how to upgrade to these new versions. You will need to follow these procedures for each of your instances.
First, login to your instance using
ssh, then switch to the
sudo su - tomcat
The configuration file
/var/lib/killbill/kpm.yml specifies the Kill Bill version (and its plugins) to be run on the instance. Once you edit this file to specify the new version number, it will be used automatically. Perform the following steps:
Edit the configuration file to update the version number
Run the command
Delete the cached directory
Restart the instance.
A similar process can be used for KAUI: update
$KPM_INSTALL_KAUI_CMD, delete the cached directory
/var/lib/tomcat/webapps2/ROOT and restart the instance.
If your installation does not seem to work correctly, the first step is to review your installation process carefully, to be sure that everything has been done and checked as described in this document. If problems persist, we will be glad to work with you to identify the problem. To help us to do this, there are several information reports that you may need to gather. These reports are somewhat technical but can be analyzed by Kill Bill personnel. This section explains how to obtain the reports that may be needed.
The procedures described here focus on obtaining detailed reports for a single EC2 instance. They should be used independently for each instance you have deployed.
System Health Check
The healthcheck report checks the health of various software components, and determines if any queues are growing improperly over time. To create this report, login to your EC2 instance and issue the following command:
This will return a series of messages giving the health status of each component.
For a detailed system information report, use the following command:
curl -u ADMIN:PASSWORD http://127.0.0.1:8080/1.0/kb/nodesInfo
Here ADMIN and PASSWORD are your KAUI administrator access credentials.
This procedure does not produce a report, but does provide important information about the status of each service.
The Kill Bill multitier option runs two services on each instance. The status of each service can be checked by the following commands:
Kill Bill service:
sudo service killbill status
sudo service kaui status
For each report there should be a line near the top with the following form:
Active: active (running) since Sat 2020-10-24 20:13:43 UTC; 1 day 1h ago
Similarly you can
stop the services using similar commands, such as
sudo service kaui stop to stop KAUI.
The system maintains a series of logfiles that should be helpful when troubleshooting is needed.
Tomcat logs are under
Kill Bill server logs:
It is not necessary to download these logs separately, though, as they will be included in the output from the diagnostic command to be discussed next.
diagnostic option of the
kpm command creates an extensive report for a given tenant that may be useful for troubleshooting. To run this command:
# Login as 'tomcat' > sudo su - tomcat # # Run the command with your access credentials: # > kpm diagnostic \ --killbill-credentials=ADMIN PASSWORD \ --bundles-dir=/var/lib/killbill/bundles \ --database-name=killbill \ --database-credentials=DBUSER DBPASS \ --killbill-api-credentials=KEY SECRET \ --kaui-web-path=/var/lib/tomcat/webapps2 \ --killbill-url=http://127.0.0.1:8080 \ --database-host=DBURL:3306
You will need to edit this command to include:
Your KAUI username and password (ADMIN PASSWORD)
Your database credentials (DBUSER DBPASS)
The key and secret key for your tenant (KEY SECRET)
Your database URL (DBURL)
The last line of the response should look like:
Diagnostic data is exported under /tmp/killbill-diagnostics-20200213-23204-u93ah5/killbill-diagnostics-02-13-20.zip
The specified zip file contains several reports of various sizes. This report can be downloaded to your computer using
sftp and forwarded to Kill Bill for analysis.