Overview
Single AMI deployments are a way to deploy the Kill Bill stack, that is both the Kill Bill server and the administrative UI (KAUI), from one AMI. We offer two options. The Single-Tier
option uses only a single EC2 instance. This is suitable for testing and experiments. The Multi-Tier
option uses two or more EC2 instances, and is designed for production deployments. Both of these deployment options rely on the same underlying AMI, but for the Multi-Tier
option there is a list of prerequisites, such as an external database, along with a startup script to be provided when launching the instance.
Single-Tier Deployments
The single-tier deployment option is designed to get you started as quickly as possible. Everything you need: Kill Bill, KAUI, and the MariaDB (MySQL) database, is bundled in the AMI and can be launched on a single EC2 instance with just one click. This makes it very suitable for test environments, but we don’t recommend it for production deployments.
The AMI also comes with a running instance of the load balancer nginx
that acts as a reverse proxy and spreads the incoming traffic to either Kill Bill or KAUI based on the incoming port. The EC2 instance will be an instance of Ubuntu Linux. The organization of the single-tier system looks like this:
In this model, the Kill Bill server and KAUI are deployed within the tomcat
application server. Requests are handled by nginx
. Requests received on port 443 are delivered to KAUI, and those received on port 8443 are delivered to Kill Bill. Both Kill Bill and KAUI communicate as needed with the mysql
database.
Getting Started
To start the installation process, point your browser to AWS Marketplace:
If you are new to AWS, you will be asked to create an account and provide billing information. Once your account is validated, you should see the following image at the top of your screen:
Click Continue to Subscribe. The next page will give the AWS Terms and Conditions:
Accept the terms if asked. You will then see a new message confirming that you have subscribed. Next, click Continue to Configuration.
The next page will give several configuration options.
In most cases you should accept the defaults. Then click Continue to Launch.
The next page will give you several options for the launch method. We recommend that you choose Launch through EC2.
All other options will disappear. Click Launch.
The next page is titled Choose Instance Type. Your instance type is already chosen, so click Review and Launch at the bottom of the page.
The following page is headed Review Instance Launch. This provides a chance to review the details of your chosen instance. Then click Launch at the bottom of the page.
Next you will see a very important popup that asks you to choose or create a key pair.
The key pair provides the credentials you will need to login to your EC2 instance. For details about key pairs, see the AWS documentation. We recommend that you create a new key pair. Give the key pair a simple, easy to remember name such as My-Key-Pair
. Then click Download Key Pair. Important: You must save the private key that will be generated in this step. If you lose this key, you will not be able to login to your instance.
When the key pair is generated, click Launch Instances. You should see the screen below:
Your instance is finally launching! To follow what is happening on the EC2 Dashboard, scroll all the way down to the bottom, and click View Instance at the bottom right. This will take you to the Instances
screen which is part of the EC2 Dashboard.
In a short time, the Instance State for your instance should indicate Running
. You will need to scroll to the right to see all of the information available about your instance.
You are almost set, but there is one more thing you should do, and that is to scroll down in the menu on the left side to select Security Groups. You should see a list of two or more groups. Select the group whose name begins with Kill Bill on AWS
, then scroll to the bottom and select the tab for Inbound Rules
. You should see:
These rules enable the ports that must be open to access KAUI and Kill Bill from a browser. To enable direct login to your instance using SSH, you need to add one more port. Click on Edit Inbound Rules. Then add a rule with the following elements: Type: SSH, Protocol: TCP, Port Range: 22, Source: 0.0.0.0/0. Your Inbound Rules should now look like this:
Initial Testing
You can now login to KAUI from your browser using the URL https://INSTANCE_IP:443, where INSTANCE_IP is the IPV4 address for your instance, given on your dashboard as Public IPV4 Address. This should display the KAUI login screen. The default credentials are: admin/{EC2 instance ID}. The first few requests might be a bit slow as Kill Bill initializes itself.
For an introduction to KAUI, see our Kaui Guide.
You can also login in directly to your instance from a terminal or command window. You may have no need to do this for a while, but you should try it and make sure it works. You may need to do this for system monitoring or troubleshooting later.
To login, use the secure shell command:
ssh -i PRIVATE_KEY.pem ubuntu@INSTANCE_IP
Here PRIVATE_KEY is the pathname where you have stored the private key that was downloaded when you generated your key pair, and INSTANCE_IP is the IPV4 address described earlier. The private key will not work unless its access controls are set to readable by the owner only.
On Windows versions before Windows 10, you may need to download a program called PuTTY to enable ssh
. On Windows 10 ssh
is available but may need to be activated through the Settings screen.
The first time you login, you will see a warning message asking if you want to add this host to your list of hosts. You should answer yes
.
For more information about the organization of your instance, see the Configuration and Troubleshooting sections below. To exit from your login, type exit
.
Multi-Tier Deployments
Overview
For production deployments, we recommend a multi-tier configuration with two or more EC2 instances and a separate database instance using the Amazon Relational Database Service (RDS)
. In place of nginx
, this configuration relies on an AWS Elastic Load Balancer (ELB)
to correctly spread the traffic among the various nodes, and also to route traffic to either Kill Server or KAUI based on the incoming port. This deployment option requires a bit more setup, so we also provide an (easier) way to deploy using CloudFormation templates. However, the multi-tier option provides more control over the deployment.
The diagram below illustrates the multi-tier configuration. This section will explain how to setup this option.
Login to AWS
To begin, log in to Amazon Web Services at https://aws.amazon.com. If you are new to AWS, you will be asked to create an account and provide billing information. You will need to sign in as a Root User. This should take you to the AWS Management Console, which provides links to all available services.
Check the upper right corner of your screen to be sure you are in the appropriate region. All resources you create will be placed in this region, and may not be accessible from other regions.
In addition, AWS places all resources within a Virtual Private Network (VPC). A default VPC will be created and used automatically in the following steps. However, if you have access to other VPCs, you will need to ensure that all Kill Bill resources are deployed in the same one.
Setup the Database
Once you are logged in, the first step is to setup the RDS instance. This process begins with the RDS dashboard, which should be available from the Services menu. When the dashboard appears, select Databases
from the left menu, and click the red button at the top right that reads Create Database
:
You will be taken to the Create Database
page. The first choice you will have is between Standard Create
, which allows you to set a full range of configuration parameters, or Easy Create
, which sets most of these parameters to defaults. We recommend Easy Create
in most cases.
The next section offers you a choice of several database types. Kill Bill can work with any database type that is mysql
or postgres
compatible. For robust production use, Amazon Aurora is probably a good choice. Here we will illustrate the simpler steps setting up a MariaDB database.
The next choice determines the instance size. We suggest the Production
option as this will provide the most robust configuration.
The last section asks you to:
-
Specify a name for your database
-
Give a username for the administrative account (we suggest that you do not use the default name)
-
Provide a password for the administrative acount (we suggest you let AWS generate one for you)
When the password is setup and confirmed, click Create Database
in the lower right corner. You will return to the main Databases screen, which should now look like this:
This display shows that your database is starting. After a few minutes, the status will change to Available
(You may need to reload the page to see this). At this time you can click on the database name to get more information, including the full name of the instance.
On the page that appears you should see a panel named Connectivity and Security
. The left side of this panel shows the full name of the endpoint, which you will need shortly, and the port number, which is normally 3306.
Lastly, on the Connectivity and Security
panel, locate and click on the link for the default VPC security group. You will need to add an inbound security rule, because the database by default does not allow external access. In the panel for this group, click on Inbound Rules
and select Edit Inbound Rules
. Next click on Add rule
. In the Type
column select MYSQL/Aurora
. The port will be set to 3306 automatically. In the Source
column, click on the search icon and select 0.0.0.0/0
. Finally, click on Save Rules
in the bottom right. Your database is ready to go.
Edit the Configuration Script
To set up the EC2 instances you will need to provide them with information needed to connect to the databases. We provide a brief configuration script to simplify this process. The template for this script is as follows:
#!/bin/bash
DB_PROPS="/var/tmp/db.props.$$"
KB_PROPS="/var/tmp/kb.props.$$"
cat <<_EOF > $DB_PROPS
#
# EDIT THE FOLLOWING DB PROPERTIES AS NEEDED:
#
DB_SERVER=DB-INSTANCE-NAME:3306
DB_USER=ADMIN-NAME
DB_PASSWORD=PASSWORD
KILLBILL_DB_NAME=killbill
KAUI_DB_NAME=kaui
_EOF
cat <<_EOF > $KB_PROPS
#
# EDIT THE FOLLOWING KB PROPERTIES AS NEEDED:
#
org.killbill.dontexist=foo
_EOF
su -l -c "cd /var/lib/tomcat/bin && /var/lib/tomcat/bin/updateProperties.sh $DB_PROPS $KB_PROPS" tomcat
First, you need to edit the database properties. DB_SERVER should be set to the full name of the DB instance, as given in the Connectivity and Security
panel (see above). The port number 3306 is required. DB_USER and DB_PASSWORD should be set to the administrator credentials you have chosen for the RDS instance.
Second, you may optionally edit any Kill Bill properties that you need to change from the standard defaults.
Launch the EC2 Instances
The next step is to launch the number of EC2 instances you want, all based on the Kill Bill single AMI.
To start the installation process, point your browser to AWS Marketplace:
You should see the following image at the top of your screen:
Click Continue to Subscribe. The next page will give the AWS Terms and Conditions:
Accept the terms if asked. You will then see a new message confirming that you have subscribed. Next, click Continue to Configuration.
The next page will give several configuration options.
In most cases you should accept the defaults. Then click Continue to Launch.
The next page will give you several options for the launch method. We recommend that you choose Launch through EC2.
All other options will disappear. Click Launch.
The next page is titled Choose Instance Type. Your instance type is already chosen, so click Configure Instance Details at the bottom of the page.
The next page will provide you with a long list of options. The first option is Number of Instances. Set the number of instances you wish to launch. Each instance will have essentially the same configuration, including the same image, the same subnet and availability zone, and the same security group.
After setting the number of instances, scroll down to the bottom of the page. The last section is titled Advanced Settings. In this section you should set the configuration file you produced above. The setting panel should look like this:
Now click Review and Launch at the bottom of the page. The following page is headed Review Instance Launch. This provides a chance to review the details of your chosen instances. Then click Launch at the bottom of the page.
Next you will see a very important popup that asks you to choose or create a key pair.
The key pair provides the credentials you will need to login to your EC2 instances. For details about key pairs, see the AWS documentation. We recommend that you create a new key pair. All your instances can use the same one. Give the key pair a simple, easy to remember name such as My-Key-Pair
. Then click Download Key Pair. Important: You must save the private key that will be generated in this step. If you lose this key, you will not be able to login to your instances.
When the key pair is generated, click Launch Instances. You should see the screen below:
Your instances are finally launching! To follow what is happening on the EC2 Dashboard, scroll all the way down to the bottom, and click View Instance at the bottom right. This will take you to the Instances
screen which is part of the EC2 Dashboard.
In a short time, the Instance State for your instance should indicate Running
. You will need to scroll to the right to see all of the information available about your instances.
Setup Security Rules
You are almost set, but there is one more thing you need to do, and that is to scroll down in the menu on the left side to select Security Groups. You should see a list of two or more groups. Select the group whose name begins with Kill Bill on AWS
, then scroll to the bottom and select the tab for Inbound Rules
. You should see:
These rules enable the ports that must be open to access KAUI and Kill Bill from a browser. However, for access through the ELB these ports will be different. In addition, to enable direct login to your instance using SSH, you need to add one more port.
Click on Edit Inbound Rules. then do the following:
-
For the rule that specifies Type: HTTPS, Port Range: 443, change the type to CUSTOM TCP and the Port Range to 3000.
-
For the rule that specifies Type: CUStOM TCP, Port Range: 8443, change the Port Range to 8080.
-
Finally, add a rule with the following elements: Type: SSH, Protocol: TCP, Port Range: 22, Source: 0.0.0.0/0.
Your Inbound Rules should now look like this:
Create the Databases
Kill Bill requires two databases, with the names killbill
and kaui
. We provide predefined schemas for these databases.
To create the databases, you will need to login to one of your instances. To login, use the secure shell command:
ssh -i PRIVATE_KEY.pem ubuntu@INSTANCE_IP
Here PRIVATE_KEY is the pathname where you have stored the private key that was downloaded when you generated your key pair, and INSTANCE_IP is the IPV4 address described earlier. The private key will not work unless its access controls are set to readable by the owner only. The ssh
command relies on the security rule providing access to Port 22, which you added in the previous step.
On Windows versions before Windows 10, you may need to download a program called PuTTY to enable ssh
. On Windows 10 ssh
is available but may need to be activated through the Settings screen.
The first time you login, you will see a warning message asking if you want to add this host to your list of hosts. You should answer yes
.
Once you are logged in, you can use the mysql
command to create the two databases killbill
and kaui
. The credentials for this command are the same ones you set up for the database and copied to the configuration file. Note that the DB-INSTANCE-NAME should not include the port number.
The password will not be echoed when it is typed.
> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p
Enter Password:
mysql> create database killbill;
mysql> create database kaui;
mysql> exit
The next step is to install the schemas. These can be found at:
-
killbill schema:
https://docs.killbill.io/latest/ddl.sql
-
kaui schema:
https://github.com/killbill/killbill-admin-ui/blob/master/db/ddl.sql
One easy way to do this is to return to your local computer (type exit
) and use the sftp
command. Then upload them to your EC2 instance home directory with the commands:
sftp -i PRIVATE_KEY.pem ubuntu@INSTANCE_IP
put killbill.ddl
put kaui.ddl
exit
Once the files are successfully uploaded, login again to your instance using the ssh
command. You can now install the schemas:
> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p killbill < killbill.ddl
Enter Password:
> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p kaui < kaui.ddl
Enter Password:
To ensure that the databases are setup correctly, login to mysql
again, then try the SHOW TABLES command:
> mysql -h DB-INSTANCE-NAME -u ADMIN-NAME -p
Enter Password:
use killbill
show tables;
use kaui
show tables;
exit
Each show tables
command should display a list of table names for the database.
You should now be able to login to KAUI from your browser using the URL http://INSTANCE_IP:3000, where INSTANCE_IP is the IPV4 address for any one of your instances, given on your dashboard as Public IPV4 Address. This should display the KAUI login screen. The default credentials are: admin/{EC2 instance ID}. The first few requests might be a bit slow as Kill Bill initializes itself. For an introduction to KAUI, see the Quick Start with Kaui guide.
In addition, the URL http://INSTANCE_IP:8080 should display a page confirming that the Kill Bill server is accessible.
Add the ELB in front of the EC2 instances
The last major task is to setup the Elastic Load Balancer in front of the EC2 instances. To begin, from the EC2 dashboard scroll down the left-hand menu and select Load Balancing / Load Balancers. Then click the Create Load Balancer button at the upper left.
You will be given a choice of several load balancer types. The type we will use is Application Load Balancer.
Click on the Create button in the Application Load Balancer box. This will bring up the page titled Step 1: Configure Load Balancer:
On this page you need to do the following:
-
Assign a name to your load balancer
-
Select a protocol for the listener. We recommend using HTTPS as discussed below.
-
Set the listener port to 443 (if using HTTPS) or 80 (if using HTTP).
-
Click Add Listener to add a second listener using the same protocol and set its port to 8443.
-
Scroll to the bottom and select at least two availability zones. IMPORTANT: You must select all of the zones that your EC2 instances are contained in. Otherwise, the load balancer will be unable to connect to these instances.
Now choose Next: Configure Security Settings. You will now see a page titled Step 2: Configure Security Settings.
If you have selected the HTTPS protocol, you will be required to create or provide an X.509 SSL Certificate. If you already have a certificate you can identify it or upload it here. Otherwise we recommend you click on Request a New Certificate from ACM. This will enable you to create a certificate using the Amazon Certificate Manager as discussed below. When your certificate is created, return to this page and proceed to the next step.
If you are using HTTP, you will see a warning message, which can be ignored.
In either case, your next step is to choose Next: Configure Security Groups. This will take you to a page titled Step 3: Configure Security Groups. This page will show you the existing security groups and offer the choice to choose an existing group or create a new one. We advise you to create a new group. The new security group should have inbound rules enabling inputs for KAUI and Kill Bill as shown here:
These rules assume you are using HTTPS. If you are using HTTP, the port for the first rule should be 80.
The next step is to identify the target instances for your load balancer, which are collected into a target group. Each listener will have a separate target group, but you can only setup one right now. The other group will be added later.
Your group will consist of all of the instances you have launched. First, create the group, give it a simple name, and set the port to 3000:
Now click on Next: Register Targets. The purpose of this step is to identify the target instances that will be part of your target group. Initially, all your instances will be listed in the bottom section. To register them, select them all and click Add to Registered. Then proceed to Next: Review.
Check all settings, then click Create. Your load balancer will be created. Close the final page to see the Load Balancer list. The initial status for your new ELB will be provisioning. After a few minutes this will change to active.
You are almost done. Your final step is to modify the second listener to use a different port number to access the Kill Bill server. From the left side menu select Target Groups. Click Create Target Group. In the page that appears, set the protocol to HTTP and set the port to 8080. Then click Next.
The next page is provided to register the targets for this group. Once again you will register all your instances as targets, but this works just a little differently than before. The list of available targets is at the top. Select all of them, then click Include as Pending Below to make these targets pending as members of the group. Finally, click Create Target Group to create the group.
Now you will need to associate this group with your load balancer’s second listener. Return to the Load Balancer console, select your load balancer, and choose the Listeners tab in the bottom information panel. Then in the second entry, click View/Edit Rules.
Your object now is to change the rule for this listener to point to your new target group. To do this:
-
Click on the pencil icon at the top
-
Click on the pencil icon to the left of the rule
-
Click on the pencil icon under the heading THEN
-
Select your new target group in the dropdown list
-
At the top of the page, click UPDATE
Go back to the listener tab and confirm that the second listener points to the new target group. Your load balancer is now ready.
Creating a Certificate
We recommend for security that you choose the HTTPS protocol to access KAUI. This protocol uses SSL/TLS encryption and authentication to verify and protect all communication with the KAUI server. To use this protocol you must create or provide an X.509 certificate. This section describes how to create a certificate using the Amazon Certificate Manager (ACM).
When you click on Request a new Certificate from ACM, as described above, you will be taken to the main page of the ACM. Initially this will probably show that you have no certificates. Click Request a Certificate.
the next page will give you the option to import a certificate, or to request a public or private certificate. Choose private, then click again on Request a Certificate. This will take you to the Request a Certificate page. You are initially at Step 1: Add Domain Names.
To validate your certificate, you must first setup a CNAME, or Canonical Name, based on a verified domain that you own (this is not an AWS domain). If you do not have a domain, you can obtain one for free, or very low cost, from an online source such as GoDaddy. The CNAME is part of your domain’s DNS profile, and your domain provider should have instructions for creating it. The values to use for the CNAME will be seen shortly. You will first setup a temporary CNAME just to enable validation of your certificate. You will then create the CNAME that you will actually use to access the system.
Enter your domain name and click Next. On the following page, select DNS Validation. The next page gives you the chance to assign one or more tags to your certificate. These are optional and can be skipped. Finally, you are given a chance to review your request. Check everything, then click Confirm and Request. The Validation page will appear.
At the bottom you will see your domain name with a status of Pending Validation. Click the triangle next to your domain name. The display will expand to show the values that you must use to create your validation CNAME.
The two strings labeled Name and Value should be copied to the two required entries for your CNAME. Copy these long strings carefully. Do not include any trailing periods. There is an option to save these strings in a file so they may be copied more easily, or stored for a later time.
Your second CNAME is much simpler. The first entry will be kaui. The second entry will be simply your domain name, e.g., killbill.com.
When your CNAMEs are set, return to ACM and click Continue. The display changes to show that your certificate is Pending Validation. The certificate should now appear also on the main ACM page. When (and if) it is validated, the status will change to Success. This may take several hours or more. A short time later it will change again to Issued.
When your certificate is successfully created, you may delete the temporary (validation) CNAME. Then proceed as described above.
Testing
When your ELB is complete you can proceed to testing. If you are using HTTPS you should be able to login to KAUI from your browser using the URL https://kaui.DOMAIN, where DOMAIN is your domain that you have used for your certificate. If you are using HTTP you will need to use the URL http://DNS_NAME where DNS_NAME is the DNS name for the KAUI load balancer as given on the Load Balancer dashboard. This should display the KAUI login screen. The default credentials are: admin/{EC2 instance ID}. The first few requests might be a bit slow as Kill Bill initializes itself.
For an introduction to KAUI, see the Quick Start with Kaui guide.
Similarly, you should be able to login directly to the Kill Bill server using the URL http://DNS_NAME:8443. Creating a CNAME for this login is optional.
Configuration
SSL Certificates
nginx
is configured to listen on port 443 for messages to be forwarded to KAUI. Port 443 is the standard port for HTTPS. By default, accessing the service from a web browser will show a Not Secure
site. In order to make the site secure, you will need to add a valid certificate. The easiest option to add the certificate is to make use of the tool certbot
, which relies on the free Certificate Authority (CA) Let’s Encrypt
This method provides an easy way to obtain and install free TLS/SSL certificates.
Certbot
is pre-installed in the latest Kill Bill AMI, but if you need a different version of certbot
, this is also available through the new package manager snap
. This package manager is also preinstalled in the Kill Bill AMI.
To validate your certificate, you must first setup a CNAME, or Canonical Name, based on a verified domain that you own. If you do not have a domain, you can obtain one for free, or very low cost, from an online source such as GoDaddy. The CNAME is part of your domain’s DNS profile, and your domain provider should have instructions for creating it.
The CNAME should link your domain to your AWS instance. If your domain is domain.com
your CNAME might look like killbill.domain.com
.
To setup your certificate, perform the following steps:
[1]. Create a CNAME linking your domain to your AWS instance.
[2]. Add your CNAME to the server block in the nginx
configuration file.
Using a text editor of your choosing, edit the file /etc/nginx/sites-enabled/killbill.conf
. You will need to use sudo to edit this file.
This file contains two server blocks. The second block contains the lines:
server {
listen 443;
server_name _;
Replace the underscore after server_name
with your CNAME:
server {
listen 443;
server_name killbill.domain.com;
Save the modified file.
[3]. Reload the configuration file with the command:
sudo nginx -s reload
[4]. Enable HTTP traffic on port 80. This is just to allow certbot
to create your certificate. To do this, go to your EC2 dashboard and add a new inbound rule as follows: Type: HTTP, Protocol: TCP, Port Range: 80, Source: 0.0.0.0/0. Your inbound rules should now look like this:
[5]. Remove the existing certbot
package:
Type the command
sudo apt-get remove certbot
When asked if you want to proceed, type Y
for yes.
[6]. Install certbot
using snap
:
The following commands will install certbot
and link it to the /usr/bin
directory:
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
[7]. Run certbot
to create and install the certificate:
sudo certbot --nginx
If you are asked, confirm the server name to be used. If all goes well, you will see a message like:
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/deployment.killbill.io/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/deployment.killbill.io/privkey.pem
Your cert will expire on 2020-05-11. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the "certonly" option. To non-interactively renew *all* of
your certificates, run "certbot renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
Of course, there are a few things that could go wrong:
[1]. certbot
will attempt to validate your CNAME through your domain provider. If the CNAME does not link correctly to the public DNS name for your AWS instance, the certificate will not be created.
[2]. You could get a message like:
Error creating new order : Cannot issue for ec2-18-234-168-57.compute-1.amazonaws.com: The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy
This could happen if you have tried to use your public DNS name directly as your CNAME, which is not allowed.
[3]. You could get a message like certificate cannot be installed
.
For help with these and other problems, try the certbot
help page.
When your certificate is installed, you should be able to access KAUI from you browser using https://
followed by your CNAME as the address. The browser should indicate that the site is secure.
When your certificate is successfully installed, you can edit your security groups again to remove the access through port 80.
The Let’s Encrypt
certifcates are only valid 90 days and will therefore neeed to be renewed. certbot
will create a cron entry under /etc/cron.d/certbot
to make this process automatic.
Kill Bill Properties
Kill Bill global properties are defined in /var/lib/killbill/config/killbill.properties
. For example, this is where you can change the MySQL credentials. Take a look at our configuration guide for more details.
This is also where you can change the default Kill Bill admin credentials: specify org.killbill.security.shiroResourcePath=/var/lib/killbill/config/shiro.ini
and create the shiro.ini
file accordingly (see our RBAC guide for more details).
TroubleShooting
After launching the EC2 instance, the full stack should come up, with all services enabled and running, that is:
-
An nginx instance receiving traffic on port
443
and8443
-
A instance of Kill Bill server listening on
127.0.0.1:8080
(and receiving external traffic through nginx on port8443
) -
A instance of Kaui listening on
127.0.0.1:3000
(and receiving external traffic through nginx on port443
) -
A local
mysql
server running on port3306
In this section, we will provide some tips to verify the operation of the system, and what to do when things are not working.
System Health
To verify the overall health of the Kill Bill server instance, you can issue the following command:
curl http://127.0.0.1:8080/1.0/healthcheck
This will return a series of messages giving the health of various system components.
A detailed system information report is available using the following command:
curl -u <ADMIN>:<{EC2 instance ID}> http://127.0.0.1:8080/1.0/kb/nodesInfo
Here <ADMIN> and <{EC2 instance ID}> are your administrator access credentials.
The Kill Bill single-tier option is composed of four services. The status of each service can be checked by the following commands:
-
Kill Bill service:
sudo service killbill status
-
KAUI service:
sudo service kaui status
-
Nginx service:
sudo service nginx status
-
Mysql service:
sudo service mysql status
For each report there should be a line near the top with the following form:
Active: active (running) since Sat 2020-10-24 20:13:43 UTC; 1 day 1h ago
Similarly you can start
or stop
the services using similar commands, such as sudo service kaui stop
to stop KAUI.
Log Files
The system maintains a series of logfiles that should be helpful when troubleshooting is needed.
Tomcat logs are under /var/lib/tomcat/logs/
:
-
KAUI logs:
/var/lib/tomcat/logs/kaui.out
-
Kill Bill server logs:
/var/lib/tomcat/logs/catalina.out
Nginx logs can be found under /var/log/nginx/
-
Access logs:
/var/log/nginx/access.log
-
Error logs:
/var/log/nginx/error.log
System Diagnostics
If support is needed, we may require some additional detailed information about your deployment. We have created a diagnostic
command for this purpose. to run this command:
# Login as 'tomcat'
> sudo su - tomcat
#
# Set your access credentials. This example
# assumes the default credentials are used:
#
> kpm diagnostic \
--killbill-credentials=admin {EC2 instance ID} \
--bundles-dir=/var/lib/killbill/bundles \
--database-name=killbill \
--database-credentials=root root \
--killbill-api-credentials=bob lazar \
--kaui-web-path=/var/lib/tomcat/webapps2 \
--killbill-url=http://127.0.0.1:8080 \
--database-host=127.0.0.1:3306
...
The diagnostic data is exported under /tmp/killbill-diagnostics-20200213-23204-u93ah5/killbill-diagnostics-02-13-20.zip
Databases
To access the mysql (MariaDB) databases, you can use the following command:
mysql -u root -proot
This enables interactive access to the database manager. There is one killbill
and one kaui
database created and used by the respective applications. To verify the tables in each database, you can type:
use killbill;
show tables;
or
use kaui;
show tables;
Standard SQL commands can be used to explore or manipulate the tables. Be sure you know what you are doing, or the databases may become corrupted!
To exit the mysql interactive mode, type exit
.
Nginx
The load balancer nginx
should normally require little attention. The configuration files are located under /etc/nginx/. The configuration file for `nginx itself is `/etc/nginx/nginx.conf
. Additional configuration files are located under /etc/nginx/sites-enabled/
. The only file normally present in this directory is /etc/nginx/sites-enabled/killbill.conf
. This file may need to be edited to enable SSL, as explained above.
Upgrades
From time to time new versions of Kill Bill and KAUI may be released. This section explains how to install these new versions.
Note: you must switch to the tomcat
user first in order to upgrade Kill Bill or Kaui (sudo su - tomcat
).
The configuration file /var/lib/killbill/kpm.yml
specifies the Kill Bill version (and its plugins) running on the instance. Once you edit this file to specify the new version number, it will be used automatically. After editing this file, simply run the command $KPM_INSTALL_KB_CMD
, delete the cached directory /var/lib/tomcat/webapps/ROOT
, and restart the instance.
A similar process can be used for KAUI: update /var/lib/kaui/kpm.yml
, run $KPM_INSTALL_KAUI_CMD
, delete the cached directory /var/lib/tomcat/webapps2/ROOT
and restart the instance.