Overview
This guide explains how to maintain your single-tier Kill Bill installation on AWS.
Login
To perform most maintenance tasks you will need to login to your EC2 instance. Be sure that SSH is enabled on Port 22 in your security group. Login procedures are described in your setup guide (See Step 7: Login to an Instance)
Troubleshooting
If a new installation does not seem to work correctly, the first step is to review your installation process carefully, to be sure that everything has been done and checked as described in your setup guide. If problems persist, we will be glad to work with you to identify the problem. To help us to do this, there are several information reports that you may need to gather. These reports are somewhat technical but can be analyzed by Kill Bill personnel. This section explains how to obtain the reports that may be needed.
Status Reports
When you launch the EC2 instance, all required services should be enabled and running, including:
-
An instance of
nginx
receiving traffic on port443
and8443
-
An instance of Kill Bill server listening on
127.0.0.1:8080
(and receiving external traffic through nginx on port8443
) -
An instance of Kaui listening on
127.0.0.1:3000
(and receiving external traffic through nginx on port443
) -
A local
mysql
server running on port3306
The status of each service can be checked by the following commands:
-
Kill Bill service:
sudo service killbill status
-
KAUI service:
sudo service kaui status
-
Nginx service:
sudo service nginx status
-
Mysql service:
sudo service mysql status
For each report there should be a line near the top with the following form:
Active: active (running) since Sat 2020-10-24 20:13:43 UTC; 1 day 1h ago
You can start
or stop
the services using similar commands, such as sudo service kaui stop
to stop Kaui. If any of these services are not active, try starting it using the appropriate command.
If all services are running, the next step is to collect several reports.
Log Files
The system maintains a series of log files that may be helpful when troubleshooting is needed. These logs may be forwarded to Kill Bill to help diagnose your problem.
Tomcat logs are under /var/lib/tomcat/logs/
:
-
Kaui log:
/var/lib/tomcat/logs/kaui.out
-
Kill Bill server log:
/var/lib/tomcat/logs/killbill.out
-
Complete tomcat log:
/var/lib/tomcat/logs/catalina.out
Some of these logs may be very large, expecially catalina.out.
System Health Check
The healthcheck report checks the health of various software components, and determines if any queues are growing improperly over time. To create this report, issue the following command:
curl http://127.0.0.1:8080/1.0/healthcheck
This will return a series of messages giving the health status of each component. These messages may look like:
"com.codahale.metrics.health.jvm.ThreadDeadlockHealthCheck":{"healthy":true},
"main.pool.Connection99Percent":{"healthy":true},
"main.pool.ConnectivityCheck":{"healthy":true},
...
"bus":{"growing":false},
"overdue-service:overdue-check-queue":{"growing":false},
...
An unexpected message may indicate a component that is not working properly.
System Information
For a detailed system information report, showing the current version of each component, use the following command:
curl -u <ADMIN>:<INSTANCE_ID> http://127.0.0.1:8080/1.0/kb/nodesInfo
Here <ADMIN> is your Kaui administrator username, and <INSTANCE_ID> is your EC2 instance ID. THe output will look like:
"nodeName":"ip 172-31-20-130.ec2.internal",
"bootTime":"2023-01-28T20:30:19.000Z",
"lastUpdatedDate":"2023-01-28T20:30:19.000Z",
"kbVersion":"0.22.26",
"apiVersion":"0.53.17",
"pluginApiVersion":"0.26.4",
"commonVersion":"0.24.15",
"platformVersion":"0.40.8",
"pluginsInfo":
[{"bundleSymbolicName":"org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
"pluginKey":null,
"pluginName":"org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
"version":null,
"state":"RUNNING",
"isSelectedForStart":true,
"services":[]},
{"bundleSymbolicName":"org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
"pluginKey":null,
"pluginName":"org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
"version":null,
"state":"RUNNING",
"isSelectedForStart":true,
"services":[]}]
Diagnostic Command
The diagnostic
option of the Kill Bill Package Manager (kpm)
command creates an extensive report for a given tenant that may be useful for troubleshooting. To run this command:
# Login as 'tomcat'
> sudo su - tomcat
#
# Run the command with your access credentials:
#
> kpm diagnostic \
--killbill-credentials=<ADMIN> <INSTANCE_ID> \
--bundles-dir=/var/lib/killbill/bundles \
--database-name=killbill \
--database-credentials=<DBUSER> <DBPASS> \
--killbill-api-credentials=<KEY> <SECRET> \
--kaui-web-path=/var/lib/tomcat/webapps2 \
--killbill-url=http://127.0.0.1:8080 \
--database-host=127.0.0.1:3306
You will need to edit this command to include:
-
Your Kaui admin username and your EC2 instance ID (<ADMIN> <INSTANCE_ID>)
-
Your database credentials (<DBUSER> <DBPASS>)
-
The public key and secret key for your tenant (<KEY> <SECRET>)
To restrict the report to a single account, you can add the line
--account-export=<ACCOUNT_ID>
replacing <ACCOUNT_ID> with the ID of the specific account to be included.
The last line of the response should look like:
Diagnostic data is exported under /tmp/killbill-diagnostics-20200213-23204-u93ah5/killbill-diagnostics-02-13-20.zip
The specified zip file contains several reports of various sizes. This report can be downloaded to your computer using sftp
and forwarded to Kill Bill for analysis.
Databases
To access the mysql (MariaDB) databases, you can use the following command:
mysql -u root -proot
This enables interactive access to the database manager. There is one killbill
and one kaui
database created and used by the respective applications. To verify the tables in each database, you can type:
use killbill
show tables;
or
use kaui
show tables;
Standard SQL commands can be used to explore or manipulate the tables. Be sure you know what you are doing, or the databases may become corrupted!
To exit the mysql interactive mode, type exit
.
Load Balancer
The load balancer nginx
should normally require little attention. The configuration files are located under /etc/nginx/
. The configuration file for nginx
itself is /etc/nginx/nginx.conf
. Additional configuration files are located under /etc/nginx/sites-enabled/
. The only file normally present in this directory is /etc/nginx/sites-enabled/killbill.conf
.
Nginx logs can be found under /var/log/nginx/
-
Access logs:
/var/log/nginx/access.log
-
Error logs:
/var/log/nginx/error.log
Configuration
Kill Bill defines a number of global properties and per-tenant properties that can be varied. These properties are explained in the Kill Bill and Kaui Configuration Guide. Default values for these properties are built into the Kill Bill code; these values can be overridden by values defined in the file /var/lib/killbill/config/killbill.properties
. For example, this is where you can change the database URL and credentials.
This file also defines the location of the shiro.ini
file, which by default is in the same directory. This file defines the Kill Bill admin credentials, along with any other users and their roles. See the Users, Roles, and Permissions Management guide for details about this file.
Upgrades
From time to time new versions of Kill Bill are released. These versions can be incorporated in your installation with minimal impact on production. This section explains how to upgrade to a new version of Kill Bill. Note that these are not the same as new versions of the AMI, which can be incorporated only by a full reinstall.
First, login to your instance using ssh
,
Next, switch to the tomcat
user:
sudo su - tomcat
The configuration file /var/lib/killbill/kpm.yml
specifies the Kill Bill version (and its plugins) to be run on the instance. Once you edit this file to specify the new version number, it will be used automatically. Perform the following steps:
-
Edit the configuration file to update the version number
-
Run the command
$KPM_INSTALL_KB_CMD
-
Delete the cached directory
/var/lib/tomcat/webapps/ROOT
-
Restart the instance.
A similar process can be used for Kaui: update /var/lib/kaui/kpm.yml
, run $KPM_INSTALL_KAUI_CMD
, delete the cached directory /var/lib/tomcat/webapps2/ROOT
and restart the instance.