Overview
This guide explains how to maintain your single-tier Kill Bill installation on AWS.
Log in
To perform most maintenance tasks you will need to log in to your EC2 instance. Be sure that SSH is enabled on Port 22 in your security group. Login procedures are described in the setup guide (See Step 7: Login to an Instance)
Troubleshooting
If a new installation does not seem to work correctly, the first step is to review your installation process carefully, to be sure that everything has been done and checked as described in the setup guide. If problems persist, we will be glad to work with you to identify the problem. To help us to do this, there are several information reports that you may need to gather. These reports are somewhat technical but can be analyzed by Kill Bill personnel. This section explains how to obtain the reports that may be needed.
Status Reports
When you launch the EC2 instance, all required services should be enabled and running, including:
-
An instance of
nginx
receiving traffic on port443
and8443
-
An instance of Kill Bill server listening on
127.0.0.1:8080
(and receiving external traffic through nginx on port8443
) -
An instance of Kaui listening on
127.0.0.1:3000
(and receiving external traffic through nginx on port443
) -
A local
mysql
server running on port3306
The status of nginx/KB/mysql services can be checked by the following commands:
-
Kill Bill service:
sudo service killbill status
-
Nginx service:
sudo service nginx status
-
Mysql service:
sudo service mysql status
For each report there should be a line near the top with the following form:
Active: active (running) since Sat 2020-10-24 20:13:43 UTC; 1 day 1h ago
You can start
or stop
the services using similar commands, such as sudo service killbill stop
to stop Kill Bill. If any service is not active, try starting it using the appropriate command.
If all services are running, the next step is to collect several reports.
Log Files
The system maintains a series of log files that may be helpful when troubleshooting is needed. These logs may be forwarded to Kill Bill to help diagnose your problem.
Tomcat/KB/Kaui logs are under /var/lib/tomcat/logs/
:
-
Kaui log:
/var/lib/tomcat/logs/kaui.out
-
Kill Bill server log:
/var/lib/tomcat/logs/killbill.out
-
Complete tomcat log:
/var/lib/tomcat/logs/catalina.out
-
Access requests to the servers:
localhost_access_log*.txt
Sometimes these logs will clearly show anomalies that will pinpoint the problem. If not you should forward your logs to Kill Bill for review.Some of these logs may be very large, especially catalina.out.
Logging Configuration
The logging configuration, including settings for log rotation and log levels, is defined in the logback.xml file
. Separate logback.xml
files are used for KB and Kaui, located at the following paths:
-
KB logback file:
/var/lib/tomcat/webapps/ROOT/WEB-INF/classes/logback.xml
-
Kaui logback file:
/var/lib/tomcat/webapps2/ROOT/WEB-INF/classes/logback.xml
These files can be modified to customize the logging settings for KB and Kaui. By default, the log level is configured as INFO
.
System Health Check
The healthcheck report checks the health of various software components, and determines if any queues are growing improperly over time. To create this report, issue the following command:
curl http://127.0.0.1:8080/1.0/healthcheck
This will return a series of messages giving the health status of each component. These messages may look like:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1412 0 1412 0 0 54307 0 --:--:-- --:--:-- --:--:-- 56480
{
"main.pool.ConnectivityCheck": {
"healthy": true
},
"org.killbill.billing.server.healthchecks.KillbillHealthcheck": {
"healthy": true,
"message": "OK"
},
"org.killbill.billing.server.healthchecks.KillbillPluginsHealthcheck": {
"healthy": true
},
"org.killbill.billing.server.healthchecks.KillbillQueuesHealthcheck": {
"healthy": true,
"bus": {
"growing": false
},
"overdue-service:overdue-check-queue": {
"growing": false
},
"notifications-retries:overdue-check-queue": {
"growing": false
},
"entitlement-service:entitlement-events": {
"growing": false
},
"invoice-service:next-billing-date-queue": {
"growing": false
},
"notifications-retries:extBusEvent-listener": {
"growing": false
},
"payment-service:janitor": {
"growing": false
},
"externalBus": {
"growing": false
},
"payment-service:retry": {
"growing": false
},
"notifications-retries:next-billing-date-queue": {
"growing": false
},
"notifications-retries:payment-bus-handler": {
"growing": false
},
"subscription-service:subscription-events": {
"growing": false
},
"invoice-service:parent-invoice-commitment-queue": {
"growing": false
},
"server-service:push-notification-queue": {
"growing": false
},
"notifications-retries:overdue-async-bus-queue": {
"growing": false
},
"overdue-service:overdue-async-bus-queue": {
"growing": false
},
"notifications-retries:payment-service": {
"growing": false
},
"notifications-retries:invoice-listener": {
"growing": false
}
},
"osgi.pool.ConnectivityCheck": {
"healthy": true
},
"shiro.pool.ConnectivityCheck": {
"healthy": true
}
}
An unexpected message may indicate a component that is not working properly.
System Information
For a detailed system information report, showing the current version of each component, use the following command:
curl -u <ADMIN>:<INSTANCE_ID> http://127.0.0.1:8080/1.0/kb/nodesInfo
Here <ADMIN> is your Kaui administrator username, and <INSTANCE_ID> is your EC2 instance ID. THe output will look like:
[
{
"nodeName": "ip-192-168-241-89.ec2.internal",
"bootTime": "2024-07-11T07:30:59.000Z",
"lastUpdatedDate": "2024-07-11T07:30:59.000Z",
"kbVersion": "0.24.10",
"apiVersion": "0.54.0",
"pluginApiVersion": "0.27.1",
"commonVersion": "0.26.2",
"platformVersion": "0.41.9",
"pluginsInfo": [
{
"bundleSymbolicName": "org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
"pluginKey": null,
"pluginName": "org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
"version": null,
"state": "RUNNING",
"isSelectedForStart": true,
"services": []
},
{
"bundleSymbolicName": "org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
"pluginKey": null,
"pluginName": "org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
"version": null,
"state": "RUNNING",
"isSelectedForStart": true,
"services": []
},
{
"bundleSymbolicName": "org.kill-bill.billing.killbill-platform-osgi-bundles-metrics",
"pluginKey": null,
"pluginName": "org.kill-bill.billing.killbill-platform-osgi-bundles-metrics",
"version": null,
"state": "RUNNING",
"isSelectedForStart": true,
"services": []
}
]
}
]
Diagnostic Command
The diagnostic
option of the Kill Bill Package Manager (kpm)
command creates an extensive report for a given tenant that may be useful for troubleshooting. See KPM Diagnostic Usage.
To run this command:
# Login as 'tomcat'
> sudo su - tomcat
#
# Run the command with your access credentials:
#
> kpm diagnostic \
--account-export=<account_id> \
--killbill-credentials=<ADMIN> <PASSWORD> \
--bundles-dir=/var/lib/killbill/bundles \
--killbill-api-credentials=<KEY> <SECRET> \
--killbill-web-path=/var/lib/tomcat/webapps \
--kaui-web-path=/var/lib/tomcat/webapps2 \
--killbill-url=http://127.0.0.1:8080
You will need to edit this command to include:
-
Your admin username and password
-
The public key and secret key for your tenant (<KEY> <SECRET>)
The last line of the response should look like:
Diagnostic data is exported under /tmp/killbill-diagnostics-20200213-23204-u93ah5/killbill-diagnostics-02-13-20.zip
The specified zip file contains several reports of various sizes. This report can be downloaded to your computer using sftp
and forwarded to Kill Bill for analysis.
Databases
To access the mysql (MariaDB) databases, you can use the following command:
mysql -u root -proot
This enables interactive access to the database manager. There is one killbill
and one kaui
database created and used by the respective applications. To verify the tables in each database, you can type:
use killbill
show tables;
or
use kaui
show tables;
Standard SQL commands can be used to explore or manipulate the tables. Be sure you know what you are doing, or the databases may become corrupted!
To exit the mysql interactive mode, type exit
.
Load Balancer
The load balancer nginx
should normally require little attention. The configuration files are located under /etc/nginx/
. The configuration file for nginx
itself is /etc/nginx/nginx.conf
. Additional configuration files are located under /etc/nginx/sites-enabled/
. The only file normally present in this directory is /etc/nginx/sites-enabled/killbill.conf
.
Nginx logs can be found under /var/log/nginx/
-
Access logs:
/var/log/nginx/access.log
-
Error logs:
/var/log/nginx/error.log
Configuration
Kill Bill defines a number of global properties and per-tenant properties that can be varied. These properties are explained in the Kill Bill and Kaui Configuration Guide. Default values for these properties are built into the Kill Bill code; these values can be overridden by values defined in the file /var/lib/killbill/config/killbill.properties
. For example, this is where you can change the database URL and credentials.
This file also defines the location of the shiro.ini
file, which by default is in the /var/lib/killbill/config
directory. The shiro.ini
file defines the Kill Bill admin credentials, along with any other users and their roles. See the Users, Roles, and Permissions Management guide for details about this file.
Upgrades
From time to time new versions of Kill Bill are released. These versions can be incorporated in your installation with minimal impact on production. This section explains how to upgrade to a new version of Kill Bill. Note that these are not the same as new versions of the AMI, which can be incorporated only by a full re-installation.
You can upgrade KB using the following steps:
-
Log in to your instance using
ssh
. -
Switch to the
tomcat
user:sudo su - tomcat
-
Update
/var/lib/killbill/kpm.yml
with the appropriate KB version. For example, to upgrade to KB0.24.9
, update this file as follows:killbill: version: 0.24.9 plugins: plugins_dir: /var/lib/killbill/bundles webapp_path: /var/lib/tomcat/webapps/ROOT.war
-
Delete the cached
webapps/ROOT
directory:rm -rf /var/lib/tomcat/webapps/ROOT
-
Run the following command to update KB:
$KPM_INSTALL_KB_CMD
-
Restart KB:
sudo service killbill restart
You can upgrade Kaui using the following steps:
-
Log in to your instance using
ssh
. -
Switch to the
tomcat
user:sudo su - tomcat
-
Update
/var/lib/kaui/kpm.yml
with the appropriate Kaui version. For example, to upgrade to Kaui3.0.6
, edit this file as follows:kaui: version: 3.0.6 # Used for the sha1.yml plugins_dir: /var/lib/kaui/bundles webapp_path: /var/lib/tomcat/webapps2/ROOT.war
-
Delete the cached
webapps2/ROOT
directory:rm -rf /var/lib/tomcat/webapps2/ROOT
-
Run the following command to update Kaui:
$KPM_INSTALL_KAUI_CMD
-
Restart KB:
sudo service killbill restart
SSL Certificate
The KB single tier AMI comes pre-configured with a self-signed certificate. This can be found at /etc/nginx/ssl/ssl-cert-snakeoil.pem
(public key) and /etc/nginx/ssl/ssl-cert-snakeoil.key
(private key). Some backend systems might not trust this certificate, leading to SSL related failures. There are two possible solutions:
-
You can update the backend to trust KB’s self-signed certificate. This involves configuring the backend application to trust the certificate. For instance, if your backend is using Java, you can add KB’s certificate to the truststore.
-
If you prefer not to modify the backend, you can obtain a valid certificate from a Certificate Authority. This certificate would be automatically trusted by most clients without requiring any additional configuration in your backend application. Let’s Encrypt is a root CA that is free to use. See Using Let’s Encrypt.