Overview

This guide explains how to maintain your single-tier Kill Bill installation on AWS.

Login

To perform most maintenance tasks you will need to login to your EC2 instance. Be sure that SSH is enabled on Port 22 in your security group. Login procedures are described in your setup guide (See Step 7: Login to an Instance)

Troubleshooting

If a new installation does not seem to work correctly, the first step is to review your installation process carefully, to be sure that everything has been done and checked as described in your setup guide. If problems persist, we will be glad to work with you to identify the problem. To help us to do this, there are several information reports that you may need to gather. These reports are somewhat technical but can be analyzed by Kill Bill personnel. This section explains how to obtain the reports that may be needed.

Status Reports

When you launch the EC2 instance, all required services should be enabled and running, including:

  • An instance of nginx receiving traffic on port 443 and 8443

  • An instance of Kill Bill server listening on 127.0.0.1:8080 (and receiving external traffic through nginx on port 8443)

  • An instance of Kaui listening on 127.0.0.1:3000 (and receiving external traffic through nginx on port 443)

  • A local mysql server running on port 3306

The status of each service can be checked by the following commands:

  • Kill Bill service: sudo service killbill status

  • KAUI service: sudo service kaui status

  • Nginx service: sudo service nginx status

  • Mysql service: sudo service mysql status

For each report there should be a line near the top with the following form:

Active: active (running) since Sat 2020-10-24 20:13:43 UTC; 1 day 1h ago

You can start or stop the services using similar commands, such as sudo service kaui stop to stop Kaui. If any of these services are not active, try starting it using the appropriate command.

If all services are running, the next step is to collect several reports.

Log Files

The system maintains a series of log files that may be helpful when troubleshooting is needed. These logs may be forwarded to Kill Bill to help diagnose your problem.

Tomcat logs are under /var/lib/tomcat/logs/:

  • Kaui log: /var/lib/tomcat/logs/kaui.out

  • Kill Bill server log: /var/lib/tomcat/logs/killbill.out

  • Complete tomcat log: /var/lib/tomcat/logs/catalina.out

Some of these logs may be very large, expecially catalina.out.

System Health Check

The healthcheck report checks the health of various software components, and determines if any queues are growing improperly over time. To create this report, issue the following command:

curl http://127.0.0.1:8080/1.0/healthcheck

This will return a series of messages giving the health status of each component. These messages may look like:

"com.codahale.metrics.health.jvm.ThreadDeadlockHealthCheck":{"healthy":true},
"main.pool.Connection99Percent":{"healthy":true},
"main.pool.ConnectivityCheck":{"healthy":true},
...
"bus":{"growing":false},
"overdue-service:overdue-check-queue":{"growing":false},
...

An unexpected message may indicate a component that is not working properly.

System Information

For a detailed system information report, showing the current version of each component, use the following command:

curl -u <ADMIN>:<INSTANCE_ID> http://127.0.0.1:8080/1.0/kb/nodesInfo

Here <ADMIN> is your Kaui administrator username, and <INSTANCE_ID> is your EC2 instance ID. THe output will look like:

"nodeName":"ip 172-31-20-130.ec2.internal",
"bootTime":"2023-01-28T20:30:19.000Z",
"lastUpdatedDate":"2023-01-28T20:30:19.000Z",
"kbVersion":"0.22.26",
"apiVersion":"0.53.17",
"pluginApiVersion":"0.26.4",
"commonVersion":"0.24.15",
"platformVersion":"0.40.8",
"pluginsInfo":
    [{"bundleSymbolicName":"org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
      "pluginKey":null,
      "pluginName":"org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
      "version":null,
      "state":"RUNNING",
      "isSelectedForStart":true,
      "services":[]},
    {"bundleSymbolicName":"org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
      "pluginKey":null,
      "pluginName":"org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
      "version":null,
      "state":"RUNNING",
      "isSelectedForStart":true,
      "services":[]}]

Diagnostic Command

The diagnostic option of the Kill Bill Package Manager (kpm) command creates an extensive report for a given tenant that may be useful for troubleshooting. To run this command:

# Login as 'tomcat'
> sudo su - tomcat
#
# Run the command with your access credentials:
#
> kpm  diagnostic \
  --killbill-credentials=<ADMIN> <INSTANCE_ID> \
  --bundles-dir=/var/lib/killbill/bundles \
  --database-name=killbill \
  --database-credentials=<DBUSER> <DBPASS> \
  --killbill-api-credentials=<KEY> <SECRET> \
  --kaui-web-path=/var/lib/tomcat/webapps2 \
  --killbill-url=http://127.0.0.1:8080 \
  --database-host=127.0.0.1:3306

You will need to edit this command to include:

  1. Your Kaui admin username and your EC2 instance ID (<ADMIN> <INSTANCE_ID>)

  2. Your database credentials (<DBUSER> <DBPASS>)

  3. The public key and secret key for your tenant (<KEY> <SECRET>)

To restrict the report to a single account, you can add the line

--account-export=<ACCOUNT_ID>

replacing <ACCOUNT_ID> with the ID of the specific account to be included.

The last line of the response should look like:

Diagnostic data is exported under /tmp/killbill-diagnostics-20200213-23204-u93ah5/killbill-diagnostics-02-13-20.zip

The specified zip file contains several reports of various sizes. This report can be downloaded to your computer using sftp and forwarded to Kill Bill for analysis.

Databases

To access the mysql (MariaDB) databases, you can use the following command:

mysql -u root -proot

This enables interactive access to the database manager. There is one killbill and one kaui database created and used by the respective applications. To verify the tables in each database, you can type:

use killbill
show tables;

or

use kaui
show tables;

Standard SQL commands can be used to explore or manipulate the tables. Be sure you know what you are doing, or the databases may become corrupted!

To exit the mysql interactive mode, type exit.

Load Balancer

The load balancer nginx should normally require little attention. The configuration files are located under /etc/nginx/. The configuration file for nginx itself is /etc/nginx/nginx.conf. Additional configuration files are located under /etc/nginx/sites-enabled/. The only file normally present in this directory is /etc/nginx/sites-enabled/killbill.conf.

Nginx logs can be found under /var/log/nginx/

  • Access logs: /var/log/nginx/access.log

  • Error logs: /var/log/nginx/error.log

Configuration

Kill Bill defines a number of global properties and per-tenant properties that can be varied. These properties are explained in the Kill Bill and Kaui Configuration Guide. Default values for these properties are built into the Kill Bill code; these values can be overridden by values defined in the file /var/lib/killbill/config/killbill.properties. For example, this is where you can change the database URL and credentials.

This file also defines the location of the shiro.ini file, which by default is in the same directory. This file defines the Kill Bill admin credentials, along with any other users and their roles. See the Users, Roles, and Permissions Management guide for details about this file.

Upgrades

From time to time new versions of Kill Bill are released. These versions can be incorporated in your installation with minimal impact on production. This section explains how to upgrade to a new version of Kill Bill. Note that these are not the same as new versions of the AMI, which can be incorporated only by a full reinstall.

First, login to your instance using ssh,

Next, switch to the tomcat user:

sudo su - tomcat

The configuration file /var/lib/killbill/kpm.yml specifies the Kill Bill version (and its plugins) to be run on the instance. Once you edit this file to specify the new version number, it will be used automatically. Perform the following steps:

  1. Edit the configuration file to update the version number

  2. Run the command $KPM_INSTALL_KB_CMD

  3. Delete the cached directory /var/lib/tomcat/webapps/ROOT

  4. Restart the instance.

A similar process can be used for Kaui: update /var/lib/kaui/kpm.yml, run $KPM_INSTALL_KAUI_CMD, delete the cached directory /var/lib/tomcat/webapps2/ROOT and restart the instance.