This guide discusses how to troubleshoot, maintain and upgrade your Kill Bill CloudFormation (CFN) installation.
Logging in to your EC2s
Many of the steps you may want to take to troubleshoot and maintain your system require login and command line access to one or more of your EC2 instances. To enable this you will need to temporarily modify the corresponding security group.
First, go to the Instances screen on your EC2 dashboard and select an instance to log in to. For most tasks you can use any one. Scroll to the right to see the security group associated with this instance. Click on the name of the group. Then select the tab labeled Inbound Rules. You should see:
Select Add Rule and add a rule with Type SSH, Source 0.0.0.0/0. The Port range will set automatically to 22. You should now see:
To login, use the secure shell command:
ssh -i PRIVATE_KEY.pem ubuntu@INSTANCE_IP
Here PRIVATE_KEY is the pathname where you have stored the private key that was downloaded when you generated your key pair, and INSTANCE_IP is the IPV4 address described earlier. The private key will not work unless its access controls are set to readable by the owner only.
On Windows versions before Windows 10, you may need to download a program called PuTTY to enable ssh
. On Windows 10 and 11, ssh
is available but may need to be activated through the Settings screen.
The first time you login, you will see a warning message asking if you want to add this host to your list of hosts. You should answer yes
.
You will now be able to explore your instance and perform various configuration and maintenance tasks. Be careful, because you automatically have root (su) privileges. To exit from your login, type exit
.
It is a good idea to remove the extra security rule when you have no need to login.
Troubleshooting
In spite of your best efforts, your installation may not succeed. Some components may not be created, or testing may produce errors. This section discusses some things that could go wrong during installation, and provides some suggestions for dealing with them.
Stack Creation Problems
There are various possible problems that could arise when creating the CFN stack. In this section
we provide some tips for debugging the issues. The stack will initially have a status of CREATE_IN_PROGRESS
during the initialization. It will then either transition to CREATE_COMPLETE
or CREATE_FAILED
. Keep in mind
that creating the stack will typically take on the order of 15-20 minutes, mostly because of the time it takes to set up the
RDS cluster, so you need to be patient. If the final status is CREATE_FAILED
you need to find out why.
Check the Resources tab on the stack creation page to see which resources failed to create. If you have set the option to prevent rollback, most of the resources will usually have created successfully. If one or more EC2 instances were created, they should now be running, and you can login to one of them (any one) for further checks.
If no instances are running, it is likely there is a problem with your template. Try once more, then contact [email protected]
.
System logs
Your system keeps a variety of logs, and some of them are useful even during startup.
CloudFormation logs are found in the /var/log
directory. First we recommend you check /var/log/cfn-init.log
, which gives a summary of the startup events. Next you may look at /var/log/cfn-init-cmd.log
which breaks down the configuration command in more detail.
Some other CFN logs are listed below. However these are more specialized and may be less helpful:
/var/log/cfn-wire.log
/var/log/cloud-init-output.log
/var/log/awslogs.logs
/var/log/xray/xray.log
The Kill Bill/Kaui server logs are located under /var/lib/tomcat/logs/
. The most interesting of these are:
-
killbill.out
: All Kill Bill server logs, configured asINFO
by default -
kaui.out
: All Kaui server logs, configured asINFO
by default -
localhost_access_log…
: Access requests to the servers
The configuration of the logging (log rotation, log level, …) can be found in /var/lib/killbill/config/logback.xml
AWS Errors
Some errors could occur due to the organization of your AWS system or possible limitations on your account. Typical issues might include:
-
Insufficient IAM Permissions
-
Limit Exceeded
-
Security Group Does Not Exist in VPC
-
RDS Cluster failed to come up
-
…
Make sure thate you have setup a valid VPC with valid subnets. Fow AWS specific issues, please refer to the AWS troubleshooting documentation.
If you think there is an issue with the CloudFormation template itself, please report any issue to [email protected]
.
If you update any configuration file, you will need to restart the service. To do this, you can run the following commands:
# Restart killbill server instance
> service killbill restart
# Restart kaui server instance
> service kaui restart
Database Problems
From any Kill Bill EC2 instance, it is possible to access the RDS Aurora database. On each node, there is a mysql
client installed
allowing database access. The database hostname can be obtained from the CFN Resources screen. This information can also be
extracted from the killbill.properties
file:
> grep 'org.killbill.dao' /var/lib/killbill/config/killbill.properties
org.killbill.dao.password=killbill
org.killbill.dao.url=jdbc:mysql:aurora://mystack-test-rdscluster-1qwiqitatcb9m.cluster-cah16olm8gkg.us-east-1.rds.amazonaws.com:3306/killbill
org.killbill.dao.user=killbill
Based on this info, the following command would allow you to get a mysql
prompt:
> mysql -h mystack-test-rdscluster-1qwiqitatcb9m.cluster-cah16olm8gkg.us-east-1.rds.amazonaws.com -u killbill -pkillbill killbill
> show tables
> ...
You should ensure that the database is up and running, and the schemas are properly installed.
Maintenance
If your installation has been running well but shows signs of slowing down or other problems, there are several commands you can use to assess its overall health. Thee commands can be used when you are logged in to any instance. Some should be run on all instances from time to time.
Service Health
Since both the Kill Bill and Kaui servers listen on port 8080, you can check if these services are running by issuing the following command:
telnet 127.0.0.1 8080
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
This check may be needed on each of the KB and Kaui instances.
For the Kill Bill servers specifically, some useful commands are:
# Check the status of various memory pools and queues
> curl http://127.0.0.1:8080/1.0/healthcheck | jq
Sample output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1164 0 1164 0 0 25866 0 --:--:-- --:--:-- --:--:-- 25866
{
"main.pool.ConnectivityCheck": {
"healthy": true
},
"org.killbill.billing.server.healthchecks.KillbillHealthcheck": {
"healthy": true,
"message": "OK"
},
"org.killbill.billing.server.healthchecks.KillbillPluginsHealthcheck": {
"healthy": true
},
"org.killbill.billing.server.healthchecks.KillbillQueuesHealthcheck": {
"healthy": true,
"bus": {
"growing": false
},
"overdue-service:overdue-check-queue": {
"growing": false
},
"entitlement-service:entitlement-events": {
"growing": false
},
"invoice-service:next-billing-date-queue": {
"growing": false
},
"notifications-retries:extBusEvent-listener": {
"growing": false
},
"payment-service:janitor": {
"growing": false
},
"externalBus": {
"growing": false
},
"payment-service:retry": {
"growing": false
},
"notifications-retries:next-billing-date-queue": {
"growing": false
},
"subscription-service:subscription-events": {
"growing": false
},
"invoice-service:parent-invoice-commitment-queue": {
"growing": false
},
"server-service:push-notification-queue": {
"growing": false
},
"overdue-service:overdue-async-bus-queue": {
"growing": false
},
"notifications-retries:invoice-listener": {
"growing": false
}
},
"osgi.pool.ConnectivityCheck": {
"healthy": true
},
"shiro.pool.ConnectivityCheck": {
"healthy": true
}
}
# Check which Kill Bill & plugin versions are present
> curl -u admin:KBADMINPASSWORD http://127.0.0.1:8080/1.0/kb/nodesInfo | jq
Sample output:
[
{
"nodeName": "ip-192-168-65-236.ec2.internal",
"bootTime": "2020-02-02T21:26:44.000Z",
"lastUpdatedDate": "2020-02-02T21:26:44.000Z",
"kbVersion": "0.22.1",
"apiVersion": "0.53.17",
"pluginApiVersion": "0.26.3",
"commonVersion": "0.23.7",
"platformVersion": "0.39.12",
"pluginsInfo": [
{
"bundleSymbolicName": "org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
"pluginKey": null,
"pluginName": "org.kill-bill.billing.killbill-platform-osgi-bundles-kpm",
"version": null,
"state": "RUNNING",
"isSelectedForStart": true,
"services": []
},
{
"bundleSymbolicName": "org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
"pluginKey": null,
"pluginName": "org.kill-bill.billing.killbill-platform-osgi-bundles-logger",
"version": null,
"state": "RUNNING",
"isSelectedForStart": true,
"services": []
}
]
}
]
Diagnostic Command
The diagnostic
option of the kpm
command creates an extensive report for a given tenant that may be useful for troubleshooting. To run this command:
# Login as 'tomcat'
> sudo su - tomcat
#
# Details about DB host can be extracted from '/var/lib/killbill/config/killbill.properties'
#
# Run the command with your access credentials:
#
> kpm diagnostic \
--killbill-credentials=ADMIN PASSWORD \
--bundles-dir=/var/lib/killbill/bundles \
--database-name=killbill \
--database-credentials=DBUSER DBPASS \
--killbill-api-credentials=KEY SECRET \
--killbill-web-path=/var/lib/tomcat/webapps \
--database-host=DBHOST
You will need to edit this command to include:
-
Your KAUI username and password (ADMIN PASSWORD)
-
Your database credentials (DBUSER DBPASS)
-
The key and secret key for your tenant (KEY SECRET)
-
Your database host (see
/var/lib/killbill/config/killbill.properties
)
The last line of the response should look like:
Diagnostic data exported under /tmp/killbill-diagnostics-20200212-26849-c0rrz3/killbill-diagnostics-02-12-20.zip
Note that there is also a --account-export=<account_id>
flag to export the data associated with a specific Kill Bill account_id
.
Upgrading
Using Newer AMIs
The Kill Bill core team will provide new AMIs whenever necessary. Here we discuss how to upgrade to these new AMIs without a complete reinstallation.
Because the CloudFormation from AWS Marketplace will always reflect the latest AMI ids, you can simply update the stack with the latest CloudFormation template and the instances in the AutoScaling groups will be updated automatically. We strongly recommend that you always test the upgrade in a test environment first.
We recommend that you rely on the CloudFormation ChangeSet
functionality to get a sense of what would be updated if the change was submitted. For more information about the CloudFormation ChangeSet
functionality see this documentation. Below is a summary of the steps:
1. Download the new CloudFormation template
Each AMI is defined by a CloudFormation template. To access the template for the latest AMI, go to the Marketplace page as described under Configure and Launch above. Check that the page lists the desired version, then scroll down to the Usage Information section. Expand the link View Cloudformation Template. Below the diagram that appears, click Download Cloudformation Template. Save the template file. This will be a long text (JSON) file with a name ending in template
.
2. Create a new ChangeSet
Go to the CloudFormation dashboard and select you current stack. Then select Stack Details from the left menu. You should see the following page:
Select Create Change Set. On the page that appears, Select Replace Current Template, then select Upload a Template File. Finally, upload the file you downloaded in Step 1.
You will now revisit several pages that you saw when the stack was created. First, you will see the page Specify Stack Details. At this time there should be no changes required. Click Next.
The next page will be the Configure Stack Options. Again, no changes required.
THe last page is the Review page. If everything looks good, scroll to the bottom. You will see the following message, that you will need to acknowledge:
Finally click Create Change Set. You can provide an optional description in the popup that appears, then select Create Change Set again. Your change set will be created. You will initially see the status CREATE_PENDING
. Wait until the status message changes to CREATE_COMPLETE
.
3. Apply the ChangeSet
It is important to remember that at this point your Kill Bill installation has not changed. Your change set is ready and waiting when you do want to use it. When that time comes, return to the cloudformation dashboard, select your stack and select the change sets tab. Select your change set, then click Execute.