How to Mantain a CloudFormation System

There are various possible problems that could arise when creating the CFN stack. In this section we provide some tips for debugging the issues. The stack will initially have a status of CREATE_IN_PROGRESS during the initialization. It will then either transition to CREATE_COMPLETE or CREATE_FAILED. Keep in mind that creating the stack will typically take on the order of 15-20 minutes, mostly because of the time it takes to set up the RDS cluster, so you need to be patient. If the final status is CREATE_FAILED you need to find out why.

Check the Resources tab on the stack creation page to see which resources failed to create. If you have set the option to prevent rollback, most of the resources will usually have created successfully. If one or more EC2 instances were created, they should now be running, and you can log in to one of them (any one) for further checks.

If no instances are running, it is likely there is a problem with your template. Try once more, then contact [email protected].

Your system keeps a variety of logs, and some of them are useful even during startup.

CloudFormation logs are found in the /var/log directory. First we recommend you check /var/log/cfn-init.log, which gives a summary of the startup events. Next you may look at /var/log/cfn-init-cmd.log which breaks down the configuration command in more detail.

Some other CFN logs are listed below. However, these are more specialized and may be less helpful:

Tomcat/KB/Kaui logs are under /var/lib/tomcat/logs/:

Sometimes these logs will clearly show anomalies that will pinpoint the problem. If not you should forward your logs to Kill Bill for review.Some of these logs may be very large, especially catalina.out.

The logging configuration, including settings for log rotation and log levels, is defined in the logback.xml file. Separate logback.xml files are used for KB and Kaui, located at the following paths:

These files can be modified to customize the logging settings for KB and Kaui. By default, the log level is configured as INFO.

The diagnostic option of the Kill Bill Package Manager (kpm) command creates an extensive report for a given tenant that may be useful for troubleshooting. See KPM Diagnostic Usage.

To run this command:

You will need to edit this command to include:

The last line of the response should look like:

Diagnostic data is exported under /tmp/killbill-diagnostics-20200213-23204-u93ah5/killbill-diagnostics-02-13-20.zip

The specified zip file contains several reports of various sizes. This report can be downloaded to your computer using sftp and forwarded to Kill Bill for analysis.

From any Kill Bill EC2 instance, it is possible to access the RDS Aurora database. On each node, there is a mysql client installed allowing database access. The database hostname can be obtained from the RDSCluster in the CFN Resources screen. On clicking RDSCluster, you can copy the DB endpoint details from the Connectivity & Security tab.

Based on this info, the following command would allow you to get a mysql prompt:

This enables interactive access to the database manager. There is one killbill and one kaui database created and used by the respective applications. To verify the tables in each database, you can type:

or

Standard SQL commands can be used to explore or manipulate the tables. Be sure you know what you are doing, or the databases may become corrupted!

To exit the mysql interactive mode, type exit.

Some errors could occur due to the organization of your AWS system or possible limitations on your account. Typical issues might include:

Make sure that you have set up a valid VPC with valid subnets. Fow AWS specific issues, please refer to the AWS troubleshooting documentation.

If you think there is an issue with the CloudFormation template itself, please report any issue to [email protected].

If you update any configuration file, you will need to restart the service. To do this, you can run the following command:

Since both the Kill Bill and Kaui servers listen on port 8080, you can check if these services are running by issuing the following command:

This check may be needed on each of the KB and Kaui instances.

The healthcheck report checks the health of various software components, and determines if any queues are growing improperly over time. To create this report, issue the following command:

curl http://127.0.0.1:8080/1.0/healthcheck

This will return a series of messages giving the health status of each component. These messages may look like:

An unexpected message may indicate a component that is not working properly.

For a detailed system information report, showing the current version of each component, use the following command:

curl -u <ADMIN>:<INSTANCE_ID> http://127.0.0.1:8080/1.0/kb/nodesInfo

Here <ADMIN> is your Kaui administrator username, and <INSTANCE_ID> is your EC2 instance ID. THe output will look like:

The Kill Bill core team will provide new AMIs whenever necessary. Here we discuss how to upgrade to these new AMIs without a complete reinstallation.

Because the CloudFormation from AWS Marketplace will always reflect the latest AMI ids, you can simply update the stack with the latest CloudFormation template and the instances in the AutoScaling groups will be updated automatically. We strongly recommend that you always test the upgrade in a test environment first.

We recommend that you rely on the CloudFormation ChangeSet functionality to get a sense of what would be updated if the change was submitted. For more information about the CloudFormation ChangeSet functionality see this documentation. Below is a summary of the steps: