Overview

Best practices

Kill Bill (whether used as a Billing System and/or Payment System) is fundamentally a backend system, so the following considerations should apply:

  • Don’t make the system visible to the outside world, instead you should have a front end system, or a reverse proxy in front of it; if you need to handle gateway notifications (e.g. IPN/Internal Payment Notification), we developed an app for that specific purpose

  • Always deploy at least 2 instances (for reliability purpose) in front of a load balancer

  • Setup your database with a master and a slave instances, and configure it to take regular snapshots

  • Aggregate your logs (for example using LogStash, GrayLog2, …​) and take a look at our Logback recipes

  • Look at the all the existing Kill Bill JMX metrics (using VisualVM, jconsole, …​) and set some alerts (for instance by using the following script)

  • Configure your system properly, especially when it comes to the settings of the bus and notification queue.

Deployment options

When deploying Kill Bill, the following pieces will need to be deployed in addition to your OS, VM or container (Docker, …​):

  • A web container (Jetty, Tomcat, …​)

  • The killbill war

  • The plugins

  • The configuration files and system properties

  • The database along with the various schemas: the Kill Bill schema is available from our download page and each stateful plugin will/should expose such a schema (for example, here is the schema for the Adyen plugin)

In order to ease the deployment, we created KPM, the Kill Bill Package Manager. KPM can fetch existing (signed) artifacts for the main killbill war and for each of the plugins, and deploy them at the right place. KPM can also deploy Tomcat.

In addition to KPM, we also provide Docker images. Those images internally rely on KPM for fetching the various artifacts and install them at the right location on the image. The docker run command allows to override existing Kill Bill system properties.

Note that one can decide to only use KPM for fetching and installing artifacts, and bypass docker to deploy on VMs or bare metal. Or alternatively one can use other tools, manual steps to fetch those artifacts from Maven Central, or by using our convenient download page.

Pre- and Post-deployment checklist

  • Test, test, test: we work hard to make the core of the platform solid. But your deployment, with your own combination of plugins and configuration (catalog, overdue, etc.) is unique. Write business-level regression tests that you can run before each upgrade.

  • Timezone of your servers and databases has to be UTC and ensure NTP is properly configured to avoid any drift between instances.

  • Make sure the servers have enough entropy: /proc/sys/kernel/random/entropy_avail should be > 3k (otherwise install haveged / rng-tools). Kill Bill should also be started with -Djava.security.egd=file:/dev/./urandom.

  • Make sure your database and queues configuration are adequate: the bus_events table should almost always be empty and the notifications table should never have any AVAILABLE entry with an effective date in the past. Otherwise, in both cases, the system will be late (invoices not generated, etc.). These two metrics should always be monitored in production (potentially a paging event).

  • Verify the integration with your payment gateway(s): very few payment transactions (if any) should be in an UNKNOWN state. Make sure to fix these manually via the Payment Admin API, if the plugin is unable to do it automatically.

  • Have a monitoring system in place (we recommend Elasticseach, Logstash, Kibana, InfluxDB and Grafana, which can be easily setup for Kill Bill) and watch your logs constantly: any WARN or ERROR entry should be reviewed, as well as stacktraces.

  • Join our mailing-list to get notified of new releases or ask questions. For inquiries regarding your specific setup, always attach your logs (killbill.out) and catalog XML configuration to your post.

Docker

Deploying Kill Bill and Kaui through Docker is the recommended way for most users, as images abstract the complexity of setting up Java, a Webapp container, plugins, etc.

Make sure to get familiar with Docker first, before attempting to install Kill Bill. The project has lots of great docs to help you get setup.

Documentation for our images is available here. Our blog has also lots of tips on how to deploy to popular cloud providers.

Manual deployment

The following sections below will give manual instructions on how to deploy Kill Bill on various Java containers. This is for advanced users only.

Installation in Jetty

Download the Jetty container

If you don’t have it yet, you can download the Jetty container here. If you’re unsure which version to use, we recommend 8.1.7.v20120910. Make sure you can unpack and start Jetty as described here.

Jetty ships with examples you can safely remove:

  • contexts/javadoc.xml

  • contexts/test.d/override-web.xml

  • contexts/test.xml

  • webapps/spdy.war

  • webapps/test.war

Deploy Kill Bill in Jetty

Place the downloaded Kill Bill war under webapps/root.war in the Jetty folder.

Update the start.ini file with Kill Bill specific configuration. Here is a working example (make sure to adapt the jdbc properties):

# Kill Bill properties
-Dorg.killbill.dao.url=jdbc:mysql://127.0.0.1:3306/killbill
-Dorg.killbill.dao.user=killbill
-Dorg.killbill.dao.password=killbill
-Dorg.killbill.billing.osgi.dao.url=jdbc:mysql://127.0.0.1:3306/killbill
-Dorg.killbill.billing.osgi.dao.user=killbill
-Dorg.killbill.billing.osgi.dao.password=killbill

# Start classpath OPTIONS
OPTIONS=Server,resources,ext,plus,annotations

# Configuration files
etc/jetty.xml
etc/jetty-annotations.xml
etc/jetty-deploy.xml
etc/jetty-webapps.xml

You should now be all set! Kill Bill can be started via the command

java -jar start.jar

Enable SSL

The following command generates a key pair and certificate directly into a keystore:

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

Save the keystore file under the etc/ directory of the Jetty distribution.

In your start.ini file, add the etc/jetty-ssl.xml configuration file and edit this file to update the KeyStorePassword, KeyManagerPassword and TrustStorePassword directives with the passwords you specified when you ran the keytool command.

Kill Bill should now be available over SSL. For more details, see the Jetty guide here.

Installation in Tomcat

Download the Tomcat container

If you don’t have it yet, you can download the Tomcat container here. The documentation is available here.

Deploy Kill Bill in Tomcat

We will assume you have installed Tomcat in a directory called CATALINA_HOME.

The Kill Bill system properties need to be added at the end of CATALINA_HOME/conf/catalina.properties. The minimal set of system properties that need to added are the ones for the jdbc parameters:

# Kill Bill properties
org.killbill.dao.url=jdbc:mysql://127.0.0.1:3306/killbill
org.killbill.dao.user=killbill
org.killbill.dao.password=killbill
org.killbill.billing.osgi.dao.url=jdbc:mysql://127.0.0.1:3306/killbill
org.killbill.billing.osgi.dao.user=killbill
org.killbill.billing.osgi.dao.password=killbill

A new entry must be added to let tomcat know about the location of the Kill Bill war. That file is called Root.xml and it must located under CATALINA_HOME/conf/Catalina/localhost (create the directories if they don’t exist). Its content must be:

<Context docBase="path_to_killbill-server.war">
</Context>

Once this is setup, you can start the tomcat container using the script CATALINA_HOME/bin/startup.sh. The server will be started in the background but logs can be followed at CATALINA_HOME/logs/catalina.out.

Installation in Glassfish

Download the GlassFish container

If you don’t have it yet, you can download the GlassFish container here.

Deploy Kill Bill in GlassFish

Configure GlassFish with your Kill Bill specific configuration in domains/domain1/config/domain.xml (make sure to adapt the jdbc properties):

<jvm-options>-Dorg.killbill.dao.url=jdbc:mysql://127.0.0.1:3306/killbill</jvm-options>
<jvm-options>-Dorg.killbill.dao.user=killbill</jvm-options>
<jvm-options>-Dorg.killbill.dao.password=killbill</jvm-options>
<jvm-options>-Dorg.killbill.billing.osgi.dao.url=jdbc:mysql://127.0.0.1:3306/killbill</jvm-options>
<jvm-options>-Dorg.killbill.billing.osgi.dao.user=killbill</jvm-options>
<jvm-options>-Dorg.killbill.billing.osgi.dao.password</jvm-options>

In the glassfish directory of the unzipped GlasshFish distribution, run the following command to deploy Kill Bill:

./bin/asadmin start-domain ; ./bin/asadmin deploy --contextroot "/" /path/to/killbill.war