Overview
Best practices
Kill Bill (whether used as a Billing System and/or Payment System) is fundamentally a backend system, so the following considerations should apply:
-
Don’t make the system visible to the outside world, instead you should have a front end system, or a reverse proxy in front of it; if you need to handle gateway notifications (e.g. IPN/Internal Payment Notification), we developed an app for that specific purpose
-
Always deploy at least 2 instances (for reliability purpose) in front of a load balancer
-
Setup your database with a master and a slave instances, and configure it to take regular snapshots
-
Aggregate your logs (for example using LogStash, GrayLog2, …) and take a look at our Logback recipes
-
Look at the all the existing Kill Bill JMX metrics (using VisualVM, jconsole, …) and set some alerts (for instance by using the following script)
-
Configure your system properly, especially when it comes to the settings of the bus and notification queue.
Deployment options
When deploying Kill Bill, the following pieces will need to be deployed in addition to your OS, VM or container (Docker, …):
-
A web container (Jetty, Tomcat, …)
-
The killbill war
-
The plugins
-
The configuration files and system properties
-
The database along with the various schemas: the Kill Bill schema is available from our download page and each stateful plugin will/should expose such a schema (for example, here is the schema for the Adyen plugin)
In order to ease the deployment, we created KPM, the Kill Bill Package Manager. KPM can fetch existing (signed) artifacts for the main killbill war and for each of the plugins, and deploy them at the right place. KPM can also deploy Tomcat.
In addition to KPM, we also provide Docker images. Those images internally rely on KPM for fetching the various artifacts and install them at the right location on the image. The docker run
command allows to override existing Kill Bill system properties.
Note that one can decide to only use KPM for fetching and installing artifacts, and bypass docker to deploy on VMs or bare metal. Or alternatively one can use other tools, manual steps to fetch those artifacts from Maven Central, or by using our convenient download page.
Pre- and Post-deployment checklist
-
Test, test, test: we work hard to make the core of the platform solid. But your deployment, with your own combination of plugins and configuration (catalog, overdue, etc.) is unique. Write business-level regression tests that you can run before each upgrade.
-
Timezone of your servers and databases has to be UTC and ensure NTP is properly configured to avoid any drift between instances.
-
Make sure the servers have enough entropy:
/proc/sys/kernel/random/entropy_avail
should be > 3k (otherwise installhaveged
/rng-tools
). Kill Bill should also be started with-Djava.security.egd=file:/dev/.urandom
. -
Make sure your database and queues configuration are adequate: the
bus_events
table should almost always be empty and thenotifications
table should never have anyAVAILABLE
entry with an effective date in the past. Otherwise, in both cases, the system will be late (invoices not generated, etc.). These two metrics should always be monitored in production (potentially a paging event). -
Verify the integration with your payment gateway(s): very few payment transactions (if any) should be in an
UNKNOWN
state. Make sure to fix these manually via the Payment Admin API, if the plugin is unable to do it automatically. -
Have a monitoring system in place (we recommend Elasticseach, Logstash, Kibana, InfluxDB and Grafana, which can be easily setup for Kill Bill) and watch your logs constantly: any
WARN
orERROR
entry should be reviewed, as well as stacktraces. -
Join our mailing-list to get notified of new releases or ask questions. For inquiries regarding your specific setup, always attach your logs (
killbill.out
) and catalog XML configuration to your post.
Docker
Deploying Kill Bill and Kaui through Docker is the recommended way for most users, as images abstract the complexity of setting up Java, a Webapp container, plugins, etc.
Manual deployment
The following sections below will give manual instructions on how to deploy Kill Bill on various Java containers. This is for advanced users only.
Installation in Jetty
Download the Jetty container
If you don’t have it yet, you can download the Jetty container here. If you’re unsure which version to use, we recommend 8.1.7.v20120910. Make sure you can unpack and start Jetty as described here.
Jetty ships with examples you can safely remove:
-
contexts/javadoc.xml
-
contexts/test.d/override-web.xml
-
contexts/test.xml
-
webapps/spdy.war
-
webapps/test.war
Deploy Kill Bill in Jetty
Place the downloaded Kill Bill war under webapps/root.war in the Jetty folder.
Update the start.ini file with Kill Bill specific configuration. Here is a working example (make sure to adapt the jdbc properties):
# Kill Bill properties
-Dorg.killbill.dao.url=jdbc:mysql://127.0.0.1:3306/killbill
-Dorg.killbill.dao.user=killbill
-Dorg.killbill.dao.password=killbill
-Dorg.killbill.billing.osgi.dao.url=jdbc:mysql://127.0.0.1:3306/killbill
-Dorg.killbill.billing.osgi.dao.user=killbill
-Dorg.killbill.billing.osgi.dao.password=killbill
# Start classpath OPTIONS
OPTIONS=Server,resources,ext,plus,annotations
# Configuration files
etc/jetty.xml
etc/jetty-annotations.xml
etc/jetty-deploy.xml
etc/jetty-webapps.xml
You should now be all set! Kill Bill can be started via the command
java -jar start.jar
Enable SSL
The following command generates a key pair and certificate directly into a keystore:
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
Save the keystore file under the etc/ directory of the Jetty distribution.
In your start.ini file, add the etc/jetty-ssl.xml configuration file and edit this file to update the KeyStorePassword, KeyManagerPassword and TrustStorePassword directives with the passwords you specified when you ran the keytool command.
Kill Bill should now be available over SSL. For more details, see the Jetty guide here.
Installation in Tomcat
Download the Tomcat container
Deploy Kill Bill in Tomcat
We will assume you have installed Tomcat in a directory called CATALINA_HOME.
The Kill Bill system properties need to be added at the end of CATALINA_HOME/conf/catalina.properties. The minimal set of system properties that need to added are the ones for the jdbc parameters:
# Kill Bill properties
org.killbill.dao.url=jdbc:mysql://127.0.0.1:3306/killbill
org.killbill.dao.user=killbill
org.killbill.dao.password=killbill
org.killbill.billing.osgi.dao.url=jdbc:mysql://127.0.0.1:3306/killbill
org.killbill.billing.osgi.dao.user=killbill
org.killbill.billing.osgi.dao.password=killbill
A new entry must be added to let tomcat know about the location of the Kill Bill war. That file is called Root.xml and it must located under CATALINA_HOME/conf/Catalina/localhost (create the directories if they don’t exist). Its content must be:
<Context docBase="path_to_killbill-server.war">
</Context>
Once this is setup, you can start the tomcat container using the script CATALINA_HOME/bin/startup.sh. The server will be started in the background but logs can be followed at CATALINA_HOME/logs/catalina.out.
Installation in Glassfish
Download the GlassFish container
If you don’t have it yet, you can download the GlassFish container here.
Deploy Kill Bill in GlassFish
Configure GlassFish with your Kill Bill specific configuration in domains/domain1/config/domain.xml (make sure to adapt the jdbc properties):
<jvm-options>-Dorg.killbill.dao.url=jdbc:mysql://127.0.0.1:3306/killbill</jvm-options>
<jvm-options>-Dorg.killbill.dao.user=killbill</jvm-options>
<jvm-options>-Dorg.killbill.dao.password=killbill</jvm-options>
<jvm-options>-Dorg.killbill.billing.osgi.dao.url=jdbc:mysql://127.0.0.1:3306/killbill</jvm-options>
<jvm-options>-Dorg.killbill.billing.osgi.dao.user=killbill</jvm-options>
<jvm-options>-Dorg.killbill.billing.osgi.dao.password</jvm-options>
In the glassfish directory of the unzipped GlasshFish distribution, run the following command to deploy Kill Bill:
./bin/asadmin start-domain ; ./bin/asadmin deploy --contextroot "/" /path/to/killbill.war