Kill Bill (whether used as a Billing System and/or Payment System) is fundamentally a backend system, so the following considerations should apply:
Don’t make the system visible to the outside world, instead you should have a front end system, or a reverse proxy in front of it; if you need to handle gateway notifications (e.g. IPN/Internal Payment Notification), we developed an app for that specific purpose
Always deploy at least 2 instances (for reliability purpose) in front of a load balancer
Setup your database with a master and a slave instances, and configure it to take regular snapshots
Look at the all the existing Kill Bill JMX metrics (using VisualVM, jconsole, …) and set some alerts (for instance by using the following script)
Configure your system properly, especially when it comes to the settings of the bus and notification queue.
When deploying Kill Bill, the following pieces will need to be deployed in addition to your OS, VM or container (Docker, …):
A web container (Jetty, Tomcat, …)
The killbill war
The configuration files and system properties
The database along with the various schemas: the Kill Bill schema is available from our download page and each stateful plugin will/should expose such a schema (for example, here is the schema for the Adyen plugin)
In order to ease the deployment, we created KPM, the Kill Bill Package Manager. KPM can fetch existing (signed) artifacts for the main killbill war and for each of the plugins, and deploy them at the right place. KPM can also deploy Tomcat.
In addition to KPM, we also provide Docker images. Those images internally rely on KPM for fetching the various artifacts and install them at the right location on the image. The docker
run command allows to override existing Kill Bill system properties.
Pre- and Post-deployment checklist
Test, test, test: we work hard to make the core of the platform solid. But your deployment, with your own combination of plugins and configuration (catalog, overdue, etc.) is unique. Write business-level regression tests that you can run before each upgrade.
Timezone of your servers and databases has to be UTC and ensure NTP is properly configured to avoid any drift between instances.
Make sure the servers have enough entropy:
/proc/sys/kernel/random/entropy_availshould be > 3k (otherwise install
rng-tools). Kill Bill should also be started with
Make sure your database and queues configuration are adequate: the
bus_eventstable should almost always be empty and the
notificationstable should never have any
AVAILABLEentry with an effective date in the past. Otherwise, in both cases, the system will be late (invoices not generated, etc.). These two metrics should always be monitored in production (potentially a paging event).
Verify the integration with your payment gateway(s): very few payment transactions (if any) should be in an
UNKNOWNstate. Make sure to fix these manually via the Payment Admin API, if the plugin is unable to do it automatically.
Have a monitoring system in place (we recommend Elasticseach, Logstash, Kibana, InfluxDB and Grafana, which can be easily setup for Kill Bill) and watch your logs constantly: any
ERRORentry should be reviewed, as well as stacktraces.
Monitor metrics at
/1.0/metricsand integrate the healthcheck at
/1.0/healthcheckwith your load balancer.
Join our mailing-list to get notified of new releases or ask questions. For inquiries regarding your specific setup, always attach your logs (
killbill.out) and catalog XML configuration to your post.
Deploying Kill Bill and Kaui through Docker is the recommended way for most users, as images abstract the complexity of setting up Java, a Webapp container, plugins, etc.
The following sections below will give manual instructions on how to deploy Kill Bill on various Java containers. This is for advanced users only.
Installation in Jetty
Download the Jetty container
Jetty ships with examples you can safely remove:
Deploy Kill Bill in Jetty
Place the downloaded Kill Bill war under webapps/root.war in the Jetty folder.
Update the start.ini file with Kill Bill specific configuration. Here is a working example (make sure to adapt the jdbc properties):
# Kill Bill properties -Dorg.killbill.dao.url=jdbc:mysql://127.0.0.1:3306/killbill -Dorg.killbill.dao.user=killbill -Dorg.killbill.dao.password=killbill -Dorg.killbill.billing.osgi.dao.url=jdbc:mysql://127.0.0.1:3306/killbill -Dorg.killbill.billing.osgi.dao.user=killbill -Dorg.killbill.billing.osgi.dao.password=killbill # Start classpath OPTIONS OPTIONS=Server,resources,ext,plus,annotations # Configuration files etc/jetty.xml etc/jetty-annotations.xml etc/jetty-deploy.xml etc/jetty-webapps.xml
You should now be all set! Kill Bill can be started via the command
java -jar start.jar
The following command generates a key pair and certificate directly into a keystore:
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
Save the keystore file under the etc/ directory of the Jetty distribution.
In your start.ini file, add the etc/jetty-ssl.xml configuration file and edit this file to update the KeyStorePassword, KeyManagerPassword and TrustStorePassword directives with the passwords you specified when you ran the keytool command.
Kill Bill should now be available over SSL. For more details, see the Jetty guide here.
Installation in Tomcat
Download the Tomcat container
Deploy Kill Bill in Tomcat
We will assume you have installed Tomcat in a directory called CATALINA_HOME.
The Kill Bill system properties need to be added at the end of CATALINA_HOME/conf/catalina.properties. The minimal set of system properties that need to added are the ones for the jdbc parameters:
# Kill Bill properties org.killbill.dao.url=jdbc:mysql://127.0.0.1:3306/killbill org.killbill.dao.user=killbill org.killbill.dao.password=killbill org.killbill.billing.osgi.dao.url=jdbc:mysql://127.0.0.1:3306/killbill org.killbill.billing.osgi.dao.user=killbill org.killbill.billing.osgi.dao.password=killbill
A new entry must be added to let tomcat know about the location of the Kill Bill war. That file is called Root.xml and it must located under CATALINA_HOME/conf/Catalina/localhost (create the directories if they don’t exist). Its content must be:
<Context docBase="path_to_killbill-server.war"> </Context>
Once this is setup, you can start the tomcat container using the script CATALINA_HOME/bin/startup.sh. The server will be started in the background but logs can be followed at CATALINA_HOME/logs/catalina.out.
Installation in Glassfish
Download the GlassFish container
If you don’t have it yet, you can download the GlassFish container here.
Deploy Kill Bill in GlassFish
Configure GlassFish with your Kill Bill specific configuration in domains/domain1/config/domain.xml (make sure to adapt the jdbc properties):
<jvm-options>-Dorg.killbill.dao.url=jdbc:mysql://127.0.0.1:3306/killbill</jvm-options> <jvm-options>-Dorg.killbill.dao.user=killbill</jvm-options> <jvm-options>-Dorg.killbill.dao.password=killbill</jvm-options> <jvm-options>-Dorg.killbill.billing.osgi.dao.url=jdbc:mysql://127.0.0.1:3306/killbill</jvm-options> <jvm-options>-Dorg.killbill.billing.osgi.dao.user=killbill</jvm-options> <jvm-options>-Dorg.killbill.billing.osgi.dao.password</jvm-options>
In the glassfish directory of the unzipped GlasshFish distribution, run the following command to deploy Kill Bill:
./bin/asadmin start-domain ; ./bin/asadmin deploy --contextroot "/" /path/to/killbill.war