Start building your own chatbot now!
This article comes from Philippe de Cuzey’s blog. Philippe de Cuzey is a data analyst working in the fields of Big Data, Hadoop and Spark based in Paris. He currently works for RueDuCommerce, a leading french e-commerce marketplace. Follow him on Twitter!

Today’s the day! You’ve worked hard and your chatbot is now done! It’s written in Python with the SAP Conversational AI Python starter kit and Flask, the Python HTTP server. You’ve developed the code on a laptop, using ngrok to conveniently expose a public URL to Bot Builder.

It’s now time to deploy the chatbot in production, to ensure permanent availability. You chose to host it on the cloud using a very popular service platform: Amazon Web Services (AWS).

Deploying on AWS is not a straightforward process, especially if (like me) you are a developer, not a system engineer, because it implies to be familiar with web hosting, and also with AWS’ extended range of products and services.


In the first place, why do we need to host the chatbot server, and which part of it exactly ?

Let’s have a look at the general architecture of a SAP Conversational AI chatbot. There are two main server components : first the chatbot conversation engine, which implements the logic of intents and manages conversations. It is a black box powered and hosted by SAP Conversational AI. On the right is the business server that we have developed and which responds to HTTP requests through webhooks.

For instance if our chatbot needs to check products inventory to answer a user request, the request will be processed through a REST API call to our server using the endpoint Url that was specified on the webhook.

It is the REST server (the red bot at the right of the webhook) that we have developed that we need to host on-premises or in the cloud.

If we chose to host on Heroku, there is a very good step by step article on the SAP Conversational AI blog. In our case we’ll go for Amazon Web Services (AWS) because it has a very broad range of cloud services for a reasonable budget.

Which AWS architecture to host our bot ?

Obviously we will need a server. In our use-case, we own a domain name, say “” and would like to access this server through a webhook base url like “”.

We have to use a HTTPS channel as per SAP Conversational AI requirement, and since self-signed certificates are not accepted, we need to attach a real certificate to our endpoint. AWS architecture guidelines don’t allow to attach a certificate directly on a AWS based website or application. We must use a front service like a load balancer (Elastic Load Balancing), a CDN (Amazon CloudFront), or an API server (Amazon API Gateway). In our case the logical choice is the load balancer, which provides also the benefit to adapt the number of servers to the load, if need arises.

AWS configuration service

To create this configuration, AWS recommends to use AWS Elastic Beanstalk, an easy-to-use service for deploying and scaling web applications and services. Elastic Beanstalk automatically handles the deployment, from capacity provisioning, load balancing, auto-scaling to application health monitoring.

Elastic Beanstalk is an additional layer on all the basic AWS services, which means we can also opt for a manual configuration of the services we need without using Elastic Beanstalk.

Using Elastic Beanstalk is great when you need scalability as this is automatically managed. Note that this implies using distributed services for each tier. If our bot needs to store data (and it certainly does, at least if we have a notion of user profile), we’ll need to use a persistent distributed storage layer, like S3, dynamoDB or RDS.

In our case, our need is a bit basic: our load will be limited so we will not need automatic load balancing, and working with a single server backed on persistent EBS storage will allow us to host our sqlite3 database directly on our server. A stronger option in production would definitely be to go for EBS or S3 persistent storage, but for the moment we’ll be happy with the root filesystem of our instance.

So in conclusion we will deploy our components (EC2 server, Elastic Load Balancer) by hand.

Let’s start !

Step 1: create instance

If you don’t have an AWS account yet, you should sign-up first. The good news is that you can benefit from the AWS Free Tier program which will provide to you a lot of services for free for up to 12 months.

Now go to the GetStarted page to see how to create your instance and launch it. The simplest way is to connect to the console and click on the Launch Instancebutton.

Although you can choose from pre-configured images (AMI), we will start with an empty Linux box. In our exemple, we will not need a lot of power or memory, so we’ll go for the default Linux AMI (HVM) of type t2.micro with 1GB of RAM and 1 CPU, because it’s free if you benefit from the Free Tier program.

One word about data persistence: there are two storage options, one is ephemeral and the other persistent. Basically, root volume (your entire virtual system disk) is ephemeral only if you choose a server backed by Amazon EC2 instance store. If your root volume is backed by EBS, everything you have on your root volume will be saved between reboots. Since our bot uses a small database to store user profiles and context, we will make sure that our root device type is “ebs”.

Choose Review and Launch to let the wizard complete the other configuration settings. You’ll notice that the wizard has created a Security Group for you. Security groups act as a firewall for associated instances, controlling both inbound and outbound traffic at the instance level. In our case we will use HTTP on port 5000 (Flask server), so as a gross initial setup you can specify inbound SSH anywhere, HTTP anywhere, HTTP on port 5000 anywhere and lastly HTTPS anywhere.

Step 2: connect to the instance

As part of the creation process, the wizard prompts you for a keypair. If you don’t already have a keypair, create one.  It  will be used to log in to your instance, which has no password. You specify the name of the key pair when you launch your instance, then provide the private key when you log in using SSH.

Proceed to launch the instance and go to the console to see it running. In the Description of the running instance, you will see it’s public IP and DNS name. Be aware that these are assigned dynamically to the instance each time it is launched, which means that if you reboot the instance you will get a new IP (and a new name). This sounds like a problem but is not since it will be accessed through the load balancer.

Click on the Connect button in the console, it will open a popin with the instructions to connect to the instance, and the ssh command line to paste in your terminal window.

If you can’t connect to the instance with SSH, one reason could be that your security group rules don’t allow SSH from your domain. Make sure inbound / outbound is open on SSH for your location.

One tip about the SSH connection : it’s timeout is very short, so you’ll find yourself having to reconnect after staying idle for a few minutes. You can change your SSH config to put a longer timeout : edit the ~/.ssh/config file on your local machine and add the following lines :

Host *
   ServerAliveInterval 50

Step 3 : configure the server

Now that you can connect to the instance, it’s time to configure it. This is quite easy.
Our server is developed in Python, and we will deploy it in a virtual environment.
Let’s install virtualenv  and then create a “recast” virtual environment :

mkdir virtualenv
cd virtualenv
virtualenv -p python2 recast

Activate it :

source ~/virtualenv/recast/bin/activate

Let’s create the project directory

mkdir mybot
cd mybot

We can now install all the python packages that we need.

So what do we need exactly ? Well any package that we had installed on our development environment: obviously the SAP Conversational AI API, and then any packages needed to implement the business logic of our bot.

sudo yum install git
git clone recast_ai
cd recast_ai
pip install -r requirements.txt

This will install Recast and Flask python librairies.

To install the other packages, from your development environment create a requirements file:

pip freeze > requirements.txt

Copying the project files to the server

You will note that although it’s possible to automate code deployment, this is out of scope for this tutorial, and for the sake of simplicity we will just deploy once by copying the files by hand.

From your development environment, make a tar of the complete hierarchy and copy it to the server instance with scp (using the keypair .pem file, just like you did with ssh) .

On the server instance, you can then untar the files in the ~/mybot/ directory. You now have the same file structure than on your development environment.

Using the requirements.txt, Let’s install the dependencies :

pip install -r requirements.txt

Server adjustments for production

Changing the Flask host

To expose our Flask app to the internet (so it can accept connections from remote sources instead of just local ones), we must set its host as “”

Edit recast/ and change the call:

1'', port=os.environ['PORT']) # for AWS
Testing the Flask server

Testing the bot can be quite cumbersome because there are a lot of components involved : the network routes, protocols, security groups, HTTPS, Flask server to name a few. So it’s a good idea to test the Flask server as an independent component. Besides, as we’ll see later on, the load balancer will need a health check route to assess if the server is up.

So in the Flask server code we will add the following route :

############ health check --- required by AWS Load Balancer
def healthCheck():;
   return "Hello World!\n"

Let’s launch the server and test it : open a second window terminal on the server instance and issue

$ curl
Hello World!

Everything’s fine.
We can move forward to the next step : HTTPS on the load balancer.

Step 4 : configure HTTPS

SAP Conversational AI needs a HTTPS endpoint for all webhooks. For our bot it will be “”. Although self-signed certificates are accepted by Flask, they are refused by SAP Conversational AI conversation server, so on your checklist you’ll need a domain name and a valid SSL certificate attached to this domain.

In case you don’t have a domain name yet, it’s easy to use Amazon Route 53 , which is their cloud DNS service. From there you can buy domain names and manage DNS entries, and they are easy to integrate with other AWS services.

As for the SSL certificate, if needed you can also create it from AWS Certificate Manager  (it’s free and auto-renewal is easy).

Step 5 : configure Load Balancer

As mentioned earlier you cannot attach directly the SSL certificate to the server instance, instead you need to go through a load balancer.

Declaring and configuring a load balancer is a quite straightforward process. We’ll configure the balancer to accept HTTPS inbound trafic, and route it to the Flask bot server on HTTP port 5000.

From the AWS console, navigate to the Load Balancing section and choose “create a load balancer”. From the 3 types of available load balancers, the one we need is the application load balancer. Hit the “create” for application load balancer. Give a name, and choose the ‘internet-facing’ scheme. Since the load balancer will route HTTPS to HTTP, choose a HTTPS listener.

You need to give 2 availability zones, make sure one of these include the zone where your server operates !

We now have to configure security settings, aka HTTPS. Attach the SSL certificate that you created or imported in he previous section (if you bought it from AWS it’s even easier).

The security group for your load balancer must allow it to communicate with registered targets on both the listener port and the health check port. Previously we created a security group with ports open for HTTP, HTTPS and SSH, we can use it just fine.

Next step is to declare routing. Just follow the steps to declare first a target group on HTTP on port 5000 (Flask inbound port), and than as target add your bot server instance.

In the Health Check section, provide the health check route that we have prepared, which is /health_check.

In summary we have declared a balancer which will receive HTTPS requests and route them to the bot server instance on HTTP port 5000.

Testing the load balancer

It’s now time to test the routing configuration. First in the EC2 management console, under “load balancing” check that the balancer is up and running and, in Target Groups, check that the status of the bot server instance is “healthy”. It means that the load balancer could reach the server on the health check route and protocol we specified, and the server responded fine.

Step 6 : attach sub-domain to Load Balancer

Last thing we need is to make sure the DNS will route any requests on mybot/ to the load balancer. Easy. just create an alias (A record) in the DNS pointing mybot/ to the Elastic Load Balancer public DNS name (you will find it in the EC2 Management Console).

Setup is complete. We can now perform a full test from the outside world !

Step 7 : end to end testing

First we will perform the health test, this time from our desktop (“the outside world”).

$ curl -X GET
Hello World!

It works !!

We can now confidently put the base URL “” in bot settings / bot builder webhook base URL in the SAP Conversational AI settings of our bot and start to use the bot in production !

Step 8 : post-install tasks

Create a server image (AMI)

When everything is working fine and the configuration is quite stable, it’s a good idea to create an image of your server that you can instantiate whenever you will need a new instance, without having to go through the whole configuration process from a standard Linux box.

This is a very simple process , from the EC2 console, in the Actions menu, selectImage -> Create Image and follow the instrutions.

Distributed file storage

Now that your server is in production, you certainly have data that is stored on the server and that is of critical importance. Server data is transient and will be deleted when the instance terminates. It is then a good practice to either host your critical data on a persistent distributed storage layer, or to schedule regular backups on persistent storage.

Furthermore, for scalability purposes, a distributed storage layer will be mandatory if you plan to scale up with multiple bot server instances behind the load balancer.

Operation Costs

Amazon AWS has a very convenient billing dashboard which details monthly costs and cost forecast.
In our configuration the monthly cost sets at 35 US Dollars (if you are free-tier, than it will be almost free).
A convenient option is to setup alerts when budget reaches a predefined threshold.


Although the whole process of setting up the bot server was a bit tedious, in the end you’ll have a very solid and scalable configuration that will allow your bot to reach out millions of users !

  • Anonymous

    “From your development environment, make a tar of the complete hierarchy and copy it to the server instance with scp (using the keypair .pem file, just like you did with ssh) .”
    Can you please elaborate on this part…..I am new to these…..Thanks in advance…

  • Anonymous

    While executing command ” $ curl “, I am getting below issue.
    curl: (7) Failed to connect to port 5000: Connection refused

    Can anyone please let me know if I am doing something wrong?

This site uses Akismet to reduce spam. Learn how your comment data is processed.