Start building your own chatbot now

By the end of this tutorial, you will be able to build a fully functional movie bot! It will be able to make movie recommendations based on several criterias. We’re using SAP Conversational AI platform to build the bot and The Movie Database for information on movies.

Here’s a demo chat with Movie Bot:

movie bot - demo GIF

Movie bot in action

WHAT ARE WE BUILDING TODAY?

Interacting with third party APIs (such as The Movie Database) allows for much more interesting use cases that simple Q/A chatbots. With Bot Skills, we added the option to call webhooks directly from the builder, which makes it even easier.

Today’s bot requires several steps:

  1. Extracting key pieces of information in a sentence
  2. Building the bot flow (triggers, requirements, actions)
  3. Creating and connecting a bot API able to fetch data from The Movie Database

You’ll need a SAP Conversational AI account, Node.JS and potentially Ngrok for testing.

Before we jump in, please check this guide instead if you are looking for a guide detailing the creation of your first bot.

This tutorial covers Node.JS only. If you would rather code with Python, check this tutorial covering a similar use case.

Let’s get to it!

I/ EXTRACTING KEY INFO FROM A SENTENCE

Intents are helpful to determine the overall meaning of a sentence. For our use case, knowing that the user wants to watch something is not enough.

We need to know what the users want to watch.

Entities are designed to solve this problem: they extract key information in a sentence.

Intents make you understand that you have to do something. Entities help you actually do something.

Let’s imagine you are a telco company providing phone and internet access, and your bot has an intent that understands when people are complaining about an outage:

movie bot - Entities extract key information from a sentence

Entities extract key information from a sentence

The entities extracted will help understand what is going wrong, where and since when.

For our movie bot, we will try to extract 3 key pieces of information:

  1. What the user wants to watch (movie vs TV show)
  2. What genre they are looking for
  3. In which language

USING GOLD ENTITIES

To help you speed up your development, SAP Conversational AI extracts several entities by default : Dates, locations, phone numbers… An exhaustive list is available here.

The Language entity will be helpful:

movie bot Gold Entities

Gold Entities – Language

See the little star next to the entity name? It differentiates a gold entity from a custom one.

We will use it to fulfil our third requirement: the movie language.

CREATING CUSTOM ENTITIES

We will create custom entities to extract the information we need. As with intents, training is very important: the more examples you add to your bot, the more accurate it gets.

Training your entities can happen through multiple intents. Entities are independent of intents.

For our movie bot, we only need one intent, discover, and 2 entities:

  • recording to identify that the user wants to watch a movie or a tv show 
  • genre

Open the intent discover and add expressions. Make sure to cover every possibility, this means a healthy mix of expressions with:

  • No entities at all: “My boyfriend wants to watch something tonight”
  • One entity: “I want to watch a movie”
  • Many entities: “Can you recommend me some French drama TV shows?”

To tag your expressions, select the text you want to tag and type your entity name:

movie bot - tagging custom entities

Tagging custom entities

You should add many more examples: 15 would be nice, but a production-ready bot would require at least 50 examples to perform well. To speed up the process you can fork the entities built within this bot [recording entity, genre entity] and then fork the discover intent from this bot.

You can see here that “French” was detected as a nationality, not a language, because that’s what it is in this context. When building the bot flow, we’ll make sure to check for these two entities.

ADDING CUSTOM ENRICHMENTS

Now that we have labeled have our entities we are going to enrich them! Open the entities panel from your bot under the training tab as showed below:

Entities management panel

Entities section

Now let’s open the genre entity. If you look at the top right of the panel you should see a toggle saying free - restricted and settings, open it so we can explain in details the different options you have access to:

Entity options panel

Entity panel

Within the entity panel you have access to different options for your entity:

  • Free vs Restricted – A free custom entity is used when you don’t have a strict list of values and want machine learning to detect all possible values whereas a restricted custom entity is used if you have a strict list of words to detect and don’t need automatic detection of the entity.
  • Fuzzy matching – Fuzzy matching is an index between 0 and 1 to indicate how close a word can be from the one in your entity list of values. If the word is above this index then the platform will tag it as the closest value within your list.
  • List of values – This is where you can add all the list of values of your entity which could be different values or synonyms

For more in depth information about entities you can read our detailed documentation.

In our case our genre entity is going to be restricted as theMovie Database API only manages a specific list of genres. Here is the list below:

[
{ id: 28, name: 'Action' },
{ id: 12, name: 'Adventure' }, 
{ id: 16, name: 'Animation' }, 
{ id: 35, name: 'Comedy' }, 
{ id: 80, name: 'Crime' }, 
{ id: 99, name: 'Documentary' }, 
{ id: 18, name: 'Drama' }, 
{ id: 10751, name: 'Family' }, 
{ id: 14, name: 'Fantasy' }, 
{ id: 36, name: 'History' }, 
{ id: 27, name: 'Horror' }, 
{ id: 10402, name: 'Music' }, 
{ id: 9648, name: 'Mystery' }, 
{ id: 10749, name: 'Romance' }, 
{ id: 878, name: 'Science Fiction' }, 
{ id: 53, name: 'Thriller' }, 
{ id: 10752, name: 'War' }, 
{ id: 37, name: 'Western' }
]

Add all the different genres to our list of values and don’t forget to also add synonyms such as SF, Sci-Fi for Science Fiction, Romantic for Romance or Animated, Cartoon for Animation. You can fetch the all list of values from there. As you can see from the JSON above, there are ids associated with the genres. The reason is because the Movie Database can’t search for a specific genre based on its English name, but rather on a custom number. What we can to achieve is to associate for each of the genre values a specific id that will be return within the JSON of the NLP API so we can pass it on to the Movie Database API. This is the purpose of custom enrichments: whenever an entity is detected, the JSON returned by the NLP API is enriched with additional information about the entity.

Within the custom enrichment panel we need to create 3 keys:

  • name – to map synonyms under a same value
  • id – to enrich with the id of the Movie Database
  • article – to add the article of the genre (we will use this later)

In order to add a custom enrichment click add new key and add the three keys listed above – for the article set the default key value to ‘a’ as most of the genres would be with ‘a’. Within name you can start adding the specific enrichment and map it to all the different values for your article, id and name such as below:

Mapping the values to a single name value

Custom enrichments for name

 

Mapping all the values to a unique id from the Movie Database

Custom enrichments for ids

Mapping all values to a unique article

Custom enrichments for article

You can fork the whole entity from this page which will include the enrichment. Now this is done let’s test is within the test console. If you send the sentence “I want to watch an animation movie” you now should now see the following custom enrichment:

  "genre": [
      {
        "value": "animated",
        "raw": "animated",
        "confidence": 0.99,
        "name": "animation",
        "id": 16,
        "article": "an"
      }

Great, now our enrichment gives us the generic name, id and the article! Let’s do the same thing for the recording entity, go back to the entities panel and click on recording then make it restricted and add all possible values and synonyms for tv show and movie (such as tv shows, shows, motion picture, film, films, movies, etc.) see the entire list here. Now go to custom enrichments and add the key type and add 2 specific values:

  • movie – for all movies synonyms
  • tv – for all tv shows synonyms

It should look like this:

Mapping all values of recordings into either movie or tv

Custom enrichments for type

Sending back our sentence “I want to watch an animation movie” we now also have the enrichment for recording:

    "recording": [
      {
        "value": "movie",
        "raw": "movie",
        "confidence": 0.99,
        "type": "movie"
      }
    ]

II/ BUILDING YOUR BOT FLOW

Since we just need to make sure all our criteria are filled before calling a Node.JS API, the build part will be rather simple.

We will just need one skill, let’s call it discover.

You can find an example of a configured skill here.

TRIGGERS

We want to trigger this skill if the intent @discover is present:

movie bot triggers

Message triggers

REQUIREMENTS

This tab helps you collect data before moving to Actions.We want to make sure the user specifies a recording, a genre, a language and a yes or no intent before moving on:

movie bot - requirements 1

Requirements

The requirements will be checked one by one. They can all be fulfilled on the first message, for example if the user says I want to watch a crime movie in English, then the Actions will be triggered immediately.

For each Requirement, you can choose to send a message if it is complete or if it is missing.

Sending messages when a requirement is complete can make your bot more lively: A crime movie? I love them too!, but are almost mandatory when the requirement is missing: You need to ask your users to fill what you need to know.

For example, I send quick replies with suggested genres if #genre is missing:

movie bot requirements 2

Conditional message if a requirement is missing

For the confirmation we are using the memory to display a dynamic message to validate the choice of the user using @yes and @no intent:

Using the memory for dynamic message through the custom enrichments of our entities

Using the memory for dynamic message

Once you have set up questions for the 4 groups of entities, go to the Actions tab.

ACTIONS

Once the requirements are fulfilled, we want to call our API to actually perform the search if the user said yes else we reset the memory and ask again what the user wants to watch.

If _memory.no is present – reset the whole memory and send a message such as “Let’s start again, what do you want to watch?”

If _memory.yes is present create a CALL WEHBOOK action. You can either type a full URL (eg: https://mydomainname.com/discover-movies), or a relative url (/discover-movies). SAP Conversational AI will use the parameter Bot base URL in you bot settings when you type a relative URL.

Next, add an action UPDATE CONVERSATION > EDIT MEMORY > RESET ALL MEMORY  to empty the memory once the call has been made.

Actions for movie bot

Actions

If you don’t have a public server, or if you want to test your bot during development, ngrok is a very handy tool: It creates a public URL for you and forwards requests to your computer.

Once you installed it, run

ngrok http 5000

And copy the Forwarding URL in HTTPS (https://XXX.ngrok.io) to your bot Settings (“Bot webhook base URL” field). All requests made to these URL will be forwarded to the port 5000 of your computer.

All your bot needs now is its API to get your movies!

III/ CREATING THE MOVIE BOT API

The NodeJS part of this bot is fairly simple: It will behave as an HTTP proxy between SAP Conversational AI and The Movie Database.

When your application receives a request from SAP Conversational AI, it sends a search query to the Movie Database with the criteria of your user and formats the JSON answer to the SAP Conversational AI message format.

Bot API diagram

Bot API diagram

Option 1 : the automatic way

You can clone the entire project directly from our Git repository : https://github.com/plieb/movie-bot-skills-training

Option 2 : the manual way

Step 1 – scaffolding your project

mkdir movie-bot && cd movie-bot
npm init
npm install --save express body-parser axios
touch index.js config.js
mkdir discover-movies && cd discover-movies
touch index.js movieApi.js
cd..

Step 2 – getting a TMDb API token

You will need a token to use the Movie Database API, go here to generate one, and edit your config.js file:

module.exports = {
  MOVIEDB_TOKEN: process.env.MOVIEDB_TOKEN || 'PURYOURTOKENHERE',
  PORT: process.env.PORT || 5000,
};

Step 3 – filling your index.js with an Express application

Let’s create an Express application to handle the requests from SAP Conversational AI. To better organize our project, as seen in Step 1, we have a folder /discover-movies/ which contains the core of our bot code (instead of putting all our files in the same folder) and we call it through loadMovieRoute.

const express = require('express');
const bodyParser = require('body-parser');

const config = require('./config');
const loadMovieRoute = require('./discover-movies');

const app = express();
app.use(bodyParser.json());

loadMovieRoute(app);

app.post('/errors', function(req, res) {
  console.log(req.body);
  res.sendStatus(200);
});

const port = config.PORT;
app.listen(port, function() {
  console.log(`App is listening on port ${port}`);
});

Step 4 – filling discover-movies/index.js

We ask SAP Conversational AI to send a POST request to /discover-movies when a user has filled his search criterias.

The main goal of our controller is to pick and format the preferences from the memory to send them to the Movie Database’s API:

const config = require('../config');
const { discoverMovie } = require('./movieApi');

function loadMovieRoute(app) {
  app.post('/discover-movies', function(req, res) {
    console.log('[GET] /discover-movies');
    const kind = req.body.conversation.memory['recording'].value;

    const genreId = req.body.conversation.memory['genre'].id;

    const language = req.body.conversation.memory['language'];
    const nationality = req.body.conversation.memory['nationality'];

    const isoCode = language
      ? language.short.toLowerCase()
      : nationality.short.toLowerCase();

    return discoverMovie(kind, genreId, isoCode)
      .then(function(carouselle) {
        res.json({
          replies: carouselle,
          conversation: {
          }
        });
      })
      .catch(function(err) {
        console.error('movieApi::discoverMovie error: ', err);
      });
  });
}

module.exports = loadMovieRoute;

Step 5 – filling discover-movies/movieApi.js

Now that we have extracted and formatted all the filters of the request, we need to send the request to the Movie Database and format the answer:

const axios = require('axios');
const config = require('../config');

function discoverMovie(kind, genreId, language) {
  return moviedbApiCall(kind, genreId, language).then(response =>
    apiResultToCarousselle(response.data.results)
  );
}

function moviedbApiCall(kind, genreId, language) {
  return axios.get(`https://api.themoviedb.org/3/discover/${kind}`, {
    params: {
      api_key: config.MOVIEDB_TOKEN,
      sort_by: 'popularity.desc',
      include_adult: false,
      with_genres: genreId,
      with_original_language: language,
    },
  });
}

function apiResultToCarousselle(results) {
  if (results.length === 0) {
    return [
      {
        type: 'quickReplies',
        content: {
          title: 'Sorry, but I could not find any results for your request :(',
          buttons: [{ title: 'Start over', value: 'Start over' }],
        },
      },
    ];
  }

  const cards = results.slice(0, 10).map(e => ({
    title: e.title || e.name,
    subtitle: e.overview,
    imageUrl: `https://image.tmdb.org/t/p/w600_and_h900_bestv2${e.poster_path}`,
    buttons: [
      {
        type: 'web_url',
        value: `https://www.themoviedb.org/movie/${e.id}`,
        title: 'View More',
      },
    ],
  }));

  return [
    {
      type: 'text',
      content: "Here's what I found for you!",
    },
    { type: 'carousel', content: cards },
  ];
}

module.exports = {
  discoverMovie,
};

Step 6 – Start the engine!

That’s all! You’re ready to test your bot.

Start your application by running: node index.js

All being well, you should see: App started on port 5000

Movie recommendation, weather, health, traffic… With third-party APIs, everything is possible! Now that you’re familiar with the workflow, we can’t wait to hear from you about what you’re building! And remember you’re very welcome to contact us if you need help, trough the comment section below or via Slack.

Want to build your own conversational bot? Get started with SAP Conversational AI !

Follow us on
  • Augusto Vera

    whenever I start the server side with “nodejs index.js”, the response is: “App started on port undefined”, and I have already set the config.js file with port 5000, I even opened it in my ubuntu firewall.

    • Nathan Grasset

      Hi Augusto,
      I made a mistake in the code snippet, the line:
      app.listen(config.port, () => console.log(`App started on port ${config.port}`));
      Should have been:
      app.listen(config.PORT, () => console.log(`App started on port ${config.PORT}`));
      instead.
      I will update the snippet very soon, thanks for your help 🙂

  • Augusto Vera

    Also I assume all code examples that start with the comment “// index.js” all go inside the indes.js file. ¿Am I right?

  • Anonymous

    Not it started correctly in port 5000 🙂

  • Anonymous

    Another question, the skill get-movies contains a webhook url action, which in turn contains this relative URL: “/discover-movies”, I understand that this should be the URL that points to the node.js code which in turn calls the moviedatabase API, so it should be replaced by the temporary URL provided by NGROK, right?

  • Anonymous

    Hi again Nathan, you have as well another typo in the scaffolding of the project when you create the files with “touch”, the “discoverMovie.js” module is spelled “dicoverMovie.js” in your example, so the main module won’t find it.

  • Augusto Vera

    Also, when you setup the NGROK, the temporary URL it provides should be placed in the web hook action of the get-movies skill in the build section of the recast.ai bot right?, so that the bot can effectively call the node.js gateway which in turn connects to the movie database API.

    • Brice Berdah

      Hello Augusto,

      You have two different URL/path to set up:

      1/ Your bot base URL: that’s t the base route that will be used for every API call made by the bot. This is your ngrok server URL.

      2/ For each API call, the relative path. You can set this up directly in the builder. Here is how it looks like for the /discover-movies route:
      https://cdn.recast.ai/blog/nodejs-chatbot/skill_step5.png

  • Nick

    I am excited to add a ChatBot to my catalog website. But independent of Messenger, Skype, or the others. I just want a little Chat Box in the lower right hand corner, that a Bot guides the visitor through with his or her comments about my products – and some suggestive selling, like the example of the movie DB, No signing in to Messenger at all. My visitors aren’t going to want to do that. I am also concerned that that might run afoul of the upcoming US and EU privacy rules/laws.

    • Brice Berdah

      We just launched the webchat feature that allow you to do just that! Now once your bot is done, you just have to copy/paste a code snippet on your website to include the chat bubble for your visitors
      Check https://github.com/RecastAI/Webchat

  • Theo

    Hello !
    I think there are code sections in “CREATING THE MOVIE BOT API” but I can’t see it… 🙁 I tried on firefox and chrome…

    • Brice Berdah

      We had some issues with our code snippets, we’re sincerely sorry for it. It’s fixed now.

  • Purav

    Hello,
    I went through all the steps and provided in tutorial and still cannot get the chatbot to show up on the webpage. Every time I type in localhost:5000 or the URL provided by ngrok, I get 404 error in my browser’s console. I double checked all of my work and went through ngrok and node.js installation on my computer, still stuck on the issue. Please help me out with this!

  • Anonymous

    Awsome! Hey where do I get info to download the completed project? I am finding a bit difficult to put the pieces together. Thank you

  • Piccosoft

    Your blog is very nice… Thanks for sharing your information…

  • Rahul Issar

    It doesn’t seem to be working for me it says “Cannot GET /” when I click on the url which is given to me from ngrok

  • Anonymous

    Hi,
    I’ve followed the tutorial. But unfortunately my web hook URL doesn’t getting any request from chat bot :(.
    URL :
    https://chikaslocalhost.com/hot-news
    It works well when i trying using browser.

  • Anonymous

    Hi, guys
    I have followed the tutorial and try to implement the “Call WebHook”. But it is strange that my web Hook URL doesn’t getting fired. My webhook url is https://ming-chatapp-demo.cfapps.sap.hana.ondemand.com/

  • Anonymous

    Or WebHook URL must be under domain “*.ngrok.io”?

  • Anonymous

    Hi,

    I am using ngrok to expose my localhost and make the webhook url publicly accessible.
    But ngrok session expires in every 8 hours. Hence, every time I need to update the URL on Recast to call my api.
    Can you suggest a work around to overcome this situation.

    • Everistus Olumese

      You can host your webserver on heroku and supply the url heroku gives you as the webhook

  • Anonymous

    how do i set up Bot webhook base URL and Callback URL ?

  • Anonymous

    I am trying this tutorial.
    And this is my Movie API URL “https://api.themoviedb.org/3/movie/550?api_key=255335f5a34bf6eb2de25ea57e744c90” , but my concern is where to put this url. In which file; index.js,discoverMovie.js config.js and i see MOVIEDB_TOKEN: ‘XXX’, in config.js. What is MOVIEDB_TOKEN? Is it PAI key?

  • Alex

    Is this tutorial still valid? I’m also getting 404.

  • Anonymous

    npm install discoverMovie.js throws 404. No package available says.
    What to do?

  • Anonymous

    Hello, there issue : “Could not parse API response as JSON”

  • Anonymous

    Is this tutorial still valid

  • Anonymous

    I keep getting the same error…”Could not parse API response as JSON”

  • BIlly

    I think this is a good addition to my blog 😀
    Thank you

  • Mike

    Awesome tutorial! thank you very much for that. No issues whatsoever during the setup process, all running smoothly… now to edit it to the bot I want it to be, many thanks.

  • Mike

    Okay, so how do we make it so when a user selects TV it actually opens the TV links not dead ones, as withing the code it is set to;

    value: `https://www.themoviedb.org/movie/${e.id}`,

    Which creates a dead end for users wanting to watch TV as it will always put movie/${e.id}`, at the end of every link… I’ve tried to do `https://www.themoviedb.org/${kind}/${e.id} but that doesn’t work, it seems that we can only have either movies or tv working with actual clickable links, unless you clone and use two seperate ones, one for movie searches and the other for tv searches that is.

    Surely there is a way to get this to work with both though?

    HELP PLEASE

  • Mr. Edmonds

    How can we get this to work with both movies and tv shows?

    Obviously the code is set to:

    type: ‘web_url’,
    value: `https://www.themoviedb.org/movie/${e.id}`,
    title: ‘View More’,

    So it will only generate links such as `https://www.themoviedb.org/movie/${e.id}`. But I would like it to be able to do both movie and tv, I tried to use `https://www.themoviedb.org/${kind}/${e.id}` but to no prevail. So unless someone can tell me what I’m missing, then it seems this bot will need to be duplicated and split into two, one for movies and the other for tv shows.

    Surely there’s a way to use both in the same bot?

    HELP PLEASE

  • PE

    Hello this has been solved, the code has been updated: https://github.com/plieb/movie-bot-skills-training you need to pass the kind as parameter to the function to use it within the URL.

  • Carlos

    Hi, I have a problem when run server with “node index.js”. Although the server starts well and says that it is listening on port 5000, when in the browser I type localhost: 5000 it shows me a “Can not GET /” error message and therefore I can not connect to NGROK. Can you please help me find out what is wrong with my code from Node.js? It is exactly the code that is in the tutorial.

    Thank you!

  • Anonymous

    Awesome tutorial !
    as a beginner found it very helpful

  • Jithin

    Hi,

    Please can you share the documentation(or relevant API spec) which explains the input and output structures required for a webhook.

    Kind Regards
    Jithin

  • Ye Man Aung

    Support for my account access issues permission.

  • adityadev

    Your blog is very nice… i got more information about your blog page… Thanks for sharing your information…

  • Navin

    Nice Blog. Can you please share the bot url to fork and check the entities ?

This site uses Akismet to reduce spam. Learn how your comment data is processed.