The Open-Source Speech and Image Recognition and Speech Synthesis Platform for Educational Needs

Quick menu:

Download Endurance chatbot platform ver. 4.0

What does the Program Look Like?

The platform software is a set of three modules that form the basis of our solution:

  • Image recognition
  • Speech recognition
  • Speech synthesis

  • In 2015 we started tо work at a platform to be used as a basis for a simple robotic interface. Our task was to combine face and voice recognition with voice synthesis.

    Modules Description

    We used the standard Microsoft SAPI as a choice for a speech synthesis module. This product with various language sets is distributed free of charge.

    Image Recognition

    Image recognition is a more complicated task, as it is necessary not just to recognize a face or an object but to recognize it in the streaming video going from the camera. The choice of the resources used by the program is very important. It does not matter for a computer but it does matter for a tablet, as the program should work efficiently and should not lock the system.
    For this solution we used the OpenCV library. To speed up the work, any face of a specified size in the camera field vision is searched for at the beginning. The search is done with the help of Haar cascades and the ready trained template from the OpenCV library.
    Then the detected face is cut out, normalized (unified) in size and light, and reproduced in black and white.
    After that a ready FaceRecognizer algorithm, trained on a number of images of the same face taken from different angles, is used to recognize a certain person.

    Voice Recognition

    We use the Google Speech API, which maintains not more than 50 recognitions a day (it’s about 15 minutes of the recognition process). It’s a great solution for open-source.
    Google has a very good recognition quality at the distance of a few meters.

    The Program Features

    The first thing the program does when activated is to detect a face in the field of camera vision. If it sees the face for the first time (it is not found in the database), it asks to enter and save the name. After that the program will always recognize this face.
    Then the program switches to the mode of speech recognition: pronounced words, sentences or commands. As soon as the program recognizes a phrase, pronounced be a person, it looks for an adequate answer in the database and voices the answer. It is necessary to put all possible answers into the program beforehand for the program to know what to say. So far, it looks like a simple text file.

    (Tim = Good afternoon, Tim. We welcome you at our conference.
    Steve = Thank you for coming, Steve. You will be now welcomed. Have a nice day!
    35310204 = Your credit repayment is due before the 30th day of the following month.)

    This is not about an artificial intellect. This is about a simple interface for image and speech recognition and speech synthesis for educational needs. To have an opportunity to create a new ready solution on the basis of this or that robotized platform.

    The Program Application

    It is possible to use the program or its separate components for development of a certain robotic platform. This kind of platforms exists nowadays as electronic kiosks.
    As we see it, this program will look more elegant if used together with our DIY SelfieBot.
    This program can have a great many ways of practical application.

    A Robo-Stand or a Registration Robot at Conferences and Exhibitions

    Problem to be solved:
    Reduction of the conference/exhibition staff costs.

    The Task: Support of conference visitors and guests with a user-friendly interface for automatic registration.

    The Solution:
    A guest comes to the registration counter and tells his name. The system recognizes him, checks him in the database and registers the guest online.

    An Example:
    — Introduce yourself, please.
    — Tim Cook.
    — Good afternoon, Tim. You are welcome to our conference.
    htmlimage (10)
    htmlimage (1)

    A Virtual Robot-Secretary (Robo-Secretary) for Offices and Office Centers

    The problem to be Solved:
    Cost reduction for a secretary work in an office center.
    The Task: Support of visitors with a user-friendly interface for automatic registration when visiting an office center.
    The Solution: A guest comes to the SelfieBot and tells his name. The info goes to the receiving party as a text or voice mail.
    An Example:
    Robot: — Please, introduce yourself.
    Guest: — My name is Sonya.
    Robot: Thank you for coming, Sonya. They will meet you now. Have a nice day!
    htmlimage (3)
    htmlimage (4)

    A Robotized Self-Service Terminal. A Robot-Terminal

    Problem to be solved:
    Cost reduction of the customers’ self-service and improvement of the service quality.

    The Task

    Customer support with a user-friendly interface for interaction with a self-service terminal. Voice communication with the terminal.
    The Solution
    A guest comes to the terminal and tells what he wants to do. No need to touch the screen.

    An Example

    Robot: Good afternoon! What would you like to do?
    Customer: I’d like to know when my loan payment is due. My account number is 35310204.
    Robot: It is due before the 30th day of the following month.

    A Robot-Assistant in a Store or a Salesroom

    The problem to be Solved
    Cost reduction for personnel and virtual information service.

    The Task

    Customer support with a user-friendly interface for interaction with an electronic shop assistant.

    The Solution
    A customer approaches the electronic assistant, asks questions and gets all the necessary information.

    An Example

    Robot: Good afternoon! We are happy to see you in our shop.
    Customer: What special discounts do you offer?
    Robot: We offer 10% discounts to our regular customers for all goods.
    Customer: Do you have new collections?
    Robot: We do, you’ll find them in the far right-hand corner.
    htmlimage (6)
    htmlimage (7)

    Robot-waiter SelfieBot in a café/restaurantе

    Problem to be solved: reduction of expenses for personnel; waiting time reduction.

    The Task: Customer support with a user-friendly interface for interaction with an electronic waiter.

    The Solution: A robot-waiter approaches a customer, asks if he is ready with the order and takes the order.

    An Example:

    — Good afternoon! We are happy to see you in our café. Are you ready with the order?
    — Yes. Please, bring me one cappuccino and a croissant.
    — Thank you for the order. You’ll get it in 5 minutes.
    htmlimage (8)
    htmlimage (9)

    More Variants of the Program Application

    The program is ideal for educational process as a separate solution or part of the integrated solution.
    The program may be used for:

  • Computer vision
  • Automatic robo-platform

  • The Program Operation Demo

    An example of the program user interface:

    Educational Process

    This program is ideal for students of high schools, colleges and universities striving to learn to work with the software designed to solve problems of the end users.

    Invitation for Cooperation

    We invite schools and colleges for cooperation. We are ready to share our project with you for non-commercial purposes and further researches.

    The project is open-source and is distributed among educational institutions free of charge.


    Educational Program

    Educational Program — «Robotics: Design Engineering and Programming»