Today, we are really proud to release the second iteration of our API!
As the link between you, your creations and our work, our JSON response has grown to be something everyone can improve. In this post I’ll present how you helped us build the JSON you want, why now is the best moment for us to release this new version, and finally the steps you have to take to fully enjoy this awesome piece of code!
Why do a second API version ?
After a year and two minor iterations, we gathered lots of feedback from our community, and we felt it was time to challenge the service we provided until now.
We decided to start this iteration from zero to put in perspective the choices we made at the beginning of SAP Conversational AI, while according importance to each and every idea we had, or had been submitted.
After several brainstorms and user testing, we came to the conclusion that not everything was to change (whew!), but we highly simplified the JSON, by reshaping its architecture.
In short, the JSON returned by the API V2 is richer in information, revolutionary by its simplicity, and much more practical. This new iteration will be the socle of groundbreaking features we’re preparing for you !
What are the changes ?
Your input is now treated as one single sentence
This breaking change is motivated by the fact that more than 80 percent of the processed inputs of SAP Conversational AI are single chunks of text. We now treat each input as one complete sentence. In the end, you get a simpler JSON, profiled for your needs.
You can now know the sentiment of your users
Something that the community asked for was the ability to get the sentiment of a sentence. Before, we provided a field called polarity which exposed whether or not the sentence we processed had a negation marker such as no or not.
Today we replaced this field by a new one that can take five values, each representing a granularity of the sentiment expressed: either very positive, positive, neutral, negative or very negative. (Unfortunately, we’re not yet detecting the emotion of the sentence, but be sure that we’d love to be able to tell you that your user is ecstatic to talk to your bot!)
You can now precisely define how to answer
In the previous API response, this field referred to the work of James Allen, whose theory of Speech Acts inspired us to classify a given sentence into several types, eg. assert, command, what, yn-query and some more. We drastically improved the detection of those acts, now raising above the 90% mark, and we reduced the numbers of classes to 4, namely: assert (asserting a fact), command (giving an order), yn-query (asking a closed question), and wh-query (asking an open question). Those four classifications of a sentence will now lie under the act field.
But the act of a question is still not enough to decide what to answer! This remark came to our mind when we stumbled upon the problematic of the Question Answering systems. As these agents are relatives of the bots, we thought that incorporating some of their logic in our response would be a great help when building a conversational agent. Helped by the taxonomy created by Xin Li and Dan Roth, we’ve worked a solution proposing an accuracy above 90%, and will allow you to further understand the request of your users, pre-filter them, or propose services they wouldn’t even imagine!
You can now find out the confidence of our intent detection
Numerous people suggested this addition, and we’re proud to announce that from now on, we’ll be providing the confidence of our machine learning algorithms in their predictions. Coupled with the usage of the strictness, which discards intents with a confidence lower than a threshold, you can control more precisely the way you will handle your interactions with your users.
We believe that transparency is the key to a great product for you, and that’s why we also added the confidence field to the entities.
As a last thought on this, let me suggest that this opens the path to self-learning. If you consider that a user answering positively to a low-confidence intent is marking the fact that the matched intent is correct, you can then add it to your training data set, and make your bot learn from its interactions.
Entities are now more adapted to all your use cases
Last but not least, we focused on enriching the entities we detect, and reworking some of them from scratch. Here are the most notable changes:
Every date or time is now formatted using the ISO-8601 reference, for consistency purposes.
In the DATETIME, we replaced the value with iso, we also fixed the typo in formatted field of the entity (I’m so sorry…), we added chronology to help you know if a date is past, present or future, and finally, an accuracy field helps you find out what was explicit and what was inferred about the datetime we detected.
The LOCATION entity is now more precise thanks the type field, made to qualify the place as a locality, a country or even a street name. The formated field is now correctly spelled as formatted. Finally, a handy Google Places ID lies under the place field, useful to get more information such as whether or not a building is open, and more.
AGE and DURATION have been merged and reworked, you can now get the duration detected in years, months, days, hours, minutes, or seconds. In addition, a cool chrono helps you visualize the time left.
SET’s next field is now formatted using the ISO 8601 specs, it’s grain is now called frequency, and an interval between the occurrence has been added. This information helped us build a rrule, or recurrent event rule, used in the different calendar apps.
INTERVAL have been introduced to complete the date entities quatuor, the key difference between a DURATION and an INTERVAL is that the latter possess a begin and an end. The timespan is calculated in seconds.
In the DISTANCE, MASS, PERCENT, SPEED, TEMPERATURE, and VOLUME entities, we replaced the value by scalar, and they respectively received a new field (meters, grams, percent, mps, celsius, and liters) representing their base SI unit. MONEY received a slightly personalized treatment: value is now amount, unit is now currency, and its base is dollars.
Concerning the LANGUAGE, we replaced the code by short and long, representing the ISO 639-1 and ISO 639-2 codes. The same treatment have been applied to NATIONALITY, and we added country to the demonyms.
We changed the deg to bearing in the CARDINAL entity. We added a new field to COLOR, called rgb, which is a pre-formatted string of the color’s rgb format.
We stopped being lazy about EMAILs and URLs, and they are now correctly parsed and enriched with a local part, a tag and a domain for the former, and a scheme, host, path, params, query, and fragment for the later.
The ORGANIZATION and JOB lost their value field, and the PERSON’s value was replaced by fullname, to prepare for a future update of its enrichment. SORT’s value was replaced by criterion, NUMBER’s value was renamed scalar and ORDINAL’s value is now called rank. PHONE now has a number instead of a value, which is the cleaned and normalized phone number.
We removed the entity MISC, because we felt it was too generalist to be usable, and enrichment was not really possible with such a large category. This removal also improved the detection of other entities (+0.1 accuracy point at most).
Updating smoothly to the new version
Every SDK has been updated to the new version of our API, starting at version 2.0.0 for the Ruby, Python, Scala, NodeJS, Golang, iOS, and Android platforms. A new SDK has been released, and helps you build PHP agents since its 1.0.0 version.
Our documentation has been updated to reflect the changes brought to the act, type, sentiment and entities, check out the glossary to get more information!
For more details, please head to the complete changelog at man.cai.tools.sap.
Don’t hesitate to reach out to us and join us on Slack to discuss those changes ! 🙂