An Interoperability Watchdog Report - Home - TAUS

A Common Translation Services APISeptember 2012

An Interoperability Watchdog Report

Enabling better translation

TAUS - Enabling better translation

Copyright © TAUS 2012 2

COPYRIGHT © TAUS 2012All rights reserved. No part of this publication may be reproduced, stored in a retrieval system of any nature, or transmitted or made available in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of TAUS. TAUS will pursue copyright infringements.

In spite of careful preparation and editing, this publication may contain errors and imperfections. Authors, editors, and TAUS do not accept responsibility for the consequences that may result thereof.

Published by TAUS BV, De Rijp, The Netherlands. For further information, please email [email protected]

RelevanceThe translation industry, unlike other sectors underpinned by technology, is defined by a lack of the type of

open standards that have fueled innovation.

In this paper, we describe a strategy for developing an open yet extensible web services API (Application

Programming Interface) that addresses most of the use cases in translation, but is also easy and inexpen-

sive for vendors to implement.

At a time when translation is moving to the cloud, the approach proposed in this paper has the potential to

significantly advance the interoperability agenda in the translation industry.

Co-authorsKlemens Waldhör, TAUS Labs

Brian McConnell

Reviewed byRahzeb Choudhury, TAUS

Hans Timmerman, AVB Translations

Chase Tingley, Spartan Consulting

Grant Straker, Straker Translations

Gustavo Lucardi, Trusted Translations

Marco Trombetti, Translated.net

Vladimir Weinstein, Google



Contents

Introduction 4

Goals and Purpose 5

API Pattern 6

Hello World Requesting A Translation 7

Defining A Common API 9

Step 1 Describe A Regular Set of Tasks 9

Step 2 Define Standard API Calls For These Tasks 10

Step 3 Define Best Practices For Extended/Custom API Calls 12

Implementing The API 12

An more detailed example using the API 12

API Specification 14

Standard HTTP Response Codes 14

Input Formats and Parameters 14

Output Formats 15

Constraints on Parameter Values 16

Call To Action 17

Translation Companies 17

SDKs 17

List Of Tables

Table 1: Example urls from www.abctranslations.com 7

Table 2: Typical callback fields 8

Table 3: Typical service fields for language service provider 9

Table 4: Supported request types 10 + 11

Table 5: Requests and their meaning 12

Table 6: Example request sequence 13

Table 7: HTTP response codes 14

Table 8: Error response codes 15

Table 9: Comment field 15


Introduction

The translation industry, unlike other sectors underpinned by technology, is defined by a lack of the type of open standards that have fueled innovation. In this paper, we describe a strategy for developing an open yet extensible web services API (Application Programming Interface) that addresses most of the use cases in translation, but is also easy and inexpensive for vendors to implement.

Most of the efforts in defining standards for translation processes have focused on file formats. This made sense when systems were not pervasively interconnected, but in today’s environment where services are increasingly hosted in the cloud, a file centric standard can create as many problems as it solves.

Web APIs, on the other hand, offer the ability to simplify the process of interacting with translation resources and services while also allowing a complex task to be broken down into several small and simple requests. This paper studies the basic set of features that should be included in a translation services API, and also proposes a reference specification for a standard API.

In this report we touch on the subject of file centric standards, such as XLIFF, only briefly to illustrate how the specifications proposed here are complementary to file based standardization initiatives.

“To me file formats (like XLIFF, for its faults) and APIs are both pieces of the same puzzle. File formats to provide standard representation of complex data types are vital any kind of interoperability - they are the nouns of a sentence, where as APIs might be thought of as the verbs.”

Chase Tingley, Spartan Consulting

The API design TAUS is proposing focuses on arresting the interoperability challenges that are emerging as the industry moves to a more user-driven/on-demand translation model than historically. The set of basic operations selected to implement as web services were selected following examination of APIs currently in use by the target profiles.

“Looks good. REST, simple and with the right parameters.” Marco Trombetti, Translated.net

“The API covers most requirements. We’ve been asked by clients and have added two things to the API that are not in there are. Other than that it would be very similar to our API.”

Grant Straker, Straker Software



Goals and Purpose

Most attempts at defining standards in the translation industry have tried to anticipate every possible use case. While this is a well-intentioned goal, the result is typically a standard or interface that requires considerable sophistication and effort to implement and test, as well as taking significant time, often years, to formulate the schema.

Popular web services, Twitter is a good example, take the opposite approach with their developer APIs. They provide a minimal set of simple interactions to cover the most common tasks. Because of this, developers can prototype new applications quickly, and can learn this type of API within a short period of time. The goal of this proposed API is to define something equally simple for translation.

This proposed API is designed for software developers who have little or no knowledge about translation services, which is to say almost all software developers. Within the industry, we tend to have debates over technology choices that are meaningless to everyone else. So what do software developers need most? A simple way to request and get translations from a service in the “cloud”, in effect a command that means “translate this into French, call me back at www.foo.com/submit when you’re done. thanks.”

The most basic API would simply define a standard way to request a translation and get the results back. With maybe a dozen other interactions, we can describe most of the things that software developers will need to do in building a multilingual website or application. Most applications will probably only use the most basic interaction, the method used to request a translation.

The design goal in this paper is simple. As a service provider, you should be able to do a partial implementation in a few hours. As a software developer, you should be able to build a “Hello World” application equally quickly, and master the overall API within a day or so without assistance.

We are taking a building with Lego blocks approach. Where complexity may follow as additional blocks are added to the simple foundation, when it is proven to work and is in use.



Copyright © TAUS 2012

API Pattern

The basic pattern used for API calls is one of two types: a simple HTTP request to which the server provides an immediate response, or a HTTP request to which the server replies with an asynchronous callback. For example, when requesting a professional translation, there will typically be a significant delay, so the server will reply with an immediate response (for example a placeholder machine translation) followed later by a callback to push the completed human translation to the client’s server.

Basic CGI InterfaceThe API calls are patterned after HTML form submissions, using the HTTP/S POST method, and thus makes implementing server side scripts trivial using existing CGI interfaces and standard libraries. This approach also eliminates the need for XML parsing, which further reduces the complexity and cost of implementing the API.

Plaintext or JSON ResponseThe server will typically reply with a HTTP response code and text message (for requests that do not return structured data), or a JSON object (to return structured data to the client, such as a list of matching translations). JSON is used instead of XML because it has less overhead, an important consideration in mobile applications, and is readily parsed using standard libraries. For example, in Python it is a trivial operation to “unpickle” a JSON object into a native Python object, with a single line of code, whereas XML requires much more effort to parse. JSON is also attractive because it is natively supported by Javascript, which will enable developers to build rich web applications that communicate directly with a translation server.

Server developers can, of course, provide additional output formats for API calls that return structured data. To borrow from the parlance of IETF specifications, the specification requires that JSON must be supported as an output format, while other formats such as XML, PO/gettext, etc may be supported as desired by the server developer. In fact many REST API implementations handle both JSON or XML, such as the TAUS Data APIs at www.tausdata.com.

6


Hello World Requesting A Translation

Let’s start by describing a “hello world” example. In this case, we want to request a professional translation from a language service provider, who will then call our system back when the translation is completed, via a callback request. We can do all of this using the same mechanism to post the contents of a form on a web page.

The client requesting the translation simply posts to a url, such as https://www.abctranslations.com/translations get with a standard list of fields, such as:

sl source language code

tl target language code

st source text

tttranslated text (if provided, the translation company will score and edit the translation, if not it will initiate a new translation)

encoding optional text encoding (utf-8 by default if omitted)

username customer’s username or API key

pw customer’s password or secret key

session session ID (optional, used in place of username, pw pair)

url URL of original document, if it is a web page

collection an optional identifier, if the request is part of a batch or collection

sla service level agreement code (e.g. 24 hours)

max-price maximum price per work unit (optional)

callback-url URL the language service provide should post the results to

edit-url URL to direct translator to (if an external or in situ editing tool is in use)

Table 1: Example urls from www.abctranslations.com

The translation server will reply with a JSON object containing a success/error code, unique record locator, and estimated time to complete the request.

7Copyright © TAUS 2012


8

When the translation or edit request is completed, the translation server will then post its results to the URL listed in the callback-url field. If no callback URL is provided, the client application will poll the request URL periodically until a translation is returned. The callback will include fields such as:

guid unique request identifier

sl source language

tl target language

st source text

tt translated text

score quality score (optional, if the translation company is editing/reviewing an existing translation)

comment translator comments (optional)

words number of words translated (optional)

cost cost of translation (optional)

currency unit of currency (optional)

Table 2: Typical callback fields

The client application will then store or process this information however it likes.

The important point here is not that the fields submitted with each form have a particular name, but simply that we agree on a standard set of fields that are needed when requesting a translation. If everyone follows this specification, then an application can request translations from many translation companies using the same procedure, and code base, to make and process these requests. Individual translation companies might extend the spec, for example to allow users to specify a particular subject keyword or domain, but the basics will be consistent across many vendors.

All we are doing is submitting data in the same way an HTML form posts to a web server. This makes it trivial for a developer to build a script to generate and process form requests, and therefore greatly reduces the time and cost to implement this interface (especially compared to XML centric approaches that require a lot of parsing on both ends of the connection), whereas this approach can be implemented using the standard CGI ser-vices and standard libraries bundled with common web programming languages (e.g. PHP, Ruby on Rails, etc).

The key advantage of this API oriented approach is that it says nothing about how a request is processed or how data is stored within the server on either side of the connection, but only defines the forms based communication between client and server. This frees system developers to use whatever languages and tools are best suited to the problem they are solving, without forcing arbitrary standards (e.g. internal file formats) upon developers.




Defining A Common API

Step 1: Describe A Regular Set of TasksThe first step in designing a standard web API for translation services is to identify the most commonly encountered tasks and use cases. Rather than try to address every possible situation, we want to cover 90-99% of real world use cases. This greatly simplifies the scope of work, while enabling service providers who need additional functionality to implement custom API calls to extend the basic functionality defined in this spec. We’ll assume that if a customer has a special need, they will develop a custom interface or extend the API to address this.

The uses we will commonly encounter when interacting with a language service provider are:

Request a translation for a text

Submit a translation to the user, or to a cloud translation memory

Request review or score for a translation

Check the status of a request

Cancel a request

Accept or reject a completed translation/task

Request a list of pending requests/tasks

Table 3: Typical service fields for language service provider

This is not intended to be a complete list, but a representative list of regular tasks. We will develop this list further based on feedback from stakeholders.




Step 2: Define Standard API Calls For These TasksEach of the tasks identified in step 1 map directly to a single API call that will result in a success/failure outcome, which in turn is easy to build as a session less HTTP request (HTML form POST transaction).

Request Comment

translation/get This API call is used to request a translation by a client from the service provider. The creation of the request will return an unique guid which will be used for identifying answers (see submit) for this request. If the requester is submitting a rich media file for translation/transcription, it should be attached as a file along with the request.

translation/submit This API call is used to submit a translation identified by the guid (specified in refid) of a corresponding translate/get request to a client's server or to a shared translation memory.

im/start This API call starts a real time translation. refid refers to the guid of the original im request.

im/callback This API call is used to transmit translations asynchronously to the client side app server. This is the preferred method compared to polling /im/translate. refid refers to the guid of the original im request.

im/translate This API handler is used to request a real-time translation for a message. The server will reply with an HTTP response code and if successful will return a JSON object. If the callback method is used for this session, it will not return a translation, that will be sent in a separate callback to /im/callback on the client’s app server. If polling is being used, the HTTP response will hang until the request is processed. We generally do not recommend using polling because it does not scale well and will fail due to time out conditions and other limits that may be beyond the developer’s control. NOTE: if a client requests real-time translation for several language pairs, it should request several real-time sessions, one for each language pair, since different translator pools may be used for each by the TRANSLATION COMPANY.

im/stop This API call is used to manually tear down a real-time translation session identified by its refid (the original guid). Even though the translation server will typically delete sessions and free resources due to inactivity, a well behaved client app should use this method so that translators are not tied up after a session as ended.

request/status This API call is used to request the status of one or more requests either identified by the refid supplied or their group-name or owner.


11Copyright © TAUS 2012

request/accept This API call is used to accept a request identified by a refid. For example, a client can accept a completed translation to trigger QA review. For a legally binding acceptance of task to be done by a translation agency the client should us the request/confirm method. The server developer can decide on the contexts in which this API call can be used.

request/reject This API call is used to reject a request identified by a refid. For example, a client can reject a completed translation to trigger QA review, or a translator could reject a task to place it back in the queue for another translator to work on. The reject method can also be used by a customer to reject the "offer" of a translation agency for a translation task. The server developer can decide on the contexts in which this API call can be used.

request/cancel This API call is used to cancel a pending request identified by a refid. The server will typically allow this action only for pending requests that are not being actively worked on. The server developer can decide on specific allow/deny rules based on task status.

request/confirm This API call is used to confirm the acceptance of a request identified by the refid request offered by a translator or translation agency. For example, a customer has sent out a request for translation, a translation agency has received the request and accepted the task. This confirm request now is the official "order" that the task should be executed by the agency. The call is similar to the accept task, the difference is that it creates a "legal" acceptance for both parties.

request/view This API call is used to request a list of matching requests currently in the translation server queue. Identification is either thru guid (supplied in refid) or group-name, owner or username. For example, this can be used to drive an AJAX web application that translators can use to view the contents of the queue.

comment/get This API call is used to request comments about a translation. The refid refers to the GUID of a (previous) translation request.

comment/submit This API call is used to submit a comment to the translation server. The refid refers to the guid of a comment/get request. This is typically used in crowd translation scenarios to allow users to discuss and comment on translations.

score/get This API call is used to request scores associated with a translation. The refid refers to the guid of a translation request.

score/submit This API call is used to submit a score to the translation server, or in a callback request to a client"s server. The refid refers to the guid of a score/get request.

Table 4: Supported request types



Step 3: Define Best Practices For Extended/Custom API CallsSince the base API is intended to address most, but not all, use cases, users may want to extend it to address specific situations. In general, the best way to do this is to enable users to submit additional fields with their requests, and to agree on standard nomenclature when adding new fields. For example, one translation company might recognize a max-price field that indicates the maximum per word price the client is willing to pay in a name your price service, while others might ignore this parameter.

Implementing The APIDepending on the services you offer, and which interfaces you want to enable, you can choose between a complete or partial implementation of the API. For example, if you do not offer real-time translation, you need not implement the /im/* API handlers. The best practice to follow is to return the HTTP response code 501 for unimplemented methods, as this will automatically inform client apps and libraries that a particular feature is not available on your system. The table below summarizes which APIs domains may be used for different types of language services.

API Domain Service Typical Use Cases

/translation/* Get and submit translations Almost all systems will implement this API

/IM/* Real-time message translation IM, SMS and chat translation services.

/comment/* View and submit comments about translations

Crowd translation services, professional translation services that allow user feedback, comments or discussion.

/score/* View and submit score for translations

Crowd and professional translation services that allow user feedback and/or professional translator QA ser-vices, as well as moderated crowd translation systems.

/request/* View and manage task queue on the translation server

This is used to expose the request queue to client applications, for example a web app that enables translators to see pending requests, or enables a client application to monitor task progress.

Table 5: Requests and their meaning

In fact, it is possible to build a minimal system by supporting only the /translation/get and /translation/submit interactions. This is sufficient to enable a requester to request and retrieve foundations from a service provider.

An more detailed example using the APIIn the following example we take a look at a typical advanced use case. The case is as follows: A customer AMED requires a document to be translated in from EN to DE and FR. The input document



is XLIFF formatted containing pre translation in the alt-trans field with 5000 segments, the result returned should be in Xliff. In addition there is the option to access the TM and term database through a web service provided by the customer. The translation should be available in about 3 weeks. As the document is from the medical domain high quality standards are required. The customer wants to use the translation API provided by the translation service transextreme. transextreme has implemented the API. transextreme itself only organizes the workflow but does not translate itself.

1. AMED issues a first request using translate/get. The fields are filled as follows: sl … EN, tl … DE, st contains the (stringified) Xliff document, no encoding (can be retrieved through the xliff document), professional … y, callback-url for submitting the translation the customers server, sla with 3 weeks, output format Xliff.

2. The transextreme servers receives the requests and returns a guid and a message stating that the request is checked.

3. AMED retrieves a guid and message from the transextreme server which functions as a task reference for future communication.

4. Transextreme now issues a request to its translation vendors who have implemented the TAUS API also, copying the original fields and replacing some fields (e.g. user, callback-url) with their own value. It also adds the field crowd … y.

5. The translation vendors (shortened) respond by accepting or rejecting the task (/request/accept or request/reject).

6. Transextreme decides which translation vendor will receive the order and confirms the order (task) with request/confirm.

7. Transextreme sends an accept request (/request/accept) to AMED clients server.

8. AMED confirms the order (request/confirm).

9. The chosen translation provider runs the translation and uses the callback-url to send the translation back to transextreme.

10. Transextreme in turn sends the translation back to AMED by using AMEDs callback url.

11. AMED checks the translation and returns a /request/accept message to the transextreme server.

12. AMED sends a second request, changing the target language field tl to FR.

13. and so on …

Table 6: Example request sequence

Clearly, the example can easily be extended and more detailed, e.g. by more detailing the acceptance / con-firmation work flow or by adding a quality check before step 9 which in turn would consist of several requests.



API Specification

Unless otherwise noted, API calls will be made as HTTP POST requests. The server will return either an HTTP error code, or if the call is successful, will reply with a JSON response. A Validation Service is avaliable at www.tauslabs.com, which developers can use as a test environment. Our thanks to Welocalize for sponsoring the development of the Validation Service.

Standard HTTP Response CodesThe server should respond with a standard set of HTTP response codes to signal success and common error conditions, as shown in the table below. A complete list of HTTP response codes can be found aten.wikipedia.org/wiki/List_of_HTTP_status_codes

Condition HTTP Code

Success 200

Authentication Error/Access Denied 401

Payment Required/Account Depleted 402

Record or Item Not Found 404

Media Type or Output Format Not Supported 415

Server Failure 500

Service/Method Not Implemented 501

Table 7: HTTP response codes

Input Formats and ParametersThe API supports two ways of submitting parameters to the server. The first method involves supplying parameters and values through the GET OR POST HTTP command. This may in some cases require sending a lot of parameters and also decoding them etc. Alternatively parameters can be supplied using a JSON object defined as an associative array. The array represents the parameters as key value pairs. The naming of the keys follows the naming scheme of the specific request. The advantage of using JSON as a single parameter is that depending on the programming language used the JSON string can be directly converted in an application specific object and this greatly enhances usability of the code.

If the parameters are supplied via JSON the name of the parameter (key) is “json”. The value is a JSON formatted string. It is up to the server to decode the string and get the required parameters.



Output FormatsAPI calls that return structured data will return a JSON object by default. We chose JSON as the required output format for several reasons: it is trivial to parse or unpickle using standard libraries in most programming languages, it is a more compact format with less overhead (an important consideration in bandwidth limited scenarios such as mobile), and it is natively supported in Javascript (which enables interactive web applications to communicate directly with an API server).

However, developers may wish to extend their servers to support other output formats, for example to generate a PO file in response to a request for translations (to enable the response to be processed via the gettext localization system). API calls that return structured data will look for an optional output field where the client can specify an alternate output format. If omitted, the server will respond with JSON. If your server does not support an alternate output format, it should return a 415 HTTP response if a client requests something other than JSON.

To maximize interoperability, we strongly recommend that developers for both client and server applications use JSON, since it is straightforward to transform into other formats such as XML.

Each call has to return a ’result ’ field. In the event of an error an ’error’ field should be returned describing the reason for the error and the ‘return’ field should be set to error (instead of ok). A more specific error message could also be supplied by using an error code number for the ‘error’ field and adding an additional ‘error-message’ field for the detailed description.

An error result is defined as follows:

result ‘error’

guid unique request identifier

errorcode ‘an error code (number etc.)

errormessage ‘the error message text

Table 8: Error response codes

A ’comment ’ field is supported for all calls; please note that this should not be confused with the ‘comment’ request.

comment ‘the comment text

Table 9: Comment field

The ‘guid’ field should also always be returned (from the one submitted) in order to be able to associate requests submitted within the application process.


Constraints on Parameter ValuesAPI calls supply parameters and values to the server. Depending on the type of the parameter certain restrictions may apply to the parameter values. This is especially true for language encoding, character encoding, and request content types.

Language encoding naming (“English”) is based on ISO 639-x and can be extended by adding the two character country encoding of ISO 3166, e.g. “EN-US” for US-English or “de-de” for “German German”. It does not matter if the coding is supplied in lower or uppercase.Character encoding naming should be restricted to standards like “UTF-8”, “UTF-16” etc. The default if no coding is supplied is “UFT-8”.File encoding naming should be restricted to standards like “text/html”, “text/xml” etc.

Parameters and return values are described in tabular form at http://tauslabs.com/interoperability/taus-translation-api/api-definitions

This API has been implemented on a demo server at http://tauslabs.com/interoperability/taus-translation-api/api-validation-service, which developers can use as a test environment.



Call To Action

This API is designed to be very easy to implement. As a service provider, you should be able to implement the most basic functions, /translation/get and /translation/submit, in a few hours, especially if you already have a web API of some sort.

We are calling on you to provide feedback on this discussion at [email protected].

Translation CompaniesAll you need to do for a partial implementation is to build a request handler for /translation/* Everything else in the API may be considered an extra. Most software developers will only use this one API call. So you can get started and try this out without committing much effort.

You can test your implementations at the TAUS API Validation Service:

http://tauslabs.com/interoperability/taus-translation-api/api-validation-service

SDKsIf you have a favorite programming language, consider building a wrapper library for the API and sharing it with the developer community.

TAUS is an innovation think tank and platform for industry-shared services, resources and research for the global translation industry.

We envision translation as a standard feature, a ubiquitous service. Like the internet, electricity, and water, translation is one of the basic needs of human civilization.

Our mission is to increase the size and significance of the translation industry to help the world communicate better.

TAUS supports entrepreneurs and principals in the translation industry to share and define new strategies through a comprehensive range of events, publications and knowledge tools.

An Interoperability Watchdog Report - Home - TAUS

Documents

Transcript of An Interoperability Watchdog Report - Home - TAUS