# Real-time ASR（Automated Speech Recognition)

# Description of the Interface

Real-time ASR based on the deep fully convolutional neural network framework (DFCNN), converting audio stream data (within 5 hours) into text stream data results in real-time. It provides a basis for information processing and data mining, which is more suitable for daily dialogue.

• Supported Audio Type: Mono 16bit inter PCM
• Supported Audio Sampling Rate: 16k

# Interface Demo

Interface demo Please click Here to download。 At present, only the demo for some development languages is provided. For other languages, please refer to the interface document below.

# Requirements for the Interface

The following requirements should be met if the stream API for Real-time ASR online is integrated.

# Interface Calling Flow

• Calculate the signature based on hmac-sha256 through the interface key, and send the Websocket protocol handshake request to the server-side. Refer to Interface Authentication below for details.
• After the handshake succeeds, the client can upload and receive data simultaneously through the WebSocket connection. After the data is completely uploaded, the client should upload the end-of-data marker once. Refer to Interface Data Transmitting and Receiving below.
• Disconnect the Websocket connection after receiving all the returned indicators of the results from the server-side.

*Note: Observe the following cautions when using Websocket

1. The WebSocket version supported by the server-side is 13. Ensure the framework used by the client supports this version.
2. The type of all frames returned by the server-side is TextMessage, which corresponds to opcode = 1 in the protocol frame of the native WebSocket. Please ensure that the frame type parsed by the client must be this type. Otherwise, try to upgrade the client frame version or replace the technical framework.
3. If there is a framing problem, namely, a JSON data packet is returned to the client by multiple frames, causing the client’s failure in parsing the JSON. In most cases, this is because the client’s framework has a problem in parsing the WebSocket protocol. Therefore, try to upgrade the framework version first, or replace the technical framework.
4. If it is necessary to close the connection after the client session is over, try to ensure that the error code sent to the server-side is the error code 1000 of WebSocket (if clients’ frameworks do not provide an interface for transmitting error code when closing, there is no need to pay attention to this requirement).

# Interface Authentication

In the handshake phase, the requester is required to sign the request, and the server-side verifies the legitimacy of the request through the signature.

# Authentication Method

By adding authentication-related parameters after the request URL. Sample URL：

ws[s]://ist-api-sg.xf-yun.com/v2/ist?authorization=aG1hYyB1c2VybmFtZT0iMTAwSU1FIiwgYWxnb3JpdGhtPSJobWFjLXNoYTI1NiIsIGhlYWRlcnM9Imhvc3QgZGF0ZSByZXF1ZXN0LWxpbmUiLCBzaWduYXR1cmU9IlVSbnk4M3o1elJsNWF1ODl1YXhUL1dGdUtWejZVNkdkWDdDV25SMGdueWc9Ig%3D%3D&date=Tue%2C+18+Dec+2018+09%3A08%3A49+UTC&host=ist-api-sg.xf-yun.com

Authentication Parameters：

· Rules for Generation of Data Parameters

The date must be based on UTC+0 or GMT zone, in RFC1123 format (Wed, 10 Jul 2019 7:35:43 AM GMT). The server-side will check the clock skew for Date, allowing the maximum deviation of 300 seconds. Once such value is exceeded, the request will be rejected.

· Rules for Generation of Authorization Parameters

1）Get the interface keys APIKey and APISecret. They are both 32-bit strings and can be viewed after creating a WebAPI platform application and adding Real-time ASR (Automatic Speech Recognition) service at the console of the iFLYTEK Open Platform.

2）The format of the parameter authorization before encoded with base64(authorization_origin) is as follows.

api_key="$api_key",algorithm="hmac-sha256",headers="host date request-line",signature="$signature"

Besides, api_key is the APIKey got from the console, the algorithm is an encryption algorithm (supports hmac-sha256 only), and headers are parameters participating in signature. A signature is a string obtained by signing the parameter participating in signature using the encryption algorithm and encoding it with base64. See below for details.

*Note: Headers are the parameters participating in signature. Please note that they are fixed-parameter names ("host date request-line"), instead of the values of these parameters.

3）The rules for the original field of signature (signature_origin) are as follows:

The original field of signature is composed of three parameters, i.e. host, date, and request-line which are concatenated in a certain format. The concatenation format is (\n is a line break character, ’:’ is followed by a space):：

host: $host\ndate: $date\n$request-line

Suppose

Requested url = wss://ist-api-sg.xf-yun.com/v2/ist
date = Fri, 25 Feb 2022 03:01:13 GMT

Then, the original field of signature (signature_origin) is:

host: ist-api-sg.xf-yun.com
date: Fri, 25 Feb 2022 03:01:13 GMT
GET /v2/ist HTTP/1.1

4）Sign the signature_origin using the hmac-sha256 algorithm in combination with apiSecret, to obtain the signed summary, i.e. signature_sha.

signature_sha=hmac-sha256(signature_origin,$apiSecret)

Where, apiSecret is the APISecret obtained from the console.

5）Encode the signature_sha with base64 to get the final signature.

signature=base64(signature_sha)

Suppose

APISecret = e6d4824ba9xxxxxxff2b66f7c6738ead	
date = Fri, 25 Feb 2022 03:01:13 GMT

Then, the signature will be

signature=Vcban+QQerK4GVKqGjmx2ZolNtoZUl808/DgrfGB/c8=

6）According to the above information, concatenate the string of authorization before it is encoded with base64（authorization_origin). See the example below:

api_key="4c18179638d2e487b50f3cfd129ffaca", algorithm="hmac-sha256", headers="host date request-line", signature="Vcban+QQerK4GVKqGjmx2ZolNtoZUl808/DgrfGB/c8="

Note: Headers are the parameters participating in signature. Please note that they are fixed-parameter names ("host date request-line"), instead of the values of these parameters.

7）Finally, encode the authorization_origin with base64 to get the authorization parameter.

authorization = base64(authorization_origin)
Example：
authorization=YXBpX2tleT0iNGMxODE3OTYzOGQyZTQ4N2I1MGYzY2ZkMTI5ZmZhY2EiLCBhbGdvcml0aG09ImhtYWMtc2hhMjU2IiwgaGVhZGVycz0iaG9zdCBkYXRlIHJlcXVlc3QtbGluZSIsIHNpZ25hdHVyZT0iVmNiYW4rUVFlcks0R1ZLcUdqbXgyWm9sTnRvWlVsODA4L0RncmZHQi9jOD0i

# Examples of Authentication URL Code

golang

//@hosturl :  like  wss://ws-api.xfyun.cn/v2/iat
//@apikey : apiKey
//@apiSecret : apiSecret
//@method: request method , GET 、POST ...
func assembleAuthUrl(hosturl string,method, apiKey, apiSecret string) string {
    ul, err := url.Parse(hosturl)
    if err != nil {
        fmt.Println(err)
    }
    //Signing Date
    date := time.Now().UTC().Format(time.RFC1123)
    //fields participating in signature: host ,date, request-line
    signString := []string{"host: " + ul.Host, "date: " + date, method + " " + ul.Path + " HTTP/1.1"}
    //String consisting of concatenated signatures
    sgin := strings.Join(signString, "\n")
    //Signature results
    sha := HmacWithShaTobase64("hmac-sha256", sgin, apiSecret)
    //Construct request parameters, and urlencoding is not required now
    authUrl := fmt.Sprintf("api_key=\"%s\", algorithm=\"%s\", headers=\"%s\", signature=\"%s\"", apiKey,
        "hmac-sha256", "host date request-line", sha)
    //Encode the request parameters with base64
    authorization:= base64.StdEncoding.EncodeToString([]byte(authUrl))

    v := url.Values{}
    v.Add("host", ul.Host)
    v.Add("date", date)
    v.Add("authorization", authorization)
    //Add the encoded string url encode after url
    callurl := hosturl + "?" + v.Encode()
    return callurl
}

func HmacWithShaTobase64(algorithm, data, key string) string {
    mac := hmac.New(sha256.New, []byte(key))
    mac.Write([]byte(data))
    encodeData := mac.Sum(nil)
    return base64.StdEncoding.EncodeToString(encodeData)
}

java:

package com.iflytek.webgatews.wsclient;


import okhttp3.*;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.io.IOException;
import java.net.URL;
import java.net.URLEncoder;
import java.nio.charset.Charset;
import java.security.MessageDigest;
import java.text.SimpleDateFormat;
import java.util.*;

/**
 * @Author:sjliu7
 * @Date:2019/7/31 15:23
 */
public class AuthUtils {

    /**
     * Generate URL,websocket interface for authentication
     * @param requestUrl
     * @param apiKey
     * @param apiSecret
     * @return final requestUrl
     */
    public static String assembleRequestUrl(String requestUrl,String method, String apiKey, String apiSecret) {
        URL url = null;
        String  httpRequestUrl = requestUrl.replace("ws://", "http://").replace("wss://","https://" );
        try {
            url = new URL(httpRequestUrl);
            SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss z", Locale.US);
            format.setTimeZone(TimeZone.getTimeZone("UTC"));
            String date = format.format(new Date());
//            date = "Thu, 19 Dec 2024 07:47:57 GMT";
            String host = url.getHost();
            StringBuilder builder = new StringBuilder("host: ").append(host).append("\n").//
                    append("date: ").append(date).append("\n").//
                    append(method).append(" "). 
                    append(url.getPath()).append(" HTTP/1.1");
            Charset charset = Charset.forName("UTF-8");
            Mac mac = Mac.getInstance("hmacsha256");
            System.out.println(builder.toString());
            SecretKeySpec spec = new SecretKeySpec(apiSecret.getBytes(charset), "hmacsha256");
            mac.init(spec);
            byte[] hexDigits = mac.doFinal(builder.toString().getBytes(charset));
            String sha = Base64.getEncoder().encodeToString(hexDigits);
            String authorization = String.format("hmac username=\"%s\", algorithm=\"%s\", headers=\"%s\", signature=\"%s\"", apiKey, "hmac-sha256", "host date request-line", sha);
            String authBase = Base64.getEncoder().encodeToString(authorization.getBytes(charset));
            return String.format("%s?authorization=%s&host=%s&date=%s", requestUrl, URLEncoder.encode(authBase), URLEncoder.encode(host), URLEncoder.encode(date));

        } catch (Exception e) {
           throw new RuntimeException("assemble requestUrl error:"+e.getMessage());
        }
    }
}

javascript:

function assembleRequestUrl(host,path,apiKey,apiSecret) {
  var url = "wss://"+host+path
  var date = new Date().toGMTString()
  var algorithm = 'hmac-sha256'
  var headers = 'host date request-line'
  var signatureOrigin = `host: ${host}\ndate: ${date}\nGET ${path} HTTP/1.1`
  var signatureSha = CryptoJS.HmacSHA256(signatureOrigin, apiSecret)
  var signature = CryptoJS.enc.Base64.stringify(signatureSha)
  var authorizationOrigin = `api_key="${apiKey}", algorithm="${algorithm}", headers="${headers}", signature="${signature}"`
  var authorization = btoa(authorizationOrigin)
  url = `${url}?authorization=${authorization}&date=${date}&host=${host}`
  return url
}

*python

from datetime import datetime
from wsgiref.handlers import format_date_time
from time import mktime
import hashlib
import base64
import hmac
from urllib.parse import urlencode
import os
import traceback
import json

class AssembleHeaderException(Exception):
    def __init__(self, msg):
        self.message = msg


class Url:
    def __init__(this, host, path, schema):
        this.host = host
        this.path = path
        this.schema = schema
        pass


# calculate sha256 and encode to base64
def sha256base64(data):
    sha256 = hashlib.sha256()
    sha256.update(data)
    digest = base64.b64encode(sha256.digest()).decode(encoding='utf-8')
    return digest


def parse_url(requset_url):
    stidx = requset_url.index("://")
    host = requset_url[stidx + 3:]
    schema = requset_url[:stidx + 3]
    edidx = host.index("/")
    if edidx <= 0:
        raise AssembleHeaderException("invalid request url:" + requset_url)
    path = host[edidx:]
    host = host[:edidx]
    u = Url(host, path, schema)
    return u


# build websocket auth request url
def assemble_ws_auth_url(requset_url, method="GET", api_key="", api_secret=""):
    u = parse_url(requset_url)
    host = u.host
    path = u.path
    now = datetime.now()
    date = format_date_time(mktime(now.timetuple()))
    print(date)
    # date = "Thu, 12 Dec 2019 01:57:27 GMT"
    signature_origin = "host: {}\ndate: {}\n{} {} HTTP/1.1".format(host, date, method, path)
    # print(signature_origin)
    signature_sha = hmac.new(api_secret.encode('utf-8'), signature_origin.encode('utf-8'),
                             digestmod=hashlib.sha256).digest()
    signature_sha = base64.b64encode(signature_sha).decode(encoding='utf-8')
    authorization_origin = "api_key=\"%s\", algorithm=\"%s\", headers=\"%s\", signature=\"%s\"" % (
        api_key, "hmac-sha256", "host date request-line", signature_sha)
    authorization = base64.b64encode(authorization_origin.encode('utf-8')).decode(encoding='utf-8')
    # print(authorization_origin)
    values = {
        "host": host,
        "date": date,
        "authorization": authorization
    }

    return requset_url + "?" + urlencode(values)

# Authentication Results

If the handshake successes, an HTTP 101 status code will be returned, indicating that the protocol upgrade is successful; if the handshake fails, different HTTP Code status codes will be returned, depending on the type of the error, along with the error message. See the detailed description of errors in the table below.

Example of returned messages upon failure of handshake:

    HTTP/1.1 401 Forbidden
    Date: Thu, 06 Dec 2018 07:55:16 GMT
    Content-Length: 116
    Content-Type: text/plain; charset=utf-8
    {
        "message": "HMAC signature does not match"
    }

# Interface Data Transmitting and Receiving

• After the handshake succeeds, the WebSocket connection will be established between the client and the server-side, through which, the client can upload and receive data simultaneously. For the uncompressed PCM format, the interval for sending audios should be 40ms, and 1280B audio bytes are sent each time; When the server has the recognition result, it will push the recognition result to the client through the WebSocket connection.
• End-of-data Marker： The last frame of audio should be ended with the marker，status=2：

{
    "data":{
        ......
        "status":2
    }
}

• If no data is sent for more than 10s, the server-side actively close the connection.

• If the connection is closed for any reason during the session, the session ends and unrecoverable. If the client continues to use the transcription service, it needs to re-establish the connection and open a new session.

• The server will use the status field as the last frame of data. When status=2, it means that all data has been sent, and the client should stop sending data and disconnect immediately after processing the frame. See the details below:

  {
  "data":{
   "result":{..},
   "status":2
  }
  }
  

# Request Parameters

All the request data is JSON string.

# Description of Common Parameter

Description of Business Parameter

# List of Parameters：

• Data Parameter Description：

# Example of Returned Parameter

Default (enable wpgs)

{
  "code": 0,
  "message": "success",
  "sid": "iat000704fa@dx16ade44e4d87a1c802",
  "context_id":"135e2a1c",
  "data": {
    "result": {
      "bg": 0,
      "ed": 0,
      "ls": false,
      "pgs": "rpl",
      "rg": [
        1,
        1
      ],
      "sn": 2,
      "ws": [
        {
          "bg": 0,
          "cw": [
            {
              "sc": 0,
              "w": "Hello"
            }
          ]
        },
        {
          "bg": 0,
          "cw": [
            {
              "sc": 0,
              "w": "Great"
            }
          ]
        }
      ]
    },
    "status": 1
  }
}

# Returned Parameters Description

• 10163: Parameter verification failed, caused by client-side. Please change the request parameters according to the description in the returned message.
• 10313: The first frame of the request parameter failed to transmit app_id, or the transmitted app_id does not match api_key. Check if the common parameter is uploaded correctly, or if the app_id parameter in common is uploaded correctly or is empty.
• 10043: Audio decoding fails. Please ensure that the transmitted audio encoding format is consistent with the requested parameter.
• 11201: The usage of the interface has exceeded the purchased maximum limit, please continue to use it after re-purchase