Sign in Subscribe
thrift

Apache Thrift with JavaScript

Some extra tips and documentation on using Apache Thrift with JavaScript, as the Thrift documentation is pretty scant.

John Hartley

• 10 min read
Apache Thrift with JavaScript

Status - Nov 2022: Write up on findings and gotchas from using Thrift with JavaScript

Background - Back to Binary on Wire

While working on a concept for integration server an a 5G Enterprise application. I found myself trying to explain many times, I saw the exposed service running. The feedback from the techncal team was, "can you please provide a specification ..".  So as my high level outlines was not doing the job, I decided to go down a level and provide a specification that:

  1. Defined data that needed to be exchanged
  2. Define the methods that would be exposed to the service clients.

Now I could to this by: defining some psuedo code style type specification and a set of methods/functions, providing a set of Java Class definintions, creating a JSON definition or an OpenAPI compliant definition. But none of these particularly appealed as:

  • Psuedo code - likely too "vague" for highly technical audience
  • Java - too language specific and not aligned with team preferred programming languages (python and go)
  • JSON - defines data not services and is not native to any programming language (ie I need to explicity encode/decode it)
  • OpenAPI - inherently flawed as it defining on "on wire" interface, while I wanted a programming API (i.e. API that can be used in native programming language, be that C/C++, Java, python, go or what ever)

Doing a little bit a searching around I landed on "Apache Thrift". This has a pretty simple philosphy:

  • Define your service with IDL (Interface Definition Language) and then
  • Compile to your preferred implementation language.

This appealed to me as I have history with programming using: SUN/ONC RPC (has interface definition language but only compiled to 'C'), CORBA (has IDL and concept of "language bindings"), DCE/RPC (has IDL and compiles to 'C') and when the world turned to SOAP and Web Service I found myself having to write a lot of code that was about pack/unpacking message on/off the wire, where previously this mechanical stuff was handled by the compiler.

So SOAP, Web Service, Restful HTTP/JSON was a step foward in terms of getting data through routers and across network, but to my mind was a step back in terms of code needing to be written and tooling required to support it.

I mention tooling, as to hide the on wire complexity of the Web Services API definition, the job got delegated to frameworks and tools that became part of the Integrated Development Environment (IDL). This was in contrast to IDL based approach of having a pretty simple command line tool to run an IDL compiler. If I need something like "Swagger" to just read the API specificati0n vs. just being able to read a definition then something is wrong.

It appears that with Apache Thrift and Google's Protocol Buffers on wire encode/decode is heading back to where it should be .... mostly invisible ;-) and binary encoding is once more possible and being used to get to significant performance benefits.

In my use use case the team used Thrift with:

  • Java - for the server implementation
  • python - for client test harness and data injection
  • JavaScript - for data load and setup utility and UI

Due to its JavaScript's thread/event loop model, I found that using Thrift with JavaScript took a bit more effort than with Java and python hence these notes.

Tips on Thrift and JavaScript

These tips on using Thrift and JavaScript are focused creating JavaScript Thrift clients as my our case the server was written in Java and the client was JavaScript. In my case I am using Ubuntu 22.04, Thrift v0.16.0 and Node v16.18.1.

Generating JavaScript for NodeJS (Server-Side JavaScript)

One of strength of Thrift is that it will let use your preferred language. This is controllered through --gen flag. For JavaScript this can take the follow options:

  • node - generate for nodejs (used)
  • es6 - generate for es6 compliance (used)
  • with_ns - generate with name space (not used)

The other Thrift item that is important is the -o output option, as node is particular about the directory structure and you cannot include local module (files) that are outside of the JavaScript file hiearchy.

I found that including name spaces (with_ns flag & "namespace js <NS>" in Thrift file) interferred with es6 generation, so my command command line was: "thrift --gen js:node,es6 -o <DIR_TREE_LOC> <MY-THRIFT-FILE>.thrift"

As I did not have name space flag on I did not use any JavaScript namespace directive in the thrift source files either.

The compiler will put the generated JavaScript code in directory: "<DIR_TREE_LOC>/gen-nodejs"

If you wish to have your Thrift client code using ES6 "strict" mode and have ES6 modules and import/export, then you will need to have JavaScript code directory structure so it seperates the Thrift generated "CommonJS module" code which uses "require" (rather then ES6 import) from the ES6 compliant code using import/export.

See by Blog - "JavaScript & Canonical Web Architecture" for overview of JavaScript Modules and how to manage these using "package.json" file.

IDL to JavaScript Type Mapping

The Thrift IDL types map pretty directly to JavaScript. As JavaScript is dynamically typed any variable can take on any type. The type is determined by the type or literal it was assigned from.

You can pass a JavaScript literal created object ("let obj = {...};"), into a thrift generated type constructor ("let thriftObj = new ttThrift.ThriftType(obj);") and it will copy all the data correctly into the object, including thrift "set", "map" & "list" collections.

The mapping for Thrift "set", "map" and "list" are:

  • "set" == JavaScript array: "['dog', 'cat', 'fish']", not a JavaScript ES6 "Set" class
  • "map" == JavaScript associative array/class: "{'dog':'barks', 'cat':'meows','fish':'bubbles'}", not a JavaScript ES6 "Map" class
  • "list" == JavaScript array "[{type:'dog',noise:'bark'},{type:'cat':noice:'meow'}]", so array again and illustrated as thrift list of pet sounds: list<petsounds>

The thrift "enum" mapping is to an associative array. Thus: "enum temp = {hot, cold, tepid}" becomes JavaScript associative array: "ttypes.temp = {'hot':0,'cold':1,'tepid':2}".

To enumerate through the available values: "for (let ek of ttypes.temp) { console.log(`enum: ${ek}`); }" (where ek == enum keys)

Handling Asynchronous Thrift Calls

In "JavaScript & Canonical Web Architecture" article in section  "JavaScript - the Language" provides a summary and example of NodeJS's event loop thread model and use of Promises to handle asychronous calls to the Thrift server.

As that example was extracted from work creating Web UI and it does not include all the details. To provide more details I use the "Thrift Calcuator" tutorial example here (slightly modified) for illustration ...

Lets start with the Thrift IDL definition, unchanged from tutorial:

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements. See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership. The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License. You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied. See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

# Thrift Tutorial
# Mark Slee (mcslee@facebook.com)
#
# This file aims to teach you how to use Thrift, in a .thrift file. Neato. The
# first thing to notice is that .thrift files support standard shell comments.
# This lets you make your thrift file executable and include your Thrift build
# step on the top line. And you can place comments like this anywhere you like.
#
# Before running this file, you will need to have installed the thrift compiler
# into /usr/local/bin.

/**
 * The first thing to know about are types. The available types in Thrift are:
 *
 *  bool        Boolean, one byte
 *  i8 (byte)   Signed 8-bit integer
 *  i16         Signed 16-bit integer
 *  i32         Signed 32-bit integer
 *  i64         Signed 64-bit integer
 *  double      64-bit floating point value
 *  string      String
 *  binary      Blob (byte array)
 *  map<t1,t2>  Map from one type to another
 *  list<t1>    Ordered list of one type
 *  set<t1>     Set of unique elements of one type
 *
 * Did you also notice that Thrift supports C style comments?
 */

// Just in case you were wondering... yes. We support simple C comments too.

/**
 * Thrift files can reference other Thrift files to include common struct
 * and service definitions. These are found using the current path, or by
 * searching relative to any paths specified with the -I compiler flag.
 *
 * Included objects are accessed using the name of the .thrift file as a
 * prefix. i.e. shared.SharedObject
 */
include "shared.thrift"

/**
 * Thrift files can namespace, package, or prefix their output in various
 * target languages.
 */

namespace cl tutorial
namespace cpp tutorial
namespace d tutorial
namespace dart tutorial
namespace java tutorial
namespace php tutorial
namespace perl tutorial
namespace haxe tutorial
namespace netstd tutorial

/**
 * Thrift lets you do typedefs to get pretty names for your types. Standard
 * C style here.
 */
typedef i32 MyInteger

/**
 * Thrift also lets you define constants for use across languages. Complex
 * types and structs are specified using JSON notation.
 */
const i32 INT32CONSTANT = 9853
const map<string,string> MAPCONSTANT = {'hello':'world', 'goodnight':'moon'}

/**
 * You can define enums, which are just 32 bit integers. Values are optional
 * and start at 1 if not supplied, C style again.
 */
enum Operation {
  ADD = 1,
  SUBTRACT = 2,
  MULTIPLY = 3,
  DIVIDE = 4
}

/**
 * Structs are the basic complex data structures. They are comprised of fields
 * which each have an integer identifier, a type, a symbolic name, and an
 * optional default value.
 *
 * Fields can be declared "optional", which ensures they will not be included
 * in the serialized output if they aren't set.  Note that this requires some
 * manual management in some languages.
 */
struct Work {
  1: i32 num1 = 0,
  2: i32 num2,
  3: Operation op,
  4: optional string comment,
}

/**
 * Structs can also be exceptions, if they are nasty.
 */
exception InvalidOperation {
  1: i32 whatOp,
  2: string why
}

/**
 * Ahh, now onto the cool part, defining a service. Services just need a name
 * and can optionally inherit from another service using the extends keyword.
 */
service Calculator extends shared.SharedService {

  /**
   * A method definition looks like C code. It has a return type, arguments,
   * and optionally a list of exceptions that it may throw. Note that argument
   * lists and exception lists are specified using the exact same syntax as
   * field lists in struct or exception definitions.
   */

   void ping(),

   i32 add(1:i32 num1, 2:i32 num2),

   i32 calculate(1:i32 logid, 2:Work w) throws (1:InvalidOperation ouch),

   /**
    * This method has a oneway modifier. That means the client only makes
    * a request and does not listen for any response at all. Oneway methods
    * must be void.
    */
   oneway void zip()

}

/**
 * That just about covers the basics. Take a look in the test/ folder for more
 * detailed examples. After you run this file, your generated code shows up
 * in folders with names gen-<language>. The generated code isn't too scary
 * to look at. It even has pretty indentation.
 */

This example has no JavaScript name space defined and so can be used to create an ES6 compliant Thrift client.

To build the example I:

  1. Create build tree with structure to that generated code has "package.json" with type == "commonjs" and Node client has "package.json" with type == "module"
  2. Build the Calculator Server using Java
  3. Updated the Calculator Client (NodeClientPromise.js) slightly to use ES6 imports, Promises and let.
  4. Generated the Calcuator JavaScript client code using: "-r -gen js:node,es6"

The resulting "slightly" updated client...

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements. See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership. The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License. You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied. See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

import * as thrift from 'thrift'
import * as Calculator from '../gen-nodejs/Calculator.js'
import def_ttypes, * as ttypes from '../gen-nodejs/tutorial_types.js';

let transport = thrift.TBufferedTransport;
let protocol = thrift.TBinaryProtocol;

let connection = thrift.createConnection("localhost", 9090, {
  transport : transport,
  protocol : protocol
});

connection.on('error', function(err) {
  assert(false, err);
});

// Create a Calculator client with the connection
let client = thrift.createClient(Calculator, connection);


client.ping()
  .then(function() {
    console.log('ping()');
  });

client.add(1,1)
  .then(function(response) {
	  console.log("1+1=" + response);
  });

let work = new ttypes.Work();
work.op = def_ttypes.Operation.DIVIDE;
work.num1 = 1;
work.num2 = 0;

client.calculate(1, work)
  .then(function(message) {
	console.log('Whoa? You know how to divide by zero?');
  })
  .catch(function(err) {
    console.log("InvalidOperation " + err);
  });


work.op = def_ttypes.Operation.SUBTRACT;
work.num1 = 15;
work.num2 = 10;

client.calculate(1, work)
  .then(function(value) {
	  console.log('15-10=' + value);
	  return client.getStruct(1);
  })
  .then(function(message) {
      console.log('Check log: ' + message.value);
  })
  .finally(function() {
	  //close the connection once we're done
      connection.end();
  });

The updated client is ES6 "strict" compliant, using:

  • import (with def_ttype to provide access the "commonjs" default definitions)
  • Promises (change "fail" to "catch" and "fin" to ES2018 "finally" statements and
  • var to let

The git diff below shows changes to tutuorial "thrift/tutorial/nodejs/NodeClientPromise.ps" make it ES6 "strict" compliant:

$ git diff NodeClientPromise.js
diff --git a/tutorial/nodejs/NodeClientPromise.js b/tutorial/nodejs/NodeClientPromise.js
index 2cdc184f9..fe8e32d5d 100644
--- a/tutorial/nodejs/NodeClientPromise.js
+++ b/tutorial/nodejs/NodeClientPromise.js
@@ -17,15 +17,14 @@
  * under the License.
  */
 
-var thrift = require('thrift');
-var Calculator = require('./gen-nodejs/Calculator');
-var ttypes = require('./gen-nodejs/tutorial_types');
-const assert = require('assert');
+import * as thrift from 'thrift'
+import * as Calculator from '../gen-nodejs/Calculator.js'
+import def_ttypes, * as ttypes from '../gen-nodejs/tutorial_types.js';
 
-var transport = thrift.TBufferedTransport;
-var protocol = thrift.TBinaryProtocol;
+let transport = thrift.TBufferedTransport;
+let protocol = thrift.TBinaryProtocol;
 
-var connection = thrift.createConnection("localhost", 9090, {
+let connection = thrift.createConnection("localhost", 9090, {
   transport : transport,
   protocol : protocol
 });
@@ -35,7 +34,7 @@ connection.on('error', function(err) {
 });
 
 // Create a Calculator client with the connection
-var client = thrift.createClient(Calculator, connection);
+let client = thrift.createClient(Calculator, connection);
 
 
 client.ping()
@@ -48,8 +47,8 @@ client.add(1,1)
          console.log("1+1=" + response);
   });
 
-work = new ttypes.Work();
-work.op = ttypes.Operation.DIVIDE;
+let work = new ttypes.Work();
+work.op = def_ttypes.Operation.DIVIDE;
 work.num1 = 1;
 work.num2 = 0;
 
@@ -57,12 +56,12 @@ client.calculate(1, work)
   .then(function(message) {
        console.log('Whoa? You know how to divide by zero?');
   })
-  .fail(function(err) {
+  .catch(function(err) {
     console.log("InvalidOperation " + err);
   });
 
 
-work.op = ttypes.Operation.SUBTRACT;
+work.op = def_ttypes.Operation.SUBTRACT;
 work.num1 = 15;
 work.num2 = 10;
 
@@ -74,7 +73,7 @@ client.calculate(1, work)
   .then(function(message) {
       console.log('Check log: ' + message.value);
   })
-  .fin(function() {
+  .finally(function() {
          //close the connection once we're done
       connection.end();
   });

The following directory structure was used to test this. This illustrates the placement of two "package.json" files to support the different "module" types ("commonjs" & "module") :

Apache Thrift - Calculator Tutorial (Updated for ES6 import & Promises)

This is required, as you can import "commonjs" modules using ES6 import with NodeJS. However when in ES6 "strict" mode you can cannot use "require" statements in ES6 "module".

Note, that ES6 "then" and "catch" and "finally" are invocations of methods on the Promise object returned from the Thrift call. The Promise provide these methods to establish callbacks in a consistent manner. So while the code may look structurally simillar to a "try/catch" statement, its behaviour is significantly different due to the asynchronous execution model.

References & Links:

Apache Thrift - the offical Apache Thrift site, with pretty spartan documentation

Programmer's Guide to Apache Thrift - Randy Abernathy - Avaiable as eDoc or printed book and has good elaboration of Thrift Server Thread Model, optional for client side only use of Thrift, useful for writing Thrift servers