Secure data synchronization: The Briteback Approach

Keeping data synchronized in real-time between clients is something that needs to be done in all apps dealing with communication or collaboration. There are lots and lots of frameworks, libraries, tools and patterns to make it happen. All with their own way to handle data flow and security. Some frameworks have a very strict structure and some will only handle transport and nothing more. In this post I'll explain how we do this in Briteback.

Briteback is an enterprise messaging app, with integrated email, online calls and calendar. There are also integrations to task management systems, etc. It works on web, Android, and iOS. Basically, what we're trying to do is to make messaging work for larger organizations in a way that really scales in terms of number of employees. This means that we have hard requirements, both in terms of security and speed.

Briteback is built with JavaScript all the way, from the tool-chains to servers to frontend. I must admit that we have some Java and Objective C in our mobile apps but we prefer not talk about that.

We are always trying to embrace new technology and are not afraid to explore new ways to build our app. We do prefer small libraries that does one thing in favor for big solve-it-all frameworks. By using small components we can easily replace parts of our systems with our own code (when we decide that there's nothing good enough out there) or with other libraries as the JavaScript world is pushing forward.

This post describes how we have solved secure data-synchronization in real-time and what libraries we use. I’m not trying to tell you what the best approach is, just how we do it. But I do think our approach works very well.

Secure data-synchronization in real-time

Secure in this context means that a user only can read and change what he or she is allowed to and that the data is in the correct format.

So data synchronization makes sure the data is the same on all clients and servers at all times. That is something you want for many reasons, not least the user experience.

What does real-time really mean? I will not even try to define it, there are already plenty of attempts (here, here, and here). But for us it means, well, fast enough (I guess what we really mean is “soft real-time” anyway).

The stack

Our stack to handle the data-synchronization from server to the UI looks like this:

Servercluster with rethinkDB, node and all that

Deepstream

https://deepstream.io
Deepstream is a library that provides fast and scalable communication between servers and clients. Under the hood it is using websockets as transport. Performance is outstanding and so is the scope and the flexibility. It handles data transport persistence and does not interfere with our way to structure the rest of our backend or frontend.

Out of the box it provides atomic data-objects called "records" which sync their state across all connected clients and servers in realtime, pub-sub and request/response.

To wrap Deepstream calls in promises we have developed a small tool available on GitHub.

NestedTypes

http://volicon.github.io/NestedTypes/
NestedTypes is a high-performance model framework, evolved from backbone but much more performant and with a lot of extra goodness such as type checking, transactions, relations, nested models and collections. Here is a great introduction.

We were using backbone in the early days and moved over to nestedTypes mostly because of the nested models.

NestedReact

https://github.com/Volicon/NestedReact
NestedReact allows you to follow a more traditional MVC approach to an application architecture while taking the full benefits of React's unidirectional data flow (including automatic "pure render" optimization). Nested react is tightly coupled with NestedTypes as it uses NestedTypes models as React states. It makes possible handling fairly complex UI state using the same technique as used inside of the data layer.

React

No further explanation is needed, one of our best technical decisions ever when we abandoned everything view-like and rewrote the entire UI with React.

A simple example

As a simple example, consider changing the user avatar, as illustrated in the gif below. Of course, our approach works well also for much more complex scenarios. Check out the other gif later in this post.

Now let's set the stage by going through a simple approach. After that I'll describe how we do it.

The naive approach

Just update the record straight on the client. The data is synchronized to the server and all other clients, perfect!

The view

<ContactAvatar  
  model={this.props.model}
  image={this.state.gravatarImage}
  onClick={this.props.model.saveAvatarUrl}
/>



Contact model
The record (Deepstream) is fetched before creating the model (NestedTypes)

initialize() {  
  this.record.subscribe('avatarUrl', url => this.avatarUrl = url);
  ...
},

saveAvatarUrl(avatarUrl) {  
  this.record.set('avatarUrl', avatarUrl);
}

Now everyone can set any attributes on every record. That's not good enough. How do we solve that?

The Deepstream solution is that you either write static rules or write your own custom permission handler where arbitrary code can be used (Deepstream tutorials). We have chosen to go with the code way since we have a very complex structure where the rules cannot be determined without checking other records. We have an organization hierarchy, roles with permissions, etc, etc.

A simple example is with the user's own contact record. We check if the id on the record is the same as the id of the logged in user, if it is we grant read permissions.

Part of our permission check for user records

switch (message.action) {  
  case DSConstants.ACTIONS.READ:
      if (user === sessionUserId) {
        return Promise.resolve();
      }

/* More advanced checks here that involve reading records.
The read permission for other contacts is more complex,  
it involves checking if the two users have any organizations  
in common, and if one of the users is a guest in the  
organization they must also share a team, etc. */

  default:
      return Promise.reject(‘No permission’);

We can also grant write permissions the same way. If we do, a user can only manipulate his or her own contact record.

  case DSConstants.ACTIONS.PATCH:
  case DSConstants.ACTIONS.UPDATE:
      if (user === sessionUserId) {
         return Promise.resolve();
      }
      return Promise.reject(‘No permission);

All good and we are done. Or, are we really? Keep reading...

So what are the benefits of the "set direct on record"-approach?

  • Easy to understand, change the data? Just set it!
  • No extra overhead on the server, most logic is on the client.
  • A user can only update the records he or she is allowed to.

But there are some problems:

  • Security
    • No data validation.
    • No control over what attributes that can be changed, everything on a record can be edited.
    • No fine-grained permission, only one record at a time. What if a user should be able to update attribute X but not Y on the very same record?
  • Complex actions involving many records will be problematic. All have to be done manually on the client. And if there are other non record-set actions associated with the action (sending a notification, etc) they have to be carried out in another way.

How can this be solved then? Let's hear it for...

The Briteback approach

We reject all client attempts to update, patch, delete or create a record (client as in frontend-client; other servers can do it). Instead we use RPCs on the backend for every client action.

The flow then looks something like this:

Client action
For example a click event in a React view

RPC on server
Check user permissions
Validate data
Update records
Perform related actions such as sending emails and notifications

NestedTypes Model
Receive updated data
Do complex updating of its own data
Automatically triggers change events to those who cares

React view
Listen for changes
Rerenders

And there we have it: Unidirectional data flow!

This is good for us mainly because of four reasons:

  • We can validate all inputs, only set legit values on legit attributes.
  • Validate the user's permission per action + data instead of per record.
  • Have complex updating functionality, where multiple records are involved, on a server instead of on clients.
  • Uniform interface towards clients, we use this pattern for everything triggered by a client.

Permissions can be granted for RPCs too in the same way as records. What we have done is that clients are permitted to call every RPC (except some server-only) and not permitted to provide any. The actual permission check then happens in the RPC callback itself instead since there we know the data and we need to get the relevant records anyway. So it is easier, and more flexible.

And here is the code for updating the avatar with this approach:

Contact model

  saveAvatarUrl(avatarUrl) {
    dsUtils.rpc.make('user:set-avatarUrl', {
      avatarUrl
    })
    .catch(err => console.error('Failed to set avatar', err));
  },



Rpc handler on server

  /**
   * Set the avatar
   * @param {Object} data
   * @param {String} data.userId 
   * userId is added in a data transform function
   * called on every RPC request
   * @param {String} data.avatarUrl
   * @param {Object} response
   */
  deepstreamClient.rpc.provide('user:set-avatarUrl', (data, response) => {
    response.autoAck = false;

    if (data.avatarUrl === undefined
        || typeof data.avatarUrl !== 'string' ) {
      // More checks here, length, is valid URL etc
      return response.error('Wrong data when trying to set avatarUrl');
    }

/* The userId is from the session and not something that is provided
from the client.  
Therefore a user can only change its own avatar and we don't have to  
do any permission checks in this particular example */

    return dsUtils.record.getRecord(`user/${data.userId}`) 
      .then(userRecord => {
        userRecord.set('avatarUrl', data.avatarUrl);
        userRecord.discard();
        response.send();
      })
      .catch(error => {
        logger.error({ filename: __filename, RPC: 'user:set-avatar' }, error);
        response.error();
      });
  });

Of course this example is really the simplest setting to set, but the principle is the same for everything just with more complex permission checks and more functionality in the RPC and in the models.

This gif shows how organizational permissions are propagated to another user and how a new messaging channel gets propagated:

Conclusion

We have achieved unidirectional data flow the entire way from the backend to the DOM and still have a powerful structure that is very easy to understand and work with.

Deepstream gives us the real-time data synchronization between servers and clients in a very performant and controlled way allowing us to focus on application logic.

NestedTypes bundled with NestedReact is the glue that connects everything on the frontend, both data from our servers served with Deepstream and other data from 3rd party APIs can be reacted on in the React views.

One additional benefit with this approach is that we don't have to care about optimistic vs pessimistic pros and cons. We do neither, the data flows from the server the same way for every client, even the one that triggers the change so we can just send the request to the server and then nothing more, no pending state. The attribute will change when the server emits the new value. We still capture errors from the RPC and maybe displays a toast or logs something but we don't have to handle the actual value for the attribute since we didn't touch it.

There are other ways to handle the permissioning with Deepstream and in other types of application choosing another solution might make more sense. For us security is of the highest importance and having permissioning and validation logic that is easy to follow and work with is one way to make sure we are not doing mistakes.

Some might argue that using models the way we do, combining data and actions is not the most modern way to do it. But it suites our application very well keeping relevant logic and data together. What do you think?



comments powered by Disqus