Introducing TOSS: Achieving Eventual Consistency of Edges in NebulaGraph

NebulaGraph has just released v.2.6. In this version, TOSS is certainly one of the important features. Here a detailed explanation about TOSS will be given.

Let's start from a GO statement

As we all know, there are two types of edges, directed and undirected edges. When traversing directed edges, you can traverse forward or reversely. NebulaGraph also supports this semantics. For example:

go from "101" over known reversely yield known.kdate, id($$);

The above statement starts from Vertex 101 to find all the corresponding neighbors reversely. However, when you insert an edge into NebulaGraph, the command will be like:

insert edge known(degree) VALUES "100" -> "101":(299792458);

Seemingly, the above statement only specifies the outgoing edge. This is because NebulaGraph will specify the incoming edge in the background when you insert an edge.

How to insert an edge into NebulaGraph

Take the INSERT statement above as an example, the background execution process contains the following:

• Nebula Console sends the INSERT request to the NebulaGraph server.

• After the NebulaGraph server receives the request, it adds an incoming edge for each outgoing edge and sends AddEdgeRequest to their hosts respectively.

• After the host (Nebula Storage server) receives AddEdgeRequest, it inserts the edge locally (via the Raft protocol) and returns the result to the NebulaGraph server.

• The NebulaGraph server then returns the results from both hosts to the Nebula Console for querying purpose.

The flow diagram is as follows:

If you are familiar with network/distributed programming, you may see the problem now. The graph service uses RPC to call both storage services. When the INSERT operation is executed enough times, one RPC succeeds while the other fails due to timeout. In other words, an INSERT operation may succeed on the outgoing edge while fail on the incoming-edge.

If now a user requires consistent property settings for both outgoing edge and incoming edge, the client has to retry the query infinitely. It is inappropriate for NebulaGraph, s a database product, to rely on the client for data atomicity.

A requirement thus comes into being, that is, to ensure the atomicity of outgoing edge and incoming edge. This means that the outgoing edge and incoming edge should be updated either successfully or they should fail at the same time. And TOSS (Transaction On Storage Side) is the feature to ensure the eventual consistency of edges upon INSERT, UPDATE, and UPSERT operations.

How to use TOSS

With the release of NebulaGraph v2.6.0, the TOSS function has also been launched. The feature is set to Disabled by default due to performance and stability considerations. You can find the enable_experimental_feature option in the NebulaGraph server configuration file and set it to True. Then you need to restart the graphd service for the feature to take effect. The command is as follows:

--enable_experimental_feature=true

Then the operations CREATE SPACE / CREATE EDGE / INSERT / UPDATE will achieve eventual consistency of edges in NebulaGraph. (Just execute the operations as before)

Note: The TOSS feature will be only applied to incremental data after it is enabled.

More information about v2.6, please refer to NebulaGraph Database Manual. It records in detail the knowledge points and specific usage of the graph database and the graph database NebulaGraph.

Welcome to join our Slack channel if you want to discuss with the rest of the NebulaGraph community!