
![NebulaGraph v2.0 GA Overview](https://user-images.githubusercontent.com/57335825/112122642-74f1fc00-8bfb-11eb-977f-557baf66990f.png)

We have seen more and more users trying to adopt the graph technology since the general availability of NebulaGraph 1.0 in June 2020. The development of NebulaGraph 2.0 was brought to the table not long after the initial release as the Nebula team have collected many problems and feature requests from our developers’ community. Following hard work for nearly one year and several beta versions, NebulaGraph 2.0 is now generally available and can be used in production as a stable version.

# Version Planning

The Nebula team has taken into consideration the following factors when planning what should be included in version 2.0:

- Suggestions and feature requests from current adopters
- Trends in the graph database and big data industry
- Feedback from the community

Finally, the team has made the decision that the development of NebulaGraph 2.0 should focus on the following aspects:

**Better query performance**. High throughput and low latency in handling large data volume is the natural architectural advantage of NebulaGraph. Most users turned to NebulaGraph because other products cannot meet the challenges brought by the growth of their data volume. Therefore, NebulaGraph 2.0 continues to maintain its advantage in large data volume query scenarios.

**Stability is guaranteed**. As more and more users use NebulaGraph in production, any slight system jitter will have a huge impact on their businesses. NebulaGraph always ensures users’ clusters in stable operations as their businesses evolve and data volume increases.

**Enhanced system openness**. NebulaGraph 2.0 added several important features and its underlying logic was optimized to make the system more open. NebulaGraph is now compatible with openCypher so that Neo4j users can use NebulaGraph with a flat learning curve. It can also integrate with more graph analysis algorithms to meet users' needs for offline analysis and graph computing.

**Easier operation and maintenance**. The complexity of operation and maintenance of distributed databases is incomparable to that of standalone ones. NebulaGraph 2.0 now provides users with more operation and maintenance tools to simplify deployment, upgrade, backup, and enable them to monitor the system and locate errors more accurately and faster.

Based on the above-mentioned principles, the team restructured and optimized NebulaGraph, trying to make it a reliable and user-friendly graph database. While some of the changes are visible to users such as better performance and easier operation, some are under the hood and are the building blocks of a more reliable and performant distributed system.

# Performance Improvement

In NebulaGraph 2.0, the data type has been unified across all subsystems such as Storage, Graph, and Client, which can significantly reduce the data type conversion between subsystems or submodules, as well as serialization and deserialization.

In the meantime, the query engine of NebulaGraph 2.0 has been restructured to introduce multiple modules such as syntax analysis, statement optimization and resource scheduling. With syntax analysis, illegal statements can be ended before being passed for further processing, which reduces unnecessary resource consumption and provides users with more readable error information. In NebulaGraph 1.0, the query statements are processed by an independent executor, which significantly limits us from optimizing query execution in the system. Therefore, in NebulaGraph 2.0, we made reasonable abstraction to computation actions in query execution and designed multiple operators. In this way, a query statement can be split into an execution plan consisting of multiple operators and can be optimized via RBO (Rule Based Optimization).  We will continue optimizing the operators and the execution plan in the future for better performance.

We have also redesigned the Scheduler to better allocate resources for execution plans.

# Stability Enhancement

We have done a huge amount of work to guarantee the stability of NebulaGraph 2.0. We have built a new testing framework at the query engine level, including unit testing, integration testing, and performance test. The integration testing framework based on BDD helps us verify integration use cases fast and significantly reduces the difficulty of use case writing. The performance testing framework based on Jmeter is now functioning. The current version can help us verify performance issues quickly. A sound testing framework ensures the stability of the entire system. We have enhanced the chaos testing framework in NebulaGraph 2.0  to ensure the reliability and stability of the storage engine.

# System Openness

In NebulaGraph 2.0, the previous Executor module has been split into multiple independent modules for better encapsulation and testing. The current query engine consists of Parser, Validator, Optimizer, Executor, and Scheduler. The new architecture not only makes NebulaGraph easier for maintenance, but also allows more openness to external feature requests.

In the past year, we have re-written most of the query engine code, making it compatible with both nGQL 1.0 and openCypher. At the same time, we are trying our best to reuse the execution mechanism and code for both nGQL and openCypher compatibility. In Validator, we have enhanced syntax verification so that errors that are captured during execution can be caught during compiling. The Optimizer has partially realized RBO, which makes adding rules to the existing framework easier. Besides, we have re-written the Executor based on asynchronous calling mechanism so that it can focus on computing and aggregating. Tasks like error handling, status saving and task scheduling are all transferred to the execution framework. The query engine is more module based and more open to extension, further reducing the threshold of community participation.

In NebulaGraph 2.0, we also added some built-in algorithms such as shortest path, full path and subgraph. We have also added a lot of scalar functions, aggregate functions, and tabulated functions. NebulaGraph now supports the “string” type vertex ID and NULL, which allow users to express more conveniently. Besides, we have implemented lots of modification and optimization at the storage engine level. The full-text indexing feature has been realized by connecting Elasticsearch with the learner role in RAFT. The multi-site high availability feature has been realized by abstracting Zone and Group in partition. Redesigning the index implementation has significantly improved data import performance with indexes. The above work has laid a solid foundation for continuous improvement of NebulaGraph for the goal of "More stable, Faster, Stronger" in the next year.

# Easy Operation & Maintenance

In terms of operation & maintenance, NebulaGraph 2.0 supports the deployment of Helm Charts. In this way, users can deploy a NebulaGraph cluster with a few simple Helm commands. Meanwhile, since inner components are able to communicate with each other via domain addresses, configuration can be simplified and users only need to provide enough nodes according to their business needs. We have provided a tool for automatic upgrade to help existing users easily migrate from other versions to NebulaGraph 2.0. We are also intensively developing Backup & Restore tools, and the Dashboard feature.

For a detailed release note, please check: [https://nebula-graph.io/posts/nebula-graph-v2.0-ga-release-note/](https://nebula-graph.io/posts/nebula-graph-v2.0-ga-release-note/)

Here is the GitHub repo for NebulaGraph 2.0: [https://github.com/vesoft-inc/nebula-graph](https://github.com/vesoft-inc/nebula-graph). Give it a try! And here is the comprehensive documentation for your reference: [https://docs.nebula-graph.io/2.0/](https://docs.nebula-graph.io/2.0/). 

If you encounter any problem, head to our discussions for help: [discussions](https://github.com/vesoft-inc/nebula/discussions). Or you can join the [Slack](https://community-chat.nebula-graph.io/) channel for discussion.


Following hard work for nearly one year and several beta versions, NebulaGraph 2.0 is now generally available and can be used in production as a stable version.

NebulaGraph v2.0 GA Overview


![Pick of the Week](https://user-images.githubusercontent.com/57335825/87520320-a4e20c00-c637-11ea-8053-7222b9c4f00a.png)

Normally the weekly issue covers NebulaGraph Updates and Community Q&As. If something major happens, it will also be covered in the additional Events of the Week section.

# Events of the Week

1. NebulaGraph v2.0.0-beta release note

NebulaGraph V2.0.0-beta has been released. This release supports full-text indexing, data statistics, and other new features. Also, NebulaGraph Studio and Go Importer now support NebulaGraph 2.x versions.

![NebulaGraph v2.0.0-beta release note](https://user-images.githubusercontent.com/57335825/102967835-c2dc6280-452d-11eb-80a1-ddf18b73308a.png)

To get more information, please read the [release note](https://discuss.nebula-graph.io/t/nebula-graph-v2-0-0-beta-release-note/793).

2. DB-Engines Ranking Has Been Updated

NebulaGraph has risen 3 places and jumped to #15 in the latest DB-Engines ranking.

![DB-Engines Ranking](https://user-images.githubusercontent.com/57335825/102968155-544bd480-452e-11eb-9d42-edc33955b78e.png)

# NebulaGraph Updates

The updates of Nebula in the last week:

- Supports the DeleteRange operation in RocksDB to greatly improves the efficiency of edge deletion. Tags: `Version 1.x`, `Optimization`. For more information, see [PR#2404](https://github.com/vesoft-inc/nebula/pull/2404).

- Fixed the issue where using `FETCH PROP ON` on timestamp properties outputs int64 results. Tags: `Version 1.x`, `bug fix`. For more information, see [PR#2389](https://github.com/vesoft-inc/nebula/pull/2332).

# Community Q&A

This week's topic is from community user @panda about schema design in NebulaGraph.

## Spark Writer Configuration Suggestions

Before using Spark Write to import data, we need to configure `application.conf`.

**@panda**: I am not familiar with graph databases. I used to use MySQL and MongoDB. Now I am having some trouble designing schemas in NebulaGraph.
Suppose we have the following schema (written in GraphQL for convenience):

```shell
// User list
type User {
    name: String,
    followings: [User],
    followers: [User],
    posts: [Post],
    topics: [Topic]
}
// Topic list
type Topic {
    name: String,
    description: String,
    user: User,
    members: [Member]
    posts: [User]
}
// Post list
type Post {
    text: String,
    member: Member,
    topic: Topic
}
// Topic member list
type Member {
    user: User,
    topic: Topic,
    name: String,
    level: Int,
    join_date: DateTime,
    posts: [Post],
}
```

If we design a schema in NebulaGraph based on the preceding data, since there is no concept of table association and we cannot set the associated fields, what is the right way to do this?

Can we retrieve the required scalar and associated fields all at once? Please advise.

Now we are using NebulaGraph v2.x.

**NebulaGraph**: You can set associated fields as edges in NebulaGraph. For example, the `followings` field indicates that a user follows another user, i.e., `User—following—>User`. So you can set the `following` field as an edge type, and insert edges (relationships) of this type to connect the vertices representing the users.

**@panda**: I know these concepts, but this is not what I'm asking. Here are what really puzzle me:

1. Tags and edges are separated and have no strong correlation, then how should we query them? Take the preceding data for example, if we want to find all the data in the post list, shall we search for the tags first, and then search for the edges one by one? Do we need to do this for all the queries?
And so does writing data? It feels super troublesome.
2. Some edges are shared by multiple tags, for example, the `user: User` edge mentioned earlier is used by both Topic and Member. Wouldn't this mess up the data? And how to distinguish them? We don't want to use other names to create new edges.
3. Associated fields usually have a one-to-one association, such as the `user: User` edge mentioned earlier, one-to-many association, such as the followings: [User] edge, and many-to-many association. How are these associations represented in NebulaGraph?
4. For associated lists, we usually need to count the quantity, normally by aggregating the queries or setting a count field alone to hold the quantity. In NebulaGraph,  is there a proper way to handle this? Do we set a count property on edges? For example `CREATE EDGE followings(count int default 0);`?

The official documents are too term-based. They are not based on actual scenes and cases, and many things are not explained in detail, which makes them difficult to understand.

If possible, please make a best practice document based on the preceding schema and introduce how to query and write data in such a case.
 
**NebulaGraph**: First of all, thank you for the scenario you provided. You can require the document from our technical writers.

Let's summarize your scenario as follows:

![schema design](https://user-images.githubusercontent.com/57335825/102968684-4b0f3780-452f-11eb-87bc-045f24fa3b95.png)

> Tags and edges are separated and have no strong correlation, then how should we query them? Take the preceding data for example, if we want to find all the data in the post list, shall we search for the tags first, and then search for the edges one by one? Do we need to do this for all the queries?

And so does writing data? It feels super troublesome.

For now, the scanning by tag feature is not ready, but it will be in the future in MATCH. For example, `MATCH(p:post) RETURN p`. Scanning by tag relies on indexes and will be quite memory-consuming. If the data volume is large, OOM may happen frequently.

To write data, simply set the properties according to the post, and no other operation is needed.

> Some edges are shared by multiple tags, for example, the `user: User` edge mentioned earlier is used by both Topic and Member. Wouldn't this mess up the data? And how to distinguish them? We don't want to use other names to create new edges.

You might have confused tags with vertices. I suggest that you read [Nebula Concepts](https://docs.nebula-graph.io) first. In fact, there is no such thing as an edge shared by different vertices. Edge IDs consist of the source vertex ID and destination vertex ID. An edge is unique if the vertices on both ends of it, its edge type, and its rank are distinct. Any difference in these attributes generates different edges. Different edges may have the same edge type, but their vertices are usually different. Even if their vertices are the same, if their ranks are different, they are different edges.
 
For example, the edge type between `user` and `topic` may be `focus`, and that between `user` and `member` may be `is`.

> Associated fields usually have a one-to-one association, such as the `user: User` edge mentioned earlier, one-to-many association, such as the `followings: [User]` edge, and many-to-many association. How are these associations represented in NebulaGraph?

Please refer to the preceding reply. In NebulaGraph tags and edges don't have direct relationships, but vertices and edges do. Tags are attached on vertices, and vertices are connected with edges. So no matter it is a one-to-one association or an N-to-N one, it is represented by edges.

> For associated lists, we usually need to count the quantity, normally by aggregating the queries or setting a count field alone to hold the quantity. In NebulaGraph, is there a proper way to handle this? Do we set a `count` property on edges? For example `CREATE EDGE followings(count int default 0);`?

nGQL supports [](https://docs.nebula-graph.io) such as `COUNT` and you don't need to set a count property for counting.

# Previous Pick of the Week

1. [Compiling NebulaGraph with ARM64](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-nov-27-2020/)
2. [NebulaGraph Studio V1.2.1-beta Has Been Released](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-nov-6-2020/)

In this weekly issue, we will share the release note of NebulaGraph v2.0.0-beta, the latest DB-Engines ranking for graph DBMS, and the best practice to design schema in NebulaGraph.

Pick of the Week at NebulaGraph - Schema design in NebulaGraph


![Pick of the Week](https://user-images.githubusercontent.com/57335825/87520320-a4e20c00-c637-11ea-8053-7222b9c4f00a.png)

Normally the weekly issue covers NebulaGraph Updates and Community Q&As. If something major happens, it will also be covered in the additional Events of the Week section.

The 48th Friday of 2020 is passing soon. Let's see what's new with NebulaGraph.

# Events of the week

- [NebulaGraph Studio v2.0.0-alpha Release Note](https://discuss.nebula-graph.io/t/nebula-graph-studio-v2-0-0-alpha-release-note/787).

NebulaGraph Studio v2.0.0-alpha supports querying and exploring data with NebulaGraph v2.0.0 alpha.
At the time, data importing and schema indexing is not compatible. We will support these functions in the follow-up releases soon. Please go to the [nebula-web-docker v2](https://github.com/vesoft-inc/nebula-web-docker) directory to try NebulaGraph Studio v2.0.0-alpha.

![NebulaGraph Studio v2.0.0-alpha](https://user-images.githubusercontent.com/57335825/100973399-57cfdb00-34ef-11eb-9c12-262e4922cac1.png)

# NebulaGraph Updates

- `FIND PATH` supports finding the outgoing, incoming, and bi-direct paths. It also supports removing the ring path. Tag: `nGQL`. PR: [https://github.com/vesoft-inc/nebula/pull/2398](https://github.com/vesoft-inc/nebula/pull/2398)
- Supports compiling NebulaGraph on ARM64. Tag: `Compiling`. PR: [https://github.com/vesoft-inc/nebula/pull/2392](https://github.com/vesoft-inc/nebula/pull/2392)

# Community Q&A

This week's topic is from community user @xrfinbj  about the relationship between Exchange and Spark Writer.

**Question:** What is the relationship between Exchange and Spark Writer?

> The code for Spark Writer is held in `nebula/src/tools/spark-sstfile-generator`. Spark Writer is a distributed data importing tool for NebulaGraph based on Spark. It converts data from various data warehouses into graph data, i.e. vertices and edges, then imports them into graph databases in batches.

> The code for Exchange is held in `nebula-java/tools/exchange`. NebulaGraph Exchange (Exchange for short) is an Apache Spark™ application used to migrate data in a cluster to NebulaGraph in a distributed environment. It supports migrating various batch data and streaming data.

> For more information, see [Import Data to NebulaGraph with Spark](https://nebula-graph.io/posts/best-practices-import-data-spark-nebula-graph/).

**NebulaGraph:** Exchange is migrated from Spark Writer and provides richer data sources based on Spark Writer, such as MySQL, Neo4j, Hive, HBase, Kafka, and Pulsar.

Fixed some problems of Exchange. For example, by default Exchange reads source data from HDFS as strings, which may be different from your graph schema defined in NebulaGraph. Exchange supports automatically matching and converting data types. With it, when a non-string data type is defined in NebulaGraph, Exchange converts the strings into data of the required data type. We will put main efforts on Exchange in the future. If you're using Spark, we suggest you using Exchange to import data.

One thing to add, the TODOs in document means the document is not done but Exchange actually provides the functions. If you're migrating data from other data sources with Exchange, refer to the [Neo4j](https://nebula-graph.io/posts/how-to-import-data-from-neo4j-to-nebula-graph/) example.

# Recommendations for You

[Import Data to NebulaGraph with Spark](https://nebula-graph.io/posts/best-practices-import-data-spark-nebula-graph/)
Starting from Spark, the writer shares some practice on indexes, cluster conflict packages.

## Previous Pick of the Week

- [Pick of the Week at NebulaGraph - NebulaGraph Studio V1.2.1-beta Has Been Released](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-nov-6-2020/)
- [Pick of the Week at NebulaGraph - vesoft Inc. Named one of the 20 Fastest-Growing OSS Startups](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-oct-23-2020/)

In this weekly issue, we are covering that NebulaGraph Studio v2.0.0-alpha supports querying and exploring data with NebulaGraph v2.0.0 alpha, compiling NebulaGraph on ARM64, and what is the difference between Nebula Exchange and Spark Writer.

Pick of the Week at NebulaGraph - Compiling NebulaGraph with ARM64


![Pick of the Week](https://user-images.githubusercontent.com/57335825/87520320-a4e20c00-c637-11ea-8053-7222b9c4f00a.png)

Normally the weekly issue covers NebulaGraph Updates and Community Q&As. If something major happens, it will also be covered in the additional Events of the Week section.

The 45th Friday of 2020 is passing soon. Let's see what's new with Graph Databases and NebulaGraph.

# Events of the week

- [NebulaGraph v2.0.0-alpha has been released](https://discuss.nebula-graph.com.cn/t/topic/1631).

This release provides more data types and enhances the expression ability of nGQL. NebulaGraph 2.x will gradually become openCypher compatible.

![NebulaGraph v2.0.0-alpha has been released](https://user-images.githubusercontent.com/57335825/100068343-3fbfd380-2dec-11eb-8eea-5e93d445c21a.png)

- [DB-Engines Ranking of Graph DBMS](https://db-engines.com/en/ranking/graph+dbms)

![DB-Engines Ranking of Graph DBMS](https://user-images.githubusercontent.com/57335825/100068211-143ce900-2dec-11eb-9470-50cb96cfb442.png)

NebulaGraph remains unchanged in the November ranking, but the score is getting pretty close to the previous two databases.

# NebulaGraph Updates

[NebulaGraph Studio V1.2.1-beta has been released](https://discuss.nebula-graph.io/t/nebula-graph-v2-0-0-alpha-release-note/754).

1. Now you can import edge types and use aliases to specify edges with the **Open in Explore** function.
1. Studio can now visualize the query results of `Find Path` statements.

 
# Community Q&A

This week's topic is from community user @85278161 about using NebulaGraph as the independent database of an online B2C mall.

**Question:** Online B2C malls use ACID databases such as MySQL and TiDB for the recommendation system, but multi-table queries can be very slow.

1. Can we use NebulaGraph independently as the database of an online B2C mall?
1. How can we use NebulaGraph together with TiDB or MySQL, while NebulaGraph functions as a database for querying relationship graphs?

**NebulaGraph:** Using relational databases and graph databases together in recommendation systems is a common technology stack. For example, you can link order numbers and phone numbers with edges in NebulaGraph and make associated queries. Whenever an order is stored in the relational database, the related vertices and edges in NebulaGraph are updated accordingly.

As for the details, it depends on what you want to recommend. Data modeling is always the key.

**Follow-up question:** Thank you. How to synchronize operations such as insertion, modification, and deletion from relational databases to NebulaGraph? Shall we perform the operations on both databases simultaneously from the server, or use some functions of the relational databases, such as logs or subscription in MySQL?

**NebulaGraph**: From the system design perspective, using subscriptions are certainly better than inserting data directly in the business layer. The former method decouples the modules from each other more, and the system can be more stable. Even if the recommendation has a problem, it will not affect the transaction. This may slightly affect the real-time performance of the data recommendation, but it should be acceptable.

# Recommendations for You

[Benchmarking the Mainstream Open Source Distributed Graph Databases](https://nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph/)
The Meituan team must select a graph database solution that can meet the business requirements and to use the solution as the basis of Meituan's graph storage and graph learning platform. In this post, the Meituan NLP team evaluated several mainstream graph databases on data import, data writing, and data query.

## Previous Pick of the Week

- [Pick of the Week at NebulaGraph - vesoft Inc. named one of the 20 Fastest-Growing OSS Startups](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-oct-23-2020/)
- [Pick of the Week at NebulaGraph - Configuration recommendations for data import](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-sept-18-2020/)

In this weekly issue, we are the release note of the NebulaGraph v2.0.0-alpha, the DB-Engines graph database ranking update in November, the question of using NebulaGraph as the independent database of an online B2C mall.

Pick of the Week at NebulaGraph - NebulaGraph Studio V1.2.1-beta Has Been Released


![Pick of the Week](https://user-images.githubusercontent.com/57335825/87520320-a4e20c00-c637-11ea-8053-7222b9c4f00a.png)

Normally the weekly issue covers NebulaGraph Updates and Community Q&As. If something major happens, it will also be covered in the additional Events of the Week section.

# Events of the Week

1. Runa Capital, an early-stage investor of Nginx and MariaDB, released the Top 20 fastest-growing open-source startups in Q3 2020

![OSS Rankings from Runa Capital](https://user-images.githubusercontent.com/57335825/97271396-bc5f9200-17ed-11eb-8baa-4108bdd9188a.png)

This list measures the GitHub star growth of each open-source project in Q3, 2020. And vesoft Inc., the creator of the NebulaGraph database, ranked the eighth with 4.5K stars (total stars of the NebulaGraph repository on GitHub as of 22/10/2020) and 351% AGR (Annualised Growth Rate in Q3, 2020). For more information, please read the Medium report: [https://medium.com/runacapital/open-source-growth-benchmarks-extention-the-ross-index-and-the-fastest-growing-startups-in-q3-2020-7aee7fa7eed7](https://medium.com/runacapital/open-source-growth-benchmarks-extention-the-ross-index-and-the-fastest-growing-startups-in-q3-2020-7aee7fa7eed7)

2. October: DB-Engines Ranking for Graph DBMS

During the last month, no big changes happened to the Top 10 ranking of Graph DBMS. However, NebulaGraph is still advancing, now ranking at the 18th.

![DB-Engines Ranking - October](https://user-images.githubusercontent.com/57335825/97271582-faf54c80-17ed-11eb-8e3f-5565d99bdda9.png)

# NebulaGraph Updates

## NebulaGraph Studio v1.2.0-beta was released

In this update, NebulaGraph Studio is furnished with a new feature: visualized schema operations. From now on, you can create graph schemas for NebulaGraph without memorizing the nGQL syntaxes, which is more convenient and less stressful for users. NebulaGraph Studio is available here: [https://github.com/vesoft-inc/nebula-web-docker](https://github.com/vesoft-inc/nebula-web-docker)

![NebulaGraph Studio - Schema 01](https://user-images.githubusercontent.com/57335825/97271722-2841fa80-17ee-11eb-82f8-c1e66827584f.png)

![NebulaGraph Studio - Schema 02](https://user-images.githubusercontent.com/57335825/97271794-43146f00-17ee-11eb-9b34-1bf68dcaf446.png)

# Community Q&A

This week's topic is about how to restore cluster data from a snapshot in NebulaGraph.

Q: How to restore the data from a snapshot on a new cluster? Which directory in the cluster should I put the snapshot file in?

NebulaGraph: After a snapshot is created, you will find that the file structure in the snapshot is the same as that in the original data directory. Therefore, you can consider that the snapshot is a backup of the original data files. If you want to use the checkpoint you created to restore data on a NebulaGraph cluster, you can write a shell script to replace the original data with the checkpoint data. If you need to repeat the data corruption problem on the cluster, keep the original data.

Further push: Thank you very much. I have another question. If I have an active/standby cluster, and the structure of the standby cluster is different from that of the active cluster. For example, the active cluster has five machines for the storage service, and the standby cluster has only four. In this case, can I use the data snapshot of the active cluster to replicate data in the standby cluster?

NebulaGraph: Generally speaking, this is not possible. The structure of the standby cluster must be the same as that of the active cluster. Data loss may be caused by inconsistent structures of active and standby clusters. For example, for an active cluster, a graph space is divided into five partitions with one replica. If the standby cluster has only four machines, the data of one partition will surely be lost, and other adaptation problems may occur. Therefore, we recommend consistent structures of the active and standby clusters during the recovery process.

Further push: OK, thank you. We originally wanted to create a graph offline and create a snapshot of it, and then copy the snapshot data on online machines. We didn't want the offline machines to be completely consistent with the online ones. However, according to your replies, the offline cluster structure must be the same as the online one.

NebulaGraph: Keeping the structures consistent is recommended. I think you may want to load online data efficiently by doing this, right? If the number of nodes in the standby cluster is less than that in the active cluster, even if there is no adaptation problem, a partition balance operation will be performed between storage nodes after the snapshot is switched, which will also consume system resources and reduce the efficiency of the entire checkpoint handover. In my opinion, the gain is not worth the loss.

Further push: One more question. Can I just copy all the files under the `/data` directory of each machine in the active cluster to the corresponding directories in the slave cluster, instead of using the snapshot to restore the data, to migrate the data?

NebulaGraph: Yes, you can. In fact, the implementation mechanism of the checkpoint is to create a hard link for data files. The advantage of this hard link is that it will not occupy too much storage space under the same file system. Therefore, if the checkpoint is created by using the hard link mechanism on the local system, it has great advantages in both storage space and backup performance. In your business, the checkpoint is actually copied to another system, which is no big difference from copying data files directly. In addition, if you want to directly copy data files, you must avoid write operations in the source cluster during the copy process.

# Recommended for You

## [Benchmarking the Mainstream Open Source Distributed Graph Databases at Meituan: NebulaGraph vs Dgraph vs JanusGraph](https://discuss.nebula-graph.io/t/benchmarking-the-mainstream-open-source-distributed-graph-databases-at-meituan-nebula-graph-vs-dgraph-vs-hugegraph/715)

Selecting a graph database solution that can meet the Meituan's business requirements is the basis of building their graph storage and graph learning platform. In this article, the NLP team at Meituan benchmarked the mainstream open-source distributed graph databases in data import, data write, and data queries.

## Previous Pick of the Week

1. [Configuration recommendations for data import](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-sept-18-2020/)
2. [DB-Engines Graph DBMS Ranking Update in September](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-sept-4-2020/)
3. [Import data from Neo4J or JanusGraph to NebulaGraph](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-28-2020/)

In this weekly issue, we are sharing NebulaGraph Studio feature updates and how to restore data from a cluster snapshot in NebulaGraph.

Pick of the Week at NebulaGraph - vesoft Inc. Named one of the 20 Fastest-Growing OSS Startups

This article shares the benchmarking tests conducted by the NLP team at Meituan. The test included data import and throughput/latency performance.

Benchmarking the Mainstream Open Source Distributed Graph Databases at Meituan: NebulaGraph vs Dgraph vs JanusGraph


> This article is written by Henson Wu from SeeYII, a financial service technology company
> It is originally published on the NebulaGraph forum:[https://discuss.nebula-graph.io/t/fast-deploy-nebula-graph-cluster-with-docker-swarm/693](https://discuss.nebula-graph.io/t/fast-deploy-nebula-graph-cluster-with-docker-swarm/693)

![Fast Deploy NebulaGraph Cluster with Docker Swarm](https://user-images.githubusercontent.com/57335825/97404495-71a84d80-18b3-11eb-8ac7-eaa79b55d844.png)

This post describes in detail how to deploy a NebulaGraph cluster with Docker Swarm. 

## Deploy a NebulaGraph Cluster

### 2.1 Environment Preparation

Prepare hosts as below.

| ip | Memory (GB) | CPU (# of Cores) |
| --- | --- | --- |
| 192.168.1.166 | 16 | 4 |
| 192.168.1.167 | 16 | 4 |
| 192.168.1.168 | 16 | 4 |

Please make sure that Docker has been installed on all the machines.

### 2.2 Initialize the Swarm Cluster

Execute the commands below on the host 192.168.1.166:

```
$ docker swarm init --advertise-addr 192.168.1.166
Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a manager.
To add a worker to this swarm, run the following command:
 docker swarm join \
 --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
 192.168.1.166:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
```

### 2.3 Add a Worker Node

Add a Swarm worker node per the notification message of the init commands. Execute the following commands on 192.168.1.167 and 192.168.1.168 respectively.

```
docker swarm join \
 --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
 192.168.1.166:2377
```

### 2.4 Authenticate the Cluster

```
docker node ls
 
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
h0az2wzqetpwhl9ybu76yxaen *   KF2-DATA-166        Ready               Active              Reachable           18.06.1-ce
q6jripaolxsl7xqv3cmv5pxji     KF2-DATA-167        Ready               Active              Leader              18.06.1-ce
h1iql1uvm7123h3gon9so69dy     KF2-DATA-168        Ready               Active                                  18.06.1-ce
```

### 2.5 Configure Docker Stack

```
vi docker-stack.yml
```

Configure Docker Stack as below:

```
version: '3.6'
services:
  metad0:
    image: vesoft/nebula-metad:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.166
      - --ws_ip=192.168.1.166
      - --port=45500
      - --data_path=/data/meta
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-166
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.166:11000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 11000
        published: 11000
        protocol: tcp
        mode: host
      - target: 11002
        published: 11002
        protocol: tcp
        mode: host
      - target: 45500
        published: 45500
        protocol: tcp
        mode: host
    volumes:
      - data-metad0:/data/meta
      - logs-metad0:/logs
    networks:
      - nebula-net

  metad1:
    image: vesoft/nebula-metad:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.167
      - --ws_ip=192.168.1.167
      - --port=45500
      - --data_path=/data/meta
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-167
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.167:11000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 11000
        published: 11000
        protocol: tcp
        mode: host
      - target: 11002
        published: 11002
        protocol: tcp
        mode: host
      - target: 45500
        published: 45500
        protocol: tcp
        mode: host
    volumes:
      - data-metad1:/data/meta
      - logs-metad1:/logs
    networks:
      - nebula-net

  metad2:
    image: vesoft/nebula-metad:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.168
      - --ws_ip=192.168.1.168
      - --port=45500
      - --data_path=/data/meta
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-168
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.168:11000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 11000
        published: 11000
        protocol: tcp
        mode: host
      - target: 11002
        published: 11002
        protocol: tcp
        mode: host
      - target: 45500
        published: 45500
        protocol: tcp
        mode: host
    volumes:
      - data-metad2:/data/meta
      - logs-metad2:/logs
    networks:
      - nebula-net

  storaged0:
    image: vesoft/nebula-storaged:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.166
      - --ws_ip=192.168.1.166
      - --port=44500
      - --data_path=/data/storage
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-166
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.166:12000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 12000
        published: 12000
        protocol: tcp
        mode: host
      - target: 12002
        published: 12002
        protocol: tcp
        mode: host
    volumes:
      - data-storaged0:/data/storage
      - logs-storaged0:/logs
    networks:
      - nebula-net
  storaged1:
    image: vesoft/nebula-storaged:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.167
      - --ws_ip=192.168.1.167
      - --port=44500
      - --data_path=/data/storage
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-167
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.167:12000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 12000
        published: 12000
        protocol: tcp
        mode: host
      - target: 12002
        published: 12004
        protocol: tcp
        mode: host
    volumes:
      - data-storaged1:/data/storage
      - logs-storaged1:/logs
    networks:
      - nebula-net

  storaged2:
    image: vesoft/nebula-storaged:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --local_ip=192.168.1.168
      - --ws_ip=192.168.1.168
      - --port=44500
      - --data_path=/data/storage
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-168
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.168:12000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 12000
        published: 12000
        protocol: tcp
        mode: host
      - target: 12002
        published: 12006
        protocol: tcp
        mode: host
    volumes:
      - data-storaged2:/data/storage
      - logs-storaged2:/logs
    networks:
      - nebula-net
  graphd1:
    image: vesoft/nebula-graphd:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --port=3699
      - --ws_ip=192.168.1.166
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-166
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.166:13000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 3699
        published: 3699
        protocol: tcp
        mode: host
      - target: 13000
        published: 13000
        protocol: tcp
#        mode: host
      - target: 13002
        published: 13002
        protocol: tcp
        mode: host
    volumes:
      - logs-graphd:/logs
    networks:
      - nebula-net

  graphd2:
    image: vesoft/nebula-graphd:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --port=3699
      - --ws_ip=192.168.1.167
      - --log_dir=/logs
      - --v=2
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-167
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.167:13001/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 3699
        published: 3640
        protocol: tcp
        mode: host
      - target: 13000
        published: 13001
        protocol: tcp
        mode: host
      - target: 13002
        published: 13003
        protocol: tcp
#        mode: host
    volumes:
      - logs-graphd2:/logs
    networks:
      - nebula-net
  graphd3:
    image: vesoft/nebula-graphd:nightly
    env_file:
      - ./nebula.env
    command:
      - --meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
      - --port=3699
      - --ws_ip=192.168.1.168
      - --log_dir=/logs
      - --v=0
      - --minloglevel=2
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == KF2-DATA-168
    depends_on:
      - metad0
      - metad1
      - metad2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.1.168:13002/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - target: 3699
        published: 3641
        protocol: tcp
        mode: host
      - target: 13000
        published: 13002
        protocol: tcp
#        mode: host
      - target: 13002
        published: 13004
        protocol: tcp
        mode: host
    volumes:
      - logs-graphd3:/logs
    networks:
      - nebula-net
networks:
  nebula-net:
    external: true
    attachable: true
    name: host
volumes:
  data-metad0:
  logs-metad0:
  data-metad1:
  logs-metad1:
  data-metad2:
  logs-metad2:
  data-storaged0:
  logs-storaged0:
  data-storaged1:
  logs-storaged1:
  data-storaged2:
  logs-storaged2:
  logs-graphd:
  logs-graphd2:
  logs-graphd3:
```

Edit the nebula.env file by adding the items below:

```
TZ=UTC
USER=root
```

### 2.6 Start the NebulaGraph Cluster

```
docker stack deploy nebula -c docker-stack.yml
```

## Cluster Configuration for Load Balancing and High Availability

The NebulaGraph clients (1.X) don't provide load balancing capability currently. They randomly select any graphd to connect the database. Therefore, you need to configure load balancing and high availability on your own if you want to use NebulaGraph in production.

![Cluster Configuration for Load Balancing and High Availability](https://user-images.githubusercontent.com/57335825/97405081-5b4ec180-18b4-11eb-8df9-ac12f3cd8be9.png)

Seen from the figure above, the entire deployment can be divided into three layers, i.e. data layer, load balance layer, and high availability layer. 

The load balance layer is responsible for distributing the requests from the client to the data layer for load balancing purpose.

The HA layer is realized by HAProxy and it ensures that the load balancing service works properly so that the entire cluster works properly.

### 3.1 Load Balancing Configuration

HAProxy uses Docker compose for configuration. Edit the three files below respectively:

1. Dockerfile

```
FROM haproxy:1.7
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
EXPOSE 3640
```

2. docker-compose.yml

```
version: "3.2"
services:
  haproxy:
    container_name: haproxy
    build: .
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
    ports:
      - 3640:3640
    restart: always
    networks:
      - app_net
networks:
  app_net:
    external: true
```

3. haproxy.cfg

```javascript
global
    daemon
    maxconn 30000
    log 127.0.0.1 local0 info
    log 127.0.0.1 local1 warning

defaults
    log-format %hr\ %ST\ %B\ %Ts
    log  global
    mode http
    option http-keep-alive
    timeout connect 5000ms
    timeout client 10000ms
    timeout server 50000ms
    timeout http-request 20000ms

# Customize your own frontends && backends && listen conf
# CUSTOM

listen graphd-cluster
    bind *:3640
    mode tcp
    maxconn 300
    balance roundrobin
    server server1 192.168.1.166:3699 maxconn 300 check
    server server2 192.168.1.167:3699 maxconn 300 check
    server server3 192.168.1.168:3699 maxconn 300 check

listen stats
    bind *:1080
    stats refresh 30s
    stats uri /stats
```

### 3.2 Start HAProxy

```
docker-compose up -d
```

### 3.3 HA Configuration

Follow the configuration steps on 192.168.1.166, 192.168.1.167, and 192.168.1.168.

1. Install Keepalived

```
apt-get update && apt-get upgrade && apt-get install keepalived -y
```

2. Modify Keepalived's configuration file, i.e. /etc/keepalived/keepalived.conf. Note that the priority item should be set to different values for the three hosts so that there is a priority among them.**

**Note: Please be noted that vip (virtual IP) is required to configure Keepalive. In the configurations below, 192.168.1.99 is the virtual IP.**

a. Configuration on 192.168.1.166

```javascript
global_defs {
    router_id lb01 #An identifier；
}
vrrp_script chk_haproxy {
    script "killall -0 haproxy"    interval 2
}
vrrp_instance VI_1 {
    state MASTER
    interface ens160
    virtual_router_id 52
    priority 999
    # Set the interval (in seconds) for sync check between MASTER and BACKUP load balancers.
    advert_int 1
    # Configure the authentication type and password
    authentication {
    # Configure the authentication type, mainly PASS and AH
        auth_type PASS
    # Set the authentication password.The MASTER and BACKUP load balancers must use the same password within the same vrrp_instance to communicate properly.
        auth_pass amber1
    }
    virtual_ipaddress {
        # The virtual IP is 192.168.1.99/24; Bounded interface is ens160; Alias for both MASTER and BACKUP load balancers is ens160:1
        192.168.1.99/24 dev ens160 label ens160:1
    }
    track_script {
        chk_haproxy
    }
}
```

b. Configuration on 192.168.1.167

```javascript
global_defs {
    router_id lb01 #An identifier；
}
vrrp_script chk_haproxy {
    script "killall -0 haproxy"    interval 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 52
    priority 888
    # Set the interval (in seconds) for sync check between MASTER and BACKUP load balancers.
    advert_int 1
    # Configure the authentication type and password
    authentication {
    # Configure the authentication type, mainly PASS and AH
        auth_type PASS
    # Set the authentication password.The MASTER and BACKUP load balancers must use the same password within the same vrrp_instance to communicate properly.
        auth_pass amber1
    }
    virtual_ipaddress {
        # The virtual IP is 192.168.1.99/24; Bounded interface is ens160; Alias for both MASTER and BACKUP load balancers is ens160:1
        192.168.1.99/24 dev ens160 label ens160:1
    }
    track_script {
        chk_haproxy
    }
}
```

c. Configuration on 192.168.1.168

```javascript
global_defs {
    router_id lb01 #An identifier；
}
vrrp_script chk_haproxy {
    script "killall -0 haproxy"    interval 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 52
    priority 777
    # Set the interval (in seconds) for sync check between MASTER and BACKUP load balancers.
    advert_int 1
   # Configure the authentication type and password
    authentication {
    # Configure the authentication type, mainly PASS and AH
        auth_type PASS
    # Set the authentication password.The MASTER and BACKUP load balancers must use the same password within the same vrrp_instance to communicate properly.
        auth_pass amber1
    }
    virtual_ipaddress {
        # The virtual IP is 192.168.1.99/24; Bounded interface is ens160; Alias for both MASTER and BACKUP load balancers is ens160:1
        192.168.1.99/24 dev ens160 label ens160:1
    }
    track_script {
        chk_haproxy
    }
}
```

Relevant commands used in Keepalived:

```javascript
# Start Keepalived
systemctl start keepalived
# Auto start Keepalived
systemctl enable keeplived
# Re-start Keepalived
systemctl restart keepalived
```

## Deploy the Cluster Offline

How to deploy the NebulaGraph cluster with Docker Swarm offline? The answer is simple. Changing the Docker image to private image repo will do.

If you have any questions regarding deploying NebulaGraph with Docker Swarm, please leave your comments in this thread: [https://discuss.nebula-graph.io/t/fast-deploy-nebula-graph-cluster-with-docker-swarm/693](https://discuss.nebula-graph.io/t/fast-deploy-nebula-graph-cluster-with-docker-swarm/693)

## You might also like:

1. [NebulaGraph Architecture — A Bird’s Eye View](https://nebula-graph.io/posts/nebula-graph-architecture-overview/)
1. [An Introduction to NebulaGraph's Storage Engine](https://nebula-graph.io/posts/nebula-graph-storage-engine-overview/)
1. [An Introduction to NebulaGraph’s Query Engine](https://nebula-graph.io/posts/nebula-graph-query-engine-overview/)

This article introduces how to deploy a NebulaGraph cluster with Docker Swarm. It also talks about how to keep high availability and load balancing of the cluster.

Fast Deploy NebulaGraph Cluster with Docker Swarm


![Pick of the Week](https://user-images.githubusercontent.com/57335825/87520320-a4e20c00-c637-11ea-8053-7222b9c4f00a.png)

Normally the weekly issue covers NebulaGraph Updates and Community Q&As. If something major happens, it will also be covered in the additional Events of the Week section.

# Events of the Week

1. Live talk: NebulaGraph in practice with WeChat

Graphs hold a promising prospect in areas such as social network recommendation, real-time computing, risk control, and security. How to store and query large-scale heterogeneous graph data efficiently with graph databases is a great challenge.

Most of the well-known graph databases are helpless in dealing with big data sets. For example, the community edition of Neo4j, which is widely used in the graph field, only provides single-replica services. And JanusGraph, while solving the storage problem of big data sets by external metadata management, KV storage, and indexing, has much-maligned performance issues.
 
How can Internet companies which are facing the challenge of big data storage and processing solve these problems with a graph database? In this live talk, Li Benli, a senior engineer in the WeChat team, shared his experience with us.

Previously Li has written an article in this regard, read the article [here](https://nebula-graph.io/posts/nebula-graph-for-large-social-network/).

2. New release: NebulaGraph 1.1.0 will be released next week

In this release, the dev team has greatly improved the stability and performance of NebulaGraph. There will also be some bug fixes. Stay tuned!

# NebulaGraph Updates

The updates of Nebula in the last week:

- The range scan for string-type indexes is no longer supported. We can only use the `==` condition in the `WHERE` clause of a `LOOKUP` statement while filtering string-type indexes, and the conditions must match all the properties of the indexes. For more information, see [PR #2283](https://github.com/vesoft-inc/nebula/pull/2283) and [PR #2277](https://github.com/vesoft-inc/nebula/pull/2277).

- Fixed an issue where stopping the meta service before the initialization of the job manager may cause meta exceptions. For more information, see [PR #2332](https://github.com/vesoft-inc/nebula/pull/2332).

- Optimized the logic of Raft. A delay is added after elections failed to ensure that there is only one election request at the same time. For more information, see [PR #2305](https://github.com/vesoft-inc/nebula/pull/2305).

# Community Q&A

This week's topic is about suggestions for the Spark Writer configuration from @nicole, a community user.

## Spark Writer Configuration Suggestions

Before using Spark Write to import data, we need to configure `application.conf`.

1. @nicole recommends that we enrich the configuration file with comments, writing comments for all parameters in the file. Put parameters with default values, such as Spark-related parameters, into comments, and write notes to remind users  to remove the comment signs to make any modification take effect.

2. For the field mapping configuration of tags and edges, @nicole wonders if we can add an option that could automatically map the fields in the source data and Neubla Graph that have the same names. For tags and edges with more than 50 properties, this option would save a lot of work.

# Recommended for You

## [Practice data migration from Neo4j to NebulaGraph](https://nebula-graph.com.cn/posts/how-to-import-data-from-neo4j-to-nebula-graph/)

In this article, you'll learn about the implementation of NebulaGraph Exchange, a data import tool based on Spark, and how to import data with it.

## Previous Pick of the Week

1. [NebulaGraph DBaaS is online](https://nebula-graph.com.cn/posts/nebula-graph-weekly-pickup-2020-09-11/)
2. [DB-Engines Graph DBMS Ranking Update in September](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-sept-4-2020/)
3. [Import data from Neo4J or JanusGraph to NebulaGraph](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-28-2020/)

In this weekly issue, we are providing some connfiguration recommendations for data import to NebulaGraph.

Pick of the Week at NebulaGraph - Configuration recommendations for data import


![Pick of the Week](https://user-images.githubusercontent.com/57335825/87520320-a4e20c00-c637-11ea-8053-7222b9c4f00a.png)

Normally the weekly issue covers NebulaGraph Updates and Community Q&As. If something major happens, it will also be covered in the additional Events of the Week section.

# Events of the Week

1. NebulaGraph DBaaS is public for beta test

![NebulaGraph DBaaS](https://user-images.githubusercontent.com/57335825/93436097-2dbb3500-f87f-11ea-8f8a-b3106dce4fcd.png)

The NebulaGraph DBaaS service is officially public for beta test this week. You are welcome to try it for free. Your need to submit a request to create an instance. After the submission, the support team will handle the request within two calendar days. Here is the entrance: [https://cloud.nebula-graph.io/](https://cloud.nebula-graph.io/)

2. Sherman Ye, the CEO of vesoft inc. was interviewed by TNP

![interview of Sherman Ye by TNP](https://user-images.githubusercontent.com/57335825/93436294-7246d080-f87f-11ea-9b40-c8356a0eaf75.png)

As the representative of the new generation graph database, NebulaGraph was invited to participate in the online database seminar hosted by TheNextPlatform, the famous technology media. The topic for this seminar is the new generation graph database. Sherman Ye， the founder and CEO of vesoft inc, was interviewed by editor-in-chief Timothy Prickett Morgan to discuss topics such as the design idea, the advantages of NebulaGraph, the market positioning of NebulaGraph, and the importance of the open source model to NebulaGraph. Check the interview on [YouTube](https://www.youtube.com/watch?v=eQQ4r2PK-kM&t=18s).

# NebulaGraph Updates

The updates of Nebula in the last week:

• Support to configure the prefix bloom filter for the RocksDB. Improved the query performance for `QueryEdgeProps` and `QueryVertexProps` and decreased the memory usage for the bloom filter. For more information, check this pull request: [https://github.com/vesoft-inc/nebula/pull/2274](https://github.com/vesoft-inc/nebula/pull/2274)

• Fixed the bug that may cause the newly-added column prop crash in `LOOKUP ON` after `alter add column`. For more information, check this pull request: [https://github.com/vesoft-inc/nebula/pull/2341](https://github.com/vesoft-inc/nebula/pull/2341)

• Refactored the RAFT leader election logic. For more information, check this pull request: [https://github.com/vesoft-inc/nebula/pull/2327](https://github.com/vesoft-inc/nebula/pull/2327)

# Community Q&A

The community Q&A topic for the last week is about index, cache, and search engine.

Q: How is the index created in NebulaGraph? For query efficiency, is it necessary to build a layer of Redis cache above the query interface layer? And it is necessary to introduce search engines such as Solr and ElasticSearch?

Nebula: Good question. I will answer them one by one.

## How is the index created in NebulaGraph?

Regarding the index storage, you can refer to this article: How Indexing Works in NebulaGraph. In addition, the index and data are stored in the same partition so that retrieving data is more easily in the partition by using the index and no network transmission is needed across different partitions.

## For query efficiency, is it necessary to build a layer of Redis cache above the query interface layer?

I personally think a Redis cache is only needed in the OLAP. Because data changes a lot in OLTP, the cache will only add extra load. At present, NebulaGraph introduced the LRU cache mechanism to its storage service, so I think it's more convenient if we put the  cache here. Of course, if you don't require high data accuracy and you can accept some data latency, you may still need the cache layer. One thing to add, when introducing a cache layer, you need to think about whether a third party system (e.g. Redis) can be used.

## Is it necessary to introduce search engines such as Solr and ElasticSearch?

I think this is a reasonable requirement. Actually, we are planning to support a third party full text search engine to strengthen fuzzy matching, wildcard matching, regular expression matching, prefix matching, suffix matching, and so on.

# Previous Pick of the Week

1. [DB-Engines Graph DBMS Ranking Update in September](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-sept-4-2020/)
2. [Import data from Neo4J or JanusGraph to NebulaGraph](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-28-2020/)
3. [Release of Studio v1.1.0-beta](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-21-2020/)

In this weekly issue, we are happy to announce that the NebulaGraph DBaaS is public for beta test now. WWe are also sharing the interview of Sherman Ye by TheNextPlatform which is a well-known technology information hub.

Pick of the Week at NebulaGraph - NebulaGraph DBaaS is Public for Beta Test


![Pick of the Week](https://user-images.githubusercontent.com/57335825/87520320-a4e20c00-c637-11ea-8053-7222b9c4f00a.png)

Normally the weekly issue covers NebulaGraph Updates and Community Q&As. If something major happens, it will also be covered in the additional Events of the Week section.

# Events of the Week

1. DB-Engines Ranking of Graph DBMS of September

In August, the ranks of the Top 9, even of the Top 20, graph DBMSs in DB-Engines Ranking had only minor changes. NebulaGraph was still ranking at 19th, but earned 0.08 more scores in August.

![db-engines ranking of graph dbms in September](https://user-images.githubusercontent.com/57335825/92688890-fbe02680-f2f2-11ea-8e3d-009d0827561b.png)

2. Data Migration from JanusGraph to NebulaGraph in 360 Finance

360 Finance, Inc. (NASDAQ: QFIN) has just migrated from JanusGraph to NebulaGraph. They shared the experience with the NebulaGraph community: why and how.

In the NebulaGraph community, Zhou Peng at 360 Finance Inc., a leading digital consumer finance company, shared his experience in data migration from JanusGraph to NebulaGraph. He illustrated how the data migration improved the performance by 20 times with only one-third resources that are for JanusGraph. For more information, visit our forum and communicate with Zhou Peng about his post [Data migration from JanusGraph to NebulaGraph](https://discuss.nebula-graph.io/t/data-migration-from-janusgraph-to-nebula-graph-practice-at-360-finance/672/2).

# NebulaGraph Updates

The updates of Nebula in the last week:

• Fixed the bug that may cause a storage crash during the process of vertex deletion in some cases. For more information, check this pull request [https://github.com/vesoft-inc/nebula/pull/2335](https://github.com/vesoft-inc/nebula/pull/2335).

• Improved the output of the `SHOW HOSTS` statement. With the enhancement, the records are returned in order. For more information, check this pull request [https://github.com/vesoft-inc/nebula/pull/2334](https://github.com/vesoft-inc/nebula/pull/2334).

![output of SHOW HOSTS statement](https://user-images.githubusercontent.com/57335825/92689346-b708bf80-f2f3-11ea-803c-f3381bba278b.png)

# Community Q&A

The community Q&A topic for the last week is about UUID() errors and the deployment of NebulaGraph server.

## About UUID() errors

During the process of vertex insertion, the `execute error: Get UUID Failed` error may occur when the built-in UUID() function is invoked.

**Possible reasons**: 

1. The network performance between the graphd service and the storaged service is not good, and an unacceptable I/O latency occurs.

2. During the deployment, network connections fail between the graphd service and the storaged service.

## About deployment of NebulaGraph server

During the deployment of a metad service for NebulaGraph, one node keeps fail.

**Server configurations**: 

• Three 16-core physical servers
• Each server has one graphd service, one storaged service, and one metad service

**Symptoms**: 

Of the metad services, two started successfully, but one failed. Here is the log:

```bash
I0824 15:32:06.928279 65490 MetaDaemon.cpp:105] Leader has not been elected, sleep 1s
I0824 15:32:07.125378 65536 RaftPart.cpp:1194] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0824 15:32:07.891225 65533 RaftPart.
```

**Reasons**: 

The firewalls were enabled for these servers, so the network connection was blocked. 

**Solution**:

Disable the firewalls and allow the ports.

# Previous Pick of the Week

1. [Import data from Neo4J or JanusGraph to NebulaGraph](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-28-2020/)
2. [Release of Studio v1.1.0-beta](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-21-2020/)
3. [Performance Testing Report Comparing NebulaGraph with Other Graph DBMSs](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-14-2020/)

In this weekly issue, we are updating the latest graph DBMS ranking change in September on the well-known website db-engines.com. In addition, we are sharing a story of data migration from JanusGraph to NebulaGraph in 360 Finance.

Pick of the Week at NebulaGraph - DB-Engines Graph DBMS Ranking Update in September


![Pick of the Week](https://user-images.githubusercontent.com/57335825/87520320-a4e20c00-c637-11ea-8053-7222b9c4f00a.png)

Normally the weekly issue covers NebulaGraph Updates and Community Q&As. If something major happens, it will also be covered in the additional Events of the Week section.

# Events of the Week

1. NebulaGraph was invited to the [TNP Database Conference](https://www.nextplatform.com/2020/08/26/the-next-database-platform/).

TheNextPlatform is a future-oriented technology platform that delivers cutting-edge technology to tech-decision makers like CIOs and senior architects. NebulaGraph, as a representative of open-source distributed Graph Database technology, was invited to the Database Event online conference hosted by the platform. Sherman Ye, founder and CEO of Vesoft Inc., which is the commercial entity behind NebulaGraph, spoke with TheNextPlaform about the value of graph databases to enterprises. Database experts from Google, MongoDB, OmniSci, and other vendors also attended the conference.

2. NebulaGraph supports importing data from Neo4J or JanusGraph.

[NebulaGraph Exchange](https://github.com/vesoft-inc/nebula-java/tree/master/tools/exchange) (formerly known as Spark Writer) now supports importing data from Neo4j and JanusGraph to NebulaGraph. After connecting to Neo4j/JanusGraph, run the following command to import data with NebulaGraph Exchange.

```bash
$SPARK_HOME/bin/spark-submit --class com.vesoft.nebula.tools.importer.Exchange --master "local[10]" target/exchange-1.0.1.jar -c /path/to/conf/neo4j_application.conf
```

NebulaGraph will provide specific guides in its official documents about the principles and configurations.

# NebulaGraph Updates

The updates of Nebula in the last week:

• Fixed an issue where `scanVertex` and `scanEdge` get no return after configuring `enable_multi_versions`. For more information, see [https://github.com/vesoft-inc/nebula/pull/2312](https://github.com/vesoft-inc/nebula/pull/2312).

• Fixed an issue where `USE SPACE` and `FIND PATH` could not be used together in some cases. For more information, see [https://github.com/vesoft-inc/nebula/pull/2303](https://github.com/vesoft-inc/nebula/pull/2303).

• Fixed an issue where a task to create a snapshot could be triggered by accident. For more information, see [https://github.com/vesoft-inc/nebula/pull/2316](https://github.com/vesoft-inc/nebula/pull/2316).

# Community Q&A

Q: How to install NebulaGraph without connecting to the Internet?

A: If you cannot connect to the Internet while compiling the NebulaGraph source code, you have to manually download the required tools and dependencies in advance, including the GCC compiler in the NebulaGraph repository, third-party libraries, and CMake. Then, copy all these to your host. You can find the step-by-step instructions below.

1. Download the following files on a host that can connect to the Internet.

_Although we use the command line to demonstrate, you can download these files through a browser._

a. Download GCC

1) For RedHat or CentOS users:

```bash
$ wget https://oss-cdn.nebula-graph.com.cn/toolset/vesoft-gcc-7.5.0-CentOS-x86_64-glibc-2.12.sh
```

2) For Debian or Ubuntu users:

```bash
$ wget https://oss-cdn.nebula-graph.com.cn/toolset/vesoft-gcc-7.5.0-Debian-x86_64-glibc-2.13.sh
```

b. Download CMake.

```bash
$ wget https://cmake.org/files/v3.15/cmake-3.15.5-Linux-x86_64.sh
```

c. Download third-party libraries.

```bash
$ wget https://oss-cdn.nebula-graph.com.cn/third-party/vesoft-third-party-x86_64-libc-2.12-gcc-7.5.0-abi-11.sh
```

2. Copy the preceding packages to your host that cannot connect to the Internet.

3. Install GCC.

- For RedHat or CentOS users:

```bash
$ sudo bash vesoft-gcc-7.5.0-CentOS-x86_64-glibc-2.12.sh
```

- For Debian or Ubuntu users:

```bash
$ sudo bash vesoft-gcc-7.5.0-Debian-x86_64-glibc-2.13.sh
```

4. Enable GCC

```bash
$ source /opt/vesoft/toolset/gcc/7.5.0/enable
```

5. Install CMake

```bash
$ sudo bash cmake-3.15.5-Linux-x86_64.sh --skip-license --prefix=/opt/vesoft/toolset/cmake
```

6. Add the bin directory of CMake to the PATH.

```bash
$ export PATH=/opt/vesoft/toolset/cmake:$PATH
```

7. Install third-party libraries.

```bash
$ sudo bash vesoft-third-party-x86_64-libc-2.12-gcc-7.5.0-abi-11.sh
```

# Recommendation for You

[Performance Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph/)

The Tencent Cloud team made a multi-dimensional comparison among three of the most popular open-source graph databases. The analysis is of great value and may be quite useful for you.

# Previous Pick of the Week

1. [Release of Studio v1.1.0-beta](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-21-2020/)
2. [Performance Testing Report Comparing NebulaGraph with Other Graph DBMSs](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-14-2020/)
3. [PRs to Improve Performance and Stability of NebulaGraph](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-7-2020/)

In this weekly issue, we are covering the release of NebulaGraph Exchange which enables you to import data from Neo4j and JanusGraph to NebulaGraph, and other NebulaGraph updates.

Pick of the Week at NebulaGraph - Import data from Neo4J or JanusGraph to NebulaGraph


![Pick of the Week](https://user-images.githubusercontent.com/57335825/87520320-a4e20c00-c637-11ea-8053-7222b9c4f00a.png)

Normally the weekly issue covers NebulaGraph Updates and Community Q&As. If something major happens, it will also be covered in the additional Events of the Week section.

# Events of the Week

Studio v1.1.0-beta was uncovered with a major feature: supporting the `LOOKUP` syntax.

![nebula graph studio v1 1 0-beta](https://user-images.githubusercontent.com/57335825/91418791-7b57fb00-e807-11ea-98d2-1a8674fa49a8.gif)

Besides, it is also armed with these updates:

On Explore:

- Supports vertex query by Index
- Supports pre-processing data to generate a VID for a vertex query

On Console:

- Supports importing results of vertex query into Explore for visualized graph exploration

Bug fixes:

- Fixed the bug that the console cannot correctly display the result for a vertex query for a boolean property

You are more than welcome to give Studio a shot and raise issues. Here is where you can get it: [https://github.com/vesoft-inc/nebula-web-docker](https://github.com/vesoft-inc/nebula-web-docker)

# NebulaGraph Updates

The updates of Nebula in the last week:

• When a query is partially successful, a warning message will appear to keep users informed of the failure. For more information, check this pull request: [https://github.com/vesoft-inc/nebula/pull/2290](https://github.com/vesoft-inc/nebula/pull/2290)

• Some configuration options are added for dynamically configuring RocksDB. For more information, check this pull request: [https://github.com/vesoft-inc/nebula/pull/2291](https://github.com/vesoft-inc/nebula/pull/2291), which is contributed by [@chenxu14](https://github.com/chenxu14)

• Fixed the bug that multiple services cannot be started on the same node in some cases. For more information, check this pull request: [https://github.com/vesoft-inc/nebula/pull/2289](https://github.com/vesoft-inc/nebula/pull/2289)

• The snapshot design is enhanced. Snapshots are created only when necessary and the performance is improved. For more information, check this pull request: [https://github.com/vesoft-inc/nebula/pull/2287](https://github.com/vesoft-inc/nebula/pull/2287)

# Community Q&A

Q: How can the UUID() function of NebulaGraph ensure the universal uniqueness of the generated VIDs?

A: A UUID has 64 bits, of which the higher 32 bits are the result of the MurmurHash function and the lower 32 bits are the timestamps of the current time. For the implementation, see [these source code](https://github.com/vesoft-inc/nebula/blob/master/src/storage/query/GetUUIDProcessor.h#L39-L42).

Further push: In a distributed environment, which shares the same timestamp, can the MurmurHash function give unique values for UUIDs?

A: Here is how the UUID generation algorithm is implemented: 

When a string is input, it is processed by the Hash function on the graphd side to get a temporary hash value, and then this value is processed by the MOD function to get the exact partition where the exact UUID calculation happens. On the storaged side, all reading and writing requests must be processed by the leader, so even though some strings may have the same hash value, they must wait for the storaged service to process the request, which will add different timestamps to the UUID, and the universal uniqueness is ensured.

We use the same MurmurHash function to process the input string and the higher 32 bits of the UUID, so after processed by the MOD function, the generated VID and the temporary Hash value mentioned above will be located in the same partition.

# Recommendation for You

[Analyzing Relationships in Game of Thrones With NetworkX, Gephi, and NebulaGraph (Part One)](https://nebula-graph.io/posts/game-of-thrones-relationship-networkx-gephi-nebula-graph/)

In this post, we will access the open source graph database NebulaGraph with NetworkX and visualize the complex character connections in _Game of Thrones_ with Gephi.

# Previous Pick of the Week

1. [Performance Testing Report Comparing NebulaGraph with Other Graph DBMSs](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-14-2020/)
2. [PRs to Improve Performance and Stability of NebulaGraph](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-aug-7-2020/)
3. [GO with int Type](https://nebula-graph.io/posts/nebula-graph-pick-of-the-week-jul-31-2020/)

In this weekly issue, we are covering the release of NebulaGraph Studio v1.1.0 Beta, and other NebulaGraph updates.