This article was written by a NebulaGraph community contributor *xrfinbj*. It introduces the process and notes of importing data from Hive to NebulaGraph with NebulaGraph Exchange. 

# Background

The company StoneWise had internal needs of using graph databases and the NebulaGraph database was confirmed to use through technical selection. We still needed to verify the query performance of the NebulaGraph database in real business scenarios. Therefore, we urgently needed to import data to NebulaGraph and validate the performance. We found that the documentation of importing data to NebulaGraph from Hive with Exchange was not very complete, so I decided to record the pitfalls we stepped in this process to avoid detours and traps.


This article was written based on NebulaGraph Documentions: [Import data from Hive](https://docs.nebula-graph.io/3.3.0/nebula-exchange/use-exchange/ex-ug-import-from-hive/)


# Environment

- NebulaGraph: nightly
- Deployment: Docker on Mac
- Hardware:
  - Disk: SSD
  - CPU、Memory: 16 G
- Data warehouse environment (Local data warehouse on Mac)
  - Hive 3.1.2
  - Hadoop 3.2.1
- Exchange: https://github.com/vesoft-inc/nebula-java/tree/v1.0/tools/exchange （Compile and generate a JAR package）
- Spark：spark-2.4.7-bin-hadoop2.7 (Configure `core-site.xml`, `hdfs-site.xml`, `hive-site.xml` in the `$HADOOP_HOME/etc/hadoop` directory, and configure `spark-env.sh` in the `$SPARK_HOME/conf` directory.)
- Scala code runner version 2.13.3 – Copyright 2002-2020, LAMP/EPFL and Lightbend, Inc.

# Configuration

1. NebulaGraph DDL

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/1.png)

2. Hive DDL

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/2.png)

3. The latest `nebula_application.conf` file.
Pay attention to these field mappings: `exec`, `fields`, `nebula.fields`, `vertex`, `source`, and `target`.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/3.png)
# Starting the Import

1. Make sure that the NebulaGraph service is started.
2. Make sure that Hive tables and data are ready.
3. Run `spark-sql cli` to see if the Hive tables and data are ready to ensure the Spark environment is OK.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/4.jpeg)

4. When the configuration is ready, execute Spark commands.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/5.png)

5. After successful import, you can check the amount of data imported with the db_dump tool.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/6.png)

# Pitfalls and Notes

- The first pitfall is that the spark-submit command does not add the `-h` option.

- The tagName in NebulaGraph is case-sensitive. the name in the configuration of tags should be the tag name of NebulaGraph.

- The Hive data type `int` and NebulaGraph `int` are not consistent. The Hive `bigint` corresponds to the NebulaGraph `int`.


## Other Notes

- INSERT operations have better performance than UPDATE because the underlying storage of NebulaGraph is KV Store. Repeated insertions are overwriting the data. 

- In addition to the information in the documentation, I also read the source code and made posts on the forum.

- I have verified the following two scenarios:

  - Importing data from Hive 2 (Hadoop 2) to NebulaGraph with Spark 2.4

  - Importing data from Hive 3 (Hadoop 3) to NebulaGraph with Spark 2.4

Note: Exchange does not support Spark 3 yet, and it reports an error after compiling, so it is hard to verify the Spark 3 environment.

There are still some questions.

- How to set the parameters `batch` and `rate.limit` in the `nebula_application.conf` file? 

- The principle of using Exchange to import data from Hive (I've started learning Spark recently.)

# Exchange Source Code Debugging

For information about Spark debugging, see [https://dzone.com/articles/how-to-attach-a-debugger-to-apache-spark](https://dzone.com/articles/how-to-attach-a-debugger-to-apache-spark).

Through the Exchange source code learning and debugging, I found that only in conjunction with the source code can I understood the descriptions about importing SST files and the Download and Ingest in the document.

Debugging the source code also helped me find some configuration problems.

Step 1:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/7.png)

Step 2:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/8.png)

Step 3: IDEA configuration

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/9.png)

Step 4: Click `Debug` in IDEA

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/10.png)

# Suggestions and Gratitude

Thanks to NebulaGraph database, which can solve many practical problems in the business. The problems I met are nothing compared to the problems met by others. The problems encountered in the middle of the process received timely feedback from the community.

---

>Are you passionate about graph databases and graph technology? Join the **NebulaGraph Open Source Community** to exchange ideas and learn practical skills with experts. [Join Slack](https://bit.ly/3SmvFu2)

>NebulaGraph is **OPEN SOURCE**：[https://github.com/vesoft-inc/nebula](https://github.com/vesoft-inc/nebula) We highly appreciate every Stargazer, and please let us know if you like NebulaGraph Database.

This blog introduces the process and notes of importing data from Hive to NebulaGraph with NebulaGraph Exchange. 

Hands-on Tutorial for Hive-Data-Import with NebulaGraph Exchange


![Validating Import Performance of Nebula Importer](https://user-images.githubusercontent.com/57335825/124109331-9a639100-da99-11eb-862f-1694c9cb09ce.png)

# Machine Specifications for Testing

| Host Name | OS  | CPU Architecture | CPU Cores | Memory  | Disk  |
| -------- | --------- | ------- | ------- | ----- | ----- |
| hadoop 10 | CentOS 7.6 | x86_64  | 32 核    | 128 GB | 1.8 TB |
| hadoop 11 | CentOS 7.6 | x86_64  | 32 核    | 64 GB  | 1 TB   |
| hadoop 12 | CentOS 7.6 | x86_64  | 16 核    | 64 GB  | 1 TB   |

# Environment of NebulaGraph Cluster

- Operating System: CentOS 7.5 +
- Necessary software for NebulaGraph Cluster, including gcc 7.1.0+, cmake 3.5.0, glibc 2.12+, and other necessary dependencies.

```
yum update
yum install -y make \
                 m4 \
                 git \
                 wget \
                 unzip \
                 xz \
                 readline-devel \
                 ncurses-devel \
                 zlib-devel \
                 gcc \
                 gcc-c++ \
                 cmake \
                 gettext \
                 curl \
                 redhat-lsb-core
```

- NebulaGraph version: V2.0.0
- Back-end storage: Three nodes, RocksDB

| Process \ Host Name   | hadoop10 | hadoop11 | hadoop12 |
| ---------------- | -------- | -------- | -------- |
| # of metad processes    | 1        | 1        | 1        |
| # of storaged processes | 1        | 1        | 1        |
| # of graphd processes   | 1        | 1        | 1        |

# Preparing Data and Introducing Data Format

| # of Vertices / File Size | # of Edges / File Size | # of Vertices and Edges / File Size |
| --------------------- | ------------------- | ------------------------ |
| 74,314,635 /4.6 G       | 139,951,301 /6.6 G    | 214,265,936 /11.2 G        |

**More details about the data:**

- edge.csv: 139,951,301 records in total, 6.6 GB

- vertex.csv: 74,314,635 records in total, 4.6 GB

- 214,265,936 vertices and edges in total, 11.2 GB

![data size](https://user-images.githubusercontent.com/57335825/124110011-59b84780-da9a-11eb-9c9a-0693c9a5bd17.png)

![vertices and edges](https://user-images.githubusercontent.com/57335825/124110122-7ce2f700-da9a-11eb-8dca-fdf9dc241789.png)

```
[root@hadoop10 datas]# wc -l edge.csv 
139951301 edge.csv
[root@hadoop10 datas]# head -10 vertex.csv 
-201035082963479683,实体
-1779678833482502384,值
4646408208538057683,胶饴
-1861609733419239066,别名: 饴糖、畅糖、畅、软糖。
-2047289935702608120,词条
5842706712819643509,词条（拼音：cí tiáo）也叫词目，是辞书学用语，指收列的词语及其释文。
-3063129772935425027,文化
-2484942249444426630,红色食品
-3877061284769534378,红色食品是指食品为红色、橙红色或棕红色的食品。
-3402450096279275143,否
[root@hadoop10 datas]# wc -l vertex.csv 
74314635 vertex.csv
[root@hadoop10 datas]# head -10 edge.csv 
-201035082963479683,-1779678833482502384,属性
4646408208538057683,-1861609733419239066,描述
-2047289935702608120,5842706712819643509,描述
-2047289935702608120,-3063129772935425027,标签
-2484942249444426630,-3877061284769534378,描述
-2484942249444426630,-2484942249444426630,中文名
-2484942249444426630,-3402450096279275143,是否含防腐剂
-2484942249444426630,4786182067583989997,主要食用功效
-2484942249444426630,-8978611301755314833,适宜人群
-2484942249444426630,-382812815618074210,用途
```

# Validating Solution

## Solution: Using Nebula Importer to import data in batch.

### Edit a YAML file for importing data.

```
version: v1rc1
description: example
clientSettings:
  concurrency: 10 # number of graph clients
  channelBufferSize: 128
  space: test
  connection:
    user: user
    password: password
    address: 191.168.7.10:9669,191.168.7.11:9669,191.168.7.12:9669
logPath: ./err/test.log
files:
  - path: ./vertex.csv
    failDataPath: ./err/vertex.csv
    batchSize: 100
    type: csv
    csv:
      withHeader: false
      withLabel: false
    schema:
      type: vertex
      vertex:
        tags:
          - name: entity
            props:
              - name: name
                type: string
  - path: ./edge.csv
    failDataPath: ./err/edge.csv
    batchSize: 100
    type: csv
    csv:
      withHeader: false
      withLabel: false
    schema:
      type: edge
      edge:
        name: relation
        withRanking: false
        props:
          - name: name
            type: string
```
#### Create schema

On Nebula Console, create a graph space, and then tags and edge types in the graph space.

```
# 1. Create a graph space.
 (admin@nebula) [(none)]> create space test2(vid_type = FIXED_STRING(64));
# 2. Switch to the specified graph space.
 (admin@nebula) [(none)]> use test2;
# 3. Create a tag.
(admin@nebula) [test2]> create tag entity(name string);
# 4. Create an edge type.
(admin@nebula) [test2]> create edge relation(name string);
# 5. View the definition of the tag.
 (admin@nebula) [test2]> describe tag entity;
+--------+----------+-------+---------+
| Field  | Type     | Null  | Default |
+--------+----------+-------+---------+
| "name" | "string" | "YES" |         |
+--------+----------+-------+---------+
Got 1 rows (time spent 703/1002 us)
# 6. View the definition of the edge type.
 (admin@nebula) [test2]> describe edge relation;
+--------+----------+-------+---------+
| Field  | Type     | Null  | Default |
+--------+----------+-------+---------+
| "name" | "string" | "YES" |         |
+--------+----------+-------+---------+
Got 1 rows (time spent 703/1041 us)
```

### Compile

Compile Nebula Importer and run shell commands.

```
# Compile Nebula Importer.
make build
# Run the shell command where a YAML configuration file is specified.
/opt/software/nebulagraph/nebula-importer/nebula-importer --config /opt/software/datas/rdf-import2.yaml
```

### View the output

![Import results](https://user-images.githubusercontent.com/57335825/124110759-23c79300-da9b-11eb-9d61-a1fd7dd60075.png)

```
# View part of logs.
2021/04/19 19:05:55 [INFO] statsmgr.go:61: Tick: Time(2400.00s), Finished(210207018), Failed(0), Latency AVG(32441us), Batches Req AVG(33824us), Rows AVG(87586.25/s)
2021/04/19 19:06:00 [INFO] statsmgr.go:61: Tick: Time(2405.00s), Finished(210541418), Failed(0), Latency AVG(32461us), Batches Req AVG(33844us), Rows AVG(87543.20/s)
2021/04/19 19:06:05 [INFO] statsmgr.go:61: Tick: Time(2410.00s), Finished(210901218), Failed(0), Latency AVG(32475us), Batches Req AVG(33857us), Rows AVG(87510.88/s)
2021/04/19 19:06:10 [INFO] statsmgr.go:61: Tick: Time(2415.00s), Finished(211270318), Failed(0), Latency AVG(32486us), Batches Req AVG(33869us), Rows AVG(87482.50/s)
2021/04/19 19:06:15 [INFO] statsmgr.go:61: Tick: Time(2420.00s), Finished(211685318), Failed(0), Latency AVG(32490us), Batches Req AVG(33873us), Rows AVG(87473.27/s)
2021/04/19 19:06:20 [INFO] statsmgr.go:61: Tick: Time(2425.00s), Finished(211959718), Failed(0), Latency AVG(32517us), Batches Req AVG(33900us), Rows AVG(87406.07/s)
2021/04/19 19:06:25 [INFO] statsmgr.go:61: Tick: Time(2430.00s), Finished(212220818), Failed(0), Latency AVG(32545us), Batches Req AVG(33928us), Rows AVG(87333.67/s)
2021/04/19 19:06:30 [INFO] statsmgr.go:61: Tick: Time(2435.00s), Finished(212433518), Failed(0), Latency AVG(32579us), Batches Req AVG(33963us), Rows AVG(87241.69/s)
2021/04/19 19:06:35 [INFO] statsmgr.go:61: Tick: Time(2440.00s), Finished(212780818), Failed(0), Latency AVG(32593us), Batches Req AVG(33977us), Rows AVG(87205.25/s)
2021/04/19 19:06:40 [INFO] statsmgr.go:61: Tick: Time(2445.01s), Finished(213240518), Failed(0), Latency AVG(32589us), Batches Req AVG(33973us), Rows AVG(87214.69/s)
2021/04/19 19:06:40 [INFO] reader.go:180: Total lines of file(/opt/software/datas/edge.csv) is: 139951301, error lines: 0
2021/04/19 19:06:42 [INFO] statsmgr.go:61: Done(/opt/software/datas/edge.csv): Time(2446.70s), Finished(213307919), Failed(0), Latency AVG(32585us), Batches Req AVG(33968us), Rows AVG(87181.95/s)
2021/04/19 19:06:42 Finish import data, consume time: 2447.20s
2021/04/19 19:06:43 --- END OF NEBULA IMPORTER ---
```

A special focus on the statistics of the statistics of results.

```
Time(2446.70s), Finished(213307919), Failed(0), Latency AVG(32585us), Batches Req
AVG(33968us), Rows AVG(87181.95/s)
2021/04/19 19:06:42 Finish import data, consume time: 2447.20s
2021/04/19 19:06:43 --- END OF NEBULA IMPORTER ---
```

#### Resource Requirements

High requirement of the machine specifications, including the number of CPU cores, memory size, and disk size.

- **hadoop 10**

![image](https://user-images.githubusercontent.com/57335825/124111076-77d27780-da9b-11eb-93a8-048f886c272e.png)
![image](https://user-images.githubusercontent.com/57335825/124111158-92a4ec00-da9b-11eb-8183-4febc8fb67ee.png)

**hadoop 11**

![image](https://user-images.githubusercontent.com/57335825/124111243-a81a1600-da9b-11eb-98d3-917d0c44773d.png)
![image](https://user-images.githubusercontent.com/57335825/124111313-bbc57c80-da9b-11eb-8dd7-b3d75418cc0d.png)

**hadoop 12**

![image](https://user-images.githubusercontent.com/57335825/124111382-ce3fb600-da9b-11eb-9917-3cbca9f5bce0.png)
![image](https://user-images.githubusercontent.com/57335825/124111443-e0b9ef80-da9b-11eb-980f-5d98ecfcb64d.png)

Recommendations on the machine specifications:

1. By comparing the memory consumption of the three machines, we found that the memory consumption is great when more than 200 million data are imported, so we recommend that the memory capacity should be as large as possible.
2. For the information about the CPU cores and disk size, see the documentation: [https://docs.nebula-graph.io](https://docs.nebula-graph.io).

### nGQL Statements Test

The native graph query language of NebulaGraph is nGQL. It is compatible with OpenCypher. For now, nGQL has not supported traversal of the total number of vertices and edges. For example, `MATCH (v) RETURN v` is not supported yet. Make sure that at least one index is available in a `MATCH` statement. If you want to create an index when related vertices, edges, or properties exist, rebuild the index after it is created to make it effective.

To test whether nGQL is compatible with OpenCypher.

```
# Test OpenCypher statements.
# Import an nGQL file.
./nebula-console -addr 191.168.7.10 -port 9669 -u user -p password -t 120  -f /opt/software/datas/basketballplayer-2.X.ngql
```

![image](https://user-images.githubusercontent.com/57335825/124111682-2080d700-da9c-11eb-8de8-5133c1551909.png)
![image](https://user-images.githubusercontent.com/57335825/124111772-37bfc480-da9c-11eb-9850-38780bb52068.png)

# Conclusion

This test validated the performance of importing a large amount of data to a three-node NebulaGraph cluster. The batch writing performance of Nebula Importer can meet the performance requirements of the production scenario. However, if the data is imported as CSV files, it must be stored in HDFS and a YAML configuration file is needed to specify the configuration of the tags and edge types for processing by tools.

Would like to know more about NebulaGraph? Join the [Slack](https://join.slack.com/t/nebulagraph/shared_invite/zt-7ybejuqa-NCZBroh~PCh66d9kOQj45g) channel!


In this article, Nebula community user, duspring, describes how he uses Nebula Importer to import 140 million edges and 70 million vertices to NebulaGraph.

Validating Import Performance of Nebula Importer

This article is written by Liu Jiahao, an engineer at the big data team of IntSig Information Co. Ltd (IntSig). He has been playing around with NebulaGraph and is one of our proud GitHub contributors. This post shares his experience importing data to [NebulaGraph](https://github.com/vesoft-inc/nebula) with Spark.

# Why NebulaGraph?

The graph-related business has grown more and more complex, and performance bottlenecks are identified in some popular graph databases. For example, a single machine has difficulties in scaling to larger graphs. In terms of performance, the native graph storage of Neo4j has irreplaceable advantages. In my survey, JanusGraph, Dgraph, and other graph databases cannot be comparable to Neo4j in this regard. JanusGraph performs very well in OLAP and can support OLTP to some extent. However, this cannot be an advantage of JanusGraph anymore, because some technologies, such as GraphFrame, are sufficient for the OLAP requirements. Besides, since Spark 3.0 starts to support Cypher, I found that comparing with the OLTP requirements of graphs, their OLAP requirements can be satisfied with more technologies. Therefore, NebulaGraph undoubtedly turns out to be a breakthrough to the low efficiency distributed OLTP databases.

I did a lot of researches and deployed some graph databases. When I did a test on the OLTP efficiency of JanusGraph and found that it cannot meet my online business requirements, I stopped requiring the equal or close performance in both OLAP and OLTP of a graph database. But it occurred to me that the architecture of NebulaGraph has all the features that can satisfy the requirements for graphs, such as:

- Distributed: NebulaGraph adopts the shared-nothing distributed architecture in storage.
- High-speed OLTP, which means the performance should be comparable to Neo4j: In NebulaGraph, its storage layer queries directly map physical addresses, which can be regarded as native graph storage.
- Highly available services, which means, without human interruptions, the database can continuously provide stable services: The services are available in case of partial failures, and the snapshot feature is available.
- Guaranteed scalability: NebulaGraph supports linear expansion, and it supports custom development because it is open source.

It seems that the architecture of NebulaGraph meets our actual requirements in the production environment. Therefore, I conducted research, deployment, and testing of NebulaGraph. In terms of deployment and performance testing, you can find some detailed information on NebulaGraph official website and some technical blogs. See [Meituan's benchmarking](https://nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph/) and [Tencent Cloud's performance test](https://nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph/). This article mainly describes my understanding and experience in NebulaGraph after my usage of a Spark application to import data into NebulaGraph.

# Test Environment

1. A Nebula Graph cluster, composed of:
    1. 3 servers of 32 cores (actually limited to 16 cores)
    2. 400 GB RAM (actually configured to 100 GB)
    3. SSD
    4. Version: Nebula Graph 1.0.0 (the test was done very early)
2. Network: 10G networking
3. Graph size: billions of vertices (with few properties), tens of billion edges (directed and no property or weighted)
4. Spark cluster: Spark 2.1.0 was used.

In this test, a total of 2 TG memory was used by Nebula Graph, which was calculated by the equation: (3*30 executor + 1 driver) * 25 GB.

# Use Spark to Batch Import Data

## Procedure

1. Package `sst.generator`, which is required by Spark to generate SST files.
2. Configure the Nebula Graph cluster, start the Nebula Graph cluster, and then create a Schema.
3. Modify the Spark configuration file (`config.conf`). For more information, see the [Spark configuration file](https://github.com/vesoft-inc/nebula-java/tree/master/tools/exchange/src/main/resources).
4. Do a check and make sure that no conflicting packages exist in the Spark cluster.
5. After Spark starts, use the configuration file and `sst.generator` to import data.
6. Verify the data.

## Some Tips to Follow

Here are some tips for you to use the Spark application:

1. I recommend that you create an index before data import.

Batch import data can be executed for only offline graphs. In NebulaGraph, you can choose to create an index for an online or offline graph service. But an index must be rebuilt when the service is offline. To prevent possible problems during the `REBUILD INDEX` process, I recommend that you create an index before data insertion. Of course, such an operation may slow down the speed of batch import of vertices.

2. I recommend that you use `int` type values as the vertex IDs.

You can use some algorithms such as the Snowflake algorithm to generate the values. If the vertex ID is not of the `int` type, you can use `policy: "uuid"` in the configuration file for vertex or edge type configuration.

3. If your Spark is deployed in the standalone mode, you can ignore the conflicting package problem, which may not occur in such a mode. If your Spark runs on clusters, this problem may occur, because some packages in `sst.generator` may conflict with some packages in the Spark cluster. To resolve the conflicts, you can shade or rename these conflicting packages.

4. Spark tuning
You can adjust the parameters to meet your business requirements to reduce memory consumption as much as possible to save resources and to speed up the parallelism.

## Data Import Performance

When the indexes are built in advance, it took about 20 hours to batch import billions of vertices (with few properties) and tens of billion edges (directed and no properties or weighted).

## Lessons Learned During Submitting GitHub PRs to NebulaGraph

I was using the Spark application with an earlier version of NebulaGraph, so problems were inevitable. During the process, I made some modifications to `SparkClientGenerator.scala`.

1. In the early stage when I was using Spark Writer ([Exchange](https://github.com/vesoft-inc/nebula-java/blob/master/doc/tools/exchange/ex-ug-toc.md) for now) to import data into NebulaGraph, some columns cannot be aligned correctly.
By reading the source code, I found that there was a bug in SparkClientGenerator.scala that the configuration file, but not the `Parquet` or `JSON` files, was read. I fixed the bug and submitted PR[#2187](https://github.com/vesoft-inc/nebula/pull/2187), my first PR to NebulaGraph. I was so happy that the PR was approved.

2. After then, when I was using `SparkClientGenerator` to use the `uuid()` or `hash()` function, I found that duplicate double quotation marks were introduced so that the batch import cannot be completed.

The extra double quotation marks were introduced during the process of data type conversion. I found that a function called `extraIndexValue` could be used to convert the user-defined values from a non-string type to the `String` type. I thought some users might want to convert non-string (for example, an array value) indexes to `uuid` or `hash`, so I changed some source code and submitted a new PR.

Unfortunately, the new PR was very tough. I made several commits but still could not get the approval. After communication with @darionyaphet, a developer at NebulaGraph, I knew that I changed the format of the source data. In his opinion, generally, when users import some data of some unsupported format, the correct response is to throw an exception, but not to directly convert the format. Yes, that makes sense. I focused too much on my business scenario and my only goal was to get the code run successfully but did not consider the general purpose of the tool.

In response to this, I submitted another PR, PR[#2258](https://github.com/vesoft-inc/nebula/pull/2258), and it was approved and merged. I learned a lot from this PR.

3. Later, I found that in nebula-python the thrift had conflicts with `fbthrift`. I thought I could shade the conflicts and submit another PR, but considering the huge modifications, I gave up and chose to raise an issue to NebulaGraph, which was fixed recently.

> Word from NebulaGraph：Welcome to submit PRs to NebulaGraph on GitHub. Here are some issues for your reference: [https://github.com/vesoft-inc/nebula/issues](https://github.com/vesoft-inc/nebula/issues)

# Summary

Before I started using NebulaGraph, I had evaluated JanusGraph thoroughly. After comparisons, I was very impressed with the relatively fewer secret bugs and the active community of NebulaGraph. After the test, NebulaGraph proved itself with its efficiency and became the first choice for distributed graph systems.

The NebulaGraph forum, groups, and GitHub projects are very active. All the users can get replies very quickly, and the PRs are reviewed effectively. I think this is the most important factor of the rapid and strong growth of this graph database. I hope I can continuously witness the growth of NebulaGraph and contribute to the improvement of the NebulaGraph ecology!


This article is written by Liu Jiahao, an engineer at the big data team of IntSig Information Co. Ltd (IntSig). He has been playing around with Nebula Graph and is one of our proud GitHub contributors. This post shares his experience importing data to Nebula Graph with Spark.

Hands-On Experience: Import Data to Nebula Graph with Spark


![Performance Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://user-images.githubusercontent.com/57335825/90609288-acf21600-e1b8-11ea-92d0-5638a45fc6c8.png)

# Who Did the Graph Database Comparison

This article describes how the Tencent Cloud team compares NebulaGraph with two other popular graph databases on the market from several perspectives.

By their nature of dealing with interconnections, graph databases are perfect for fraud detection and building knowledge graphs in the security field. To better serve the Tencent Cloud business scenarios, the Tencent Cloud Security team has to select a highly performant [graph database](https://www.nebula-graph.io/posts/what-is-a-graph-database) which fits the business development well, which is how this performance comparison comes into play.

# Whom to Compare With

## Neo4j

Neo4j is the most widely adopted graph database in the industrial world. It has a Community edition and an Enterprise edition. For comparison in this article, the team has chosen the Community edition.

## HugeGraph (A fork of JanusGraph)

HugeGraph is a distributed graph database developed by Baidu. It is forked from JanusGraph. HugeGraph is developed to address the needs of anti-fraud, threat intelligence collection, and underground economy attack with graph storage and analysis capabilities. It has pretty good read and write performance.

## NebulaGraph

NebulaGraph is an open source distributed graph database developed by vesoft Inc. It features the capability of dealing with super large datasets with hundreds of billions of vertices and trillions of edges.

# Hardware Environments

| Item | Specs |
| --- | --- |
| CPU | Intel Xeon(R) Gold 6133 CPU @ 2.5GHz X86_64 |
| # of Physical CPUs | 2 |
| # of Physical Cores | 20 |
| # of Logical CPUs | 80 |
| Memory | 260 GB |

# Graph Database Performance Comparison Test Results


The Tencent Cloud Security team has used graph data at different orders of magnitudes for testing purpose. The test has been performed against various metrics, including data import efficiency, one-hop query, two-hop query, and shared friends query.

The results are as below:

| Graph Data Size | Platform | Data Import | One-Hop  Query | Two-Hop  Query | Shared Friends Query |
| --- | --- | --- | --- | --- | --- |
| 10 Million Edges | Neo4j | 26s | 6.618s | 6.644s | 6.661s |
|  | HugeGraph | 89s | 16ms | 22ms | 72ms |
|  | NebulaGraph | 32.63s | 1.482ms | 3.095ms | 0.994ms |
| 100 Million Edges | Neo4j | 1min21s | 42.921s | 43.332s | 44.072s |
|  | HugeGraph | 10min | 19ms | 20ms | 5s |
|  | NebulaGraph | 3min52s | 1.971ms | 4.34ms | 4.147ms |
| 1 Billion Edges | Neo4j | 8min34s | 165.397s | 176.272s | 168.256s |
|  | HugeGraph | 65min | 19ms | 651ms | 3.8s |
|  | NebulaGraph | 29min35s | 2.035ms | 22.48ms | 1.761ms |
| 8 Billion Edges | Neo4j | 1h23min | 314.34s | 393.18s | 608.27s |
|  | HugeGraph | 16h | 68ms | 24s | 541ms |
|  | NebulaGraph | ~30min | Less than 1s | Less than 5s | Less than 1s |

Seen from the above table, in terms of data import, NebulaGraph is a bit slower than Neo4j when the data size is small. However, when the data size is large, NebulaGraph is much faster than the other two. For the three graph queries, NebulaGraph shows clearly better performance compared to Neo4j and HugeGraph.

Here is a chart overview of the comparison:

![Graph Database Performance Comparison Chart](https://user-images.githubusercontent.com/57335825/90246630-dfcb9100-dde9-11ea-8ddc-3334e6614c8e.png)

## Graph Query Language Comparison

### Neo4j Cypher

One-Hop Friends Query

```Shell
match ({vid:11111}) -> (u)
return u;
```

Two-Hop Friends Query

```Shell
match ({vid:11111}) -> () -> u
return u;
```

Shared Friends Query

```Shell
match ({vid:11111}) -> (u) <- ({vid:22222})
return u;
```

### NebulaGraph nGQL

One-Hop Friends Query

```Shell
GO FROM 11111 OVER relation
```

Two-Hop Friends Query

```Shell
GO 2 STEPS FROM 11111 OVER relation
```

Shared Friends Query

```Shell
GO FROM 11111 OVER relation INTERSECT
GO FROM 22222 OVER relation
```

# How NebulaGraph Works
1. [NebulaGraph Architecture — A Bird’s Eye View](https://www.nebula-graph.io/posts/nebula-graph-architecture-overview)
2. [Benchmark: NebulaGraph vs Dgraph vs JanusGraph](https://www.nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph)
3. [Deploy the Graph Database on Kubernetes](https://www.nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes)

[![docker](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/docker-banner.png)](javascript:window.open('https://bit.ly/3XHS873'))


This article describes how the Tencent Cloud team compares NebulaGraph with two other popular graph databases on the market, i.e. Neo4j and JanusGraph, from several perspectives. 

Tag