Snapchat, the camera-first social media platform, promotes authentic connections between its users through ephemeral photos, videos, and AR experiences. As of June of 2024, Snapchat reported having more than 432 million daily active users worldwide, ranking among the top mobile platforms in the US. **The user base growth poses growing challenges for the underlying infrastructure, requiring efficient tools to handle complex data workloads to navigate its vast, dense, interconnected social graph.** Snapchat’s Infrastructure team was seeking an efficient database solution to manage and analyze the massive interconnected user data, aiming to enhance their friend recommendation use case.

# The Challenge: Performance Bottleneck Restricts Real-time Friend Recommendations

To create better recommendations for friend suggestions, the underlying system needs to compute heuristics that leverage users and relationships, stored in aggregate and anonymized as metadata in vertices and edges. During the retrieval stage of the recommendation pipeline, these heuristics are used to compute affinity to the query user, and often require multiple hops, a task where graph databases excel compared to traditional relational databases. Moreover, the graph model is more natural for users to express data relations. At the end of 2022, Snapchat's Infrastructure team began exploring several graphDB solutions to examine their feasibility and performance.

# The Solution: NebulaGraph’s High Availability and Performance Make It the Ideal Choice

**After a thorough evaluation of various graph databases, the team decided to adopt NebulaGraph as it integrates seamlessly with their in-house environment and uniquely satisfies their production requirements of serving diverse online requests.**

NebulaGraph stood out as the team’s top choice with the following technical advantages:
- **High availability and resilience to failures**: NebulaGraph commits to delivering reliable performance with a 99.95% uptime, ensuring smooth operation during updates and upgrades.. Its high availability stands to remain unaffected in the face of failures such as node or network breakdowns, and even zone failures. This resilience is critical, considering the common occurrence of failures in large-scale clusters.
- **High performance**: NebulaGraph was among the very few top-performing databases, distinguished by its ability to store and process graphs with trillions of vertices and edges, while delivering impressive QPS and latency. It ensures highly simultaneous access, swift graph traversals, and optimized memory utilization.
- **Horizontal scalability**: NebulaGraph's shared-nothing distributed architecture offers linear scalability. This feature allows Snapchat to independently scale computing and storage, achieving desired performance at a cost-effective rate.
- **Seamless integration with Kubernetes**: NebulaGraph, with its cloud-native nature, can be effortlessly hosted within a Kubernetes environment, including AWS EKS, GCP GKE, Azure AKS, and more. Moreover, NebulaGraph holds full compatibility with the Kubernetes Operator Framework. This compatibility significantly reduces Snapchat's maintenance costs while ensuring uninterrupted service. The ability to run seamlessly in a Kubernetes environment further enhances its high-availability, offering customers peace of mind.
- **AAA security compliance**: NebulaGraph enhances Snapchat's network security through authentication, authorization, and accounting, safeguarding their user data. NebulaGraph supports both local and LDAP authentication that authorizes each user a role comprising various privileges, including metadata and data read/write access, and even fine-grained vertex and edge level permission management. Additionally, NebulaGraph logs user activities and database operations for future auditing. It also secures client-to-NebulaGraph and intra-NebulaGraph node traffic with TLS encryption. To protect data privacy, NebulaGraph supports TTL (Time-To-Live), ensuring that expired data is automatically and permanently removed from the database. Users can also delete data by dropping and clearing the database, providing control over data retention and privacy.

# The Result: Enhanced Performance, Reduced Costs and Simplified Maintenance

With the power of NebulaGraph, Snapchat has rolled out new friend recommendation workloads in production with some highlighted results, including:
- **Performance Amplification**: NebulaGraph's efficient handling of massive data sets has enabled Snapchat to employ new distributed online graph query algorithms, previously unattainable in traditional KV-store databases, while ensuring user data is stored in aggregate and anonymized as metadata in vertices and edges. This has led to a noticeable increase in new friendships formed through friends-of-friends recommendations. 
- **Large Graph Support**: NebulaGraph’s sharded architecture allowed ingestion of 12TB graphs and real-time queries at p90 <20ms. 
- **High Availability**: NebulaGraph has maintained 99.95% uptime since its initial deployment, and has shown it’s capable of handling large-scale ingestion, online query, through multiple traffic spike events. This has led to reduced operational overhead for the Snapchat Infrastructure team.
- **Cost-Effective Hardware**: Leveraging more affordable machine types for scaling, Snapchat can host NebulaGraph more economically compared to using a single, powerful machine with equivalent capacity.

**“We evaluated 9 graph DB solutions over the last year and found NebulaGraph capable of meeting our data size, query SLO, and QPS needs. NebulaGraph’s native Kubernetes support also makes it a nice fit to Snapchat’s underlying multi-cloud service mesh infrastructure. Since launch, we've thrown the book at NebulaGraph, adding 3 production clusters with more complex workloads and spiky online traffic patterns. Each time, it's scaled effortlessly, exceeding our expectations.” - Martin Qian, Sr. Manager Infrastructure at Snap.**

# Looking Ahead: Evolving the Entire Infrastructure with NebulaGraph

Moving beyond friend recommendations, Snapchat's Infrastructure team plans to leverage NebulaGraph in three new use cases:
1.	ML Edge Feature Storage: Eliminating the overhead of materializing costly multi-hop ML features, NebulaGraph can be used to power real-time inference and training queries.
2.	In-app User Engagement: Storing and querying relationships between users, content, lenses, locations and more will enable training graph convolution neural nets and relationship embeddings, boosting personalization.
3.	Off-app Analytics: Tackling identity resolution for Ads Attribution, and further data exploration opportunities beyond the app.

Additionally, NebulaGraph enables data sovereignty by allowing Snapchat to manage its own deployments of database instances. This shared-nothing data model ensures all of Snapchat's user data is maintained within Snapchat Infrastructure's controlled environments. This guarantees that enterprise data sovereignty is upheld and maximizes adoption flexibility.

This blog introduces why and how Snapchat leverages NebulaGraph for enhancing their real-time friend recommendations.

Snapchat Enhances Friend Recommendations with NebulaGraph

This global payments fintech unicorn offers a financial service interface for international transactions and global funds transfer for over 100,000 corporate entities over the world. This platform was established in 2015 and it serves as a hub for customers to receive payments from close to 180 countries and send money to over 150 countries, totaling over $50 billion in worldwide remittances as of January 2024.

# Challenges: Upgrading Risk Control System Amid Rising Financial Fraud

The company used to leverage a conventional relational database for their risk control system, yet the relational database has restrictions on data model representation. Besides, it was inefficient in processing multiple-table joins, leading to failed identification of risk in fraud conduct within financial transactions.

The company well understands that a successful financial platform relies on not only its ability to facilitate seamless transactions but also its capacity to guard assets and transactions, particularly with today’s sophisticated financial scams. Upon retrospective analysis of the fraudulent incidents, the company found that graph databases outperforms relational ones in revealing hidden data correlations, making them an optimum solution for such scenarios.

# The Solution: NebulaGraph Boosted Their Fraud Detection Capabilities

The company evaluated several enterprise graph database solutions, including ArangoDB, Neo4j, TuGraph and NebulaGraph with regard to architectures, performance, and scalability. The selection saw the choice of NebulaGraph because of its shard-nothing distributed approach, horizontal scalability as well as professional technical support.

- Global Compliance with Distributed Architecture
Since the company operates globally, it is faced with many data compliance requests from various regional authorities such as anti-money laundering and consumer privacy protection. This requirement was met by NebulaGraph which has a shard-nothing distributed architecture design separating storage from computation.
- Superior Performance for Real-time Anti-fraud Detection
An essential goal of the selection was to support real-time anti-fraud operations. In such a context, the underlying graph database must be able to handle over 1,000 concurrent requests and offer millisecond-level response times. NebulaGraph's powerful native graph engine enables lightning fast QPS and TPS with millisecond latency, handles high levels of concurrency, making it an excellent fit for the company’s needs.
- Seamless Scalability for Future Growth
In consideration of the future expansion plans, the company laid emphasis on scalability. NebulaGraph provides robust horizontal scalability due to its shared-nothing distributed storage design. It was thus perceived as an ideal solution which could effectively handle their expanding business and data in the future.
- Premium Technical Support for Critical Infrastructure
Stability is a must-have for any foundational software and the company needed the provider of graph databases to have reliable technical support in graph technology. NebulaGraph, the leader of the distributed graph databases, boasts a deep bench of top experts in the graph technology industry. Thus, their professional technical support capabilities have been highly praised by the company from the initial project engagement to the final rollout.

# The Achievement: Secure Transactions through Risk Control Platform Built on NebulaGraph

NebulaGraph plays a vital role in this entire risk control platform across three sub-domains namely Onboarding Evaluation, Transaction Risk Review and Post-Transaction Analysis, all aimed at guaranteeing safety of transactions made on the system.

- Onboarding Evaluation: NebulaGraph is used by the company to analyze the potential relationships between new users’ network information, device fingerprints, and user profiles. Through a complete review of all this, it automatically filters out those with some high-risk connections.

![image](https://www-cdn.nebula-graph.io/blogs/risk-control-platform.png)

- Transaction Risk Review: The system reviews each occurring transaction carefully based on a model of risk assessment. When transactions have been flagged for suspicion and necessitate manual review, NebulaGraph can be used to conduct detailed association analysis of both the transaction entity and its counterparty. This capability significantly aids their audit personnel in making informed decisions.

- Post-Transaction Analysis: Historical behavior of company clients is periodically scanned using data mining and machine learning techniques to detect unusual users or transactions. With NebulaGraph’s native support towards graph clustering algorithms, clusters of connected accounts can be identified very effectively. Such suspicious account clusters could be homogeneously labeled as well as discovered thereby being subjected to comprehensive manual scrutiny in order to mitigate future risks.

In addition to contributing to the risk control platform, NebulaGraph has been responsible for creating a holistic user knowledge graph for the company. This graph acts as an information base for RAG and assists by giving out short summaries of customer firms’ equity, operational status and partner data through LLM in the most user-friendly manner. These summaries are important points of reference for compliance departments that facilitate quick comprehension of their different businesses operating conditions.

In summary, NebulaGraph has played an essential role in the company’s pursuit to establish a secure, efficient and reliable financial system. Through its capacity of analyzing complex interactions and behaviors, it has enabled the firm to mitigate risks at all stages of users journey from on-boarding up to post event analysis.


A global fintech unicorn combats financial fraud with NebulaGraph, ensuring secure transactions.

A Global Fintech Unicorn Safeguards Financial Transactions with NebulaGraph

Akulaku is a fintech company serving Southeast Asia through e-commerce, consumer finance, and insurance brokerage services. It has a global business reach that extends to Indonesia, the Philippines, Vietnam, Malaysia, and Europe.

# The Problem: Challenges with Neo4j’s Scaling Up
Akulaku is an early adopter of graph technology, mainly for the scenarios including graph relationship mining and visualization analysis, as well as using graph queries to replace complex queries. Initially, Akulaku used a standalone version of Neo4j, primarily for relevance feature query. As the company's business grew, it needed to develop a series of real-time graph algorithms for financial real-time risk control scenarios. This required a graph database that could support data scales of up to a billion vertices and tens of billions of edges. While Neo4j has high query efficiency, its scalability could not meet Akulaku's business development needs, even though the team had tried the distributed edition of Neo4j. Therefore, the Akulaku team was eager to find a replacement for Neo4j to keep up with business development needs.

# Selection and Evaluation Criteria: Scalability, Data Import Capability and Query Efficiency

Overall, the Akulaku team primarily focused on three aspects of the replacement solution: scalability, data import capability, and query efficiency. Akulaku evaluated three graph database products: NebulaGraph, Dgraph, and JanusGraph, concluding that while Dgraph and JanusGraph each had their strengths, neither could simultaneously meet the three aforementioned requirements.

## Dgraph was ruled out due to inefficient data import

The Akulaku team had initially considered Dgraph, attracted by the well-written related academic papers. However, they found that Dgraph's engineering implementation did not meet their expectations, particularly its batch import function. Given Akulaku's large scale of graph data, primarily for financial risk control scenarios, with graph scales reaching up to a billion vertices and tens of billions of edges, they required a solution capable of rapid data import for initialization. During testing, they discovered that when the data imported exceeded a certain scale, it would cause memory leakage issues. This was a significant concern, leading to the decision to rule out Dgraph.

## Low query efficiency leads to the decision against JanusGraph

JanusGraph was also considered as a potential solution by the Akulaku team, given its advantage of integrating other storage engines in the backend. However, Akulaku's requirement for high query efficiency after data import and initialization was critical. Upon testing JanusGraph, they found that the query efficiency did not meet their expectations. This led them to continue their search for a solution that could better meet their efficiency needs.

# Benefits of NebulaGraph
After several rounds of technical selection and performance testing, Akulaku finally chose NebulaGraph. "Overall, the greatest value Nebula brings to Akulaku is its exceptional import performance and scalability," says Huang Hong, tech lead at Akulaku's Risk Control Team.

## Outstanding performance ensures high throughput and low latency

NebulaGraph has a powerful native graph engine that provides low latency read-write and high throughput, ensuring high concurrent access, rapid graph traversal, and efficient memory usage. This makes NebulaGraph capable of storing and processing super-large scale graph data with trillions of edges and points. 

## Easy scalability with linear expansion and shrinkage

NebulaGraph’s shared-nothing architecture supports expansion and shrinkage without stopping the database service. "NebulaGraph's high performance and excellent scalability allow us to process large amounts of data quickly and effectively. Its import speed is incredibly fast, with QPS reaching 110,000, of course, with asynchronous writing. This data is much better than other graph databases", said Hong. 

# The Solution: Leveraging NebulaGraph for the Real-Time Anti-Fraud Platform

NebulaGraph empowers Akulaku’s real-time anti-fraud platform via two engines: the graph database platform itself and the real-time graph computing engine. Since the main application scenario of Akulaku is anti-fraud, which requires high effectiveness, a series of real-time graph algorithms are added to the graph computing engine. NebulaGraph’s application scenarios towards the business-side could be summarized below.

- Scenario 1: Visualization Analysis of Fraud Incidents

A normal application of graph databases in the anti-fraud field is the visualization analysis of fraud incidents, as NebulaGraph leveraged by Akulaku. Through the visualization tools of NebulaGraph, associated relationships can be clearly displayed and analyzed for further exploration of the attributes of associated vertices. This multi-level, multi-angle relationship mining can reveal potential fraud patterns, enhancing the efficiency and accuracy of anti-fraud efforts.

- Scenario 2: Calculating the Association of Device IDs

Fraudsters may continually change identity elements to evade anti-fraud strategies, but usually, they only change one element, while other elements still have a certain association with that element. NebulaGraph can accurately identify the actual mapping relationships between these elements through set association rules, thereby revealing the true identity of the fraudsters, further enhancing risk control efficacy.

- Scenario 3: Graph Model Training and Deployment

Another significant value of NebulaGraph lies in graph model training and deployment. Graph analysis can play a vital role in scenarios like credit granting, order placing, label propagation, etc. Using NebulaGraph, Akulaku team can extract local subgraphs for computation and conduct real-time group mining, anomaly structure discovery based on vertice labels or specific business attribute labels, following certain rules. This flexibility and real-time nature make NebulaGraph possess great application potential in risk control scenarios.

# Looking Ahead: Optimizing the Platform to Fully Unlock the Graph Potential

NebulaGraph has been applied to many scenarios in Akulaku's real-time anti-fraud platform. In the future, the Akulaku team will focus on enhancing the platform's stability. Moreover, they will further optimize the platform, reducing the difficulty of graph models, backtesting, and model deployment.



Akulaku is a fintech company serving Southeast Asia through e-commerce, consumer finance, and insurance brokerage services. This article introduces why and how Akulaku building a real-time intelligent risk control platform based on NebulaGraph

Akulaku Building a Real-time Intelligent Risk Control Platform Based on NebulaGraph

This article was written by a NebulaGraph community contributor *xrfinbj*. It introduces the process and notes of importing data from Hive to NebulaGraph with NebulaGraph Exchange. 

# Background

The company StoneWise had internal needs of using graph databases and the NebulaGraph database was confirmed to use through technical selection. We still needed to verify the query performance of the NebulaGraph database in real business scenarios. Therefore, we urgently needed to import data to NebulaGraph and validate the performance. We found that the documentation of importing data to NebulaGraph from Hive with Exchange was not very complete, so I decided to record the pitfalls we stepped in this process to avoid detours and traps.


This article was written based on NebulaGraph Documentions: [Import data from Hive](https://docs.nebula-graph.io/3.3.0/nebula-exchange/use-exchange/ex-ug-import-from-hive/)


# Environment

- NebulaGraph: nightly
- Deployment: Docker on Mac
- Hardware:
  - Disk: SSD
  - CPU、Memory: 16 G
- Data warehouse environment (Local data warehouse on Mac)
  - Hive 3.1.2
  - Hadoop 3.2.1
- Exchange: https://github.com/vesoft-inc/nebula-java/tree/v1.0/tools/exchange （Compile and generate a JAR package）
- Spark：spark-2.4.7-bin-hadoop2.7 (Configure `core-site.xml`, `hdfs-site.xml`, `hive-site.xml` in the `$HADOOP_HOME/etc/hadoop` directory, and configure `spark-env.sh` in the `$SPARK_HOME/conf` directory.)
- Scala code runner version 2.13.3 – Copyright 2002-2020, LAMP/EPFL and Lightbend, Inc.

# Configuration

1. NebulaGraph DDL

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/1.png)

2. Hive DDL

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/2.png)

3. The latest `nebula_application.conf` file.
Pay attention to these field mappings: `exec`, `fields`, `nebula.fields`, `vertex`, `source`, and `target`.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/3.png)
# Starting the Import

1. Make sure that the NebulaGraph service is started.
2. Make sure that Hive tables and data are ready.
3. Run `spark-sql cli` to see if the Hive tables and data are ready to ensure the Spark environment is OK.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/4.jpeg)

4. When the configuration is ready, execute Spark commands.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/5.png)

5. After successful import, you can check the amount of data imported with the db_dump tool.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/6.png)

# Pitfalls and Notes

- The first pitfall is that the spark-submit command does not add the `-h` option.

- The tagName in NebulaGraph is case-sensitive. the name in the configuration of tags should be the tag name of NebulaGraph.

- The Hive data type `int` and NebulaGraph `int` are not consistent. The Hive `bigint` corresponds to the NebulaGraph `int`.


## Other Notes

- INSERT operations have better performance than UPDATE because the underlying storage of NebulaGraph is KV Store. Repeated insertions are overwriting the data. 

- In addition to the information in the documentation, I also read the source code and made posts on the forum.

- I have verified the following two scenarios:

  - Importing data from Hive 2 (Hadoop 2) to NebulaGraph with Spark 2.4

  - Importing data from Hive 3 (Hadoop 3) to NebulaGraph with Spark 2.4

Note: Exchange does not support Spark 3 yet, and it reports an error after compiling, so it is hard to verify the Spark 3 environment.

There are still some questions.

- How to set the parameters `batch` and `rate.limit` in the `nebula_application.conf` file? 

- The principle of using Exchange to import data from Hive (I've started learning Spark recently.)

# Exchange Source Code Debugging

For information about Spark debugging, see [https://dzone.com/articles/how-to-attach-a-debugger-to-apache-spark](https://dzone.com/articles/how-to-attach-a-debugger-to-apache-spark).

Through the Exchange source code learning and debugging, I found that only in conjunction with the source code can I understood the descriptions about importing SST files and the Download and Ingest in the document.

Debugging the source code also helped me find some configuration problems.

Step 1:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/7.png)

Step 2:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/8.png)

Step 3: IDEA configuration

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/9.png)

Step 4: Click `Debug` in IDEA

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/10.png)

# Suggestions and Gratitude

Thanks to NebulaGraph database, which can solve many practical problems in the business. The problems I met are nothing compared to the problems met by others. The problems encountered in the middle of the process received timely feedback from the community.

---

>Are you passionate about graph databases and graph technology? Join the **NebulaGraph Open Source Community** to exchange ideas and learn practical skills with experts. [Join Slack](https://bit.ly/3SmvFu2)

>NebulaGraph is **OPEN SOURCE**：[https://github.com/vesoft-inc/nebula](https://github.com/vesoft-inc/nebula) We highly appreciate every Stargazer, and please let us know if you like NebulaGraph Database.

This blog introduces the process and notes of importing data from Hive to NebulaGraph with NebulaGraph Exchange. 

Hands-on Tutorial for Hive-Data-Import with NebulaGraph Exchange


![Validating Import Performance of Nebula Importer](https://user-images.githubusercontent.com/57335825/124109331-9a639100-da99-11eb-862f-1694c9cb09ce.png)

# Machine Specifications for Testing

| Host Name | OS  | CPU Architecture | CPU Cores | Memory  | Disk  |
| -------- | --------- | ------- | ------- | ----- | ----- |
| hadoop 10 | CentOS 7.6 | x86_64  | 32 核    | 128 GB | 1.8 TB |
| hadoop 11 | CentOS 7.6 | x86_64  | 32 核    | 64 GB  | 1 TB   |
| hadoop 12 | CentOS 7.6 | x86_64  | 16 核    | 64 GB  | 1 TB   |

# Environment of NebulaGraph Cluster

- Operating System: CentOS 7.5 +
- Necessary software for NebulaGraph Cluster, including gcc 7.1.0+, cmake 3.5.0, glibc 2.12+, and other necessary dependencies.

```
yum update
yum install -y make \
                 m4 \
                 git \
                 wget \
                 unzip \
                 xz \
                 readline-devel \
                 ncurses-devel \
                 zlib-devel \
                 gcc \
                 gcc-c++ \
                 cmake \
                 gettext \
                 curl \
                 redhat-lsb-core
```

- NebulaGraph version: V2.0.0
- Back-end storage: Three nodes, RocksDB

| Process \ Host Name   | hadoop10 | hadoop11 | hadoop12 |
| ---------------- | -------- | -------- | -------- |
| # of metad processes    | 1        | 1        | 1        |
| # of storaged processes | 1        | 1        | 1        |
| # of graphd processes   | 1        | 1        | 1        |

# Preparing Data and Introducing Data Format

| # of Vertices / File Size | # of Edges / File Size | # of Vertices and Edges / File Size |
| --------------------- | ------------------- | ------------------------ |
| 74,314,635 /4.6 G       | 139,951,301 /6.6 G    | 214,265,936 /11.2 G        |

**More details about the data:**

- edge.csv: 139,951,301 records in total, 6.6 GB

- vertex.csv: 74,314,635 records in total, 4.6 GB

- 214,265,936 vertices and edges in total, 11.2 GB

![data size](https://user-images.githubusercontent.com/57335825/124110011-59b84780-da9a-11eb-9c9a-0693c9a5bd17.png)

![vertices and edges](https://user-images.githubusercontent.com/57335825/124110122-7ce2f700-da9a-11eb-8dca-fdf9dc241789.png)

```
[root@hadoop10 datas]# wc -l edge.csv 
139951301 edge.csv
[root@hadoop10 datas]# head -10 vertex.csv 
-201035082963479683,实体
-1779678833482502384,值
4646408208538057683,胶饴
-1861609733419239066,别名: 饴糖、畅糖、畅、软糖。
-2047289935702608120,词条
5842706712819643509,词条（拼音：cí tiáo）也叫词目，是辞书学用语，指收列的词语及其释文。
-3063129772935425027,文化
-2484942249444426630,红色食品
-3877061284769534378,红色食品是指食品为红色、橙红色或棕红色的食品。
-3402450096279275143,否
[root@hadoop10 datas]# wc -l vertex.csv 
74314635 vertex.csv
[root@hadoop10 datas]# head -10 edge.csv 
-201035082963479683,-1779678833482502384,属性
4646408208538057683,-1861609733419239066,描述
-2047289935702608120,5842706712819643509,描述
-2047289935702608120,-3063129772935425027,标签
-2484942249444426630,-3877061284769534378,描述
-2484942249444426630,-2484942249444426630,中文名
-2484942249444426630,-3402450096279275143,是否含防腐剂
-2484942249444426630,4786182067583989997,主要食用功效
-2484942249444426630,-8978611301755314833,适宜人群
-2484942249444426630,-382812815618074210,用途
```

# Validating Solution

## Solution: Using Nebula Importer to import data in batch.

### Edit a YAML file for importing data.

```
version: v1rc1
description: example
clientSettings:
  concurrency: 10 # number of graph clients
  channelBufferSize: 128
  space: test
  connection:
    user: user
    password: password
    address: 191.168.7.10:9669,191.168.7.11:9669,191.168.7.12:9669
logPath: ./err/test.log
files:
  - path: ./vertex.csv
    failDataPath: ./err/vertex.csv
    batchSize: 100
    type: csv
    csv:
      withHeader: false
      withLabel: false
    schema:
      type: vertex
      vertex:
        tags:
          - name: entity
            props:
              - name: name
                type: string
  - path: ./edge.csv
    failDataPath: ./err/edge.csv
    batchSize: 100
    type: csv
    csv:
      withHeader: false
      withLabel: false
    schema:
      type: edge
      edge:
        name: relation
        withRanking: false
        props:
          - name: name
            type: string
```
#### Create schema

On Nebula Console, create a graph space, and then tags and edge types in the graph space.

```
# 1. Create a graph space.
 (admin@nebula) [(none)]> create space test2(vid_type = FIXED_STRING(64));
# 2. Switch to the specified graph space.
 (admin@nebula) [(none)]> use test2;
# 3. Create a tag.
(admin@nebula) [test2]> create tag entity(name string);
# 4. Create an edge type.
(admin@nebula) [test2]> create edge relation(name string);
# 5. View the definition of the tag.
 (admin@nebula) [test2]> describe tag entity;
+--------+----------+-------+---------+
| Field  | Type     | Null  | Default |
+--------+----------+-------+---------+
| "name" | "string" | "YES" |         |
+--------+----------+-------+---------+
Got 1 rows (time spent 703/1002 us)
# 6. View the definition of the edge type.
 (admin@nebula) [test2]> describe edge relation;
+--------+----------+-------+---------+
| Field  | Type     | Null  | Default |
+--------+----------+-------+---------+
| "name" | "string" | "YES" |         |
+--------+----------+-------+---------+
Got 1 rows (time spent 703/1041 us)
```

### Compile

Compile Nebula Importer and run shell commands.

```
# Compile Nebula Importer.
make build
# Run the shell command where a YAML configuration file is specified.
/opt/software/nebulagraph/nebula-importer/nebula-importer --config /opt/software/datas/rdf-import2.yaml
```

### View the output

![Import results](https://user-images.githubusercontent.com/57335825/124110759-23c79300-da9b-11eb-9d61-a1fd7dd60075.png)

```
# View part of logs.
2021/04/19 19:05:55 [INFO] statsmgr.go:61: Tick: Time(2400.00s), Finished(210207018), Failed(0), Latency AVG(32441us), Batches Req AVG(33824us), Rows AVG(87586.25/s)
2021/04/19 19:06:00 [INFO] statsmgr.go:61: Tick: Time(2405.00s), Finished(210541418), Failed(0), Latency AVG(32461us), Batches Req AVG(33844us), Rows AVG(87543.20/s)
2021/04/19 19:06:05 [INFO] statsmgr.go:61: Tick: Time(2410.00s), Finished(210901218), Failed(0), Latency AVG(32475us), Batches Req AVG(33857us), Rows AVG(87510.88/s)
2021/04/19 19:06:10 [INFO] statsmgr.go:61: Tick: Time(2415.00s), Finished(211270318), Failed(0), Latency AVG(32486us), Batches Req AVG(33869us), Rows AVG(87482.50/s)
2021/04/19 19:06:15 [INFO] statsmgr.go:61: Tick: Time(2420.00s), Finished(211685318), Failed(0), Latency AVG(32490us), Batches Req AVG(33873us), Rows AVG(87473.27/s)
2021/04/19 19:06:20 [INFO] statsmgr.go:61: Tick: Time(2425.00s), Finished(211959718), Failed(0), Latency AVG(32517us), Batches Req AVG(33900us), Rows AVG(87406.07/s)
2021/04/19 19:06:25 [INFO] statsmgr.go:61: Tick: Time(2430.00s), Finished(212220818), Failed(0), Latency AVG(32545us), Batches Req AVG(33928us), Rows AVG(87333.67/s)
2021/04/19 19:06:30 [INFO] statsmgr.go:61: Tick: Time(2435.00s), Finished(212433518), Failed(0), Latency AVG(32579us), Batches Req AVG(33963us), Rows AVG(87241.69/s)
2021/04/19 19:06:35 [INFO] statsmgr.go:61: Tick: Time(2440.00s), Finished(212780818), Failed(0), Latency AVG(32593us), Batches Req AVG(33977us), Rows AVG(87205.25/s)
2021/04/19 19:06:40 [INFO] statsmgr.go:61: Tick: Time(2445.01s), Finished(213240518), Failed(0), Latency AVG(32589us), Batches Req AVG(33973us), Rows AVG(87214.69/s)
2021/04/19 19:06:40 [INFO] reader.go:180: Total lines of file(/opt/software/datas/edge.csv) is: 139951301, error lines: 0
2021/04/19 19:06:42 [INFO] statsmgr.go:61: Done(/opt/software/datas/edge.csv): Time(2446.70s), Finished(213307919), Failed(0), Latency AVG(32585us), Batches Req AVG(33968us), Rows AVG(87181.95/s)
2021/04/19 19:06:42 Finish import data, consume time: 2447.20s
2021/04/19 19:06:43 --- END OF NEBULA IMPORTER ---
```

A special focus on the statistics of the statistics of results.

```
Time(2446.70s), Finished(213307919), Failed(0), Latency AVG(32585us), Batches Req
AVG(33968us), Rows AVG(87181.95/s)
2021/04/19 19:06:42 Finish import data, consume time: 2447.20s
2021/04/19 19:06:43 --- END OF NEBULA IMPORTER ---
```

#### Resource Requirements

High requirement of the machine specifications, including the number of CPU cores, memory size, and disk size.

- **hadoop 10**

![image](https://user-images.githubusercontent.com/57335825/124111076-77d27780-da9b-11eb-93a8-048f886c272e.png)
![image](https://user-images.githubusercontent.com/57335825/124111158-92a4ec00-da9b-11eb-8183-4febc8fb67ee.png)

**hadoop 11**

![image](https://user-images.githubusercontent.com/57335825/124111243-a81a1600-da9b-11eb-98d3-917d0c44773d.png)
![image](https://user-images.githubusercontent.com/57335825/124111313-bbc57c80-da9b-11eb-8dd7-b3d75418cc0d.png)

**hadoop 12**

![image](https://user-images.githubusercontent.com/57335825/124111382-ce3fb600-da9b-11eb-9917-3cbca9f5bce0.png)
![image](https://user-images.githubusercontent.com/57335825/124111443-e0b9ef80-da9b-11eb-980f-5d98ecfcb64d.png)

Recommendations on the machine specifications:

1. By comparing the memory consumption of the three machines, we found that the memory consumption is great when more than 200 million data are imported, so we recommend that the memory capacity should be as large as possible.
2. For the information about the CPU cores and disk size, see the documentation: [https://docs.nebula-graph.io](https://docs.nebula-graph.io).

### nGQL Statements Test

The native graph query language of NebulaGraph is nGQL. It is compatible with OpenCypher. For now, nGQL has not supported traversal of the total number of vertices and edges. For example, `MATCH (v) RETURN v` is not supported yet. Make sure that at least one index is available in a `MATCH` statement. If you want to create an index when related vertices, edges, or properties exist, rebuild the index after it is created to make it effective.

To test whether nGQL is compatible with OpenCypher.

```
# Test OpenCypher statements.
# Import an nGQL file.
./nebula-console -addr 191.168.7.10 -port 9669 -u user -p password -t 120  -f /opt/software/datas/basketballplayer-2.X.ngql
```

![image](https://user-images.githubusercontent.com/57335825/124111682-2080d700-da9c-11eb-8de8-5133c1551909.png)
![image](https://user-images.githubusercontent.com/57335825/124111772-37bfc480-da9c-11eb-9850-38780bb52068.png)

# Conclusion

This test validated the performance of importing a large amount of data to a three-node NebulaGraph cluster. The batch writing performance of Nebula Importer can meet the performance requirements of the production scenario. However, if the data is imported as CSV files, it must be stored in HDFS and a YAML configuration file is needed to specify the configuration of the tags and edge types for processing by tools.

Would like to know more about NebulaGraph? Join the [Slack](https://join.slack.com/t/nebulagraph/shared_invite/zt-7ybejuqa-NCZBroh~PCh66d9kOQj45g) channel!


In this article, Nebula community user, duspring, describes how he uses Nebula Importer to import 140 million edges and 70 million vertices to NebulaGraph.

Validating Import Performance of Nebula Importer

This article is written by Liu Jiahao, an engineer at the big data team of IntSig Information Co. Ltd (IntSig). He has been playing around with NebulaGraph and is one of our proud GitHub contributors. This post shares his experience importing data to [NebulaGraph](https://github.com/vesoft-inc/nebula) with Spark.

# Why NebulaGraph?

The graph-related business has grown more and more complex, and performance bottlenecks are identified in some popular graph databases. For example, a single machine has difficulties in scaling to larger graphs. In terms of performance, the native graph storage of Neo4j has irreplaceable advantages. In my survey, JanusGraph, Dgraph, and other graph databases cannot be comparable to Neo4j in this regard. JanusGraph performs very well in OLAP and can support OLTP to some extent. However, this cannot be an advantage of JanusGraph anymore, because some technologies, such as GraphFrame, are sufficient for the OLAP requirements. Besides, since Spark 3.0 starts to support Cypher, I found that comparing with the OLTP requirements of graphs, their OLAP requirements can be satisfied with more technologies. Therefore, NebulaGraph undoubtedly turns out to be a breakthrough to the low efficiency distributed OLTP databases.

I did a lot of researches and deployed some graph databases. When I did a test on the OLTP efficiency of JanusGraph and found that it cannot meet my online business requirements, I stopped requiring the equal or close performance in both [OLAP and OLTP](https://www.nebula-graph.io/posts/OLTP-vs-OLAP-what-are-the-differences) of a graph database. But it occurred to me that the architecture of NebulaGraph has all the features that can satisfy the requirements for graphs, such as:

- Distributed: NebulaGraph adopts the shared-nothing distributed architecture in storage.
- High-speed OLTP, which means the performance should be comparable to Neo4j: In NebulaGraph, its storage layer queries directly map physical addresses, which can be regarded as native graph storage.
- Highly available services, which means, without human interruptions, the database can continuously provide stable services: The services are available in case of partial failures, and the snapshot feature is available.
- Guaranteed scalability: NebulaGraph supports linear expansion, and it supports custom development because it is open source.

It seems that the architecture of NebulaGraph meets our actual requirements in the production environment. Therefore, I conducted research, deployment, and testing of NebulaGraph. In terms of deployment and performance testing, you can find some detailed information on NebulaGraph official website and some technical blogs. See [Meituan's benchmarking](https://nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph/) and [Tencent Cloud's performance test](https://nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph/). This article mainly describes my understanding and experience in NebulaGraph after my usage of a Spark application to import data into NebulaGraph.

# Test Environment

1. A Nebula Graph cluster, composed of:
    1. 3 servers of 32 cores (actually limited to 16 cores)
    2. 400 GB RAM (actually configured to 100 GB)
    3. SSD
    4. Version: Nebula Graph 1.0.0 (the test was done very early)
2. Network: 10G networking
3. Graph size: billions of vertices (with few properties), tens of billion edges (directed and no property or weighted)
4. Spark cluster: Spark 2.1.0 was used.

In this test, a total of 2 TG memory was used by Nebula Graph, which was calculated by the equation: (3*30 executor + 1 driver) * 25 GB.

# Use Spark to Batch Import Data

## Procedure

1. Package `sst.generator`, which is required by Spark to generate SST files.
2. Configure the Nebula Graph cluster, start the Nebula Graph cluster, and then create a Schema.
3. Modify the Spark configuration file (`config.conf`). For more information, see the [Spark configuration file](https://github.com/vesoft-inc/nebula-java/tree/master/tools/exchange/src/main/resources).
4. Do a check and make sure that no conflicting packages exist in the Spark cluster.
5. After Spark starts, use the configuration file and `sst.generator` to import data.
6. Verify the data.

## Some Tips to Follow

Here are some tips for you to use the Spark application:

1. I recommend that you create an index before data import.

Batch import data can be executed for only offline graphs. In NebulaGraph, you can choose to create an index for an online or offline graph service. But an index must be rebuilt when the service is offline. To prevent possible problems during the `REBUILD INDEX` process, I recommend that you create an index before data insertion. Of course, such an operation may slow down the speed of batch import of vertices.

2. I recommend that you use `int` type values as the vertex IDs.

You can use some algorithms such as the Snowflake algorithm to generate the values. If the vertex ID is not of the `int` type, you can use `policy: "uuid"` in the configuration file for vertex or edge type configuration.

3. If your Spark is deployed in the standalone mode, you can ignore the conflicting package problem, which may not occur in such a mode. If your Spark runs on clusters, this problem may occur, because some packages in `sst.generator` may conflict with some packages in the Spark cluster. To resolve the conflicts, you can shade or rename these conflicting packages.

4. Spark tuning
You can adjust the parameters to meet your business requirements to reduce memory consumption as much as possible to save resources and to speed up the parallelism.

## Data Import Performance

When the indexes are built in advance, it took about 20 hours to batch import billions of vertices (with few properties) and tens of billion edges (directed and no properties or weighted).

## Lessons Learned During Submitting GitHub PRs to NebulaGraph

I was using the Spark application with an earlier version of NebulaGraph, so problems were inevitable. During the process, I made some modifications to `SparkClientGenerator.scala`.

1. In the early stage when I was using Spark Writer ([Exchange](https://github.com/vesoft-inc/nebula-java/blob/master/doc/tools/exchange/ex-ug-toc.md) for now) to import data into NebulaGraph, some columns cannot be aligned correctly.
By reading the source code, I found that there was a bug in SparkClientGenerator.scala that the configuration file, but not the `Parquet` or `JSON` files, was read. I fixed the bug and submitted PR[#2187](https://github.com/vesoft-inc/nebula/pull/2187), my first PR to NebulaGraph. I was so happy that the PR was approved.

2. After then, when I was using `SparkClientGenerator` to use the `uuid()` or `hash()` function, I found that duplicate double quotation marks were introduced so that the batch import cannot be completed.

The extra double quotation marks were introduced during the process of data type conversion. I found that a function called `extraIndexValue` could be used to convert the user-defined values from a non-string type to the `String` type. I thought some users might want to convert non-string (for example, an array value) indexes to `uuid` or `hash`, so I changed some source code and submitted a new PR.

Unfortunately, the new PR was very tough. I made several commits but still could not get the approval. After communication with @darionyaphet, a developer at NebulaGraph, I knew that I changed the format of the source data. In his opinion, generally, when users import some data of some unsupported format, the correct response is to throw an exception, but not to directly convert the format. Yes, that makes sense. I focused too much on my business scenario and my only goal was to get the code run successfully but did not consider the general purpose of the tool.

In response to this, I submitted another PR, PR[#2258](https://github.com/vesoft-inc/nebula/pull/2258), and it was approved and merged. I learned a lot from this PR.

3. Later, I found that in nebula-python the thrift had conflicts with `fbthrift`. I thought I could shade the conflicts and submit another PR, but considering the huge modifications, I gave up and chose to raise an issue to NebulaGraph, which was fixed recently.

> Word from NebulaGraph：Welcome to submit PRs to NebulaGraph on GitHub. Here are some issues for your reference: [https://github.com/vesoft-inc/nebula/issues](https://github.com/vesoft-inc/nebula/issues)

# Summary

Before I started using NebulaGraph, I had evaluated JanusGraph thoroughly. After comparisons, I was very impressed with the relatively fewer secret bugs and the active community of NebulaGraph. After the test, NebulaGraph proved itself with its efficiency and became the first choice for distributed graph systems.

The NebulaGraph forum, groups, and GitHub projects are very active. All the users can get replies very quickly, and the PRs are reviewed effectively. I think this is the most important factor of the rapid and strong growth of this [graph database](https://www.nebula-graph.io). I hope I can continuously witness the growth of NebulaGraph and contribute to the improvement of the NebulaGraph ecology!


This article is written by Liu Jiahao, an engineer at the big data team of IntSig Information Co. Ltd (IntSig). He has been playing around with Nebula Graph and is one of our proud GitHub contributors. This post shares his experience importing data to Nebula Graph with Spark.

Hands-On Experience: Import Data to Nebula Graph with Spark


![Performance Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://user-images.githubusercontent.com/57335825/90609288-acf21600-e1b8-11ea-92d0-5638a45fc6c8.png)

# Who Did the Graph Database Comparison

This article describes how the Tencent Cloud team compares NebulaGraph with two other popular graph databases on the market from several perspectives.

By their nature of dealing with interconnections, graph databases are perfect for fraud detection and building knowledge graphs in the security field. To better serve the Tencent Cloud business scenarios, the Tencent Cloud Security team has to select a highly performant [graph database](https://www.nebula-graph.io) which fits the business development well, which is how this performance comparison comes into play.

# Whom to Compare With

## Neo4j

Neo4j is the most widely adopted graph database in the industrial world. It has a Community edition and an Enterprise edition. For comparison in this article, the team has chosen the Community edition.

## HugeGraph (A fork of JanusGraph)

HugeGraph is a distributed graph database developed by Baidu. It is forked from JanusGraph. HugeGraph is developed to address the needs of anti-fraud, threat intelligence collection, and underground economy attack with graph storage and analysis capabilities. It has pretty good read and write performance.

## NebulaGraph

NebulaGraph is an open source distributed graph database developed by vesoft Inc. It features the capability of dealing with super large datasets with hundreds of billions of vertices and trillions of edges.

# Hardware Environments

| Item | Specs |
| --- | --- |
| CPU | Intel Xeon(R) Gold 6133 CPU @ 2.5GHz X86_64 |
| # of Physical CPUs | 2 |
| # of Physical Cores | 20 |
| # of Logical CPUs | 80 |
| Memory | 260 GB |

# Graph Database Performance Comparison Test Results


The Tencent Cloud Security team has used graph data at different orders of magnitudes for testing purpose. The test has been performed against various metrics, including data import efficiency, one-hop query, two-hop query, and shared friends query.

The results are as below:

| Graph Data Size | Platform | Data Import | One-Hop  Query | Two-Hop  Query | Shared Friends Query |
| --- | --- | --- | --- | --- | --- |
| 10 Million Edges | Neo4j | 26s | 6.618s | 6.644s | 6.661s |
|  | HugeGraph | 89s | 16ms | 22ms | 72ms |
|  | NebulaGraph | 32.63s | 1.482ms | 3.095ms | 0.994ms |
| 100 Million Edges | Neo4j | 1min21s | 42.921s | 43.332s | 44.072s |
|  | HugeGraph | 10min | 19ms | 20ms | 5s |
|  | NebulaGraph | 3min52s | 1.971ms | 4.34ms | 4.147ms |
| 1 Billion Edges | Neo4j | 8min34s | 165.397s | 176.272s | 168.256s |
|  | HugeGraph | 65min | 19ms | 651ms | 3.8s |
|  | NebulaGraph | 29min35s | 2.035ms | 22.48ms | 1.761ms |
| 8 Billion Edges | Neo4j | 1h23min | 314.34s | 393.18s | 608.27s |
|  | HugeGraph | 16h | 68ms | 24s | 541ms |
|  | NebulaGraph | ~30min | Less than 1s | Less than 5s | Less than 1s |

Seen from the above table, in terms of data import, NebulaGraph is a bit slower than Neo4j when the data size is small. However, when the data size is large, NebulaGraph is much faster than the other two. For the three graph queries, NebulaGraph shows clearly better performance compared to Neo4j and HugeGraph.

Here is a chart overview of the comparison:

![Graph Database Performance Comparison Chart](https://user-images.githubusercontent.com/57335825/90246630-dfcb9100-dde9-11ea-8ddc-3334e6614c8e.png)

## Graph Query Language Comparison

### Neo4j Cypher

One-Hop Friends Query

```Shell
match ({vid:11111}) -> (u)
return u;
```

Two-Hop Friends Query

```Shell
match ({vid:11111}) -> () -> u
return u;
```

Shared Friends Query

```Shell
match ({vid:11111}) -> (u) <- ({vid:22222})
return u;
```

### NebulaGraph nGQL

One-Hop Friends Query

```Shell
GO FROM 11111 OVER relation
```

Two-Hop Friends Query

```Shell
GO 2 STEPS FROM 11111 OVER relation
```

Shared Friends Query

```Shell
GO FROM 11111 OVER relation INTERSECT
GO FROM 22222 OVER relation
```

# How NebulaGraph Works
1. [NebulaGraph Architecture — A Bird’s Eye View](https://www.nebula-graph.io/posts/nebula-graph-architecture-overview)
2. [Benchmark: NebulaGraph vs Dgraph vs JanusGraph](https://www.nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph)
3. [Deploy the Graph Database on Kubernetes](https://www.nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes)

[![docker](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/docker-banner.png)](javascript:window.open('https://bit.ly/3XHS873'))


This article describes how the Tencent Cloud team compares NebulaGraph with two other popular graph databases on the market, i.e. Neo4j and JanusGraph, from several perspectives. 

Tag