This article attempts to explore the traces of these contributors using graph visualization, "peeping" into the relationship between contributors and projects in the Open Source NebulaGraph Community and the PR traces they leave behind.

Exploring Contributor Relationships in the Open Source NebulaGraph Community: A Graph Visualization Approach

The growth of e-commerce has led to an overwhelming number of choices for consumers, making it increasingly challenging for online retailers to showcase relevant products and services in order to maintain customer loyalty and retain customers.

One of the most effective solutions to this problem is the use of recommendation engines. 
It has emerged as a key technology to bridge the gap, providing personalized suggestions that improve customer satisfaction and increase sales by analyzing the user’s previous behavior. [Graph databases](https://www.nebula-graph.io/posts/what-is-a-graph-database) are particularly well-suited for recommendation engines as they can handle large amounts of interconnected data with ease. In this article, we will explore how recommendation engines work in e-commerce, their different types, and the advantages of using graph databases like NebulaGraph for building powerful recommendation systems.

# What is a Recommendation Engine?

A recommendation engine is an algorithmic system that analyzes data on user behavior, preferences, and historical interactions with a product or service, to make personalized recommendations on content, products, or services that a user is likely to enjoy or find relevant.
Recommendation engines are commonly used by companies that offer personalized content or services, such as online retailers, music or video streaming platforms, and social media sites. By using data on a user's past behavior and preferences, a recommendation engine can make accurate predictions on what the user might be interested in, and present them with personalized recommendations.
Recommendation engines play a crucial role in the customer journey, helping businesses improve user experience, increase sales, and retain customers.

# Types of Recommendation Engines for Ecommerce

There are several types of recommendation engines, each with their own approach to providing personalized recommendations. The three most common types of recommendation engines are:

1. **Collaborative filtering**: Collaborative filtering is a technique that recommends items based on the behavior of similar users. This approach analyzes user behavior data such as purchases, ratings, and viewing history, to find similarities between users and recommend items that users with similar behavior patterns have enjoyed. Collaborative filtering can be further divided into two categories: user-based and item-based filtering.
2. **Content-based filtering**: Content-based filtering recommends items based on the user's past behavior and preferences. This approach uses data about the user's past behavior, such as their viewing history or search queries, to recommend items that are similar in content or characteristics to the user's past interactions. For example, if a user has watched a lot of action movies, a content-based recommendation engine might recommend other action movies.
3. **Hybrid filtering**: Hybrid filtering combines both collaborative and content-based filtering to make recommendations. This approach can provide more accurate recommendations by taking into account both the user's past behavior and preferences as well as the behavior of similar users.

Other types of recommendation engines include knowledge-based filtering, which recommends items based on a user's explicitly stated preferences, and context-aware filtering, which takes into account additional data such as the user's location or time of day to make recommendations.

And when it comes to recommendation engines used in e-commerce, common types are:

- **Personalized Recommendations**: These engines provide individualized suggestions based on a user's behavior, preferences, and history.
- **Popular Items**: These recommend the most popular items within a specific category, time frame, or location.
- **Similar Items**: These engines suggest items that are similar to a specific item, based on their features or user behavior.
- **Complementary Items**: Also known as cross-selling recommendations, these engines suggest items that complement a specific item, such as accessories or related products.

# Why Graph Databases Are Well-suited for Recommendation Engines

Graph databases excel in modeling complex relationships between data entities, making them an ideal choice for recommendation engines. Some key advantages of graph databases for recommendation engines include:

- Relationship-centric data modeling: Graph databases represent data as nodes and edges, making it easy to model relationships between users, products, and other relevant data points.
- High query performance: Graph databases provide fast and efficient query execution, enabling [real-time recommendations](https://www.nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation).
- Flexibility and schema evolution: Graph databases allow for schema changes without affecting existing data or queries, making it easy to incorporate new data sources and recommendation algorithms.
- Advanced analytics capabilities: Graph databases support advanced analytics, such as community detection and centrality measures, that can improve the accuracy and relevance of recommendations.

# NebulaGraph: A Recommended Solution for E-commerce Recommendation Engines

NebulaGraph is an [open-source, distributed graph database](https://www.nebula-graph.io/) that offers high performance, scalability, and ease of use. It is an excellent choice for building recommendation systems for e-commerce because of the following features:

- Efficient querying: NebulaGraph supports a flexible, SQL-like query language called nGQL, allowing for efficient querying and traversing of complex relationships between data entities.
- High scalability: NebulaGraph can handle large-scale graphs and store billions of nodes and edges with low latency, making it ideal for e-commerce platforms with massive amounts of user and item data.
- Data consistency: NebulaGraph ensures strong data consistency through the Raft consensus algorithm, which guarantees that updates are consistently applied across all replicas.
- Easy integration: NebulaGraph provides comprehensive APIs and connectors for popular programming languages, making it simple to integrate with e-commerce platforms and other data processing tools.

# Designing a Recommendation Engine with NebulaGraph Database

Building a recommendation engine with [NebulaGraph](https://nebula-graph.io) involves several steps, including:

1. Data modeling and schema design: Model users, products, and other relevant data points as nodes and their relationships as edges.
2. Importing and processing e-commerce data: Collect data on user behavior, preferences, and product information, and store it in NebulaGraph.
3. Implementing recommendation algorithms using NebulaGraph's query language (nGQL): Develop queries to identify patterns and relationships between users and items and generate personalized recommendations.
4. Testing and validating recommendations: Ensure that the generated recommendations are relevant and accurate by comparing them with actual user behavior and feedback.
5. Integrating the recommendation engine with the e-commerce platform: Use NebulaGraph's APIs and connectors to seamlessly integrate the recommendation engine with the existing e-commerce system, enabling real-time recommendations for users.
6. Continuously updating and refining the recommendation engine: Regularly update the data in NebulaGraph and fine-tune the recommendation algorithms to keep the recommendations fresh and relevant.

# Conclusion

Recommendation engines are essential for e-commerce platforms to provide personalized experiences and improve customer satisfaction. Graph databases, with their ability to model complex relationships and offer high query performance, are well-suited for building such engines. NebulaGraph, an open-source, distributed graph database, is a recommended solution for e-commerce recommendation engines due to its scalability, efficiency, and ease of use. By implementing a recommendation engine with NebulaGraph, e-commerce businesses can significantly enhance their user experience, increase sales, and retain customers in the competitive online marketplace.

# How NebulaGraph Works
1. [NebulaGraph Architecture — A Bird’s Eye View](https://www.nebula-graph.io/posts/nebula-graph-architecture-overview)
2. [Benchmark: NebulaGraph vs Dgraph vs JanusGraph](https://www.nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph)
3. [Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph)
4. [Deploy the Graph Database on Kubernetes](https://www.nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes)
5. [Graph Database For Recommendation Systems](https://www.nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation)


Recommendation engines are essential for e-commerce platforms to provide personalized experiences and improve customer satisfaction. Graph databases, with their ability to model complex relationships and offer high query performance, are well-suited for building such engines. 

How Recommendation Engine for Ecommerce Works Using Graph Database

Memory management in databases is an important module in database kernel design, and measurable and controllable memory is an important guarantee for the stability of the database. Similarly, memory management is also crucial for NebulaGraph Database.
The multi-degree association query feature of graph databases often leads to a huge demand for memory in the execution layer of graph databases. This article mainly introduces the new feature Memory Tracker in NebulaGraph v3.4, aiming to achieve fine-grained control of memory usage by the Memory Tracker module, reduce the risk of system OOM kill for graphd and storaged, and improve the kernel stability of NebulaGraph graph database.


# Available Memory


Before diving into Memory Tracker, let's introduce some relevant background knowledge: available memory.

## Process Available Memory

Here, we briefly introduce how the system determines available memory under various modes.

**Physical Machine Mode**

The database kernel reads the system directory /proc/meminfo to determine the actual memory and available memory in the current environment. Memory Tracker takes "actual physical memory" as the "maximum memory that the process can use".

**Container/cgroup Mode**

In the `nebula-graphd.conf` file, there is a configuration item `FLAG_containerized` to determine whether the database is running in a container. After setting `FLAG_containerized` (false by default) to `true`, the kernel reads the files under the relevant `cgroup path` to determine how much memory the current process can use. There are two versions of `cgroup` - v1 and v2, and v2 is used as an example here.

| FLAG                               | Default Value                       | Explanation                           |
| :--------------------------------- | :---------------------------- | :----------------------------- |
| FLAG_cgroup_v2_memory_max_path     | /sys/fs/cgroup/memory.max     | Determine the maximum memory usage by reading the path |
| FLAG_cgroup_v2_memory_current_path | /sys/fs/cgroup/memory.current | Determine the current memory usage by reading the path |

For example, to control the memory quota for graphd and storaged on a single machine separately, follow these steps:：

Step 1：Set `FLAG_containerized=true`；

Step 2：Create `/sys/fs/cgroup/graphd/`，`/sys/fs/cgroup/storaged/`，and configure `memory.max` under their respective directories；

Step 3：Add relevant configuration in `etc/nebula-graphd.conf`，and `etc/nebula-storaged.conf`

```
--containerized=true
--cgroup_v2_controllers=/sys/fs/cgroup/graphd/cgroup.controllers
--cgroup_v2_memory_stat_path=/sys/fs/cgroup/graphd/memory.stat
--cgroup_v2_memory_max_path=/sys/fs/cgroup/graphd/memory.max
--cgroup_v2_memory_current_path=/sys/fs/cgroup/graphd/memory.current
```

## Memory Tracker Available Memory

After obtaining the "process available memory", the system needs to convert it into memory that Memory Tracker can track. There is a conversion formula between "process available memory" and "Memory Tracker available memory".

```
memtracker_limit = ( total - FLAGS_memory_tracker_untracked_reserved_memory_mb ) *  FLAGS_memory_tracker_limit_ratio
```
![usable_memory](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-blog/memory-tracker/usable_memory.png)

| **FLAG**                                    | **Default Value** | **Explanation**                                                     | **Dynamic Change** |
| :------------------------------------------ | ---------- | ------------------------------------------------------------ | -------------- |
| memory_tracker_untracked_reserved_memory_mb | 50 M        | Memory Tracker manages memory allocated through new/delete, but there may be other ways a process consumes memory besides this method; for example, by calling low-level malloc/free. Memory not tracked by this flag is subtracted during calculations. | Yes            |
| memory_tracker_limit_ratio                  | 0.8        | Memory Tracker allows you to specify the proportion of memory it can use. In some scenarios, we may need to reduce this value to prevent out-of-memory (OOM) issues. | Yes            |

Here we'll elaborate on the usage of `memory_tracker_limit_ratio`:

- In a mixed deployment environment where multiple graphd or storaged instances are deployed together, it needs to be reduced. For example, if graphd only occupies 50% of the memory, you need to manually change it to 0.5 in `nebula-graphd.conf`;
- Value range: In addition to the (0,1] value range for `memory_tracker_limit_ratio`, there are two special values defined:
   - `2`：Dynamically adjusts available memory based on the current system runtime environment as perceived by the database kernel. This method is not real-time and may have some degree of inaccuracy;
   - `3`：The limit will be set to an extremely large value, effectively disabling Memory Tracker;

# How Memory Tracker is Designed and Implemented

Next, let's discuss the design and implementation of Memory Tracker. The overall Memory Tracker design consists of several sub-modules, including **Global new/delete operator**, **MemoryStats**, **system malloc**, and **Limiter**. In this section, we'll focus on the Global new/delete operator and MemoryStats modules.


![memory_tracker](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-blog/memory-tracker/memory_tracker.png)

## Global new/delete operator

Memory Tracker overloads the global new/delete operator, taking control of memory allocation and deallocation, allowing memory quota allocation management before actual memory allocation takes place. This process is broken down into two steps:

- Step 1: Report memory allocation through MemoryStats；
- Step 2: Invoke jemalloc for actual memory allocation behavior；

> **jemalloc**：Memory Tracker does not change the underlying malloc mechanism, still using jemalloc for memory allocation and deallocation.

## MemoryStats

Global memory usage statistics are managed through GlobalMemoryStats and ThreadMemoryStats for global memory and thread-internal memory, respectively;

## ThreadMemoryStats

`thread_local`  variable, execution engine threads maintain their MemoryStats in their ThreadMemoryStats, including "memory Reservation information" and "whether to allow exceptions to be thrown on memory exceeded";

- **Reservation**

Each thread reserves a 1 MB memory quota to avoid frequent requests to GlobalMemoryStats for quota. When applying or returning, ThreadMemoryStats will use a larger memory block as the exchange unit with the global scope. 

- **alloc**：

Only when the locally reserved 1 MB memory is used up, will it ask the global scope for the next 1 MB. This approach is used to minimize the frequency of requesting memory quota from the global scope;


- **dealloc**：

The returned memory is first added to the thread's reserved memory, and when the reserve quota exceeds 1 MB, 1 MB is returned, and the rest is kept;

```
 // Memory stats for each thread.
 struct ThreadMemoryStats {
   ThreadMemoryStats();
   ~ThreadMemoryStats();
 
   // reserved bytes size in current thread
   int64_t reserved;
   bool throwOnMemoryExceeded{false};
 };
```

- **throwOnMemoryExceeded**  

Whether to throw an exception when the thread exceeds the memory quota. Only when `throwOnMemoryExceeded` is set to true will `throw std::bad_alloc` occur. See the `Catch std::bac_alloc` section for scenarios where `throw std::bad_alloc` needs to be disabled.

### GlobalMemoryStats

Global memory quota, maintaining limit and used variables.

- **limit**：limit: Calculated from the runtime environment and configuration information, it represents the maximum memory Memory Tracker can manage. Limit serves the same purpose as the Limiter module. For detailed memory conversion, see the "Memory Tracker Available Memory" section above;

- **used**：Atomic variable, summarizing all reported used memory from threads (including reserved portions). If used + try_to_alloc > limit, and when `throw OnMemoryExceeded` is set to true, `std::bad_alloc` exception will be thrown.


## Catch std::bac_alloc

Since Memory Tracker overloads new/delete and affects all threads, including third-party threads, throwing `bad_alloc` may cause unintended behavior in some third-party threads. To prevent such issues, we use an active memory detection approach in the code path, enabling memory detection in operators, RPC modules, etc.

## Memory detection in operators

In graph/storage operators, add `try...catch` (performing calculations/allocating memory in the current thread) and `thenError` (asynchronously submitting calculation tasks via `folly::Executor`) to detect Memory Tracker throwing `std::bad_alloc`. 

The database then returns an error code through Status, causing the query to fail;
When debugging memory, you can enable the `FLAGS_memory_tracker_detail_log` configuration item in the `nebula-graphd.conf` file and reduce the `memory_tracker_detail_log_interval_ms` to observe memory usage before and after the query;

**folly::future asynchronous execution**

```c++
thenValue([this](StorageRpcResponse<GetNeighborsResponse>&& resp) {
    memory::MemoryCheckGuard guard;
    // memory tracker turned on code scope
    return handleResponse(resp);
})
.thenError(folly::tag_t<std::bad_alloc>{},
    [](const std::bad_alloc&) {
    // handle memory exceed
})
```

**Synchronous execution**

```c++
memory::MemoryCheckGuard guard; \
try {
    // ...
} catch (std::bad_alloc & e) { \
    // handle memory exceed
}
```

## Memory detection in RPC

RPC mainly addresses memory quota control issues during the serialization/deserialization of Request/Response objects. Since storaged responses are all encapsulated in the DataSet structure, the problem is transformed into memory detection during DataSet serialization and deserialization.

**Serialization**： DataSet object construction is in the NebulaGraph operator result return logic, and by default, memory detection is enabled;

**Deserialization**：Explicitly enabled through `MemoryCheckGuard`, exceptions can be caught in `StorageClientBase::getResponse's onError`;

## Error Codes

To help identify which module has issues, NebulaGraph has added relevant error codes, indicating memory exceeded exceptions in graphd and storaged respectively:

```
E_GRAPH_MEMORY_EXCEEDED = -2600, // Graph memory exceeded
E_STORAGE_MEMORY_EXCEEDED = -3600, // Storage memory exceeded
```

# How NebulaGraph Works
1. [NebulaGraph Architecture — A Bird’s Eye View](https://www.nebula-graph.io/posts/nebula-graph-architecture-overview)
2. [Benchmark: NebulaGraph vs Dgraph vs JanusGraph](https://www.nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph)
3. [Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph)
4. [Deploy the Graph Database on Kubernetes](https://www.nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes)


This article mainly introduces the new feature Memory Tracker in NebulaGraph v3.4, aiming to achieve fine-grained control of memory usage by the Memory Tracker module, reduce the risk of system OOM kill for graphd and storaged, and improve the kernel stability of NebulaGraph graph database.

Memory Tracker:  Memory Management Practice in NebulaGraph Database

Graph databases have been the fastest growing database category since 2013.

Graph Database Market Overview

A graph database is a specialized type of NoSQL database that stores the connections between entities (nodes) and the relations associated with them (edges).

What is a Graph Database and What are the Benefits of Graph Databases

With the rapid development of network information technology, data is gradually developing towards multi-source heterogeneity. Inside the multi-source heterogeneous data lies countless inextricable relations. And this kind of relations, together with the the network structure, are sure essential for data analysis. Unfortunately, when it comes to large scale data analysis, the traditional relational databases are poor in association detection and expressions. Therefore, graph data has attracted great attention for its powerful ability in expressions. Graph computing uses a graph model to express and solve the problem. Graphs can integrate with multi-source data types. In addition to displaying the static basic features for data, graph computing also finds its chance to display the graph structure and relationships hidden in the data. Thus graph becomes an important analysis tool in social network, recommendation system, knowledge graph, financial risk control, network security and text retrieval.

# Practical Algorithms

In order to support the business needs of large-scale graph computing, [Nebula Graph](https://github.com/vesoft-inc/nebula) provides [PageRank](https://en.wikipedia.org/wiki/PageRank) and [Louvain](https://en.wikipedia.org/wiki/Louvain_method) community-discovered graph computing algorithms based on [GraphX](https://github.com/apache/spark/tree/master/graphx). Users can execute these algorithm applications by submitting Spark tasks. In addition, users can also write Spark programs by using Spark Connector to call other built-in graph algorithms in GraphX, such as LabelPropagation, ConnectedComponent, etc.

## PageRank

PageRank is an algorithm raised by Google to rank web pages in their search engine results. PageRank is a way of measuring the importance of website pages.

### Introduction to PageRank

PageRank was named after Larry Page and Sergey Brin from Stanford in the United States. PageRank works by counting the number and quality of links to a page to determine a rough estimate of how important the website is. The underlying assumption is that more important websites are likely to receive more links from other websites.

### Applications of PageRank

#### Content recommendation based on similarity

With the help of the PageRank, you can recommend similar content to users based on their browse history and view time when analyzing social applications such as Twitter and Facebook.

#### Analyze the social influence of users

You can analyze the social influence of users according to their PageRank values in social network analysis.

#### Importance research for papers

Judge a paper's quality according to its PageRank value. Actually the PageRank algorithm comes from the idea of judging the quality of the papers. In addition, PageRank also finds its usage in data analysis and mining.

### Implementation PageRank

PageRank in GraphX is implemented based on the Pregel computing model. The algorithm contains three procedures:

1. Set a same initial PageRank value for every vertex (web page) in the graph;
1. The first iteration: Send a message along the edge. Each vertex receives all the vertices information along its related edges and gets a new PageRank value;
1. The second iteration: Put the PageRank values you get in the first iteration to the formulas corresponding to different algorithm models and get the new PageRank values.

## Louvain method

The Louvain method for community detection is a method to extract communities from large networks. The method is an aggregation algorithm for graphs.

### Introduction to Louvain

The inspiration for Louvain is the optimization of modularity as the algorithm progresses. If a node joins a certain community and maximizes the modularity of the community compared to other communities, then the node belongs to the community. If a node doesn't increase the modularity after joining other communities, it should stay in the current community.

#### Modularity

##### Modularity formula

The modularity Q measures the density of links inside communities compared to links between communities. Modularity is defined as:

$$Q = \frac{1}{2m} \times \sum_{i,j}[A_{ij} - \frac{k_ik_j}{2m}]\delta(c_i,c_j)  \\
 conditions: \delta(u,v) = \begin{cases}
                 1 &\text{when u == v} \\
                 0 &\text {}else
                 \end{cases}
$$                 

where

- $A_{ij}$ represents the edge weight between nodes $i$ and $j$.
- $K_i$ reprsents the sum of the weights of the edges attached to nodes $i$.
- $C_i$ is the community of the node $i$.
- $m$ is the sum of all of the edge weights in the graph.

**Deformation for the modularity formula**

In this formula, the formula is only meaningful when node i and node j belong to the same community. Therefore the formula measures the closeness within a certain community. The simplified deformation of this formula is as follows:

$ Q = \frac{1}{2m} \times \sum{i,j}[A{i,j} - \frac{k_ik_j}{2m}]\delta(c_i,c_j) \\
= \frac{1}{2m}[\sum{i,j}A{i,j} - \frac{\sum_ik_i \sum_jk_j}{2m}]\delta(c_i,c_j) \\
= \frac{1}{2m} \sum_c[\sum{in} - \frac{{\sum{tot}}^{2}}{2m}] $

where

- $\Sigma_{in}$ represents the sum of the weights of the edges in community c.
- $\Sigma_{k_n}$ represents the sum of the weighs of the edges connected with the nodes in community c (because $i$ belongs to community c), including the edges inside and outside the community attached to the node $i$.
- $\Sigma_{k_j}$ represents the sum of the weighs of the edges connected with the nodes in community c (because $j$ belongs to community c), including the edges inside and outside the community attached to the node $j$.
- $\Sigma{tot}$ replaces $\Sigma{k_i}$ and $\Sigma_{k_j}$. (Namely the sum of the weights of the edges in community c plus the sum of the weights of the edges in other community attached to community c. )

**Calculate the modularity change**

In the Louvain method, there is no need to calculate the specific modularity for each community. You only need to compare the modularity changes after adding a certain node to the community. That is to say you only need to calculate △Q.

When inserting node i to a certain community, the modularity change is:

$$\Delta{Q} = [\frac{\sum{in} + k{i,in}}{2m} -({\frac{\sum{tot} + k_i}{2m}})^{2}] - [\frac{\sum{in}}{2m} - (\frac{\sum_{tot}}{2m})^{2} - (\frac{k_i}{2m})^{2}] \\
= [\frac{k{i,in}}{2m} - \frac{\sum{tot}k_i}{2m^2}]$$

where

- $k_{i,in}$ is the sum of the weights of the edges between $i$ and other nodes in the community (This is 2 times of the actual weight in the new community, because all the edges attached to node i in the community are calculated twice).
- $K_i$ is the weighted degree of $i$. Thus when implementing the algorithm, you only need to calculate the $[k{i,in} - \sum{tot} \times \frac{k_i}{m}]$.

### Applications of Louvain

- Financial risk control

In financial risk control scenarios, you can use the method to identify fraud gangs based on the users behaviors.

- Social network

Louvain divides the social network based on the breadth and strength of nodes association in a graph. Louvain also analyzes complex networks and the closeness among a group of people.

- Recommendation system

Community detection based on the users’ interests. More accurate and effective customized recommendation can be implemented based on the community and collaborative filtering algorithm.

### Implement Louvain

The Louvain method contains two stages. The procedure is actually the iteration of the two stages.

**Stage 1:** Continuously traverse the nodes in the graph, and compare the modularity changes introduced by nodes in each neighbor community. Then add a single node to the community that can maximize the modularity. (For example, node v is added to communities A, B, and C respectively. The modularity increments of the three communities are -1, 1, 2. Then node v should be added to community C.)

**Stage 2:** Process based on the first stage. The nodes belonging to the same community are merged into a super node to reconstruct the graph. That is,  the community is regarded as a new node in the graph. At this time, the weight of the edges between the two super nodes are the sum of the weight of the edges attached to the original nodes in the two super nodes. That is, the sum of the weight of the edges in the two communities.

Following are details for the two stages.

![Louvain](https://user-images.githubusercontent.com/57335825/99624951-03ebcf00-29e4-11eb-9000-911efdf519d1.png)

In the first stage, the nodes are traversed and then added to the communities they belong to. In this way, we get the middle graph and the four communities.

In the second stage, the nodes in the community are merged into a super node. The community nodes have self-connected edges whose weight is twice the sum of the weights of the edges connected between all nodes in the community. The edge weight between the communities is the sum of the weights of the edges connecting the two communities. For example, the red community and the light green community are connected by (8,11), (10,11), (10,13). So the weight of the edges between the two communities is 3.

> Note: The weight inside the community is twice the weight of the edges between all internal nodes, because the concept of Kin is the sum of the edges of all nodes in the community and node i. When calculating the Kin for a certain community, each edge is actually counted twice.

The Louvain method continuously iterates stage 1 and stage 2 until the algorithm is stable (the modularity for the graph does not change) or the iterations reach the maximum.

# Practice the proceeding algorithms

## Demo environment

- Three virtual machines:
   - Cpu name: Intel(R) Xeon(R) Platinum 8260M CPU @ 2.30GHz
   - Processors: 32
   - CPU Cores: 16
   - Memory Size: 128G
- Software environment:
   - Spark: spark-2.4.6-bin-hadoop2.7 a cluster with three nodes
   - yarn V2.10.0: a cluster with three nodes
   - NebulaGraph V1.1.0: distributed deployed, used the default configurations

## Dataset for testing

1. Create graph space

```shell
CREATE SPACE algoTest(partition_num=100, replica_factor=1);
```

2. Create schema

```shell
CREATE TAG PERSON()
CREATE EDGE FRIEND(likeness double);
```

3. Import data

Use Spark Writer to import data offline into NebulaGraph.

4. Test results

The resource allocation fot the Spark job is `--driver-memory=20G --executor-memory=100G --executor-cores=3`.

- PageRank algorithm execution time on a dataset with hundred million nodes is 21 minutes.
- Louvain algorithm execution time on a dataset with hundred million nodes is 1.3 hours.

## How to use NebulaGraph algorithm

1. Download and pack the nebula-algorithm project to a jar package.

```shell
$ git clone git@github.com:vesoft-inc/nebula-java.git
$ cd nebula-java/tools/nebula-algorithm
$ mvn package -DskipTests
```

2. Configure the file `src/main/resources/application.conf`.

```xml
{
  # Spark relation config
  spark: {
    app: {
        # not required, default name is the algorithm that you are going to execute.
        name: PageRank

        # not required
        partitionNum: 12
    }

    master: local

    # not required
    conf: {
        driver-memory: 8g
        executor-memory: 8g
        executor-cores: 1g
        cores-max:6
    }
  }

  # NebulaGraph relation config
  nebula: {
    # metadata server address
    addresses: "127.0.0.1:45500"
    user: root
    pswd: nebula
    space: algoTest
    # partition specified while creating nebula space, if you didn't specified the partition, then it's 100.
    partitionNumber: 100
    # nebula edge type
    labels: ["FRIEND"]

    hasWeight: true
    # if hasWeight is true，then weightCols is required， and weghtCols' order must be corresponding with labels.
    # Noted: the graph algorithm only supports isomorphic graphs,
    #        so the data type of each col in weightCols must be consistent and all numeric types.
    weightCols: [“likeness”]
  }

  algorithm: {
    # the algorithm that you are going to execute，pick one from [pagerank, louvain]
    executeAlgo: louvain
    # algorithm result path
    path: /tmp

    # pagerank parameter
    pagerank: {
        maxIter: 20
        resetProb: 0.15  # default 0.15

    }

    # louvain parameter
    louvain: {
        maxIter: 20
        internalIter: 10
        tol: 0.5
   }
  }
}
```

3. Make sure that you have installed and started the Spark service on your machine.
4. Submit the nebula algorithm application:

```shell
spark-submit --master xxx --class com.vesoft.nebula.tools.algorithm.Main /your-jar-path/nebula-algorithm-1.0.1.jar -p /your-application.conf-path/application.conf
```

If you are interested in the content, welcome to try nebula-algorithm.

# References

- NebulaGraph: [https://github.com/vesoft-inc/nebula](https://github.com/vesoft-inc/nebula)
- GraphX: [https://github.com/apache/spark/tree/master/graphx](https://github.com/apache/spark/tree/master/graphx)
- nebula-algorithm: [https://github.com/vesoft-inc/nebula-java/tree/v1.0/tools/nebula-algorithm](https://github.com/vesoft-inc/nebula-java/tree/v1.0/tools/nebula-algorithm)

> Hi, I’m Nicole, the software engineer in NebulaGraph. Let me know if you have any thoughts and suggestions on this article. Welcome to talk to me on our [forum](https://discuss.nebula-graph.io/). Thanks!

# You Might Also Like

- [Exploring the S&P 100 Index Stocks Using Graph Machine Learning](https://nebula-graph.io/posts/stock-interrelation-analysis-nebula-graph-machine-learning/)
- [Analyzing Relationships in Game of Thrones With NetworkX, Gephi, and NebulaGraph (Part Two)](https://nebula-graph.io/posts/game-of-thrones-relationship-networkx-gephi-nebula-graph-part-two/)


In order to support the business needs of large-scale graph computing, NebulaGraph provides PageRank and Louvain community-discovered graph computing algorithms based on GraphX. Users can execute these algorithm applications by submitting Spark tasks. In addition, users can also write Spark programs by using Spark Connector to call other built-in graph algorithms in GraphX, such as LabelPropagation, ConnectedComponent, etc.

Practicing Graph Computation with GraphX in NebulaGraph

In the modern business environment where data inevitably rules, traditional relational databases are no longer enough to keep up with the increasing complexity and interconnectivity of information. Enter graph databases, the revolutionary technology that has taken the tech industry by storm - for a good reason. 

With the ability to store and manage vast amounts of complex and interconnected data, graph databases are rapidly becoming the go-to solution for businesses and organizations of all sizes. 

According to a [report](https://www.databridgemarketresearch.com/reports/global-graph-database-market) by Data Bridge Market Research, the global market for graph databases was around USD 1938.20 million in 2022. The same report projects that the industry will grow at a compounded annual rate of 18.20% and reach USD 7384.79 million by 2030.

But which is the best graph database for enterprise? There are a few good players, and this article is dedicated to looking at Neo4j vs TigerGraph vs Dgraph vs NebulaGraph. Whether your intention is to use [open source graph database](https://nebula-graph.io/) options or commercial ones, you'll find this useful. 

Meanwhile if you are just beginning out with graph databases, we recommend that you also make sure to go through this valuable [guide on how to choose a graph database](https://www.nebula-graph.io/posts/how-to-select-a-graph-database). 


# Benefits of graph databases for enterprises

These benefits represent the outstanding role that graph databases play in helping enterprises to get the most out of their big data.


## **Data Integrity**

Graph databases ensure data integrity by allowing developers to define constraints that enforce data relationships and eliminate anomalies or inconsistencies. This makes them ideal for use cases where data quality and accuracy are critical, such as financial transactions or healthcare records. 

For example, a financial institution could use a graph database to ensure the accuracy and completeness of customer data across multiple accounts and transactions.


## **Semantic Reasoning**

Semantic reasoning is a process by which a computer program or system is able to derive new facts or insights from existing data by analyzing the relationships between entities and inferring new information based on those relationships. This is done by applying logical rules and algorithms that take into account the meaning and context of the data being analyzed, as opposed to just processing the data as raw information.

In the context of graph databases, semantic reasoning can be used to gain deeper insights into the relationships between entities in the graph, even when those relationships are not explicitly stated. By analyzing the semantics of the data and applying logical rules, a graph database can infer new relationships and properties between entities, leading to a richer and more comprehensive understanding of the data. This can be particularly useful in fields such as life sciences, where researchers may be trying to identify new connections between genes and diseases, or in finance, where analysts may be trying to detect patterns in market data that are not immediately apparent.


## **Data Integration**

Graph databases can be used to integrate data from different sources, enabling users to analyze data from disparate sources in a single location. This enables enterprises to gain a more complete picture of their data, and also helps to identify relationships and patterns that might not be apparent when analyzing individual data sources in isolation. 

For example, an e-commerce company can use a graph database to integrate customer data from multiple sources, such as social media profiles, web browsing history, and purchase history, in order to better understand customer behavior and preferences.


## **Collaboration**

Collaboration is a crucial element for success in many enterprises, but it can often be challenging due to siloed data and disparate systems. Graph databases offer a solution to this problem by providing a unified and comprehensive view of data that can be accessed by multiple teams and departments within an organization.

For example, a pharmaceutical company may have separate teams working on drug development, clinical trials, and marketing. Each team may have its own systems and data sources, making it difficult to share information and collaborate effectively. However, by using a graph database to store all the data in a unified structure, each team can easily access and analyze the same information, regardless of where it originated. This can help to break down silos and improve collaboration across the organization, leading to more efficient drug development and faster time-to-market.

Furthermore, graph databases can enable real-time collaboration by providing a single source of truth that is always up-to-date. Changes made by one team or department are immediately visible to others, allowing for faster decision-making and more agile processes.


## **Regulatory Compliance**

One of the main advantages of graph databases in regulatory compliance is their ability to maintain a complete and auditable view of data. The ability to store information about data lineage and changes means that graph databases can track data as it moves through an enterprise's systems, allowing enterprises to demonstrate compliance with regulations such as GDPR and HIPAA.

For example, consider a healthcare provider that uses a graph database to manage patient records. By simply tracking changes to patient records over time, the provider can demonstrate compliance with regulations that require the maintenance of an audit trail. In the event of an audit or investigation, the provider can easily provide a complete record of all changes.

Another way that graph databases can help enterprises meet regulatory compliance requirements is by providing a granular level of access control. This is achieved through defining roles and permissions for each user, ensuring that only authorized personnel have access to sensitive data. This can help enterprises comply with regulations that require the protection of personal data.


# Overview of Neo4j, TigerGraph, Dgraph, NebulaGraph

Here is a brief view of each of the graph databases we are focussing on:


## **Neo4j**

Neo4j has become one of the most popular graph databases on the market. 

According to the parent  company Neo4j, Inc, the solution is currently being used by over 900 enterprises and enjoys a strong community of over 200,000 members. Over 30 million downloads and 150,000 instances are executed every year.


## **TigerGraph**

TigerGraph was first released in 2012 by TigerGraph Inc, though in stealth mode. It came out of stealth in 2017.

Information appearing on [Techtarget](https://www.techtarget.com/searchbusinessanalytics/news/252492072/TigerGraph-taking-advantage-of-graph-database-growth) indicates that TigerGraph may have had about 4000 users in TigerGraph Cloud by 2021. However, it's not clear if these are pure enterprise customers or a mix of enterprise and individuals.


## **Dgraph**

Written purely in Go, Dgraph came to the market in 2017 but was first introduced in 2015 by Dgraph Labs, Inc. According to Dgraph, their community is now at 21, 000 members.

Dgraph Enterprise, which is our focus, is actually built atop the open source offering.


## **NebulaGraph**

NebulaGraph is also an open source graph database whose first version, Nebula Graph v0.1.0-alpha, was first released in 2019. Check out the entire [timeline of NebulaGraph](https://docs.nebula-graph.io/3.0.2/20.appendix/history/).  

The [NebulaGraph Enterprise](https://www.nebula-graph.io/enterprise) version gives companies access to enhanced visual tools, Analytics, enterprise-level security and dedicated support. Big companies including Tencent, Oppo, Vivo, and WeBank are already using NebulaGraph Enterprise. Others like [360 DigiTech have migrated to NebulaGraph](https://www.nebula-graph.io/posts/why-migrate-janusgraph-nebula-graph) from their previous graph solutions.  

NebulaGraph has now passed 200,000 pull requests and is already at over 8000 Github stars (in 2023). This is remarkable considering NebulaGraph is in fact the youngest of all the rest here. 


#  Comparison across key graph database features and capabilities

The above graph databases can offer numerous benefits to your enterprise. But how do you choose between them? 

Below is a detailed comparison of the features and capabilities of these databases, which will help you make an informed decision when choosing the right one for your enterprise.

We would also like to clarify that the details here are mostly based on publicly available information and also from the graph databases own assertions, and not on practical tests that we have done ourselves. 


## **Performance**



* **Neo4j:** Offers several high performance features, such as optimized indexing and query planning. Also,[ according to InfoWorld,](https://www.infoworld.com/article/3679269/neo4j-50-improves-on-scalability-performance-and-agility.html) the latest database version includes autonomous clustering (for connecting related nodes).
* **TigerGraph:** According to TigerGraph's performance benchmark report, the database is up to 377 times faster than other graph databases in two-hop path queries.
* **Dgraph:** Dgraph offers several performance features, such as optimized read and write speeds and concurrent caching.  
* **NebulaGraph:** NebulaGraph is designed to be automatically highly performant for large-scale graph data.Here is the detailed [NebulaGraph performance report](https://www.nebula-graph.io/posts/nebula-graph-v2.6.0-performance-report).  The Tencent Cloud Security team tested NebulaGraph and Neo4j alongside HugeGraph. They found that while Neo4j outperformed NebulaGraph slightly with smaller data sets, NebulaGraph proved significantly faster than the others with larger data sets. Here are the [comprehensive results](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph) of the test.


## **Data modeling**



* **Neo4j:** Neo4j is a property graph database and thus represents data in nodes, edges, and properties. The relationships are directional – either unidirectional or bidirectional.
* **TigerGraph:** Like Neo4j, TigerGraph is a directional property graph representing data in nodes, edges, and properties. The edges are equally directed.
* **Dgraph:** Dgraph is also a property graph database, which models data into nodes and edges. The edges are also directional and contain properties that define the relationship of the data sets. It supports reverse traversals, allowing you to follow the data flow from the end to the start nodes.
* **NebulaGraph:** Users can create and modify their data models on-the-fly, without the need for any pre-defined schemas or constraints.This allows users to model their data in a way that best fits their specific use case, without being constrained by pre-existing schemas. It also supports property graphs. 


## **Query language**



* **Neo4j:** Neo4j uses its own query language, Cypher, which features simple and human-readable syntax and commands.
* **TigerGraph:** Uses the GSQL language, which has highly-expressive syntax, leading to improved completeness of the retrieved data. The language also supports parallelism, which improves query speeds.This language has some similarities to SQL.
* **Dgraph:** Uses the GraphQL language, which is highly expressive yet direct. GraphQL will enable you to define the shape of your data, leading to improved modeling.
* **NebulaGraph:** Provides a powerful query language known as [nQL](https://docs.nebula-graph.io/3.0.0/3.ngql-guide/1.nGQL-overview/1.overview/), which is similar in syntax to SQL and is designed for both developers as well as operations teams.


## **Ecosystem**



* **Neo4j:** Neo4j is equipped with a wide collection of libraries, tools, and drivers, enabling  seamless integration. For example, the connector tools make it easy to integrate Neo4j to popular data tools including data warehouses such as Google BigQuery.  
* **TigerGraph:** The TigerGraph ecosystem is powered by RESTful APIs and JSON output. Also has several integration connectors, allowing you to import your data from multiple sources.
* **Dgraph:** Uses proprietary software tools to offer data integration. According to a 2021 news report on DevOps, Dgraph Labs (the parent company) contracted software firm Capventis to help expand its data integration tools. One of the solutions acquired was Glu,  a software that allows the database to unite data from multiple sources during analysis.
* **NebulaGraph:** [NebulaGraph’s ecosystem](https://docs.nebula-graph.io/3.2.0/20.appendix/6.eco-tool-version/) allows users to seamlessly integrate NebulaGraph with other systems. For example, it integrates with Apache Spark for big data processing. This allows users to perform graph computations on NebulaGraph data using Spark's powerful parallel processing capabilities. Check out this comprehensive guide on [using NebulaGraph in Apache Spark](https://www.nebula-graph.io/posts/work-with-nebula-graph-apache-spark). 


## **Analytics**



* **Neo4j:** Neo4j has optimized graph algorithms, which help reveal hidden patterns and improve data visualization. The database also supports AI and machine learning integration, which allows for more optimal analytics.
* **TigerGraph:** Uses algorithms with parallel computation, allowing faster and improved data retrievals. It also supports AI and Machine Learning for cloud data.
* **Dgraph:** Also offers excellent analytic features, making it popular amongst businesses and enterprises that process big data. GraphQL is a potent analytical tool that offers fast and optimal data visualization with numerous analytical information.
* **NebulaGraph:** Offers an enterprise analytics platform known as [NebulaGraph Analytics](https://docs.nebula-graph.io/3.4.1/graph-computing/nebula-analytics/) which is purely dedicated to graph analytics. 


## **Scalability**



* **Neo4j:** Neo4j is famous for, among other things, offering impressive scalability capabilities. According to a press release published on PRNewswire in 2021, the database can support up to 200 billion nodes and over one trillion relationships.
* **TigerGraph:** TigerGraph is also becoming increasingly popular due to its scalability features. According to a news report on Silicon Angle, TigerGraph can support up to 70 billion nodes and 500 billion edges (relationships).
* **Dgraph:** Supports horizontal scaling. According to a news report published on GlobalNewsWire, the database was used by a company in China to store 48 billion triple datasets.
* **NebulaGraph:** Also offers horizontal scalability. You can  expand the cluster by adding more nodes or services without impacting performance. Moreover, scaling out NebulaGraph is a breeze as it doesn't require reconfiguring existing nodes - all you need is just sufficient bandwidth. 


## **Security**



* **Neo4j:** Offers a comprehensive access-control security mode, which makes it possible to restrict access to individual nodes and properties. The database also has control permission features for data reading, writing, and traversing. Additionally, it allows for integrating third-party data security measures such as data encryption, BOLT and HTTPS support, and SSL certificates.
* **TigerGraph:** Equally offers several security capabilities, including network-access control, authentication, and encryption. According to a news report on Datanami, TigerGraph has of late made several updates, including adding new security features to its cloud platform. The latest updates are expected to bolster the database's stability, performance, and security.
* **Dgraph:** Offers diverse security features, such as network access control and data encryption. The database also allows you to create access control to the GraphQL endpoint applications, thus limiting the people who can access and query data. 
* **NebulaGraph:** Supports role-based [access control](https://db-engines.com/en/system/NebulaGraph%3BNeo4j%3BTigerGraph) to ensure the confidentiality, integrity, and availability of user data. The access control mechanism is [implemented](https://docs.nebula-graph.io/3.2.0/7.data-security/1.authentication/1.authentication/) through local authentication or LDAP authentication. 


# Conclusion

The graph database you choose can enormously impact the kind of value your enterprise gets from data. This is why the selection process demands careful reflection to avoid the common pitfalls that businesses often encounter. 

One of the most prevalent missteps is choosing a database solution that has limited scalability, thereby restricting the ability to quickly adapt to future data management needs. Another mistake is settling for a database solution that underperforms or has inadequate security features, jeopardizing the integrity and confidentiality of the data.

With the above comparison and the resources provided, you should be able to make an informed decision and choose a graph database that is best suited for your enterprise's needs. 


The graph database you choose can enormously impact the kind of value your enterprise gets from data. With the comparison and the resources provided, you should be able to make an informed decision and choose a graph database that is best suited for your enterprise’s needs.

Best Graph Database for Enterprise: Neo4j vs TigerGraph vs Dgraph vs NebulaGraph Comparison



>This ariticle was written by Ajeet Singh Raina and Wey Gu, originally published on [Docker blog](https://www.docker.com/blog/distributed-cloud-native-graph-database-nebulagraph-docker-extension/).

Graph databases have become a popular solution for storing and querying complex relationships between data. As the amount of graph data grows and the need for high concurrency increases, a distributed graph database is essential to handle the scale.

Finding a distributed graph database that automatically shards the data, while allowing businesses to scale from small to trillion-edge-level without changing the underlying storage, architecture of the service, or application code, however, can be a challenge. 

In this article, we’ll look at NebulaGraph, a modern, open source database to help organizations meet these challenges.


# Meet NebulaGraph
[NebulaGraph](https://nebula-graph.io) is a modern, open source, cloud-native graph database, designed to address the limitations of traditional graph databases, such as poor scalability, high latency, and low throughput. NebulaGraph is also highly scalable and flexible, with the ability to handle large-scale graph data ranging from small to trillion-edge-level.

NebulaGraph has built a thriving community of more than [1000 enterprise users](https://nebula-graph.io/cases) since 2018, along with a rich ecosystem of tools and support. These benefits make it a cost-effective solution for organizations looking to build graph-based applications, as well as a great learning resource for developers and data scientists.

The NebulaGraph cloud-native database also offers [Kubernetes Operators](https://github.com/vesoft-inc/nebula-operator) for easy deployment and management in cloud environments. This feature makes it a great choice for organizations looking to take advantage of the scalability and flexibility of cloud infrastructure.

# Architecture of the NebulaGraph database
NebulaGraph consists of three services: the Graph Service, the Storage Service, and the Meta Service (Figure 1). The Graph Service, which consists of stateless processes (nebula-graphd), is responsible for graph queries. The Storage Service (nebula-storaged) is a distributed (Raft) storage layer that persistently stores the graph data. The Meta Service is responsible for managing user accounts, schema information, and Job management. With this design, NebulaGraph offers great scalability, high availability, cost-effectiveness, and extensibility.

![Nebulagraph database architecture infographic](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-database-architecture-infographic.png)
Figure 1: Overview of NebulaGraph services.

# Why NebulaGraph?
NebulaGraph is ideal for graph database needs because of its architecture and design, which allow for high performance, scalability, and cost-effectiveness. The architecture follows a separation of storage and computing architecture, which provides the following benefits:

- **Automatic sharding**: NebulaGraph automatically shards graph data, allowing businesses to scale from small to trillion-edge-level data volumes without having to change the underlying storage, architecture, or application code.
- **High performance**: With its optimized architecture and design, NebulaGraph provides high performance for complex graph queries and traversal operations.
- **High availability**: If part of the Graph Service fails, the data stored by the Storage Service remains intact.
- **Flexibility**: NebulaGraph supports property graphs and provides a powerful query language, called Nebula Graph Query Language (nGQL), which supports complex graph queries and traversal operations. 
- **Support for APIs**: It provides a range of APIs and connectors that allow it to integrate with other tools and services in a distributed system.

# Why run NebulaGraph as a Docker Extension?
In production environments, NebulaGraph can be deployed on Kubernetes or in the cloud, hiding the complexity of cluster management and maintenance from the user. However, for development, testing, and learning purposes, setting up a NebulaGraph cluster on a desktop or local environment can still be a challenging and costly process, especially for users who are not familiar with containers or command-line tools.

This is where the NebulaGraph Docker Extension comes in. It provides an elegant and easy-to-use solution for setting up a fully functional NebulaGraph cluster in just a few clicks, making it the perfect choice for developers, data scientists, and anyone looking to learn and experiment with NebulaGraph.

## Getting started with NebulaGraph in Docker Desktop
### Setting up
Prerequisites: [Docker Desktop 4.10](https://docs.docker.com/desktop/) or later.

### Step 1: Enable Docker Extensions
You’ll need to enable Docker Extensions under the Settings tab in Docker Desktop. Within Docker Desktop, confirm that the **Docker Extensions** feature is enabled (Figure 2). Go to **Settings > Extensions** and select **Enable Docker Extensions**.

![Docker desktop extensions settings](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/docker-desktop-extensions-settings-980x557.png)
Figure 2: Enabling Docker Extensions within the 
Docker Desktop.

All Docker Extension resources are hidden by default, so, to ensure its visibility, go to **Settings > Extensions** and check the **Show Docker Extensions system containers**.

### Step 2: Install the NebulaGraph Docker Extension
The NebulaGraph extension is available from the Extensions Marketplace in Docker Desktop and on [Docker Hub](https://hub.docker.com/extensions/weygu/nebulagraph-dd-ext). To get started, search for NebulaGraph in the Extensions Marketplace, then select **Install** (Figure 3).

![Nebulagraph extensions marketplace](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-extensions-marketplace-980x624.png)
Figure 3: Installing NebulaGraph from the Extensions Marketplace.

This step will download and install the latest version of the NebulaGraph Docker Extension from Docker Hub. You can see the installation process by clicking Details (Figure 4).


![Nebulagraph installation progress docker desktop](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-installation-progress-docker-desktop-980x624.png)
Figure 4: Installation progress.

### Step 3: Waiting for the cluster to be up and running
After the extension is installed, for the first run, it normally takes fewer than 5 minutes for the cluster to be fully functional. While waiting, we can quickly go through the **Home** tab and **Get Started** tab to see details of NebulaGraph and NebulaGraph Studio, the WebGUI Utils.

We can also confirm whether it’s ready by observing the containers’ status from the **Resources** tab of the Extension as shown in Figure 5.

![Nebulagraph resources docker desktop](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-resources-docker-desktop-980x624.png)
Figure 5: Checking the status of containers.

### Step 4: Get started with NebulaGraph
After the cluster is healthy, we can follow the Get Started steps to log in to the NebulaGraph Studio, then load the initial dataset, and query the graph (Figure 6).

![Nebulagraph get started docker desktop](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-get-started-docker-desktop-980x624.png)
Figure 6: Logging in to NebulaGraph Studio.

## Step 5: Learn more from the starter datasets 
In a graph database, the focus is on the relationships between the data. With the starter datasets available in NebulaGraph Studio, you can get a better understanding of these relationships. All you need to do is click the Download button on each dataset card on the welcome page (Figure 7).

![starter datasets](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-starter-datasets-980x937.png)
Figure 7: Starter datasets.

For example, in the ```demo_sns``` (social network) dataset, you can use the following query to find new friend recommendations by identifying second-degree friends with the most mutual friends:

```
1 Einstein
```

![Nebula console demo sns](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebula-console-demo-sns-980x782.png)
Figure 8: Query results shown in the Nebula console.

Instead of just displaying the query results, you can also return the entire pattern and easily gain insights. For example, in Figure 9, we can see LeBron James is on two mutual friend paths with Tim:

![Nebula console demo sns results graph](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebula-console-demo-sns-results-graph-980x853.png)
Figure 9: Graphing the query results.


Another example can be found in the ```demo_fraud_detection``` (graph of loan) dataset, where you can perform a 10-degree check for risky applicants, as shown in the following query:
```Cypher
MATCH 
p_=(p:`applicant`)-[*1..10]-(p2:`applicant`)
WHERE id(p)=="p_200" AND 
p2.`applicant`.is_risky == "True"
RETURN p_ LIMIT 100
```

The results shown in Figure 10 indicate that this applicant is suspected to be risky because of their connection to ```p_190```.

![Nebula console demo fraud detection graph](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebula-console-demo-fraud-detection-graph-980x783.png)
Figure 10: Results of query showing fraud detection risk.

By exploring the relationships between data points, we can gain deeper insights into our data and make more informed decisions. Whether you are interested in finding new friends, detecting fraudulent activity, or any other use case, the starter datasets provide a valuable starting point.

We encourage you to download the datasets, experiment with different queries, and see what new insights you can uncover, then share with us in the NebulaGraph community.

# Try NebulaGraph for yourself

To learn more about NebulaGraph, visit our [website](https://nebula-graph.io), [documentation site](https://docs.nebula-graph.io/), star our [GitHub repo](https://github.com/vesoft-inc/nebula), or join our [community chat](https://community-chat.nebula-graph.io/).

[![docker](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/docker-banner.png)](javascript:window.open('https://bit.ly/3XHS873'))



NebulaGraph Docker Extension provides an elegant and easy-to-use solution for setting up a fully functional NebulaGraph cluster in just a few clicks, making it the perfect choice for developers, data scientists, and anyone looking to learn and experiment with NebulaGraph.

Distributed Cloud-Native Graph Database with NebulaGraph Docker Extension

Organizations worldwide are facing a huge problem due to the exponential growth of data. According to the [Institute of Electrical and Electronics Engineers](https://cmte.ieee.org/futuredirections/2021/01/29/latest-data-on-data/), about 168 zettabytes of data will have been created by 2025. This is massive, and it’s not yet peak level. What levels of data are we looking at in a decade and beyond? 

These are mind boggling figures. But there is one more trend that is also worrying: Traditional databases are not very good at handling the overwhelming data volume .  A [study](https://www.researchgate.net/publication/360155336_Handling_Big_Data_in_Relational_Database_Management_Systems) published on ResearchGate reveals just how limited relational databases are, particularly when dealing with big data.

Because of this growing frustration with traditional databases in the midst of enormous data, organizations are turning to modern database solutions. And with this need, graph databases have now come of age. 

The unique thing, which also happens to be the greatest advantage of graph databases, is that they are designed to store and manage large sets of interconnected data points, such as social networks, supply chains, and financial transactions. As we have previously indicated when [breaking down the concept of a graph database](https://www.nebula-graph.io/posts/what-is-a-graph-database), these databases are particularly useful for applications that require real-time processing and analysis of data, such as [fraud detection](https://www.nebula-graph.io/posts/fraud-detection-with-graph-machine-learning)**,** [recommendation engines](https://www.nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation), and [social network analysis](https://www.nebula-graph.io/posts/social-networks-with-graph-database-1) tools.

Now, like any other database out there, graph databases too need a way (language) to communicate with their clients (users). For graph databases, these languages are known as graph query languages. Whether you are working with an [open source graph database](https://nebula-graph.io/) or a commercial one, you’ll need a super good graph query language by your side. 

Let’s break it all down. 


# **What is a graph query language?**

So what exactly is a graph query language?

A graph query language is a programming language designed to work with graph databases. It allows users to specify the structure of the graph they want to query, as well as the rules for traversing the graph and retrieving the relevant data. 

Graph query languages are typically declarative, meaning that users specify what they want to retrieve, rather than how to retrieve it. In other words the language instructs the database on what to do, which is what unlocks the numerous benefits that they offer.

Furthermore, graph query languages can recognize and relate the current data to other data sets in real-time.


# **How does a graph query language work?**

Graph query languages work by allowing users to specify patterns in the graph that they want to query. Users can specify the nodes and edges that they are interested in, as well as any conditions that must be met for the query to return results. The language then processes the query and returns the results in a format that can be easily consumed by the user.

For instance, let's say a financial institution wants to detect fraudulent activity related to credit card transactions. The institution can store all transactions in a graph database, where each transaction is a node and edges connect transactions to their associated accounts, customers, and merchants.

Using a graph query language, a developer can write queries to identify patterns that may indicate fraud, such as transactions that involve multiple cards, transactions from new or suspicious merchants, and transactions that occur outside of a customer's usual geographic region.

The output of this query could be further analyzed to identify potentially fraudulent transactions involving multiple cards and merchants, which could then be flagged for further investigation.

For example, a simple query might look for all nodes in the graph that are labeled "person" and return their names and ages. The query might also specify that the results should only include nodes that are connected to other nodes labeled "organization." The query language would then process the query and return a list of all the matching nodes and their associated information.


# **Popular graph query languages**

You’ll notice that most of the graph query languages are developed by players in the graph database market, which is only natural. A graph database provider would be best placed to develop a query language that works best for their solution. 

Below is a look at some of the top ones that your business can consider:


## **nGQL**

[nGQL](https://docs.nebula-graph.io/3.4.0/3.ngql-guide/1.nGQL-overview/1.overview/) is NebulaGraph’s query language. It is a declarative language that allows users to query the graph database using various patterns and criteria. It’s quite friendly for both developers and professionals on the operations side of things. 

nGQL provides a flexible and expressive syntax for querying graph data. It supports a wide range of operations, such as pattern match, traversal,  graph mutation, aggregation, access control, filtering, composite queries , and index.  

The beauty about this query language is that it’s constantly being improved to accommodate new features and support more environments. It’s compatible with openCypher 9, but not completely - does not support mutations and controls syntax.


## **Gremlin**

Gremlin is run by Apache TinkerPop. It allows users to specify complex graph traversals using a simple and intuitive syntax, enabling powerful queries that would be difficult or impossible to express using the traditional SQL. 

One of the key strengths of Gremlin is its ability to traverse graphs of any size, complexity, and shape, making it well-suited for analyzing highly connected data. Gremlin is also highly extensible, with a large and active community of developers contributing to its development and supporting a wide range of graph databases and tools.


## **SPARQL**

SPARQL is designed specifically for querying RDF (Resource Description Framework) data. As you may already know,  RDF is the model used to represent and link data on the web, and SPARQL allows users to query this data in a standardized and expressive way. It provides a flexible and powerful way to extract information from RDF datasets, allowing users to retrieve data based on complex patterns and relationships.

SPARQL was developed by the RDF Data Access Working Group of the World Wide Web Consortium (W3C), a global community that develops web standards and guidelines. The language was first introduced in 2008 and has since been updated to include additional features and functionality. 

The W3C continues to manage and maintain the SPARQL standard, and the language is widely used in academic, research, and industry applications for querying and analyzing RDF data. Several open-source and commercial SPARQL query engines are available, providing users with a range of options for working with RDF data.


## **Cypher**

Cypher was developed by Neo4j and is maintained as an open-source project under the openCypher initiative. openCypher is a community-driven project that aims to promote the adoption and standardization of the Cypher query language across different graph database platforms. 

The openCypher project is managed by a team of developers and contributors from various organizations, including Neo4j. 


## **AQL**

AQL (ArangoDB Query Language) is used for querying and manipulating data in the ArangoDB multi-model database. It is also declarative in nature, in which you express requests based on the data you want – as opposed to how to retrieve it. 

The language is designed to be simple and easy to learn, with a SQL-like syntax that makes it accessible to a wide range of users. AQL supports a variety of data manipulation operations, including filtering, sorting, grouping, and aggregating data, as well as complex graph traversals and joins between different collections and data models.


## **GraphQL**

GraphQL is specifically tailored for API-based data. With GraphQL, you can send a query to your API and retrieve only the data you need, making data retrieval simpler and more efficient.

One of the key benefits of using GraphQL is that it allows for highly predictable queries, which makes for faster and more reliable apps. 

Rather than relying on the server to control the data, GraphQL puts the control in your hands, allowing you to optimize data retrieval for your specific needs.


# **What makes graph query languages crucial for enterprises?**

How can the above graph query languages, and all the others we have not mentioned,  help your business? Of course the developers within the company have to select the right query language for the job, then implement it appropriately depending on the intended outcomes. 

Below is a look at some of the top benefits of graph query languages for businesses:


## **1. Simplified data queries**

Graph query languages allow you to retrieve data much more quickly and correlate it to other data sets in the database. 

The data is stored as nodes (data entities) and edges (data relationships) instead of tables, rows, and columns. The nodes and edges are easy to correlate, which helps simplify data queries, even when dealing with large data sets.


## **2. Better Data integrity**

Graph database query languages help guarantee data accuracy, completeness, and consistency. They allow you to retrieve data from multiple disparate sources and represent it in an easy-to-follow visual pattern. 

This means you are able to see how your data is connected and how changes to one set can affect other data sets.


## **3. Enhanced engagement efficiency**

With a graph query language, you can see how your requested data relates to other data in the database - eliminating the need for manual data exploration.

Additionally, these languages allow for precise data querying without over-fetching, which is a common issue with relational databases. According to an MDPI [study](https://www.mdpi.com/2076-3417/12/13/6490), graph databases outperformed relational databases when dealing with complex data queries.


## **4. Better data modeling**

Graph databases represent data much more naturally in easy-to-follow graphs when compared to a relational database. When using a graph query language, you can be able to model your data much better and identify relationships and patterns. It’s so easy to query and analyze data in a way that is intuitive and closer to the way humans think. For example, it's easier to model social networks, supply chain relationships, and other complex systems.

Graph query languages also make it easy to query data that is constantly changing, which is difficult to do with traditional query languages. As new nodes and edges are added or deleted, a graph database query language efficiently query the data and gibe returns that reflect these changes.

You can even automate the data queries and analysis process, leading to faster predictions.


## **5. Complex pathfinding**

Pathfinding queries, such as finding the shortest path between two nodes or all the paths between two nodes that meet certain criteria, are essential in applications such as transportation planning, logistics, and network optimization. 

Graph query languages make it possible to perform these complex queries pathfinding with ease.


# **Choosing the right graph database query language**

How do you choose between the many graph query languages we have covered here? Consider the factors below:


## **1. The type of graph database and data you are dealing with**

As we went about describing the various query languages, you may have noticed that some of them are tailored for specific functions or environments. 

Therefore, your choice of language may be heavily influenced by the choice of graph database you are using. Of course the choice of database will be influenced by what you want to achieve, which could be operational or user/customer centric. 

Consider these items:


## **2. Performance**

A key reason why graph query languages and databases have gained popularity is due to their ability to enhance data querying efficiency. 

As a result, it's crucial that you select a language that provides speedy returns on queries, particularly when dealing with large data sets requiring real-time data analysis.


## **3. Ease of use**

Maybe this is quite obvious but it's important that we mention it. A good graph query language should be easy to use. 

For example, declarative languages allow you to request data without indicating how the data will be retrieved – the language's engine does this for you. Such a feature makes it easy to use the language without writing complex algorithms.


## **4. Check out the security and privacy**

We are right in the middle of data security and privacy concerns - all over the world.  Thus, you need to be aware of any potential security risks that come with a specific language. 

For example, according to [DevOps](https://devops.com/does-graphql-introduce-new-security-risks/), GraphQL comes with potential security concerns, which you need to be aware of and prepare appropriate mitigation strategies. 


# **Concluding perspective**

A study published on [Science Direct](https://www.sciencedirect.com/science/article/pii/S2405844020311373) shows how graph queries were used to uncover hidden corruption of various individuals named in the Panama Papers. If you can take a moment to imagine the huge role that graph query language played in this groundbreaking investigation, then you immediately appreciate the power of these languages. 

Going forward, developers working with graph query languages should keep in mind the importance of data quality and consistency. As graphs become more complex, ensuring data accuracy and completeness will be critical to producing reliable outcomes.

One area where graph query languages may see significant growth is in the integration with AI and machine learning techniques, allowing for more sophisticated querying and analysis of graph data.

A graph query language is a programming language designed to work with graph databases. It allows users to specify the structure of the graph they want to query, as well as the rules for traversing the graph and retrieving the relevant data.

Graph Query Language: What You Should Know

This is the performance report for NebulaGraph Community Edition v3.4.0.

In this version, NebulaGraph has added Memory Tracker and has enabled concurrent queries. 

NebulaGraph Database v3.4.0 Benchmark Report

NebulaGraph Database recently released its latest [Enterprise Edition v3.4.0](https://docs.nebula-graph.io/3.4.0/20.appendix/release-notes/nebula-ent-release-note/), further enhancing the stability and ease-of-use of the core product. In this version, the performance of deep queries is significantly optimized, more than 60% better than before, enabling enterprise users to explore the correlation value in massive data more efficiently.

# Guaranteed Security of Data Assets

As the significance of data storage and data mining increasingly matters, data security has become a top concern for enterprises. To meet this demand, NebulaGraph supports incremental backup in v3.4.0  on top of existing data recovery & backup features, ensuring data security and business continuity.

![incremental-backup](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/1.png)

In addition, NebulaGraph also offers more fine-grained permission management in the new version through tag/edge-type level access control so that different roles in the same graph can only see the data within their limits of authority, thus protecting team collaboration security.

# Enhanced Stability and Ease-of-Use

Facing explosive data volume growth, enterprises attach great importance to real-time data queries when selecting a database.
With the latest NebulaGraph Enterprise edition, enterprises can leverage its optimized capability in high availability and stability to explore a sheer volume of connected data with milliseconds latency.


NebulaGraph Enterprise v3.4.0 adds a new black box monitoring tool, which can capture the system status at the moment of abnormality, such as core dump and crash, and pinpoint the issue with the parsing toolkits.

The new version also optimizes memory management and adds Memory Tracker to support more refined memory management control. Users can accurately track the memory resource application of each operator in each query and control it accordingly so as to reduce the probability of OOM.

Furthermore, functions like Kill Session are also added to the latest version to avoid link non-release caused by incorrect operation, which makes it more convenient for DBAs to maintain. The full-text index also becomes more efficient by streamlined deployment and usage constraints.

# Upgraded Graph Visualization Experience

NebulaGraph is committed to building an end-to-end, easy-to-use graph technology ecosystem to accelerate the application and popularization of graph technology. Now it has developed a series of graph visualization products, such as NebulaGraph Studio, NebulaGraph Explorer and NebulaGraph Dashboard, for developers and data scientists to easily explore graph data and intuitively manage NebulaGraph clusters.

NebulaGraph Dashboard, as a multi-cluster visual operation and maintenance tool developed for NebulaGraph, provides monitoring, alerting, backup, diagnosis, upgrade and expansion, which make the work of DBAs and operations personnel easier.

![NebulaGraph-dashboard](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/nebulagraph-dashboard-overview.jpeg)


In NebulaGraph Enterprise Edition v3.4.0, NebulaGraph Dashboard adds incremental backup, runtime log, audit log, and job management functions and displays and configures multiple services. Armed with these features, DBAs and developers are able to view, pause and restore the workspace whenever needed. The new version also has a feedback portal, and users can quickly seek professional support when encountering challenging issues.

Meanwhile, other visualization tools -  NebulaGraph Studio and NebulaGraph Explorer,  are also optimized in terms of schema creation statement view, slow query optimization and significant improvement of visualization presentation effect.

# About NebulaGraph Database

As a high-performance native distributed graph database, NebulaGraph Database excels in handling mega data sets with hundreds of billions of nodes and trillions of edges, while maintaining millisecond query latency. Relying on its shared-nothing, distributed architecture and computation-and-storage separation, it is highly available, highly secure, and easy to scale out, largely meeting the security requirements of enterprises. Besides,  it can be deployed on the public cloud and on-premises as needed.

# Talk to a Graph Database Expert

Get a customized solution for your organization and see how NebulaGraph can help you:

[**Contact NebulaGraph**](https://www.nebula-graph.io/contact)



NebulaGraph Database recently released its latest Enterprise Edition v3.4.0, enabling enterprise users to explore the correlation value in massive data more efficiently.

NebulaGraph Announces General Availability of Enterprise Edition v3.4.0

Along with the release of [NebulaGraph v3.4.0](https://www.nebula-graph.io/posts/nebulagraph-v3.4.0-release-notes), its graph visualization products NebulaGraph Dashboard, NebulaGraph Explorer, and NebulaGraph Studio have also been upgraded to their latest versions. In this article, we will take a quick look at the new features of each visualization product.

# NebulaGraph Dashboard

NebulaGraph Dashboard is a powerful visualization and operations tool for the NebulaGraph database, designed to make the work of DBAs and operations personnel easier. With Dashboard, you can quickly create and import clusters and monitor them in real time, receiving alarms as needed. Additionally, the dashboard offers various operations such as cluster upgrades, scaling, backup and recovery, and service start and stop, making cluster management more straightforward and intuitive.

![NebulaGraph-Dashboard-Overview](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/nebulagraph-dashboard-overview.jpeg)

## New Features of NebulaGraph Dashboard Enterprise Edition

* When system failures occur, troubleshooting problems by reviewing runtime logs is often necessary. The 3.4.0 version of NebulaGraph Dashboard allows viewing NebulaGraph runtime logs, quickly identifying the issues.

![run-time-log](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/2.png)

* In addition to runtime logs, NebulaGraph Dashboard's audit log feature classifies and stores all operations the Graph service receives, making it easy to review NebulaGraph audit logs.

![audit-log](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/3.png)

* Furthermore,  NebulaGraph Dashboard allows for visual management of long-running tasks, also known as jobs, within a specified graph space, including the ability to view, stop, resume, and view the details of individual jobs.

![job-management](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/4.png)

* Additionally, you can also incrementally back up existing backup files on S3 through the Dashboard.

![incremental-backup](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/1.png)


## Enhancements in NebulaGraph Dashboard Community Edition v3.4.0

- Feature
  - Support the built-in [dashboard.service](../../nebula-dashboard/2.deploy-dashboard.md) script to manage the Dashboard services with one-click and view the Dashboard version.
  - Support viewing the configuration of Meta services.

- Enhancement
  - Adjust the directory structure and simplify the [deployment steps](../../nebula-dashboard/2.deploy-dashboard.md).
  - Display the names of the monitoring metrics on the overview page of `machine`.
  - Optimize the calculation of monitoring metrics such as `num_queries`, and adjust the display to time series aggregation.
  

# NebulaGraph Explorer

NebulaGraph Explorer is a web-based visual graph exploration tool. Based on the NebulaGraph Database for visual interaction with graph data, it supports fast finding friends and neighbors,  analyzing suspicious targets, and performing graph computational analysis from complex relationships and large amounts of data.

![NebulaGraph-Explorer](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/nebulagraph-explorer-overview.png)

![NebulaGraph-Explorer-3D](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/nebulagraph-explorer-overview-3d.png)

* The new version of NebulaGraph Explorer supports viewing the schema creation statement for the specified graph space, and you can download the statement or copy it to the console for execution.

*  Removed the timeout limit for viewing slow queries.

This iteration of NebulaGraph Explorer places a major emphasis on product usability and visualization presentation, spending a significant amount of time polishing the functions of the explorer, hoping to help users more quickly and easily explore relationships between data. For more details, please see the [release notes](https://docs.nebula-graph.io/3.4.0/20.appendix/release-notes/explorer-release-note/).

# NebulaGraph Studio

Finally, the open-source web-based graph visualization tool NebulaGraph Studio. It now supports viewing the creation statements of Schema just like NebulaGraph Explorer. And a feedback page is integrated for easier issue collection. Besides, we have made optimization in multiple aspects and fixed some bugs, making it better than ever.


![NebulaGraph-Studio](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-3.4.0/nebulagraph-studio-overview.png)

It now supports viewing the creation statements of Schema just like NebulaGraph Explorer. And a feedback page is integrated for easier issue collection. Besides, we have made optimization in multiple aspects and fixed some bugs, making it better than ever.

Get started free with the latest version of NebulaGraph Studio, click the link to download [https://github.com/vesoft-inc/nebula-studio/releases/tag/v3.6.0](https://github.com/vesoft-inc/nebula-studio/releases/tag/v3.6.0)


Are you interested in experiencing the entire NebulaGraph Database Enterprise Edition in one go? Click [here](http://nebula-graph.io/visulization-tools-free-trial) to have a 30-day free trial right away!

Along with the release of NebulaGraph v3.4.0, its graph visualization products NebulaGraph Dashboard, NebulaGraph Explorer, and NebulaGraph Studio have also been upgraded to their latest versions. In this article, we will take a quick look at the new features of each visualization product.

Posts

Tag