
This sharing is mainly from the perspective of (potential) users and discusses about what considerations the technical team will have when selecting a database. At the very beginning, our RoyalFlush Knowledge Graph team chose Neo4j when the knowledge graph strategy was started. However, during the using process, it turned out that Neo4j could not handle our growing business well. Therefore, the technical team came up with a solution: adding a multi-level cache between the database and the application layer. Although caching can alleviate the problem to a certain extent and improve the operation performance, it actually causes more problems.

So, we began to think about whether there is a graph database that can not only meet the current business needs, but also is capable of handling the future business growth. Thus, our Knowledge Graph team starts the journey of selecting an ideal graph database.

# How to select a graph database

![How to Select a Graph Database: Best Practices at RoyalFlush](https://user-images.githubusercontent.com/90186547/142156266-3654804f-ec19-4241-a8ef-80f365dbf5e0.jpg)

This speech can be divided into three parts: How to choose, What to choose, and Why. In addition, our RoyalFlush Knowledge Graph team and related businesses will be introduced at the end.

When selecting a Graph Database, we will consider the following aspects: 

* Inclusive functions with outstanding performance
* Commercially mature products
* Active community
* Large amount of data to handle the current business and future business growth
* Open-source preferred
* Technical support
* Domestic preferred

Selecting a graph database is similar to select a stock. We will take information analysis, fundamental analysis, and technical analysis into consideration. In the world of the graph database, they mean:

* Information analysis: Whether the graph database meets the above seven demands
* Fundamental analysis: Whether the graph database is stable and durable
* Technical analysis: Whether the graph database meets the technical requirements

## Information analysis

![Information analysis](https://user-images.githubusercontent.com/90186547/142158108-5b94ca33-1f96-450e-a6e7-5eea8578785d.png)

NOTE: The screenshot is the DB-Engine ranking in Sep. 2021 when this talk was given.

Generally, we will go to DB-Engine for the information about the graph database. You can see that Neo4j ranks first here. After all, it is the earliest graph database that is relatively mature. After a round of research, we found that Neo4j did a great job in marketing. If you search "graph database" on Baidu or Google, their ads will show up.

In this DB-Engine ranking, you can see other famous graph databases like JanusGraph (ranked seventh). So, among so many databases, how do we determine the database we will ultimately use?

![Information analysis](https://user-images.githubusercontent.com/90186547/142160038-9f889dbe-a35c-4716-8eb9-e7226bfcac3d.png)

As for the ranking rules of DB-Engine, it is based on the search results of Google and Bing, keyword trends in Google Trends, discussions on Stack Overflow, DBA Stack Exchange, the number of posts on recruitment websites like Indeed and Linkedin, and the amount of discussions on social networks like Twitter.

We can find a problem: the above websites are either inaccessible in China or relatively niche websites.

Therefore, the ranking on DB-Engine is actually biased for some domestic products.  At least its data certainly cannot reflect the true conditions. If some domestic products are also included, the rankings of HugeGraph and NebulaGraph in this DB-Engine ranking will definitely ascend a lot.

After knowing what is DB-Engine, let's filter the information on this DB-Engine ranking:

![Information analysis](https://user-images.githubusercontent.com/90186547/142160285-7deea852-f897-4d6d-acfd-c114505fed35.png)

![Information analysis](https://user-images.githubusercontent.com/90186547/142160472-c21dfe95-7f45-426f-8c4d-eb9adde62077.png)

From the view of the information analysis, these databases do not meet the needs of our RoyalFlush Knowledge Graph team. This result also proves that the DB-Engine ranking results are not so accurate. For example, some databases ranked before NebulaGraph are already inactive or no longer updated products. The above screenshots are all from DB-Engine.

## Fundamental analysis

![Fundamental analysis](https://user-images.githubusercontent.com/90186547/142160638-0bc6d130-144e-4b32-8ac6-f9baba294be6.png)

![Fundamental analysis](https://user-images.githubusercontent.com/90186547/142160840-e988a22c-ef6e-4a71-bbb7-638dc4193264.png)

After the information analysis, we filter out 6 graph database products including Neo4j, Orient DB, Dgraph, JanusGraph, HugeGraph, Nebula, etc. These 6 products are relatively active and support multiple languages.

According to the earliest release time of the product, Neo4j is ranked first, which was released in 2007, followed by OrientDB, Dgraph, JanusGraph, HugeGraph, and NebulaGraph which released in 2019. The higher the ranking of the database product, the higher its language support level, and the more complete its system functions. This is also where the new databases need to be perfected and improved.

![Fundamental analysis](https://user-images.githubusercontent.com/90186547/142160840-e988a22c-ef6e-4a71-bbb7-638dc4193264.png)

The functions of many products like HugeGraph and Neo4j are only supported in the Enterprise Edition. We also need to know the content of their Enterprise Edition and the corresponding pricing.

![Fundamental analysis](https://user-images.githubusercontent.com/90186547/142161062-be705952-f83e-4192-b977-2376f0e9bf0e.png)

The above picture is more interesting. It points out the order of magnitude supported by various graph databases in different scenarios. For example, Neo4j supports billions of nodes, trillions of relationships with millisecond response time and unlimited elasticity. OrientDB is more interesting. It supports unlimited data. I don't know how it achieves this value. Dgraph did not clarify the amount of data, but said that it provides Google production level scale and throughput, with low enough latency to be serving real-time user queries, over terabytes of structured data.

As for the rest two, one is HugeGraph, open sourced by Baidu, and the other is NebulaGraph. Nebula is still relatively dominant. It is capable of hosting graphs with hundreds of billions of vertices and trillions of edges, and serving queries with millisecond-latency.

## Technical analysis

![Technical analysis](https://user-images.githubusercontent.com/90186547/142164284-9c1fd737-bc25-4259-b390-1bf542f6f09c.png)

After the information analysis and fundamental analysis, there is actually no clear winner. Let's look at the technical analysis. Since HugeGraph and JanusGraph have similar architectures and the technical support of OrientDB is plain, we will only focus on the following three graph databases: Neo4j, HugeGraph, and NebulaGraph.

Neo4j only supports isomorphic deployment and does not support heterogeneous deployment, so it cannot take advantage of each machine. For example, some machines may be computational-oriented, and some machines have strong IO. If you follow Neo4j's deployment settings, it requires that each machine is in the same condition. The analysis and performance improvement of Neo4j mainly relies on the virtual machine composed of two fabrics to provide regional partitioning capabilities (the green part of the above figure).

At the architecture level, the separated architecture  of storage and computing of NebulaGraph and HugeGraph has more advantages. At least users can configure the machines according to the realities of the business department.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142164746-cd271108-a0f6-4dbb-8bc8-9529d1c74145.png)

HugeGraph supports a variety of storage engines in the storage area, which can be connected to a variety of third-party engines, and then a computing framework is added on this basis. Above the computing framework is the interface layer, such as Huge Console, Huge Loader, etc.

In computing, the storage engine at the bottom mainly provides OLTP and OLAP. Here the storage engine of HugeGraph is not self-provided, but mainly depends on other storage engines. It may not be as good as Nebula's own storage engine. For example, the third-party storage engine we use is HBase. The real-time performance of HBase is very unsatisfactory. At this time, using it as a storage engine will affect the performance of the upper layer.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142167861-1aa1c3c1-e48e-4fe7-bb81-078becaf4cbd.png)

NebulaGraph also applies an architecture that separates storage and computing. Each layer can be expanded infinitely. You can expand the storage/computing layer if it is full. Relatively speaking, this architecture is relatively advanced.

After talking about the architecture, RoyalFlush also refers to the test results of other technical teams:

![Technical analysis](https://user-images.githubusercontent.com/90186547/142168255-8099bcf8-837a-48b5-a0a3-33fc9f81e485.png)

This is the performance test of v1.x carried out by the Tencent Cloud Security Team.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142168458-5d451e57-5947-4ee3-8b22-d205ec60b8eb.png)

This is the performance comparison of Nebula, Dgraph, and HugeGraph carried out by the Meituan NLP team. Here, HugeGraph uses HBase as the storage engine, while Dgraph and NebulaGraph uses RocksDB. As mentioned before, HBase is at a disadvantage in real-time performance. If all three databases use the KV storage engine, there will be no such a large gap in the results.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142168642-ac5b9dff-33e1-42d2-9db9-a1c8a01315fe.png)

The following is the performance test carried out by RoyalFlush. We not only compare different graph databases, but also compare two query languages (native nGQL and nGQL that is compatible with openCypher) of NebulaGraph.

From the data, we can see that the native query language nGQL has better performance (nebula-ngql and nebula-cypher in the figure above). In the multi-hop query, you can see that for queries that are more than 5 hops, Neo4j stays in a timeout state.

In terms of performance, Nebula still has a relatively large advantage over Neo4j.

![Technical analysis](https://user-images.githubusercontent.com/90186547/142173955-941d1da2-ed1e-45f1-87ca-5529927e5497.png)

Considering factors such as architecture, performance, community support, etc., we will still choose a graph database that is more suitable for us, namely NebulaGraph. Although some minor problems were found in the using process:

* Chinese identifiers are not supported.
* Syntax error prompts are not user-friendly.
* Parameterized queries are not supported in Java.
* The source vertex ID must be specified in GO queries.

In general, NebulaGraph still meets our business requirements.

# RoyalFlush · the Knowledge Graph team

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142174441-0c30845c-cd49-40f6-9cf3-4e3c6f8ea5c1.png)

Finally, let me introduce our RoyalFlush Knowledge Graph Team. Our main serving businesses are intelligent dialogue system, system recommendation, search engines, investment strategies, and some businesses such as multi-mode integration, stock tips, investment consulting and education, which may expand later.

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142174763-4ec054b2-fb6a-4d76-aa05-0e827e1dba83.png)

At present, we maintain several graphs. One is the graph of the industry chain, which means that the data are public. The graph of the industry chain is constructed through semantic extraction.

NOTE: The right part in the above figure is an example of Graph used in the industry chain, which shows leading enterprises rankings, qualified enterprises rankings, and non-listed benchmarking enterprises rankings.

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142174956-347116c4-3798-46f3-99aa-3ee8c7a6d6ce.png)

We also maintain the graph of the supply chain,

NOTE: The right part in the above figure is an example of graph used in the supply chain, which displays acquired clients and co-supplier of an enterprise. 

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142175115-24130468-0091-4c14-8090-5b0de14f7523.png)

the graph of personal relationships,

NOTE: The right part in the above figure is a true case of graph used in personal relationships. 

![RoyalFlush · the Knowledge Graph team](https://user-images.githubusercontent.com/90186547/142175262-16ec088d-498c-4a35-999b-d0592d6398c4.png)

and the graph of enterprises, so as to meet the financial investment needs of RoyalFlush. The above is all that I'd like to share today. Thank you!

NOTE: The right part in the above figure is an example of graph used in enterprises, which shows infinite upward and downward penetration of equity relations, he beneficiary and actual controller of an enterprise etc.

Welcome to join our [Slack channel](https://join.slack.com/t/nebulagraph/shared_invite/zt-7ybejuqa-NCZBroh~PCh66d9kOQj45g) if you want to discuss with the rest of the NebulaGraph community!





This sharing is mainly from the perspective of (potential) users and discusses about what considerations the technical team will have when selecting a database.

How to Select a Graph Database: Best Practices at RoyalFlush

The performance report runs against GO, LOOKUP and MATCH queries in NebulaGraph v2.6.0. Meanwhile, it has compared the performance of v2.6.0 with that of v2.5.0.

NebulaGraph v2.6.0 Performance Report

This article shares the benchmarking tests conducted by the NLP team at Meituan. The test included data import and throughput/latency performance.

Benchmarking the Mainstream Open Source Distributed Graph Databases at Meituan: NebulaGraph vs Dgraph vs JanusGraph


![How to Reduce Docker Image Size](https://user-images.githubusercontent.com/56643819/74205747-8dcb7680-4cb3-11ea-97c0-7c8bf36dd0e1.png)

If there are top ten buzzwords in the technology industry in the year of 2019, **container** is sure to be one of them. With the popularity of Docker, more and more scenarios are using Docker in the front-end field. This article shows how do we use Docker in the visualization interface of **NebulaGraph**, a distributed open source graph database.

## Why Using Docker

Docker is widely used in daily front-end development. NebulaGraph Studio (A visualization tool for **NebulaGraph**) uses Docker based on the following considerations:

- **Unified operating environment**: There are several services behind our tools, such as existing services from different technology stacks, and pure front-end static resources.
- **Low user cost**: Currently, cloud services are under development. We want a smooth experience of the combined services, and Docker makes this possible with the ability of starting and applying all the services locally just in one step.
- **Quick deployment**: This front-end project is inspired by the [NebulaGraph Docker Image](https://github.com/vesoft-inc/nebula-docker-compose).

## Building Docker Image

We need to build an image first before hosing our services with Docker. Here we need a configuration file named [Dockerfile](https://docs.docker.com/engine/reference/builder/) that contains descriptions of the building steps. To be brief, we need to copy the project into the image and set the startup method:

```shell
# Select base image
FROM node:10
# Set work directory
WORKDIR /nebula-web-console
# Copy the current project to the /nebula-web-console directory of the image
ADD . /nebula-web-console
# Download front-end dependency in the image
RUN npm install
# Run the building
RUN npm run build
EXPOSE 7001
# Deployment commands executed when the image starts
CMD ["npm", "run", "docker-start"]
```

## Reducing Docker Image

The above configuration file will build a Docker image with a size of about 1.3GB, which looks a bit scary because downloading is too time-consuming even with a fast network. That is totally unacceptable.

After some research, we learned some tips that help reduce Docker image size.

### Using Smaller Base Image

Docker base image (for example, the above mentioned `node:10`) is the basic image on which you add layers and create a final image containing your applications. There are **multiple versions** of the Node.js image on [DockerHub](https://hub.docker.com/_/node), and each of them share a different **internal environment**. For example, the [alpine](https://yeasy.gitbooks.io/docker_practice/cases/os/alpine.html) version is a more simplified Linux system image that removes some tools like bash, curl etc. to decrease size.

Based on our needs, we change the base image to [alpine](https://yeasy.gitbooks.io/docker_practice/cases/os/alpine.html) and rebuild to reduce the docker image from 1.3GB to 500MB+. So if the docker image you are building is too large, you can **replace the basic image**.

### Multi-stage Build

[Multi-stage](https://docs.docker.com/develop/develop-images/multistage-build/) build in docker is a new feature introduced in docker 17.05. It is a method to reduce the image size, create a better organization of docker commands, and improve the performance while keeping the dockerfile easy to read and understand.

#### Docker Building Principle

In short, multi-stage build is dividing the dockerfile into multiple stages to pass the required artifact from one stage to another and eventually deliver the final artifact in the last stage. This way, our final image won't have any unnecessary content except our required artifact. Let's consider an example:

```shell
# Set up the image generated in the first step and name it builder
FROM node:10-alpine as builder
WORKDIR /nebula-web-console
# Copy the current project to the image
ADD . /nebula-web-console
# Start building
RUN npm install
RUN npm run build
....

# Start the second step build
FROM node:10-alpine
WORKDIR /nebula-web-console
# Copy the product of the first step image to the current image. Only one image layer is used here, which saves the number of image layers in the previous building step.

COPY --from=builder . /nebula-web-console
CMD ["npm", "run", "docker-start"]
```

### .dockerignore

Similar to the well known `.gitignore` that ignores unnecessary (such as document files, git files, node_modules, etc) files when using `COPY` or `ADD` command to copy or add files, we can use [.dockerignore](https://docs.docker.com/engine/reference/builder/#dockerignore-file) to specify files to be ignored.

### Merging Multiple Layers Into One

When building a Docker image with a Dockerfile, each operation adds a new layer based on the previous step image. We can use `&` to merge multiple operations to reduce layers. For example:

```shell
# The two operations represent two layers
RUN npm install
RUN npm run build
```

Merge the above command to one:

```shell
# It becomes a single layer with &
RUN npm install && npm run build
```

### Regular Front-End Optimization

- Compress ugly code and remove source code
Finish this step when building image so that the image size is further reduced.
- Only downloading code needed for production with node_modules
Finish this step when deploying, be sure to download only third party dependence code for production: `npm install --production`
- Place public source on CDN
If the image is expected to run in network environment, place large public files ( pictures and third party libraries, etc.) on the the CDN server so that some resources are separated and the image size is further reduced.
- ...

The above suggestions are only for your reference. You can migrate more regular front-end optimizations to the image given that image building itself is an environment for running code.

## Summary

The above is our experience on reducing the Docker image of the NebulaGraph Studio. Please leave us a comment if you have any questions. Welcome try [NebulaGraph Studio](https://github.com/vesoft-inc/nebula-web-docker).

## References

- [How to reduce Docker Image sizes using multi-stage builds](https://blog.logrocket.com/reduce-docker-image-sizes-using-multi-stage-builds/)
- [NebulaGraph Docker](https://github.com/vesoft-inc/nebula-docker-compose)
- [NebulaGraph Studio](https://github.com/vesoft-inc/nebula-web-docker)

## You might also like

1. [NebulaGraph Architecture — A Bird’s View](https://nebula-graph.io/posts/nebula-graph-architecture-overview/)
1. [An Introduction to NebulaGraph's Storage Engine](https://nebula-graph.io/posts/nebula-graph-storage-engine-overview/)
1. [An Introduction to NebulaGraph’s Query Engine](https://nebula-graph.io/posts/nebula-graph-query-engine-overview/)

This post shows you several ways to reduce Docker image size. These tips helped NebulaGraph devs reduce the image from 1.3G to 0.3G.

Tag