In the era of information overload, sifting through vast amounts of data to provide accurate search results in an engaging and comprehensible manner has become an uphill battle. Traditional search enhancement techniques often fall short when it comes to complex queries and the high demand brought by cutting-edge technologies like ChatGPT. This is where Graph RAG steps in – a game-changing concept crafted by NebulaGraph that promises to revolutionize the field.

**“Graph RAG” is a concept pioneered by NebulaGraph**. This technology harnesses the power of knowledge graphs in conjunction with Large Language Models (LLMs) to provide search engines with a more comprehensive contextual understanding. It assists users in obtaining smarter and more precise search results at a lower cost.

NebulaGraph's revolutionary Graph RAG (Retrieval-Augmented Generation) technique, which is based on knowledge graphs, is a breakthrough in this area. Graph RAG combines knowledge graphs with a large language model to provide more cost-effective, intelligent, and precise search results. **Capable of deep integration with large language models such as Llama Index and LangChain, NebulaGraph is the first in the industry to propose the concept of Graph RAG.** Besides, Graph RAG has also achieved excellent results in the field of vector database integration.

In this article, we will introduce the concept of Graph RAG and compare it with other RAG techniques. You are also welcome to visit the NebulaGraph‘s website to [Try Demos](https://www.nebula-graph.io/demo).



## Challenges Faced by Traditional Search Enhancement Techniques

Obtaining accurate search results is a challenge, especially with complex or long-tail queries when traditional search engines rely on keyword matching, which often does not meet the users' actual needs. The bottleneck of traditional search augmentation techniques is the lack of training data and text understanding, which makes them less effective when it comes to generating accurate, diverse and relevant information.

To address this problem, RAG came into being. **RAG (Retrieval-Augmented Generation) refers to the process of enhancing search results through RAG models.** Specifically, it is a technique that combines retrieval and language generation to enhance the generation process. By using RAG, traditional search engines can generate more accurate and relevant search results.

However, the RAG technique still faces challenges in terms of training data and text understanding. These challenges include:

- Train data: the RAG technique requires a large amount of data and computational resources for training and generating models, especially when handling multiple languages and complex tasks.
- Text understanding: RAG needs to understand the intent of the query, but for complex queries or polysemous queries, RAG may have ambiguity or uncertainty, which affects the quality of the generated content.

Therefore, the problem of how to find more powerful retrieval enhancement techniques to obtain search results that better meet the searcher's expectations more efficiently is even more urgent.

## What is Graph RAG?

**Graph RAG** is proposed by NebulaGraph, which is a retrieval enhancement technique based on knowledge graphs. It uses a knowledge graph to show the relationship between entities and relationships and then uses the large language model LLM (Large Language Model) for retrieval enhancement.

As mentioned in the previous [blog](https://www.nebula-graph.io/posts/graph-llm), the graph database is naturally suitable for storing and expressing complex context information by organizing and connecting information in a graphical format. By using graph technology to build a knowledge graph to enhance in-context learning, users can provide more context information to help large language models (LLM) better understand the relationship between entities and improve their expression and reasoning abilities.

Graph RAG equates the knowledge graph to a large-scale vocabulary, and entities and relationships correspond to words. In this way, Graph RAG can jointly model entities and relationships as units during retrieval, which can more accurately understand the query intent and provide more accurate search results.

## Demo: Graph RAG vs. Vector RAG vs. Text2Cypher

In the following sections, we'll explore two demos that provide a visual comparison of Graph RAG with Vector RAG and Text2Cypher.

### Graph RAG vs. Graph + Vector RAG

The GIT below compares the results returned by Vector RAG (vector retrieval) with Graph and Vector RAG (graph-enhanced vector retrieval).

![Graph RAG vs. Vector RAG](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/1.gif)

- Results on the left: Vector RAG 
- Results on the right: Vector RAG with Graph

As an example, let's consider the data set of "Guardians of the Galaxy 3". When we search for "Peter Quill's related information," traditional retrieval techniques like vector retrieval engines only provide simple information on his identity, plot, and actors. However, with the Graph RAG-enhanced search results, we can obtain more information about the protagonist's skills, character goals, and identity changes.

This example highlights how Graph RAG effectively complements the shortcomings of traditional methods like embedding and vector retrieval and provides more in-depth knowledge and information relevant to the query. By combining knowledge graphs with large language models, Graph RAG can understand the relations between entities, equate knowledge graphs to large-scale vocabularies and better comprehend the intent of complex queries, leading to more accurate and relevant search results.

### Graph RAG vs. Text2Cypher 

Another interesting approach to knowledge graph-based LLM is Text2Cypher, which is a natural language generation graph query. This method does not rely on entity subgraph retrieval but translates tasks or questions into an answer-oriented graph query, which is essentially the same as what we commonly call Text2SQL.

Text2Cypher and Graph RAG differ mainly in their retrieval mechanisms. Text2Cypher generates graph pattern queries based on the knowledge graph schema and the given task, while (Sub)Graph RAG obtains relevant subgraphs to provide context. Both have their advantages, and you can get a more intuitive understanding of their characteristics through the following demo.

![Graph RAG vs. Text2Cypher](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/2.gif)

- Results on the left: Text2Cypher 
- Results on the right: Graph RAG



From the above demo, it's evident that the two graph query modes exhibit pronounced differences in visualization. Retrieval using Graph RAG presents more comprehensive results. Users not only receive fundamental introductory information but also gain access to a range of outcomes based on associative searches and contextual inferences, such as "Peter Quill is the leader of the Guardians of the Galaxy," "This role implies that he will return in the sequel," and insights into the character's personality.

## NebulaGraph  : Build Your Enterprise-Specific Knowledge Graph Applications in One Step

**NebulaGraph database has seamlessly integrated with large language model frameworks like Llama Index and LangChain.**Therefore, developers can focus on LLM orchestration logic and pipeline design without having to deal with many details of abstraction and implementation and generate high-quality and low-cost enterprise-level large language model applications in one stop.

The advent of Graph RAG presents a breakthrough for processing and retrieving information on a massive scale. By fusing knowledge graphs and graph storage within the LLM stack, Graph RAG offers an unparalleled level of context learning. With the utilization of NebulaGraph, building Graph RAG is highly simplified and requires minimal effort, based on just three lines of code. Even more complex RAG logic, such as Vector RAG with graph can be easily integrated.

![Graph+Vector RAG](https://www-cdn.yueshu.cn/yueshu-website/images/blogs/%E8%AF%AD%E8%A8%80%E6%A8%A1%E5%9E%8B/demo-3.png)

As graph technology and deep learning algorithms continue to advance, the adoption of Graph RAG in data processing and retrieval is set to become increasingly widespread. We invite you to [contact us](https://www.nebula-graph.io/contact) for a free trial of NebulaGraph database and to effortlessly create your knowledge graph application!



NebulaGraph database's revolutionary Graph RAG (Retrieval-Augmented Generation) technique, which combines knowledge graphs with a large language model to provide more cost-effective, intelligent, and precise search results.

NebulaGraph Launches Industry-First Graph RAG: Retrieval-Augmented Generation with LLM Based on Knowledge Graphs

This year's SIGMOD'22 was held in Philadelphia, and I was honored to attend as a speaker on behalf of the company. During this conference, I communicated a lot with students, professors, and companies. On the one hand, I publicized our products; on the other hand, I got many valuable suggestions and opinions. Next, I'll share some of these suggestions, as well as my overall thinking.

# Graph learning and storage technology have attracted much attention

‍

**There are two hotspots that most participants pay attention to at this conference, which reflects the new trend in the industry.&nbsp;**

**First, the importance of machine learning is further increasing, and graph learning is becoming a new hotspot**. In recent years, more and more conferences that focus on specific systems set sections about machine learning to study the use of machine learning in optimizing systems or algorithms. Among the sub-fields of machine learning, a new force, i.e., graph learning, which is machine learning on graphs, suddenly rises. Just like other sub-fields of machine learning, graph learning also aims at solving problems in the real world. **Through the method of embedding, graph learning extracts feature vectors from a graph, and then uses machine learning methods to solve complex problems in the graph, including classification, subgraph matching, an  
d link classification.**		

Judging by the tendency of conferences and guidance from the National Science Foundation (NSF), this trend is likely to continue. We have also received a lot of inquiries from students and professors about whether NebulaGraph supports graph learning. From a graph database provider's perspective, graph learning is one of the NebulaGraph application scenarios. We should provide not only the graph database core, but also libraries, interfaces, and even optimizations for upper-layer applications to ensure the smooth and efficient operation of upper-layer applications. On the other hand, if NebulaGraph can help academia make some breakthroughs in areas like graph learning, it'll be great for our company, and the industry as a whole.
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/thoughts-after-attending-the-sigmod22-conference/figure1.jpeg)

**In addition to machine learning, I think another hotspot is memory, including in-memory databases, persistent memory, etc**. Non-volatile memory, or persistent memory, has become more and more mature through the joint efforts of academia and industry for more than ten years. Research on how to use persistent memory is also very hot, and there are numerous articles on it at top events of almost all systems.

**In the database field specifically, research topics include how to design an in-memory database, how to mix durable memory with SSD, DRAM, and other types of storage, and how to solve the problem of memory segregation.** NebulaGraph, as a graph database provider, should be embracing the changes that new hardware brings, and exploring persistent memory-based storage systems as well.

# The advantages and development direction of the graph

**In addition to the new trends, my biggest takeaway and thought from the conference was the comparison between graph databases (GDBMS) and relational databases (RDMBS). **There were quite a few people at the conference who were wondering why the graph database was not implemented based on a relational database, because relational databases have been optimized over the years, and they are really armed to the teeth. For example, Professor Peter Boncz (one of the founders of LDBC) gave a keynote address at the EDBT/ICDT conference earlier this year on The (Sorry) State of Graph Database Systems [1]. He stated that compared with RDBMS, GDBMS still had a lot to improve in terms of computation and storage, especially regarding AP queries like subgraph matching. He proposed a performance benchmark for subgraph matching scenarios. In his experimental environment, none of the graph database software came close to the performance of relational databases Hyper and Umbra in the subgraph matching scenario (see the following figure).
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/thoughts-after-attending-the-sigmod22-conference/figure2.png)

Certainly, the setup of this experiment is worthy of discussion. Still, Peter, as a founder of LDBC, did not mean to disparage graph databases but to encourage graph database providers to improve their products by using subgraph matching, a query scenario in which graph databases are not well optimized.

From the broader picture, NebulaGraph, as a graph database provider,** must know the differences between graph databases and relational databases and grasp the advantages of graph databases**. For example,** GQL is easier to use and more efficient when executing graph queries than SQL**. Emil Eifrem, CEO of Neo4J, has a perfect example [3]: For a very common query in an AP system, if using SQL, it needs a very long statement consisting of 23 SELECTs, 21 WHEREs, 11 JOINs, and 9 UNIONs. But if you use a graph query language, you only need a MATCH and a WHERE. For SQL, the query task may be impossible to complete or easy to go wrong, but for a graph query language, the task is easy to complete, saving countless manpower and resources.

**In terms of performance**, in addition to the natural advantages of graph databases over relational databases in association relation queries, i**t is also necessary to gain an advantage over relational databases in a variety of other graph-related queries (not limited to subgraph matching mentioned above). **This is actually the core enlightenment of Peter's proposal of subgraph query as a benchmark for us.		

‍

# Thoughts about the underlying storage structure

![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/thoughts-after-attending-the-sigmod22-conference/figure3.png)


At last, we also gained a number of suggestions for the underlying storage. NebulaGraph currently applies RocksDB, which is based on LSM-Tree, as the underlying storage engine. However, it has been controversial whether LSM-Tree is suitable for the workload of the graph database, especially, whether it is the best choice with the lowest cost on the cloud. Possible alternatives are:

- Bε-tree File System, or betrFS.[ https://www.betrfs.org/](https://www.betrfs.org/)

- LiveGraph. [https://marcoserafini.github.io/papers/LiveGraph.pdf](https://marcoserafini.github.io/papers/LiveGraph.pdf)

- B+ Tree. We will follow up the research on this aspect.	

All in all, the SIGMOD'22 trip was fruitful. Look forward to SIGMOD next year. See you in Seattle!

[1] [https://conferences.inf.ed.ac.uk/edbticdt2022/?contents=invited_talks.html](https://conferences.inf.ed.ac.uk/edbticdt2022/?contents=invited_talks.html)

[2] [https://docs.google.com/presentation/d/1xUbooiL8rIMzkp9G9EXmN4tIMMWt2mC53Q0u4mslq5g/edit#slide=id.ge0f13c0edc_0_47](https://docs.google.com/presentation/d/1xUbooiL8rIMzkp9G9EXmN4tIMMWt2mC53Q0u4mslq5g/edit#slide=id.ge0f13c0edc_0_47)

[3] [https://people.eecs.berkeley.edu/~istoica/classes/cs294/15/notes/21-neo4j.pdf](https://people.eecs.berkeley.edu/~istoica/classes/cs294/15/notes/21-neo4j.pdf)

‍

This year's SIGMOD'22 was held in Philadelphia, and I was honored to attend as a speaker on behalf of the company. During this conference, I communicated a lot with students, professors, and companies. On the one hand, I publicized our products; on the other hand, I got many valuable suggestions and opinions. Next, I'll share some of these suggestions, as well as my overall thinking.

Thoughts after attending the SIGMOD'22 conference

This article is transcribed from NebulaGraph's recent webinar about data visualization in NebulaGraph, where three NebulaGraph engineers talked about how data visualization in implemented in [NebulaGraph](https://nebula-graph.io/), the world's most powerful graph database management system.

# What is visualization?

From my personal understanding, visualization is the use of certain data or text that we usually see in a relatively abstract way, such as icons or charts, for secondary display and more convenient for users to interact. Simply speaking, the common buttons and icons on web pages, such as the "Google" button in blue after typing in https://www.google.com/ and the auxiliary function icons in the lower right corner, are all simple and basic visual displays. Without these small icons, you may need to call the terminal to execute a certain command to complete the corresponding operation conveniently. The visualization can greatly reduce the operation cost as well as make it easier for users to understand the corresponding operation.

In the above example, we are talking about UI visualization. Speaking of UI, let's read the memory of more than ten years ago. As you know, in the early years, computer technology was not yet developed, and we all looked at the command line. And now, why computers can fly into ordinary people's homes? It is because the computer or computer has an interface, a UI, something that meets human intuition. So, visual things that improve efficiency are called visualization. If we were to give a definition, visualization would probably have two points.

1.  The above example and the GUI, such visualization is visible to everyone, but in my opinion is not the biggest highlight that reflects visualization.
2.  The design, or the art, which is the soul of the visualization. Because with this thing, it will give meaning to the visualization of life. In layman's terms, whether something is good or not, often not you can make it out it is good; but if it is good interaction, good experience, good visual effect, can meet the human intuition, dig out some information, it is considered a good visualization. So, I think visualization still needs two parts, one is in the physical sense, that is, the first part of the rendering, and the second part is its design experience, which involves software and hardware.

In fact, the concept of visualization can be viewed in two dimensions: broad and narrow.

In the broad sense, visualization can be understood as taking a complex piece of information and making it simpler and more intuitive to present to the user through certain processing, so that the user can receive some information more easily. This belongs to the category of visualization, like GUI, which is actually a kind of visualization, abstracting the underlying commands in a visual way, so that users can interact with the machine more easily.

In a narrow sense, it is the application of the scenario. For example, along with the development of the Internet, the amount of data we deal with will become larger and larger, we have to present these data through a certain way, the better the presentation, the better the data expression can be understood. How to display more data more smoothly on the basis of the current hardware with certain performance bottlenecks may be the meaning of the current visualization. In this way, visualization in a narrow sense, it may be charts, Canvas animations, or the rise of VR technology, browser rendering, 3D rendering in the meta-universe, these visualization researchable directions.

Specifically, because Nebula is a distributed graph database, the visualization we mention is more about data visualization. For example, Nebula Dashboard is a visual operation and maintenance monitoring product, the visualization practice in this area is not only icons and animations, but also how to make the complex operation and maintenance information easier for users to control the cluster through a simple way.

![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/visualization-in-nebula-graph/figure1.gif)

# Visualization technologies

At different stages and in different scenarios, we choose different visualization technologies to better serve our products and business.

As far as visualization technologies are concerned, the common visual rendering technologies for front-end display are SVG, Canvas, CSS and WebGL, and what to use to draw SVG and Canvas is another topic. D3, as often mentioned by the Nebula community, is not a visualization technology, but essentially a visualization algorithm tool that can be used on the web or in a terminal application. D3 itself has some peripheral tools to do simple SVG wrapping, which is more on the upper layer of application tools, such as the charting library [ECharts](https://echarts.apache.org/en/index.html) is an example.

Wang Yang, the product leader of Nebula Dashboard, said that the common visual rendering technologies such as CSS and SVG mentioned above will be customized according to the application scenarios and computing resources faced. For example, in order to facilitate third-party access, the designers of the visualization library will abstract the syntax and make the syntax sugar more scientific and standardized; for scenarios with high performance requirements, they will encapsulate the tools in order to obtain high performance. Here he mentioned Ant open source visualization engine: G2 https://github.com/antvis/g2/, G2 was inspired by the idea of the book "The Grammar of Graphics" to design this set of visualization based on the theory of graphics grammar underlying engine, G2's set of grammar implementation is also more recognized in the industry to the scientific specification. Compared with other visualization libraries, G2 is more like a complete product in terms of syntax design.

When it comes to G2, the core developer of the visual graph exploration tool Nebula Explorer, Miao Zhuang, said that although the syntax of G2 is standardized and scientific, it is not a very smooth visualization tool if you really want to do data visualization, because it is based on the theory of graph syntax and requires certain expertise, so there will be a certain threshold. More often than not, G2 may still be used as a visual charting library rather than a visualization library.

# Performance issues of visualization

Nebula Explorer is a graph exploration tool and it will face the problem of node data presentation. The community also often asks how many node data can be rendered by a single canvas page. In response to the visualization performance issue, Miao Zhuang, the core product developer, said: visualization performance is not an absolute performance, it is often accompanied by the user experience. The user experience and performance are somehow inversely proportional, the better the performance, the better the user experience, and the less information will be presented; while presenting more data, it will consume more hardware resources. This is a balance problem.

And there are two ways to improve the performance of visualization: one is to improve the hardware configuration, before the very popular terrain - distributed concurrency is not enough what to do? Add a machine, there is nothing that can not be solved by adding a machine. But the simple and brutal method of adding machines is not a good policy, although the physical limitations are solved but the cost of adding the problem, in addition to the cost of the machine and the corresponding system maintenance costs. A machine, or more than one machine to join the system will certainly bring the complexity of the system increases. In addition, adding machines is not a universal solution, some users may not be able to add machines as they wish. Another is to optimize the algorithm, compared to adding machines, method 2 is more difficult, the current algorithm is very good, and continue to optimize only the marginal decreasing effect.

# Nebula visualization tool optimization practice

Nebula Explorer optimization performance is also from the above two hardware and software direction to start. In terms of hardware, the hardware performance of different users is different, so Explorer will choose better hardware to achieve cool effects during the development process; while for users with poor hardware performance, it provides two modes: high quality and normal quality, adapting to the user's hardware to achieve a better rendering effect. Although Explorer's 3D graphics exploration mode looks cool and "eats" computing resources, the hardware requirements for this feature are not as high as you might think, and any phone's GPU configuration can run Explorer 3D mode.

In fact, in addition to the performance optimization of hardware and software, there is also a strategic optimization of product design. For example, some of the more difficult performance problems can be circumvented. In terms of performance optimization, there is a time-honored principle, which is to ensure that 60 frames are rendered in 1 second. Even in special scenarios where 60 frames a second is not possible, achieving more than 40 frames a second will keep the system smooth. This is like an optimized implementation of React, which breaks up a lot of big tasks into smaller ones, and then plugs in smaller ones when the system is free. Another way to think about this is that if the problem cannot be solved at the code level, you can modify the corresponding interaction. For example, a long time-consuming task is running, and if the page is not clickable, you will feel very laggy, right? However, adding a loading animation at this time will greatly reduce the user's waiting anxiety and make the user feel less laggy.

Here is a personal experience: when Nebula Dashboard first launched the visualization screen, we stepped through the performance pitfalls. When the visualization screen was first launched, I just thought it was a simple screen that might not involve too many performance problems. As a result, once the CPU was spinning like crazy, I was confused: Nebula Dashboard visualization screen only had 7 or 8 charts. Although the interface was cool, there were actually only a dozen network requests, so we ruled out the problem that the CPU was consuming too many resources in the request calculation. In addition, when the big screen was online, there were not many concurrent cases... After a round of checking, we finally located that the SVG was consuming too much resources. So we adjusted the animation implementation scheme: for animations with high CPU requirements, we downgraded them and reduced the number of rendering and rendering frequency. This is a bit like image compression, you throw a 1 MB image to the compression site, the compression may be 200 KB, at this time the user's naked eye is not aware of it.

# Visualization technology selection

As mentioned above, hardware and software can improve the performance of visualization, but a good choice of technology can also affect the performance of your system.

Before Nebula Explorer v2.2, due to the technical selection and business logic organization, once we encountered hundreds of nodes displaying problems, we would encounter lag and the animation was obviously not smooth. After that, Nebula Explorer was refactored and replaced by G6, a [graph visualization](https://www.nebula-graph.io/posts/graph-database-visualization) engine based on G2, which performed poorly in Nebula Explorer, probably due to G6's over-focus on business scenarios and lack of validation for high-performance scenarios. After switching to Force Graph, the performance has improved by about 10 times. Of course, it is not that the performance of G6 is not good, but the visualization requirements of Nebula Explorer itself are not MATCH with G6.

On the contrary to Nebula Explorer, Nebula Dashboard does not have thousands of points of data display, but a relatively fixed number of graphs and the corresponding operation execution. For monitoring charts, Grafana is currently the most recognized product in the market, and Grafana itself is an open source product. So, we researched Grafana's implementation and found that it uses the lightweight uPlot https://github.com/leeoniya/uPlot. Because uPlot is less than 40kb, it features time series charting and is easy to do custom development. In addition to monitoring, Nebula Dashboard also has an important function for cluster management and maintenance operations. This is a departure from Grafana's focus on monitoring, and the three criteria for selecting Nebula Dashboard are: first, easy to develop and start; second, active open source community; third, many base libraries to improve development efficiency. In the end, G2 was chosen as the underlying visualization tool for Nebula Dashboard in order to improve development efficiency.

# Visual Product Design

Nebula Explorer has two graph exploration modes, 2D and 3D, which involve how the pages are laid out. In the Nebula Explorer design, algorithms are used to achieve aggregation and hashing of specific groups of points; and the full graph bird's eye view mode is displayed in 3D mode, because the mainstream computer is 1920*1080, and a plane can display up to a million pixels, then we can use 3D mode to present more data than 2D when we change the angle.

‍

This article is transcribed from NebulaGraph's recent webinar about data visualization in NebulaGraph, where three NebulaGraph engineers talked about how data visualization in implemented in NebulaGraph, the world's most powerful graph database management system.

How data visualization is implemented in NebulaGraph


This article is based on a talk given by Dr. Min Wu, a senior expert at Vesoft Inc. Dr. Wu talked about the status quo of the global graph database market, the design and features of NebulaGraph as a distributed graph database, as well as NebulaGraph’s open-source community.

# The Global Graph Database Market

Let’s start with some numbers. Markets and Markets anticipates the graph database market will reach $2.4 billion by 2023 from $821.8 million in 2018. 

Graph database is still in a rising trend and it is one of Gartner’s top 10 data and analytics trends in 2021. This is because graph databases can be used in many more areas than traditional databases, including computing, processing, deep learning, and machine models.

![The Global Graph Database Market](https://user-images.githubusercontent.com/90186547/153827006-decd54de-76fa-4123-83e3-739512013fce.png)

Graph databases gained the most in popularity in the past 10 years, according to data compiled by DB-Engines. The data is based on social media mentions, Stack Overflow questions, and search trends.

![The Global Graph Database Market](https://user-images.githubusercontent.com/90186547/153827206-90b5b420-99b4-466c-904c-78250115ee1e.png)

# Advantages of Graph Databases

One of the most significant advantages of graph databases is that they are intuitive. If you want to express the character relationships of Game of Thrones, you can use both traditional tabular databases and graph databases. But as it is shown below, using graph data is much more intuitive, though they both express the same data model.

![Advantages of Graph Databases](https://user-images.githubusercontent.com/90186547/153827594-7c9f5e95-ae9d-499a-bb5b-f41cb48d3578.png)

(Tabular data)

![Advantages of Graph Databases](https://user-images.githubusercontent.com/90186547/153983950-58cd84d4-8bed-400e-8daa-ce1e04549172.png)

(Graph data)

In another example, we can compare how to search in SQL databases and graph databases. For example, here is how to find out how many posts and comments were created in a given time frame and rank the results in SQL databases and graph databases.

SQL:

```
--PostgreSQL 

WITH RECURSIVE post_all (psa_threadid

                      , psa_thread_creatorid, psa_messageid

                      , psa_creationdate, psa_messagetype

                       ) AS (

    SELECT m_messageid AS psa_threadid

         , m_creatorid AS psa_thread_creatorid

         , m_messageid AS psa_messageid

         , m_creationdate, 'Post'

      FROM message

     WHERE 1=1 AND m_c_replyof IS NULL -- post, not comment

       AND m_creationdate BETWEEN :startDate AND :endDate

  UNION ALL

    SELECT psa.psa_threadid AS psa_threadid

         , psa.psa_thread_creatorid AS psa_thread_creatorid

         , m_messageid, m_creationdate, 'Comment'

      FROM message p, post_all psa

     WHERE 1=1 AND p.m_c_replyof = psa.psa_messageid

     AND m_creationdate BETWEEN :startDate AND :endDate

)

SELECT p.p_personid AS "person.id"

     , p.p_firstname AS "person.firstName"

     , p.p_lastname AS "person.lastName"

     , count(DISTINCT psa.psa_threadid) AS threadCount

END) AS messageCount

     , count(DISTINCT psa.psa_messageid) AS messageCount

  FROM person p left join post_all psa on (

       1=1   AND p.p_personid = psa.psa_thread_creatorid

   AND psa_creationdate BETWEEN :startDate AND :endDate

   )

 GROUP BY p.p_personid, p.p_firstname, p.p_lastname

 ORDER BY messageCount DESC, p.p_personid

 LIMIT 100;
```

Here is how to realize the same query using the Cypher graph query language:

```
--Cypher
MATCH (person:Person)<-[:HAS_CREATOR]-(post:Post)<-[:REPLY_OF*0..]-(reply:Message)
WHERE  post.creationDate >= $startDate   AND  post.creationDate <= $endDate
  AND reply.creationDate >= $startDate   AND reply.creationDate <= $endDate
person. RETURN
id,   person.firstName,   person.lastName,   count(DISTINCT post) AS threadCount,
  count(DISTINCT reply) AS messageCount
ORDER BY
  messageCount DESC,  person.id ASC
LIMIT 100
```

In addition, the graph ecosystem is diversified. The following is the [graph technology landscape in 2020](https://graphaware.com/graphaware/2020/02/17/graph-technology-landscape-2020.html), and we can expect more graph related technologies coming along in 2021.

![Advantages of Graph Databases](https://user-images.githubusercontent.com/90186547/153828477-22d4c36c-63ea-438e-9ee6-ed3c618cd37a.png)

# NebulaGraph

Now, let's take NebulaGraph, a distributed graph database, as an example to talk about the evolution of graph technology. I will also share the challenges the team had faced when developing NebulaGraph and how we had solved them.

When we started designing the blueprint of NebulaGraph in late 2018, the team had set four goals for the database. They are scalability, production-ready, OLTP(Online Transaction Processing), and open source. The four goals are still influencing the roadmapping of NebulaGraph until today.

**Scalability**

Scalability is the No.1 design principle of NebulaGraph. This is because we believe the data that businesses will process in the future must be massive and there will be no way that single machines can handle them. That’s why we designed NebulaGraph in a way that it is capable of handling graph data with trillions of vertices and edges.

**Production-ready**

NebulaGraph is also designed to be production-ready on the first day, including the design of its query language, visualization, programmability, and DevOps.

**OLTP**

OLTP (online transactional processing) enables the real-time execution of large numbers of database transactions by large numbers of users, typically over the internet. One of the priorities of NebulaGraph’s design is OLTP. This makes NebulaGraph an online, high-concurrency, and low-latency graph database.

**Open source** 

NebulaGraph is also devoted to building an open-source community and integrating with the big data world, supporting graph computing and training frameworks like Tencent Plato and Spark GraphX.

![NebulaGraph](https://user-images.githubusercontent.com/90186547/153829025-447f8153-0154-443e-96b6-aaf54f620877.png)

# The Nebula Core

The above figure shows the ecosystem built around NebulaGraph. The section with red background is the NebulaGraph Core, which consists of three parts called Meta, Graph, and Storage.

NebulaGraph’s query language is our in-house nGQL, which is also compatible with openCypher. We have also developed clients in languages including Java, C++, Python, and Go. Then on the top we have a number of SDKs that can work with frameworks like Spark, Flink, GraphX, Tencent Plato. 

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153829317-ad4a7330-8b3b-461c-b611-fe017856336c.png)

Let’s dive into the Nebula core, which, as I mentioned above, consists of the meta, graph, and storage services. The meta service deals with metadata, while the storage service stores the data, and the graph service is in charge of querying. The three modules run on their independent processes, ensuring the separation of compute and storage.

The meta service manages the schema. NebulaGraph is not a schema-free database, and it requires the properties of vertices and edges to be pre-configured. The meta service also manages storage spaces, long-duration tasks, and data cleaning.

NebulaGraph can handle data with trillions of vertices and edges. This means the system must segment the data in storage and handling. NebulaGraph uses the segmentation of edges and stores vertices in partitions. Each partition may have a few replicas and run on different machines. The query engine is stateless, meaning that all query data should either be retrieved from the meta service or the storage service and there is no communication between query services.

The above is about NebulaGraph’s separation of compute and storage. Now let’s talk about data characteristics. We have mentioned that NebulaGraph is not a schema-free database and that all the data stored is pre-defined by Data Definition Languages (DDLs). We call types of vertices Tag and types of edges EdgeType. Vertices are defined using a 2-tuple consisting of the vid and the tag. Edges are defined using a 4-tuple consisting of the endpoints, EdgeType, and rank.

NebulaGraph supports primitive data types like boolean, int, and double as well as composite types like list, set, or graph data types like path and subgraph. If there is long string data stored in the database, it is usually indexed by Elasticsearch.

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153829515-9259bd0c-d127-43ee-9724-0d2561619296.png)

Here is some additional information about the storage engine. For the query engine graphd, the external interface exposed by the storage engine is a distributed graph service, but it can also be used as a distributed key-value (KV) service if necessary. In the storage engine, partitions adopt the Raft consensus protocol. NebulaGraph stores vertices and edges in separated partitions. The following figure is about how KV is implemented using the storage partitioning.

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153829730-a73d0aa3-ab99-404e-b355-06e1b98111d6.png)

In NebulaGraph, each edge is stored as two pieces of data. As mentioned above, the storage layer relies on VID and guarantees strong consistency using the Raft protocol.

NebulaGraph uses ElasticSearch for full-text index. From NebulaGraph v2.x, our R&D team has optimized the write performance of Nebula's indexing capability. Since v2.5.0, NebulaGraph has started to support the combination of data expiration TTL and indexing. And from v2.6.0, NebulaGraph started to support the TOSS (Transaction on storage side) function to achieve the eventual consistency of edges. That is to say, edges are either successfully written or failed at the same time when they are inserted or modified. 

NebulaGraph uses its in-house nGQL as its query language. In June 2021, the International Organization for Standardization has drafted the standard for the syntax and semantics of GQL and there is a consensus between major graph database vendors.

From NebulaGraph v2.0, nGQL started to be compatible with openCypher, which was an open-source version of Neo4j’s Cypher query language. Now, nGQL supports the Doctrine Query Language (DQL) of openCypher and has developed its own vanilla nGQL syntax style in Data Manipulation Language (DML) and Data Definition Language (DDL).

We also mentioned that NebulaGraph is born to be production-ready. So it supports a wide range of operation features like data isolation, user permission and authentication, and replica configuration. Also, NebulaGraph supports clustering. Nebula Operator, which was released in April 2021, started to support Kubernetes.

As for the performance of NebulaGraph, most performance tests are carried out by users in the community, such as engineers from tech companies like Meituan, WeChat, 360 DigiTech, and WeBank, etc. The figures below are performance reports compiled by users.

As for the performance of NebulaGraph, most performance tests are carried out by users in the community, such as Meituan, WeChat, 360 DigiTech, WeBank. The figures below are performance reports compiled by users.

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153830033-15cfdf2f-fa39-457f-96b4-b83bebdab759.png)

![The Nebula Core](https://user-images.githubusercontent.com/90186547/153830090-2b9d1624-00ea-42d0-a6b4-737719683b41.png)

We have mentioned that one of the priorities of NebulaGraph’s design is OLTP. But that doesn’t mean it neglects analytical processing (AP). NebulaGraph has integrated AP frameworks like Apache Spark’s GraphX and Plato developed by Tencent.

Generally speaking, NebulaGraph performance has improved significantly when the deep traversal is conducted.

NebulaGraph meets users' AP (Access Point) requirements of OLTP (Online Transactional Processing ) by docking with Spark's Graph X and supports Plato, the graph computing engine of Tencent's WeChat team. Plato docking is actually the data connection between the two engines, which needs to change the internal data format of NebulaGraph to be that of Plato and then Partition to map them one by one. 

# The NebulaGraph Community

![The NebulaGraph Community](https://user-images.githubusercontent.com/90186547/153830303-56568605-7547-4c00-a3ff-9d6e7b78e398.png)

NebulaGraph became open source in May 2019. Its v1.0 GA was released in June 2020, even though some companies had applied NebulaGraph in production before that. NebulaGraph first entered DB-Engines’ graph database management system ranking two years ago and now it ranks 15th on the list.

![DTCC2021](https://user-images.githubusercontent.com/90186547/153830477-11d7f607-aecb-468e-b7f5-f0ab360ff71b.png)

NOTE: The screenshot is the DB-Engine ranking in Apr. 2021 when this talk was given.

NebulaGraph is also one of the top open source players in China. This following report, which was published by the X-Lab of [East China Normal University](https://en.wikipedia.org/wiki/East_China_Normal_University), ranks the companies according to the community popularity of their open source products. Vesoft Inc., the maker of NebulaGraph, ranks the eighth, before TikTok parent Bytedance, and just one position after Huawei.

![The NebulaGraph Community](https://user-images.githubusercontent.com/90186547/153830897-9cc1d126-1735-48c4-a8c3-10db68eee805.png)

Here are some of my thoughts about open source graph databases. Open source is very common in the graph database industry because it is a relatively new area and only got traction in recent years. This is why NebulaGraph chose to be open source from day one. Open source software can also attract more developers to use and gain valuable feedback from adopters.

If you encounter any problems in the process of using NebulaGraph, please refer to [NebulaGraph Database Manual](https://docs.nebula-graph.io/2.6.0/pdf/NebulaGraph-EN.pdf) to troubleshoot the problem. It records in detail the knowledge points and specific usage of the graph database and the graph database NebulaGraph.

Join our [Slack channel](https://join.slack.com/t/nebulagraph/shared_invite/zt-7ybejuqa-NCZBroh~PCh66d9kOQj45g) if you want to discuss with the rest of the NebulaGraph community!


This article introduced the status quo of the global graph database market, the design and features of NebulaGraph as a distributed graph database, as well as NebulaGraph’s open-source community.