As organizations collect and store more and more data, the need to understand and analyze complex relationships within that data has grown tremendously. Graph databases, which are designed specifically to store and query interconnected and complex data, are uniquely suited to this task. The [benefits of graph databases](https://www.nebula-graph.io/posts/why-use-graph-databases) are immense and this explains why they have become a popular choice for many businesses. For example, a [graph database](https://nebula-graph.io/) offers fast performance on complex queries, making them useful for use cases such as fraud detection, [recommendation engines](https://www.nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation), and social network analysis.

One key aspect that users are increasingly seeking when working with complex data is visualization, as it makes it easier to quickly interpret what their data is telling them. 

Let’s explore graph database visualization in particular,  the benefits and use cases.  


# **What is graph database visualization?**

Graph database visualization is the representation of the data stored in a graph database in a visual format, for example a chart or diagram. This makes it easier to understand and analyze the data, because it's easier to see patterns and relationships when they're represented visually. 

For example, a social network company can use a bar chart to show how many connections each user has, or a line graph to show how the number of connections has changed over time. We might also use a scatter plot to show how different users are connected to each other, or a network diagram to show how different groups of users are connected. 

![graph-database-on-social-network](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/graph-database-on-social-network-02.jpeg)

When you visualize your data in a graph database, it makes your work much easier on many fronts. The structure becomes user-friendly and you gain a significant boost in efficiency when working with the data. We'll tackle the advantages shortly.


# **How graph database visualization works**

To best understand how graph database visualization works, it’s important to briefly go over how data is stored and used in a graph database. We are aware that a graph is basically a collection of nodes and edges, where each node represents an individual data point like a person or transaction, and each edge indicates how two nodes are related; for instance, a person makes a transaction. This data in the graph is stored in a graph database, such as NebulaGraph. 

Now, whenever a user wants to retrieve some data, they perform a query. By default, the results of the query normally come in non-visual format such as CSV, JSON, XML, Text Files, spreadsheet, etc. But with visualization, the results will come in a visual format specified by the user - as long as it’s supported by the Graph Database that is being used. Some of the formats can include network diagrams, interactive visualization, heat map, scatter plot, etc.

With this level of visualization, it becomes easy even for novices and non-technical users to understand complex data and make meaningful deductions from it. It means, for example, that a team of data experts can comfortably collaborate on a project with other team members who are not data experts themselves. A business development manager with little data knowledge can make use of such visualization to draw incredible insights that can be used to make valuable decisions. 

A graph visualization tool allows you to specify the layout of the display, the color and size of the nodes and edges, and other formatting options. These tools employ algorithms to work out node positions and showcase the data in either two or three dimensional formats. When you interact with the visualized graph, you can zoom in and out, pan around, and click on individual nodes and edges to see more information about them. It’s that simple. 


# **Advantages of graph database visualization**

If we had to give a concise description of the many benefits of graph database visualization, it would be "enlightening”. Check out this breakdown of the top advantages.


## **1. Straightforward view**

When data is presented in visual format, it becomes much more intuitive and easier to grasp. This is because the brain is naturally better at processing and understanding visual information. 

With graph database visualization, you can see the data in a way that is more similar to how you would see and understand the real-world objects or relationships that the data represents. 


## **2. You can easily discover more revelations**

Visual data formats allow users to easily find patterns and relationships that may not be immediately apparent in non-visual formats.. 

The visual appeal of the data also makes it easier for users to explore it more extensively. It encourages users to be more involved with the info, so they'll spend more time there and uncover other perspectives as a result. The fact that the data is visually appealing makes it even easier for them to explore it more extensively - leading to even further discoveries!


## **3. You can easily explain findings**

Graph database visualization allows users to easily share and explain their findings by highlighting important nodes and adding notes or comments. 

By adding annotations or highlights to specific nodes, you can can draw attention to trends or patterns in the data and help your colleagues to quickly recognize the insights that you have discovered


## **4. Friendly for non-experts**

Even users who have little or no technical knowledge in data analysis will find it so easy to make use of graph visualization.  

Many graph visualization tools are designed to be user-friendly and intuitive, with features such as interactive elements, hover-over text, and filtering options that allow users to explore the data and customize their views. 


## **5. You can zoom in and out**

Graph database visualization enables you to zoom in or out and explore the data at varying levels of detail. You can narrow in to pinpoint some areas of the graph or widen your view to observe what it looks like overall. 

This can be beneficial when you're attempting to recognize how dissimilar parts of the data interconnect or require finer details. Say, if you're analyzing a large network of people, you could zero in on particular clusters of people to see how they are interrelated or broaden your view to see the complete structure of the network.


# **Graph database visualization use cases**

Any context that requires understanding and analyzing complex data relationships is a good candidate for graph database visualization. Here are some common use cases:  


## **1. Fraud detection and investigation**

Fraud is a serious problem that is already having significant consequences for individuals, businesses, and society as a whole. It can take many forms, including identity theft, and[ financial fraud](https://www.nebula-graph.io/posts/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph) such as investment fraud, insurance fraud, bank fraud, mortgage fraud, and charity fraud, to name a few.

Graph database visualization can help to detect all these forms of fraud by identifying patterns and anomalies in data that may indicate fraudulent activity. It can be particularly useful for identifying suspicious patterns or behaviors, such as consistent trades on the same stocks or unusual money transfers between accounts.

![fraud-detection-using-graph-visulization](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/fraud-detection-using-knowledge-and-graph-database/figure3.png)

## **2. Supply chain management**

Supply chain management is a complex process with many moving parts. Graph visualization can help supply chain managers to coordinate and optimize many activities and entities in the production and distribution of goods and services.

You can so easily visualize data for example the flow of materials, information, and financial transactions as a graph, enabling you to quickly respond to bottlenecks and identify opportunities for optimization.


## **3. Cybersecurity**

Cybersecurity has become a top priority for organizations in all industries, particularly due to these reasons:



* **Growing number of cyber threats:** There has been a dramatic increase in the number and sophistication of cyber threats in recent years, ranging from malware and phishing attacks, to ransomware and data breaches.
* **Increased regulation:** Many industries are now subject to various regulatory requirements that mandate certain cybersecurity practices and standards. For example, the healthcare industry is subject to HIPAA regulations, which require the protection of personal health information, and the financial industry is subject to various regulations, such as the [Gramm-Leach-Bliley Act](https://www.ftc.gov/business-guidance/privacy-security/gramm-leach-bliley-act), which require the protection of personal financial information.

Companies can use graph visualization to address the above cybersecurity challenges in the following ways:



* **Network security:** If you work in network security, try to represent the relationships between different entities (e.g., devices, users, and IP addresses) in your network as a graph and see how easy it is to identify anomalies that may indicate a security breach. For example, if a group of devices that normally communicate with each other suddenly start communicating with a new, unknown device, this could be a red flag and warrants further investigation.
* **Threat intelligence:** Visualize the relationships between different types of cyber threats (e.g., malware, phishing attacks, etc.) and the entities (e.g., IP addresses, domains, etc.) associated with them. Armed with this visual presentation, you will easily understand how different threats are related and how they may be used together in coordinated attacks.
* **Incident response:** In the event of a security breach, an organization can use graph visualization to understand the scope and impact of the incident. Simply represent the affected assets, the entities involved, and the timeline of events as a graph, and you could be on your way to quickly identifying the root cause of the breach. Next is to deploy the appropriate response plan.


## **4. Knowledge management**

Knowledge management is the process of creating, sharing, using and managing the knowledge and information of an organization. It involves capturing, distributing, and effectively using knowledge to support the goals and objectives of the organization. The goal of knowledge management is to improve efficiency and effectiveness by making the most of an organization's collective expertise and experience. 

A graph database can be used to represent the relationships between different concepts and ideas in a body of knowledge. Visualizing these relationships can help users understand the structure of the knowledge and how different concepts are related to one another.


## **5. IT operations management**

IT operations management refers to the process of managing the day-to-day operations of an organization's IT infrastructure and systems. This includes tasks such as monitoring and maintaining the availability and performance of IT systems, and managing the configuration and deployment of hardware or software.

One use case of graph database visualization in IT operations management is identifying and troubleshooting issues. IT operations teams can do this by creating a visual representation of the relationships between different components and events. This allows them to easily see how the different components may be impacting each other, enabling them to more effectively identify and resolve issues


## **6. Drug discovery**

Drug discovery is the process of identifying and developing new drugs to treat specific diseases or conditions. This process involves a number of activities, including identifying potential drug targets, synthesizing and testing potential drug candidates, and conducting clinical trials to evaluate the safety and effectiveness of the drug.

Researchers can use graph database visualization to identify potential drug targets by creating a visual representation of the relationships between different biological pathways.

For example, consider a researcher who is studying a particular disease, such as cancer. They may create a graph that represents the relationships between different proteins and genes that are known to be involved in the disease, as well as the pathways in which they are involved. By analyzing this graph visually, the researcher can identify potential targets for new drugs that may be able to disrupt the disease process.


## **7. Recommendation engines**

A recommendation engine is a system that makes personalized recommendations to users by identifying patterns in data. The recommendations are based on the user's past behavior, as well as the behavior of similar users. Recommendation engines are commonly used in e-commerce, media, and social media to help users discover new products, articles, or friends that they might be interested in.

Here are a couple of ways that organizations are putting graph database visualization to work in recommendation engines:



* **E-commerce:** Many e-commerce companies use graph database visualization to help recommend products to users based on their past purchases and the purchases of similar users..
* **Social media:** Social media platforms often use graph database visualization to make content or friend recommendations to users based on their connections and interactions with other users.
* **Media:** Media companies can use graph database visualization to recommend articles, videos, or other content to users based on their past consumption and the consumption of similar users. They use graph visualization to identify patterns in data and highlight groups with similar interests.
* **Gaming:** Gaming companies can use graph database visualization to recommend games or in-game items to users based on their playstyles and the playstyles of similar users. It helps identify patterns in data and highlight groups with similar interests.


# **Graph database visualization with NebulaGraph**

NebulaGraph offers an intuitive enterprise graph visualization tool known as [NebulaGraph Explorer](https://explorer.nebula-graph.io/welcome), which is browser-based. You can use this tool in conjunction with the NebulaGraph core to visualize complex interactions in your graph data. As we indicated earlier, you need not be a graph database expert in order to comfortably use a visualization tool, and so you’ll be glad to know that NebulaGraph Explorer is easy to use even for a non-technical user. 

The [NebuaGraph Manual](https://docs.nebula-graph.io/2.5.0/pdf/NebulaGraph-EN.pdf) has all the friendly help you need to get started with this amazing graph database visualization tool - please check it out. Simply go to the table of contents, look for Nebula Explorer and click. Whether you're working in e-commerce, media, social media, or any other industry, Nebula Explorer will help you get the most out of your data.


# **Conclusion**

We sincerely hope you now understand how beneficial graph database visualization can be for anyone working with data, regardless of the level of expertise or technical skill. From subject matter experts and data scientists to analysts, developers and non-technical users - anyone who wants to explore their data and discover hidden insights can benefit from graph visualization.

Finally, it's important to stay focused on the data itself, rather than just being captivated by the aesthetics of the visuals. We say this because it is easy to be mesmerized by the beauty or elegance of a visualization and overlook the underlying data and what it's actually trying to tell you. This can lead to a superficial understanding of the data and may prevent you from fully grasping the trends and patterns that are present in the data. The key is to move beyond just looking at the visualization, and instead really delve into the data points which make it up, while also taking into account any context or questions which it is meant to answer.


 Dealing with complex interconnected data? Learn how to effectively visualize your graph database to gain deeper insights and better understand your data.

Graph Database Visualization | Graph-Based Analytics and Visualization with NebulaGraph

Imagine a world where transactions, accounts, and customers are connected to a vast web of data. This is the world that fraudsters operate in, using a variety of techniques to stay one step ahead of detection. But imagine if businesses could see the same web of data, connecting the dots between seemingly unrelated transactions and customers to uncover fraudulent activity.

Enter graph machine learning, where it’s possible to map out the complex web of relationships between users, accounts, and transactions, making it easier to spot patterns of fraudulent activity. With this, we’ll catch the fraud in near real-time and stop it before damage happens.

The worldwide fraud detection market is rapidly expanding and is projected to reach[ USD 129.17 billion](https://www.fortunebusinessinsights.com/industry-reports/fraud-detection-and-prevention-market-100231) by 2029, as compared to USD 30.65 billion in 2022. This highlights a great potential for growth and, with the large scale and intricate data involved, graph-based fraud detection is going to play a huge role.

This post explores the concept of Graph Machine Learning and its potential to revolutionize[ fraud detection](https://www.nebula-graph.io/posts/fraud-detection-using-knowledge-and-graph-database). 


# **What is graph machine learning?**

Graph Machine Learning is a type of ML that specializes in building algorithms and models to process and analyze graph-structured data in a[ graph database](https://nebula-graph.io). Graphs are made of nodes (or vertices) and edges, with the edges connecting the nodes and representing relationships between them. 

The goal of graph ML is to make predictions or get insights from the graph structure by training models to do tasks such as node classification, link prediction or community detection. These tasks can be accomplished through different techniques such as graph convolutional networks, graph attention networks and random walk-based methods. These models can be trained on large, complex graphs using either semi-supervised or unsupervised representation learning techniques.


# **Using graph machine learning  for fraud detection**

Graph machine learning is used for fraud detection by analyzing the connections and relationships between entities in a network. It can be applied to a wide range of datasets, such as financial transactions, communication logs, and social media interactions, to uncover patterns and anomalies that indicate fraudulent activity.

One common approach is to construct a graph based on the relationships between entities, such as financial accounts, individuals, or IP addresses. This graph can then be analyzed to identify groups of entities that exhibit suspicious behavior, such as abnormal patterns of communication or financial transactions.

The typical process of getting to prepare a graph ML model for use in fraud detection can be divided into several phases. Here is a visual presentation of how the process generally looks like:  

Let’s discuss the process of creating a fraud detection solution with graph machine learning.


# **Stage 1. Understanding the data**

It all starts by understanding the kind of data you are dealing with. Below are some of the common types of data types you’re likely to encounter in a typical organization that is concerned with fraud (both private and government)

**Transaction data:** This type of data includes details relating to individual transactions, like the date, time, price, and vendor. We can use this data to identify suspicious activity, like strange spending habits or purchases that don't match up with a customer's usual behavior.

**Customer data:** Encompasses details about a customer, like their name, address, phone number, and email address. This info can be leveraged to recognize fraud linked to an individual customer, like out of the blue changes to their contact info.

**Network data**: Information about networks and devices used for transactions like IP addresses and device fingerprints. This type of data can help detect fraudulent activities that are associated with a particular network or device - for example, a group of transactions that all come from the same IP address.

**Demographic data:**  Includes information about the customer, such as age, gender, income, and job position. Through such data, we can recognize patterns of fraud that are characteristic of particular demographics.

**Behavioral data**: Contains info on how customers utilize a system/service, e.g. logins, time spent in sessions, clicks and navigation. This data can help detect fraud patterns related to a customer’s behavior, like an account being accessed by numerous IPs and devices, or an account originating from an abnormal or suspicious location.

**External data:** Consists of information sourced from external sources like social media and public records.This kind of info can supplement the other forms of data above, help confirm customer identity, and detect fraudulent activity, like recognizing bogus or stolen identities.

It is essential to train the model with a labeled dataset that includes fraudulent and non-fraudulent activities. The data's accuracy, relevance, and variety are key factors that determine how well the model will perform.


## **How to preprocess the data and get it ready for analysis**

The specific steps and techniques for preprocessing our data will depend on the nature of the data, the goals of the analysis, and the type of model. 

These are the common steps in pre-processing:   



1. **Data cleaning:** This step involves identifying and rectifying errors or inconsistencies in the data. For instance, this could include deleting duplicate entries, updating incorrect values, or inputting missing information..
2. **Feature engineering:** Entails developing fresh features or variables from the current data. For instance, you could devise a new feature that represents the average transaction amount of a customer or shows if a customer has ever committed a fraudulent transaction.
3. **Data scaling:** The process of transforming data from a given range to a new range, in order to bring all the features to the same level of magnitude so that no feature dominates others.
4. **Data splitting:** Involves dividing a dataset into two or more subsets,e.g a training set, a validation set and a test set, in order to train models and evaluate their performance. The training set is used to train the model,  the validation set is used to evaluate the model performance  and optimize the parameters, and the test set is used to evaluate the final performance of the model.
5. **Outlier detection and treatment:** This is all about identifying and eliminating any data points that are drastically different from the rest. It’s important for a fraud detection system since as much as outliers may suggest suspicious activity, they can also affect the model's overall accuracy in performance if not taken care of.
6. **Data transformation:** Involves applying mathematical functions to the data to change its distribution and make it more suitable for the models. For example, logarithmic transformation can be applied to decrease the skewness of the data and to make it more homoscedastic, or power transformation can be applied to make the data more normal.


# **Stage 2: Building  the graph model**

Several approaches can be used for fraud detection with graph machine learning, such as community detection, link prediction, and anomaly detection. 

Community detection machine learning algorithms can be used to identify clusters or groups of individuals or entities that are likely to be involved in fraudulent activities. 

Link prediction algorithms can identify potential connections or relationships between individuals or entities that may indicate fraud. 

Anomaly detection algorithms can identify patterns or behaviors that deviate from normal patterns and may indicate fraud.

To build a simple model, let's say you are using a graph database like Nebula Graph and want to build a graph model to detect fraudulent transactions. You can start by importing your transaction data into your Nebula Graph database and create nodes for the entities involved in the transactions, such as individuals, accounts, and merchants. You will then create relationships between the nodes to represent the transactions and add properties to the nodes and relationships to store additional information such as amounts, dates, and locations.

Once you have your data imported and modeled in the graph, you can use[ nGQL](https://docs.nebula-graph.io/1.2.0/manual-EN/1.overview/1.concepts/2.nGQL-overview/), Nebula Graph's query language, to run community detection algorithms like Louvain or info map to identify communities of related nodes that may indicate fraudulent activities. You can also use nGQL to link prediction algorithms to find potential relationships between nodes that you may want to investigate further.


# **Stage 3: Evaluating the model**

Evaluation is crucial to ensure that our graph model can accurately detect fraud. Some common metrics you can use to evaluate the performance of a graph model include precision, recall, and F1 score.

Precision measures the proportion of true positives (correctly identified fraud) to all positive predictions (both true and false positives). Recall measures the proportion of true positives to all actual fraud cases. The F1 score is the harmonic mean of precision and recall that measures the model's overall accuracy.

To assess the model's effectiveness, it is essential to separate the data into training and testing sets so that the model can be trained on the training set and then evaluated on the testing set. This will accurately represent the model's performance on unseen data. Unseen data in this case refers to any data that the model has not been trained on. The idea here is that the model has been trained on a portion of the data (the training set), and then its performance is evaluated on a separate portion of the data (the test set) that it has not seen before. In other words we are essentially using the test set as a proxy for new, unseen data that the model will eventually encounter when it is deployed in the real world.

You can fine-tune the model to improve its performance by adjusting its parameters and algorithms or incorporating new data into the model. You can achieve this by running several tests to identify the parameters and algorithms that yield the best performance. One way to fine-tune the model is to use hyperparameter tuning, which involves adjusting the model's parameters to find the best combination that yields the best performance. To achieve this, we can deploy techniques like grid search, random search, or Bayesian optimization.

Another way to improve the model's performance is by incorporating new data, such as additional fraud cases or new feature data, into the model. This can increase the model's accuracy and make it more robust.


# **Stage 4: Deploying the model**

Just like a typical software or app, we can deploy our model on premise or in the cloud. Both options have advantages and disadvantages, and the best choice will certainly depend on the project's specific preferences.



* **Deploying on-premise:** Here we’ll deploy the model within the organization's infrastructure. This allows for more control over the deployment and security of the model, as well as the ability to customize the environment to the model's specific needs. However, this option requires more resources and maintenance from the organization’s IT team.
* **Deploying in the cloud:** The most popular options for this option include cloud platforms like AWS, Azure, or Google Cloud. This allows for easier scalability and reduces the need for infrastructure maintenance by the organization.


## **Example: How to deploy the model using Docker**

With Docker, you can containerize your fraud detection graph ML model and deploy it to your favorite cloud platform. Follow these steps:



1. **Create a Docker container:** The first step is to create a Docker container that includes all the necessary dependencies and software for the model. This includes the graph mode itself, necessary libraries, and configurations.
2. **Deploy the container:** Once the container is set up, you can deploy it either on-site or in the cloud. With tools like Docker Compose, you can easily launch multiple containers at the same time.
3. **Test the deployment:** Before deploying the model to production, it is essential to test it and be sure it’s going to function properly once deployed. You can do this by running a series of tests against the model and evaluating the results.
4. **Monitor and maintain the model:** So far so good? Well, things can go wrong at any time. So it's important that we constantly monitor our model's accuracy. Promptly add dependencies as the need arises, as well as new data to ensure the model's performance is always at peak.

Docker allows for easy deployment in different environments. It also makes it easy to maintain the model and update its dependencies.

**Also Read:**[ Graph-based intrusion detection](https://www.nebula-graph.io/posts/intrusion-detection-with-graph-technology)


# **Which types of fraud can graph machine learning detect?**

Some common types of fraud that can be detected using graph machine learning systems include:



* **Money laundering:** This is a type of financial fraud involving the movement of illegally obtained money through a series of transactions in order to conceal its origin..
* **Insider trading:** Involves the use of non-public information for financial gain. Graph machine learning can be used to identify connections and patterns of communication between individuals that may indicate the sharing of insider information.
* **Ponzi schemes:** The use of new investors' funds to pay returns to earlier investors.
* **Identity theft:** The unauthorized use of another person's personal information for financial gain. 
* **Social engineering:** The manipulation of people into divulging sensitive information or performing actions that can be used for financial gain..
* **Credit card fraud:** Credit card fraud typically involves someone obtaining credit card information and using it to buy goods and services without authorization. This type of fraud can cause a financial loss for the victim, as well as damage their credit score. 
* **Insurance fraud:** When false information is given to an insurer in order to obtain a payment that would otherwise not be granted. This fraudulent activity can range from making false claims about injuries or damage, to exaggerating medical problems in order to obtain higher payments.  

**Also Read:**[ Detecting insurance fraud with graph technology](https://www.nebula-graph.io/posts/insurance-fraud-detection-with-graph)


# **Conclusion**

Traditionally, organizations have used rule-based systems and linear models to detect fraud.  But as fraudsters become more sophisticated, these methods are becoming increasingly ineffective.

The sophistication of fraud means that we should stop approaching detection as a guessing game. It’s high time we started to uncover hidden connections and patterns within data, and detect even the most complex and well-concealed fraud schemes.

As we have seen, this is possible with graph machine learning. Graph models can reveal the subtle connections and relationships that exist within data, allowing us to see the bigger picture and gain a deeper understanding of data. With this capability, we can detect a developing fraudulent activity and stop it even before it’s actualized. It's not just a fraud detection software, it's a game-changer.

Think of it this way, with graph machine learning, fraud detection is no longer a reactive measure. It's a proactive approach that can keep your organization one step ahead of the fraudsters.


# **FAQ Section**


## **Why is graph machine learning used for fraud detection?**

Graph machine learning is used for fraud detection because it can accurately identify complex and hidden patterns plus connections that are not easily detectable by traditional methods. Graphs can represent the relationships and interactions between entities in a more intuitive way, and graph-based models can be used to analyze the structure of these relationships to detect patterns or anomalies that may indicate fraudulent activity.


## **How does graph machine learning work in fraud detection?**

Graph machine learning fraud detection works by creating a graph that represents the relationships and interactions between entities, and then creating fraud detection algorithms to analyze the graph for patterns or anomalies that may indicate fraudulent activity.


## **What are the challenges of using graph machine learning in fraud detection?**

Some of the common challenges associated with using graph machine learning for fraud detection include the high computational cost of analyzing large and complex graphs, the need for high-quality data to effectively train the model, and the need for high level experts to interpret the results.


## **How can graph machine learning prevent fraud in addition to detection?**

Graph machine learning can be used to prevent fraud by identifying at-risk individuals or transactions and taking proactive measures to prevent such activities. Graph machine learning can also be used to monitor known fraudulent actors to prevent further fraudulent activities.

# How NebulaGraph Works
1. [NebulaGraph Architecture — A Bird’s Eye View](https://www.nebula-graph.io/posts/nebula-graph-architecture-overview)
2. [Benchmark: NebulaGraph vs Dgraph vs JanusGraph](https://www.nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph)
3. [Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph)
4. [Deploy the Graph Database on Kubernetes](https://www.nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes)


Graph machine learning is used for fraud detection by analyzing the connections and relationships between entities in a network. It can be applied to a wide range of datasets, such as financial transactions, communication logs, and social media interactions, to uncover patterns and anomalies that indicate fraudulent activity.

Fraud Detection with Graph Machine Learning

Have you ever pondered why cyber attackers always appear to be one step ahead of security teams?

It’s because their work is much less extensive. 

While those who are tasked with defending networks and systems need to identify and block every potential vulnerability, attackers just have to search for a single flaw - no matter how small it is. From this single entry point, they can cause massive damage.

This discrepancy in focus gives attackers greater agility and malleability, allowing them to swiftly take advantage of newly found vulnerabilities.

On the other hand, security teams often find themselves spread thin as they struggle to keep up with an array of constantly evolving cyber threats. They have to make sure they can handle every attack or threat, continually searching all possible sources - which is an overwhelming and extremely time consuming undertaking.

But with the entry of the [graph database](https://nebula-graph.io/) era, we can change this and put the attackers behind. 

Let’s find out. 


# **Graphs for cybersecurity: what does it mean?**

It means that for the first time, security teams can get ahead of the cyber attackers.

Many of us might actually be surprised to learn that cyber attackers often view an organization's network from the perspective of a graph. So they mostly concentrate on getting access to just one entry point (node). Using this one entry point, they can construct an attack path to easily penetrate the entire infrastructure. 

What about security teams? They have historically relied on lists of alerts from their security systems. Since the lists are simply linear, they don’t offer a complete overview.

On the other hand if you were to use the graph approach, it means you'll essentially have a holistic view of the entire infrastructure and you can constantly add to the graph as more items are added to the network. In other words while the attackers are busy looking for an entry point, you have the entire view to yourself. You are now one step ahead as opposed to the traditional list approach where you lag behind. 

The concept is really simple: Make use of graph data structures and visualization techniques to represent then analyze it using [graph analytics](https://www.nebula-graph.io/posts/graph-analytics-and-graph-databases) to unearth cybersecurity issues. This includes data such as network traffic, log files, and malware samples. By representing this data as nodes and edges in a graph, analysts can quickly identify patterns and relationships that may not be immediately obvious in other forms of representation.

The nodes in the graph represent the data points, such as IP addresses, domains, files, or processes. The edges represent the relationships between the nodes, such as a connection between an IP address and a domain. 


# **How to use graphs for cybersecurity ( how it works)**

The data from various sources such as network traffic, log files, and malware samples is transformed into a format that can be used to create the nodes and edges of the graph. Once the data is in this format, it can be loaded into a **graph database** as **NebulaGraph**.

Once the data is in the graph, it can be queried, analyzed and visualized. For example, an attack analyst can use graph algorithms to identify patterns and relationships, such as the shortest path between two nodes, which can help them trace the origin of an attack or track the spread of malware. You can also use [graph visualization](https://www.nebula-graph.io/posts/visualization-in-nebula-graph) tools to create interactive diagrams that allow you to explore the data and identify anomalies or potential threats.

Additionally, graph databases can be integrated with other cybersecurity tools such as [intrusion detection](https://www.nebula-graph.io/posts/intrusion-detection-with-graph-technology) systems, network monitoring tools, and threat intelligence platforms. This allows analysts to correlate data from multiple sources, providing them with a more complete picture.

Very briefly, these are the general steps you will undertake when using graphs for cybersecurity: 



1. **Data collection:** As mentioned earlier, you’ll basically collect data from all the critical points in the infrastructure such as network logs, system events, and security devices. Network logs provide information about the traffic flowing through the network, including IP addresses, ports, and protocols. System events provide information about the activity on a particular system, including user logins, file access, and system updates. Security devices, such as firewalls and intrusion detection systems, provide information about threats and attacks that have been detected on the network.
2. **Data preprocessing:** The goal of data preprocessing is to prepare the collected data for further analysis. This typically involves a series of tasks such as data cleaning, data integration, data transformation, data reduction, and data discretization which is the conversion of continuous data into categorical data.
3. **Graph construction:** The preprocessed data is used to build a graph that can be used to analyze the data and identify patterns that could point to security issues. The process of graph construction typically involves several sub-steps, such as node creation, edge creation, attribute assignment, clustering, and visualization.
4. **Graph analysis:** Involves using graph-based algorithms to analyze the graph built in the previous step and identify patterns, anomalies, and potential threats. There are different types of graph-based algorithms that can be used for graph analysis, such as centrality measures,clustering algorithms,link analysis algorithms, and anomaly detection algorithms.
5. **Threat hunting:** Actively searching for potential threats in the graph, rather than passively waiting for alerts or alarms to be triggered. The process of threat hunting typically involves approaches including hypothesis generation, data exploration, verification,and mitigation: 
6. **Remediation:** Can involve anything from implementing a patch or software update to reconfiguring an entire system. 
7. **Continuous monitoring and improving:** This could involve adding or removing nodes, changing access control policies, implementing new algorithms, and more.


# **Why should your organizations consider switching to graphs for cybersecurity?**

Graphs offer an all-encompassing, comprehensive look at the network. This perspective has been absent, and as we've observed, this is precisely why security teams are constantly fighting fires.

It's telling that large companies like [Cisco have already started incorporating graphs in their security products](https://blogs.cisco.com/security/big-data-in-security-part-iii-graph-analytics) - clearly, there's something special about graphs that the major players have uncovered

Fortunately it’s not a secret. It’s all about these strengths: 


<table>
 <tr>
 <td>Holistic view
 </td>
 <td>The ability to see the interconnectedness of different elements within a network or system. This includes understanding how different nodes and edges relate to one another, and how they may impact the overall security. 
 </td>
 </tr>
 <tr>
 <td>Visualization
 </td>
 <td>The ability to represent data in a variety of ways, such as node-link diagrams, heat maps, and matrix plots. These different visualizations can be used to represent different aspects of the data, such as network topology, user activity, and threat detection.
 </td>
 </tr>
 <tr>
 <td>Real-time threat intelligence 
 </td>
 <td>Graphs allow for real-time analysis and correlation of data from multiple sources, such as network logs, system events, and threat intelligence feeds. With this, security teams can detect and respond to threats more rapidly.
 </td>
 </tr>
 <tr>
 <td>Prediction
 </td>
 <td>This can involve using machine learning algorithms.
 </td>
 </tr>
 <tr>
 <td>Investigation
 </td>
 <td>While it's not possible to see everything especially when you are dealing with huge data sets, the ability to narrow down to small points within the data and investigate, is a powerful element of a security solution that is powered by graphs. It means security professionals can pick a few nodes and engage them at a higher level.
 </td>
 </tr>
 <tr>
 <td>Compliance
 </td>
 <td>Graph-based security solutions can help organizations comply with industry and government regulations, such as HIPAA, PCI-DSS, and NIST. For example, you’ll be able to easily segment and access controls that are required by compliance bodies. 
 </td>
 </tr>
 <tr>
 <td>Concentration
 </td>
 <td>This appealing nature of graphs can help analysts stay focused and motivated as they work through large amounts of data. It can also make it easier to communicate their findings to other stakeholders.
 </td>
 </tr>
</table>


All these elements contribute to a cost-effective cybersecurity solution, meaning you can detect and block more threats or attacks at a much more affordable rate than the traditional method of solely relying on lists.


# **Conclusion**

Our brains mostly think best in terms of images, patterns, and relationships. This makes visual representation of data, such as graphs, more intuitive, easier to understand and analyze. Graphs in particular are an effective way to represent complex data and relationships, making it easier to identify patterns, anomalies, and potential threats. 

This is why using graphs for cybersecurity is proving disruptive. It’s all your security team needs to get ahead of the ever elusive cyber criminals. 

# How NebulaGraph Works
1. [NebulaGraph Architecture — A Bird’s Eye View](https://www.nebula-graph.io/posts/nebula-graph-architecture-overview)
2. [Benchmark: NebulaGraph vs Dgraph vs JanusGraph](https://www.nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph)
3. [Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph)
4. [Deploy the Graph Database on Kubernetes](https://www.nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes)

Make use of graph data structures and visualization techniques to represent then analyze it using graph analytics to unearth cybersecurity issues. This includes data such as network traffic, log files, and malware samples. By representing this data as nodes and edges in a graph, analysts can quickly identify patterns.

How to Use Graph Database for Cybersecurity


# What is a distributed system?

Generally, a distributed system is a set of computer programs that work together across multiple independent servers to achieve a common goal. Those servers refer to those commodity servers instead of mainframes. The hardware for cross-server collaboration here is mostly based on Ethernet devices or higher-end RMDA devices. 

# Why do we need a distributed system?

The main reason to build a distributed system is to replace the cost of expensive hardware devices with software technology and inexpensive hardware devices. Especially in most private server rooms, not public cloud or supercomputing conditions, procurement costs are an important basis for business decisions. 

In addition to reducing costs, another benefit of distributed technology is its scalability. By adding several servers to the original number of servers, and then combining the scheduling and distribution ability of the distributed system, the new servers can be used to provide additional services.  

Compared with purchasing more servers in equal numbers or purchasing higher configuration servers, the distributed technology allows you to purchase servers on demand, which reduces the risk of over-provisioning and improves the utilization rate of the hardware resources.

# Basic problems with distributed systems

In distributed technologies, since data storage and computation need to be implemented across multiple independent servers, a series of underlying technologies must be involved. In this article, we discuss only two problems: **One is the data copy or replica problem, and another is how to distribute the storage and computation of large data to independent servers.**

## The problem of data replicas 

The hardware reliability and maintenance of commodity servers are much lower than that of mainframes. Because loose network cables, damaged hard drives, and power failures occur almost every hour in large machine rooms. It is a basic problem for a distributed software system to solve or avoid these hardware problems. One common solution is to replicate data on multiple servers. Once some data replicas are lost, the system can still provide services by using the remaining data replicas. What's more, when the access load of the system is too large, the system can also provide more services by adding more replicas. In addition, some technologies are needed to ensure that the data replicas are consistent with each other, that is, the data of each replica on different servers is the same. For graph databases, the data replica problem also exists. The way to solve this problem is similar to the way to solve the data replica problem in relational databases or big data systems.


## The problem of data partitioning

The hardware, memory, and CPU of a single server are limited. If the data is too large, it is impossible to store all the data on a single server. **Therefore, the TB-level or even PB-level data must be distributed to multiple servers, and we call this process data partitioning**. When a request is to access multiple data partitions, the distributed system needs to distribute the request to each correct data partition and then combine the results.

# The problem of data partitioning in graph databases - graph partitioning 

In graph databases, the distribution process is imaginatively called graph partitioning. **A big graph is partitioned into multiple small graphs, and the storage and computation of each small graph are stored on different servers**. 
Compared with the partitioning problem in relational databases and big data systems, the graph partitioning problem deserves more special attention.

Let's take a look at a static graph structure, such as the CiteSeer dataset, which is a citation network of scientific papers, consisting of 3312 papers and their citations between them. It is a small-scale dataset, so it can be stored on a single server.

The Twitter 2010 dataset, is a social network of Twitter users, consisting of 12.71 million vertices and 0.23 billion edges. It is relatively easy to store this dataset on a single mainstream server produced in 2022. However, it may be necessary to purchase very expensive high-end servers produced 10 years ago to do so.

However, the WDC (Web Data Commons) dataset, consists of 1.7 billion vertices and 64 billion edges. It is difficult or impossible to store such a large-scale dataset on a current mainstream server.

On the other hand, since the data of human beings is growing faster than Moore's Law and the number of connections or relationships between data is exponentially higher than the speed of data production, the data partitioning problem seems to be an inevitable problem for the graph database system. But it sounds no different from the way data is partitioned or hashed in the mainstream distributed technologies. After all, data is partitioned into multiple big data systems.

Wait, is it that easy to partition a graph?

No, it is not. In the field of graph databases, the graph partitioning problem is a trade-off among technologies, products, and engineering.  

## Three problems faced by graph partitioning

**The first problem** - what should be partitioned? In big data or relational database systems, **row-based partitioning or column-based partitioning** is performed based on records or fields, or partitioning is performed based on data IDs, and these are intuitive in terms of semantics and technology. However, the strong connectivity of the graph data structure makes it difficult to partition the graph data. One vertex may be connected to many other vertices through multiple edges, and the other vertices may also be connected to many other vertices through their neighboring edges. It is just like the web pages that are almost linked to each other. So for a graph database, what should be partitioned that can make the semantics intuitive and natural? (In RDBMS, this is equivalent to how to partition the data when there are a large number of foreign keys in the table.) Of course, there also exist some natural semantic partitioning methods. For example, under the COVID-19 epidemic, the transmission chain of various strains in China and other countries are two different network structures.  

Then, a second problem is introduced.

**The second problem** - how to ensure that the data of each partition is roughly balanced after the data is partitioned. The naturally formed graphs conform to the power low that a minority of 20% vertices are connected to other 80% vertices, and these minority vertices are called super nodes or dense nodes. This means that the minority vertices are associated with most of the other vertices. Therefore it can be expected that the load and hotspots of the partition containing the super nodes are much higher than that of the other partitions containing the other vertices.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/2.png)

The above image shows the visual effect of the association network formed by hyperlinks to websites on the Internet, where the super websites (nodes) are visible.

**The third problem** - how to evaluate and perform repartitioning when the original partitioning methods are gradually outdated as the graph network grows and the graph distribution and connection patterns change? The following image shows the visual effect of the connections among 86 billion neurons in the human brain. With learning, exercise, sleep, and aging, neuronal connections are constantly changing at the weekly level. The original partitioning method may not be able to keep up with the changes at all.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/3.png)

Of course, many other details need to be considered. In this article, we try to avoid using too many technical terms. 

Unfortunately, there is no silver bullet for the graph partitioning problems from a technical point of view and each product has to make its trade-offs. 

Here are some examples of the trade-offs made by different products.

# Partitioning methods in different graph database products

## 1. Distributed but unpartitioned

Neo4j 3.5 adopts the unpartitioned distributed architecture.

**The reason to use a distributed system is to ensure the consistency and the read availability of the written data in multiple replicas.**

**That means the full amount of graph data is stored on each server, and the size of data cannot be larger than the capacity of a single server's memory and hard disk. We can ensure the failure of a single server in the process of data writing by adding multiple write replicas. We can improve the read performance by adding multiple read replicas (The write performance is not improved).**

This solution can avoid the above-mentioned three problems of graph data partitioning and in theory, there is nothing wrong with calling such a solution a distributed graph database.

To add to that, since the data is complete on each server, ACID transactions are relatively easy to implement.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/4.png)

For details about the Neo4j 3.5 causal cluster architecture, see https://medium.com/neo4j/querying-neo4j-clusters-7d6fde75b5b4.


## 2. Distributed and partitioned by users

The distributed and partitioned by users architecture is typically represented by Neo4j 4.x Fabric. Depending on the user's business case, users can specify that subgraphs can be placed on a (group of) server(s). For example, in one cluster, the subgraph of the product E is placed on the server E and the subgraph of the product N is placed on server N. (Of course, for the availability of the service itself, these servers can also be placed in the Causal Cluster mentioned in the above-mentioned image.) In this process, for both the write and read operations, the user needs to specify a server or a group of servers to operate.

This solution leaves the above-mentioned three problems to the user for decision at the product level. Such a solution is also called a distributed graph database.

What's more, this solution can guarantee the ACID transactions in server E. However, there are a certain amount of edges that connect the vertices in server E and the vertices in other servers, so the ACID transactions of these edges cannot be guaranteed technically.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/5.png)

For details about the Neo4j 4.x Fabric architecture, see https://neo4j.com/developer/neo4j-fabric-sharding/.


## 3. Non-equivalent distributed, partitioned, and coarse-grained replicated

This solution allows multiple replicas and graph data partitioning, and these two processes require a small number of users to be involved.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/6.png)

For details about TigerGraph's partitioning solution, see https://www.youtube.com/watch?v=pxtVJSpERgk.

In TigerGraph's solution, vertices and edges are scattered in multiple partitions after being encoded.

The first two problems among the above-mentioned problems can be partially solved by encoding vertices and edges. Users can make decisions on whether to read or compute the data in a distributed system or a single server. 

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/7.png)
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/8.png)

The above image shows TigerGraph's single read mode and parallel computing mode.

However, such a set of partitions must be replicated in full and identical copies (so the granularity of the scale-out is the entire graph instead of a partition), which requires a larger amount of storage space.


## 4. Full-equivalent distributed, partitioned, and fine-grained replicated

There are also solutions whose architectural design purpose ranks graph scalability or resiliency relatively at the highest priority of the overall system design. The assumption is that data is being generated faster than Moore's Law, and the interactions and relationships between data are exponentially higher than the rate of data generation. Therefore it is necessary to be able to handle the such explosive growth of data and provide services quickly.

In this solution, the obvious characteristic is the separation design of the storage layer and the computing layer, each having the ability for fine-grained scalability.

Data is partitioned in the storage layer with a hash or consistent hash solution. Hashing is performed based on the IDs of vertices or primary keys. This solution just solves the first problem.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/9.png)

For details about the architecture of [NebulaGraph](https://nebula-graph.io), see the paper *Wu, Min, et al. "Nebula Graph: An open-source distributed graph database." arXiv preprint arXiv:2206.07278 (2022)*. 

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/10.png)

For details about the architecture of ByteGraph, see Li C, Chen H, Zhang S, et al. ByteGraph: A High-Performance Distributed Graph Database in ByteDance[J]. 

To deal with the super node and load balancing problem (the second problem), another layer of the B-tree data structure is introduced. It splits super nodes into multiple processing units, balances data among threads, and scales out the computing layer.

For the third problem, the solution is to use a fine-grained partitioning method so that the scale-out of some partitions can be performed.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/distribution-and-partitioning-in-graph-database/11.png)

For sure, this solution is also called [a distributed graph database](https://nebula-graph.io).

The above-mentioned four solutions make different trade-offs at the product and technical levels, with a focus on suitable business scenarios. These solutions can all be called distributed graph databases.

# Extended reading


- The problem of graph partitioning

  There has been a lot of research on how to partition graphs on a single machine. Here is the recommended tool METIS http://glaros.dtc.umn.edu/gkhome/metis/metis/overview.

  Graph partitioning is a very difficult problem in a distributed system. The following articles are recommended for further reading.

  - https://dgraph.io/blog/post/db-sharding/ 

  - https://hal.inria.fr/hal-01401338/document 

  - https://docs.janusgraph.org/advanced-topics/partitioning/ 


- About Pseudo-Distributed Mode

  Generally, this mode uses only a single node and simulates the distributed mode. It is recommended to use this mode for studying and debugging.

- An overview of the graph databases

    - Sahu, S., Mhedhbi, A., Salihoglu, S., Lin, J., ¨Ozsu, M.T.: The ubiquity of large graphs and surprising challenges of graph processing: an extended survey. VLDB J. 29(2-3), 595–618 (2020) 

    - Sakr, S., Bonifati, A., Voigt, H., Iosup, A., Ammar, K., Angles, R., Yoneki: The future is big graphs: a community view on graph processing systems. Communications of the ACM 64(9), 62–71 (2021) 

# Disclaimer

1. The above-mentioned solutions are the author's understanding and should not be used as a reference for making investment decisions. If there are any errors, please feel free to point them out.

2. To reduce the difficulty of reading, the author has simplified the description of the solutions, which does not mean that the original solutions are so simple.


This article summarizes the designs of the distribution and partitioning for several different types of graph databases to help you understand the differences between them.

Difference Between Distribution and Partitioning in Graph Databases

We will now explore new techniques for pinpointing important content using common neighbors, pushing information flow based on friend relationships and geographic location, and using spatio-temporal relationship mapping to query the relationship between people. 

7 Fundamental Use Cases of Social Networks with NebulaGraph Database | EP 3

We have had a thorough introduction to graph data modeling and data import in the application of social network with graph technology. 

[**7 Fundamental Use Cases of Social Networks with NebulaGraph Database | Episode 1**](https://www.nebula-graph.io/posts/social-networks-with-graph-database-1)

Now, let me introduce more use cases in this episode.

# Identifying key people

Identifying influencers in social networks involves using a variety of metrics and methods to identify individuals who have a lot of influence in a given network. This is useful for many business scenarios, such as for marketing or researching the spread of information in a network.

There are many ways to identify them, and the specific methods and information, relationships, and perspectives considered also depend on the type of these key individuals, and the purpose of acquiring them.

Some common methods include looking at the number of followers a person has or the amount of content consumed, their reader engagement on their posts, videos, and the reach of their content (retweets, citations). These methods are also doable on the graph but are rather mundane, so I won't give examples. Here, we can try to derive these key people on the graph using graph algorithms that evaluate and calculate the importance of nodes.

## PageRank

PageRank is a very classic graph algorithm that iterates through the number of relationships between points on a graph to get a score (Rank) for each point. It was originally proposed by Google founders Larry Page and Sergey Brin and used in the early Google search engine to sort search results, where Page can be a pun on Larry Page's last name and Web Page.

PageRank has long been abandoned as too simple in modern, complex search engines, but it still shines in other graph-structured web scenarios, where we can roughly assume that all links are of similar importance and run the algorithm to find those key users in social networks.

In NebulaGraph, we can use NebulaGraph Algorithm, NebulaGraph Analytics to run PageRank on large full graphs, while in the daily analysis, validation, and design phases, we don't need to run results on full data, but on very small subgraphs (up to tens of thousands), we can easily run various graph algorithms in the browser to derive methods that can be used for production.

Today, we will use the built-in in-browser graph algorithm function of NebulaGraph Explorer to execute PageRank (the specific method is omitted here, you can refer to the [documentation](https://docs.nebula-graph.io/3.3.0/nebula-explorer/graph-explorer/graph-algorithm/ ), but it's really just a matter of mouse clicks).

![PageRank](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/PageRank.png)

We can see from the above that among all the green players (people) after PageRank calculation, "player.name: Tim Duncan" is the largest one, and the relationship associated with it does seem to be quite a lot, so we select him on the graph, then right-click to invert, select all the points except Tim Duncan, use the backspace key to delete all the other points, and then explore 1 to 5 steps in both directions with him as the starting point. In one step, we get Tim Duncan's subgraph.

![TimDuncan](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/TimDuncan.png)

As you can see from the subgraphs, Tim Duncan is associated with a very large number of other players, while some other very popular players have served with him in the very popular Spurs team, which confirms the way PageRank is evaluated.

Now let's see if the algorithm will come to the same conclusion for the other dimensions of determination.

## Betweenness Centrality

As you can see from the subgraphs, Tim Duncan is associated with a very large number of other players, while some other very popular players have served with him in the very popular Spurs team, which confirms the way PageRank is evaluated.

Now let's see if the algorithm will come to the same conclusion for the other dimensions of determination.

![Betweeness_centrality](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Betweeness_centrality.png)

From its five-hop subgraph, it can be seen that unlike the star-shape of the key figure Tim Duncan obtained from PageRank before, Dejounte Murray's subgraph shows clusters, where it is sensory, intuitive to imagine that Dejounte Murray is really on the necessary path of the minimal path between many nodes, while Tim Duncan seems to be associated with more important connecters.

![DejounteMurray](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/DejounteMurray.png)

In practical application scenarios, we usually have to understand the definitions in different ways, experiment with different execution results, and analyze to find the structural features that affect the key people we care about, and use them to choose different algorithms for different needs.

# Detect communities and clusters

Community detection in social networks is a technique to discover community structure by analyzing social relationships. A community structure is a set of nodes that are closely connected to each other in a social network, graph, and these nodes usually have similar characteristics or interests. For example, a community structure may manifest itself as a group of users who are clustered together based on common topics or interests.

The purpose of community detection is to identify the boundaries of different communities and determine the nodes in each community by analyzing the social network. This process can be done by using various algorithms such as label propagation algorithm, weakly connected component algorithm and Louvain algorithm. By discovering the community structure, we can better understand the structure and characteristics of social networks, and help social network service providers to better infer and predict behaviors in social networks, and help in good social network governance, advertisement placement, marketing, etc.

Since our dataset is fake-generated, the results I get under different algorithms do not show the real meaning, so this chapter just shows the results after community identification using several graph algorithms, in real-world cases, we should also use domain knowledge or other technical means on top of that to collaboratively give the portraits and labels of different groups and communities.

Effect of label propagation algorithm.

![LPA](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/LPA.png)



Louvain algorithm:

![Louvain](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Louvain.png)

WCC algorithm:

![WCC](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Louvain.png)

In later sections, we could in better chance verify these algorithms again on smaller and simpler subgraphs, with somewhat more interpretable results.

# Friend Closeness

With the community detection algorithm, it is actually possible to obtain friends with similar interests and close associations to some extent, in a global calculation. So how do we get the other close friends of a given user? We can get this information by counting the number of friends this user has in common with him in order to get this information!

Let's take "Tim Duncan" for example, we know that his two-degree friends (friends of friends: `(:player{name: "Tim Duncan"})-[:follow]-(f:player)-[:follow]-(fof:player)`) are also his friends: Mutual Friend, then it is reasonable to believe that those who have more friends in common with Tim Duncan may have a higher closeness to him.

```cypher
MATCH (start:`player`{name: "Tim Duncan"})-[:`follow`]-(f:`player`)-[:`follow`]-(fof:`player`),
(start:`player`)-[:`follow`]-(fof:`player`)
RETURN fof.`player`.name, count(DISTINCT f) AS NrOfMutualF ORDER BY NrOfMutualF DESC;
```

This query shows that "Tony Parker" and Tim have 5 friends in common and are the closest.

| fof.player.name   | NrOfMutualF |
| :---------------- | :---------- |
| Tony Parker       | 5           |
| Dejounte Murray   | 4           |
| Manu Ginobili     | 3           |
| Marco Belinelli   | 3           |
| Danny Green       | 2           |
| Boris Diaw        | 1           |
| LaMarcus Aldridge | 1           |
| Tiago Splitter    | 1           |

Here, let's verify this result through visualization!

First, let's see who the common friends (f:) are for each of the friends.

```cypher
MATCH (start:player{name: "Tim Duncan"})-[:`follow`]-(f:player)-[:`follow`]-(fof:player),
(start:player)-[:`follow`]-(fof:player)
RETURN fof.player.name, collect(DISTINCT f.player.name);
```

The result:

| fof.player.name   | collect(distinct f.player.name)                              |
| :---------------- | :----------------------------------------------------------- |
| Boris Diaw        | ["Tony Parker"]                                              |
| Manu Ginobili     | ["Dejounte Murray", "Tiago Splitter", "Tony Parker"]         |
| LaMarcus Aldridge | ["Tony Parker"]                                              |
| Tiago Splitter    | ["Manu Ginobili"]                                            |
| Tony Parker       | ["Dejounte Murray", "Boris Diaw", "Manu Ginobili", "Marco Belinelli", "LaMarcus Aldridge"] |
| Dejounte Murray   | ["Danny Green", "Tony Parker", "Manu Ginobili", "Marco Belinelli"] |
| Danny Green       | ["Dejounte Murray", "Marco Belinelli"]                       |
| Marco Belinelli   | ["Dejounte Murray", "Danny Green", "Tony Parker"]            |

 Then we visualize the result on Explorer.

- First, let's find out all of Tim's 2-degree friend paths

```cypher
MATCH p=(start:player{name: "Tim Duncan"})-[:`follow`]-(f:player)-[:follow]-(fof:player)
RETURN p
```

- Then we render the node size by degree in which we select Tim and Tony and find all paths between them for follow type edge, bidirectional, up to 2 hops:

We can see that they are the closest of friends to each other and that their mutual friends are also in the paths.

`["Dejounte Murray", "Boris Diaw", "Manu Ginobili", "Marco Belinelli", "LaMarcus Aldridge"]`

![closest_friend](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/closest_friend.png)

## Small groups in your friends

At this point, as mentioned earlier, the non-authenticity of this dataset itself makes the results of the community discovery algorithm unable to get the insightful connotation of it. Now we can follow this small subgraph to see how groups and communities can be distinguished among Tim's friends.

- Weakly connected components can split Tim's friends into two or three parts that are not connected to each other, which is very much in line with the intuitive understanding and definition of connected components.

![Tim_wcc](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_wcc.png)

- Label propagation, we can control the number of iterations on-demand to delineate different degrees of division by random propagation, which results in a certain degree of differentiation.

  20 iterations

  ![Tim_LPA](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_LPA_1000.png)

  1000 iterations

  ![Tim_LPA_1000](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_LPA_1000.png)

- Louvain, a more efficient and stable algorithm, basically under this subgraph we can get a very intuitive division with a very small number of iterations.

![Tim_Louvain](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_Louvain.png)

# New friend recommendation

Following the previous idea of 2-degree friends (friends of friends), we can easily add those 2-degree friends who are not yet friends as recommended friends, and the sorting rule is the number of friends they have in common with each other:

```cypher
MATCH (start:player{name: "Tim Duncan"})-[:`follow`]-(f:player)-[:`follow`]-(fof:player)
WHERE NOT (start:player)-[:`follow`]-(fof:player) AND fof != start
RETURN fof.player.name, count(DISTINCT f) AS NrOfMutualF ORDER BY NrOfMutualF DESC;
```

| fof.player.name   | NrOfMutualF |
| :---------------- | :---------- |
| LeBron James      | 2           |
| James Harden      | 1           |
| Chris Paul        | 1           |
| Yao Ming          | 1           |
| Damian Lillard    | 1           |
| JaVale McGee      | 1           |
| Kevin Durant      | 1           |
| Kyle Anderson     | 1           |
| Rudy Gay          | 1           |
| Russell Westbrook | 1           |

Obviously, LeBron is the most recommended! And look at who these mutual friends are.

| fof.player.name   | collect(distinct f.player.name)    |
| :---------------- | :--------------------------------- |
| James Harden      | ["Dejounte Murray"]                |
| LeBron James      | ["Danny Green", "Dejounte Murray"] |
| Chris Paul        | ["Dejounte Murray"]                |
| Yao Ming          | ["Shaquille O'Neal"]               |
| Damian Lillard    | ["LaMarcus Aldridge"]              |
| JaVale McGee      | ["Shaquille O'Neal"]               |
| Kevin Durant      | ["Dejounte Murray"]                |
| Kyle Anderson     | ["Dejounte Murray"]                |
| Rudy Gay          | ["LaMarcus Aldridge"]              |
| Russell Westbrook | ["Dejounte Murray"]                |


Again, let's look for LeBron James in the subgraph we just created! And find the two-step, two-way path between them, and sure enough, it only goes through `["Danny Green", "Dejounte Murray"]` and, without a direct connection.

![Tim_newFriend](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_newFriend.png)

Now, the system could send reminders to both sides: "HEY, maybe you two should make new friends!"

Identifying influencers in social networks involves using a variety of metrics and methods to identify individuals who have a lot of influence in a given network. This is useful for many business scenarios, such as for marketing or researching the spread of information in a network.

7 Fundamental Use Cases of Social Networks with NebulaGraph Database | EP 2

Two months ago, my colleague brought in a graph of Twitter interactions  and asked me if I could write a post on graph technology to explore social network analysis and see how those connections are recognized through graph technology.

![graph-database-on-social-network](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/graph-database-on-social-network-02.jpeg)

![graph-database-on-social-network-3](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/graph-database-on-social-network-03.jpeg)

So, how are typical social relationship graphs like the one above, or even more complex social network-based graphs, generated? In the following episodes, I will use the NebulaGraph database to conduct the social network analysis, and you can find step-by-step guides to try all the methods in Playground.

# Social Network Analysis on Graph Database

We can use a graph database in a social network system to represent users and their connections or relationships. Graph databases allow efficient querying of relationships between users, making various business implementations on social networks based on connection findings, statistics, and analysis feasible and efficient.

For example, graph databases can identify "influential users" in a network, recommend new connections (friendships, favorite content ) based on commonalities between users, or find different groups of people and communities to profile users. Graph databases are ideal for social networking systems where user relationships constantly change because they can support complex multi-hop queries and real-time writes and updates.

There have been quite a few examples of practical applications in social networks, including:

- Finding key people
- Identifying clusters of people and communities
- Determining the closeness between two users
- Recommending new friends
- Pinpointing important content using common neighbors
- Push information flow based on friend relationships and geographic location
- Use spatio-temporal relationship mapping to query the relationship between people, get the people who intersected in time and space, and the provinces visited

I will dive into each application with hands-on demonstrations to present graph databases' competence in a couple of episodes. Stay tuned!

# Graph Modeling

To showcase the SNS graph use cases, I'll build most of the examples on a typically small social network. I started by adding extra data on top of the NebulaGraph default dataset, basketballplayer:

Three new tags of vertices:

- address
- place
- post

Five new types of edges:

- created_post
- commented_at
- lived_in
- belong_to

It looks like this:

![schema_sketch](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/schema_sketch.png)

# Importing the data

## Load the default dataset

- In the Command Line Console, we could execute:play basketballplayer` to load the default dataset.

- Or, if we do so from NebulaGraph Studio/Explorer, click the Download from the Demos in the welcome page:

  ![baskertballplayer_studio_starter](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/baskertballplayer_studio_starter.png)

## Add the SNS Graph schema

First, the DDL for those new tags and edges:

```sql
CREATE TAG IF NOT EXISTS post(title string NOT NULL);
CREATE EDGE created_post(post_time timestamp);
CREATE EDGE commented_at(post_time timestamp);
CREATE TAG address(address string NOT NULL, `geo_point` geography(point));
CREATE TAG place(name string NOT NULL, `geo_point` geography(point));
CREATE EDGE belong_to();
CREATE EDGE lived_in();
```

## Load the data

Then we load the DML part to insert vertices and edges:

```sql
INSERT VERTEX post(title) values \
    "post1":("a beautify flower"), "post2":("my first bike"), "post3":("I can swim"), \
    "post4":("I love you, Dad"), "post5":("I hate coriander"), "post6":("my best friend, tom"), \
    "post7":("my best friend, jerry"), "post8":("Frank, the cat"), "post9":("sushi rocks"), \
    "post10":("I love you, Mom"), "post11":("Let's have a party!");

INSERT EDGE created_post(post_time) values \
    "player100"->"post1":(timestamp("2019-01-01 00:30:06")), \
    "player111"->"post2":(timestamp("2016-11-23 10:04:50")), \
    "player101"->"post3":(timestamp("2019-11-11 10:44:06")), \
    "player103"->"post4":(timestamp("2014-12-01 20:45:11")), \
    "player102"->"post5":(timestamp("2015-03-01 00:30:06")), \
    "player104"->"post6":(timestamp("2017-09-21 23:30:06")), \
    "player125"->"post7":(timestamp("2018-01-01 00:44:23")), \
    "player106"->"post8":(timestamp("2019-01-01 00:30:06")), \
    "player117"->"post9":(timestamp("2022-01-01 22:23:30")), \
    "player108"->"post10":(timestamp("2011-01-01 10:00:30")), \
    "player100"->"post11":(timestamp("2021-11-01 11:10:30"));

INSERT EDGE commented_at(post_time) values \
    "player105"->"post1":(timestamp("2019-01-02 00:30:06")), \
    "player109"->"post1":(timestamp("2016-11-24 10:04:50")), \
    "player113"->"post3":(timestamp("2019-11-13 10:44:06")), \
    "player101"->"post4":(timestamp("2014-12-04 20:45:11")), \
    "player102"->"post1":(timestamp("2015-03-03 00:30:06")), \
    "player103"->"post1":(timestamp("2017-09-23 23:30:06")), \
    "player102"->"post7":(timestamp("2018-01-04 00:44:23")), \
    "player101"->"post8":(timestamp("2019-01-04 00:30:06")), \
    "player106"->"post9":(timestamp("2022-01-02 22:23:30")), \
    "player105"->"post10":(timestamp("2011-01-11 10:00:30")), \
    "player130"->"post1":(timestamp("2019-01-02 00:30:06")), \
    "player131"->"post2":(timestamp("2016-11-24 10:04:50")), \
    "player131"->"post3":(timestamp("2019-11-13 10:44:06")), \
    "player133"->"post4":(timestamp("2014-12-04 20:45:11")), \
    "player132"->"post5":(timestamp("2015-03-03 00:30:06")), \
    "player134"->"post6":(timestamp("2017-09-23 23:30:06")), \
    "player135"->"post7":(timestamp("2018-01-04 00:44:23")), \
    "player136"->"post8":(timestamp("2019-01-04 00:30:06")), \
    "player137"->"post9":(timestamp("2022-01-02 22:23:30")), \
    "player138"->"post10":(timestamp("2011-01-11 10:00:30")), \
    "player141"->"post1":(timestamp("2019-01-03 00:30:06")), \
    "player142"->"post2":(timestamp("2016-11-25 10:04:50")), \
    "player143"->"post3":(timestamp("2019-11-14 10:44:06")), \
    "player144"->"post4":(timestamp("2014-12-05 20:45:11")), \
    "player145"->"post5":(timestamp("2015-03-04 00:30:06")), \
    "player146"->"post6":(timestamp("2017-09-24 23:30:06")), \
    "player147"->"post7":(timestamp("2018-01-05 00:44:23")), \
    "player148"->"post8":(timestamp("2019-01-05 00:30:06")), \
    "player139"->"post9":(timestamp("2022-01-03 22:23:30")), \
    "player140"->"post10":(timestamp("2011-01-12 10:01:30")), \
    "player141"->"post1":(timestamp("2019-01-04 00:34:06")), \
    "player102"->"post2":(timestamp("2016-11-26 10:06:50")), \
    "player103"->"post3":(timestamp("2019-11-15 10:45:06")), \
    "player104"->"post4":(timestamp("2014-12-06 20:47:11")), \
    "player105"->"post5":(timestamp("2015-03-05 00:32:06")), \
    "player106"->"post6":(timestamp("2017-09-25 23:31:06")), \
    "player107"->"post7":(timestamp("2018-01-06 00:46:23")), \
    "player118"->"post8":(timestamp("2019-01-06 00:35:06")), \
    "player119"->"post9":(timestamp("2022-01-04 22:26:30")), \
    "player110"->"post10":(timestamp("2011-01-15 10:00:30")), \
    "player111"->"post1":(timestamp("2019-01-06 00:30:06")), \
    "player104"->"post11":(timestamp("2022-01-15 10:00:30")), \
    "player125"->"post11":(timestamp("2022-02-15 10:00:30")), \
    "player113"->"post11":(timestamp("2022-03-15 10:00:30")), \
    "player102"->"post11":(timestamp("2022-04-15 10:00:30")), \
    "player108"->"post11":(timestamp("2022-05-15 10:00:30"));

INSERT VERTEX `address` (`address`, `geo_point`) VALUES \
    "addr_0":("Brittany Forge Apt. 718 East Eric  WV 97881", ST_Point(1,2)),\
    "addr_1":("Richard Curve Kingstad  AZ 05660", ST_Point(3,4)),\
    "addr_2":("Schmidt Key Lake Charles  AL 36174", ST_Point(13.13,-87.65)),\
    "addr_3":("5 Joanna Key Suite 704 Frankshire  OK 03035", ST_Point(5,6)),\
    "addr_4":("1 Payne Circle Mitchellfort  LA 73053", ST_Point(7,8)),\
    "addr_5":("2 Klein Mission New Annetteton  HI 05775", ST_Point(9,10)),\
    "addr_6":("1 Vanessa Stravenue Suite 184 Baileyville  NY 46381", ST_Point(11,12)),\
    "addr_7":("John Garden Port John  LA 54602", ST_Point(13,14)),\
    "addr_8":("11 Webb Groves Tiffanyside  MN 14566", ST_Point(15,16)),\
    "addr_9":("70 Robinson Locks Suite 113 East Veronica  ND 87845", ST_Point(17,18)),\
    "addr_10":("24 Mcknight Port Apt. 028 Sarahborough  MD 38195", ST_Point(19,20)),\
    "addr_11":("0337 Mason Corner Apt. 900 Toddmouth  FL 61464", ST_Point(21,22)),\
    "addr_12":("7 Davis Station Apt. 691 Pittmanfort  HI 29746", ST_Point(23,24)),\
    "addr_13":("1 Southport Street Apt. 098 Westport  KY 85907", ST_Point(120.12,30.16)),\
    "addr_14":("Weber Unions Eddieland  MT 64619", ST_Point(25,26)),\
    "addr_15":("1 Amanda Freeway Lisaland  NJ 94933", ST_Point(27,28)),\
    "addr_16":("2 Klein HI 05775", ST_Point(9,10)),\
    "addr_17":("Schmidt Key Lake Charles AL 13617", ST_Point(13.12, -87.60)),\
    "addr_18":("Rodriguez Track East Connorfort  NC 63144", ST_Point(29,30));

INSERT VERTEX `place` (`name`, `geo_point`) VALUES \
    "WV":("West Virginia", ST_Point(1,2.5)),\
    "AZ":("Arizona", ST_Point(3,4.5)),\
    "AL":("Alabama", ST_Point(13.13,-87)),\
    "OK":("Oklahoma", ST_Point(5,6.1)),\
    "LA":("Louisiana", ST_Point(7,8.1)),\
    "HI":("Hawaii", ST_Point(9,10.1)),\
    "NY":("New York", ST_Point(11,12.1)),\
    "MN":("Minnesota", ST_Point(15,16.1)),\
    "ND":("North Dakota", ST_Point(17,18.1)),\
    "FL":("Florida", ST_Point(21,22.1)),\
    "KY":("Kentucky", ST_Point(120.12,30)),\
    "MT":("Montana", ST_Point(25,26.1)),\
    "NJ":("New Jersey", ST_Point(27,28.1)),\
    "NC":("North Carolina", ST_Point(29,30.1));

INSERT EDGE `belong_to`() VALUES \
    "addr_0"->"WV":(),\
    "addr_1"->"AZ":(),\
    "addr_2"->"AL":(),\
    "addr_3"->"OK":(),\
    "addr_4"->"LA":(),\
    "addr_5"->"HI":(),\
    "addr_6"->"NY":(),\
    "addr_7"->"LA":(),\
    "addr_8"->"MN":(),\
    "addr_9"->"ND":(),\
    "addr_10"->"MD":(),\
    "addr_11"->"FL":(),\
    "addr_12"->"HI":(),\
    "addr_13"->"KY":(),\
    "addr_14"->"MT":(),\
    "addr_15"->"NJ":(),\
    "addr_16"->"HI":(),\
    "addr_17"->"AL":(),\
    "addr_18"->"NC":();

INSERT EDGE `lived_in`() VALUES \
    "player100"->"addr_4":(),\
    "player101"->"addr_7":(),\
    "player102"->"addr_2":(),\
    "player103"->"addr_3":(),\
    "player104"->"addr_0":(),\
    "player105"->"addr_5":(),\
    "player106"->"addr_6":(),\
    "player107"->"addr_1":(),\
    "player108"->"addr_8":(),\
    "player109"->"addr_9":(),\
    "player110"->"addr_10":(),\
    "player111"->"addr_11":(),\
    "player112"->"addr_12":(),\
    "player113"->"addr_13":(),\
    "player114"->"addr_14":(),\
    "player115"->"addr_15":(),\
    "player116"->"addr_16":(),\
    "player117"->"addr_17":(),\
    "player118"->"addr_18":();
```

## First glance at the data

Let's start with the stats of the data.

```sql
[basketballplayer]> SUBMIT JOB STATS;

+------------+
| New Job Id |
+------------+
| 10         |
+------------+
[basketballplayer]> SHOW STATS;
+---------+----------------+-------+
| Type    | Name           | Count |
+---------+----------------+-------+
| "Tag"   | "address"      | 19    |
| "Tag"   | "place"        | 14    |
| "Tag"   | "player"       | 51    |
| "Tag"   | "post"         | 10    |
| "Tag"   | "team"         | 30    |
| "Edge"  | "belong_to"    | 19    |
| "Edge"  | "commented_at" | 40    |
| "Edge"  | "created_post" | 10    |
| "Edge"  | "follow"       | 81    |
| "Edge"  | "lived_in"     | 19    |
| "Edge"  | "serve"        | 152   |
| "Space" | "vertices"     | 124   |
| "Space" | "edges"        | 321   |
+---------+----------------+-------+
Got 13 rows (time spent 1038/51372 us)
```

We could get all of the data:

```sql
MATCH ()-[e]->() RETURN e LIMIT 10000
```

As the data volume is quite small, we could render them all in the canvas of NebulaGraph Explorer:

![match_all](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/match_all.png)

Above are the two fundamental steps to embark on the journey of social network exploration. I'll introduce other applications in the next episode:
- Finding Key People
- Identifying clusters of people and communities
- Determining the closeness between two users

We can use a graph database in a social network system to represent users and their connections or relationships. Graph databases allow efficient querying of relationships between users, making various business implementations on social networks based on connection findings, statistics, and analysis feasible and efficient.

7 Fundamental Use Cases of Social Networks with NebulaGraph Database | EP 1

Qingteng Cloud Security is a host security unicorn company, which focuses on the security of cloud native applications. At present, its main business is the host product named Wanxiang and the container product named Honeycomb. It leads the industry and supports 8 million agents in total. The company's next-generation real-time intrusion detection system based on NebulaGraph and combined with graph technology has been initially put into the market and participated in the 2022 Network Protection, which has achieved good response.

In this topic, we will briefly introduce the intrusion detection system based on graph, hoping to have more people to join in the mining of graph and security application combination.

# The status and challenge of intrusion detection

## Mainstream intrusion detection system

Intrusion detection has always been a major direction of security research. The two sets of products of Qingteng, Wanxiang and Honeycomb, are host-based and container-based intrusion detection products, both of which have similar principles, as shown below.

![Intrusion detection system architecture diagram](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-1.png)

The agent is placed on the host and container side, receiving rules from the server side, combining the collected raw events (process/network connection/file read and write, etc.), and reporting them to the server side as alarms after hitting the rules written by security experts, such as the MD5 of the process file, execution command line, machine learning, and other features. The full amount of raw events are also reported to the server side log collection system and stored in the SIEM system.

Once the alarm is reported to the server, the security staff will take the information related to the alarm to SIEM system to check the events before and after the time of occurrence, log in to the relevant machine to check the relevant information if necessary, and make a comprehensive judgment on whether the current attack alarm is valid and do the relevant disposal.

This system is the current mainstream architecture of intrusion detection, but there are many problems:

1. For single-point detection based on single events and rules, it may result in false positives due to too loose rules or missed positives due to too strict rules.
2. Too many alarms may be triggered by the same attack, and security staff have a heavy workload in tracing the source.
3. The original events related to the same alarm need to be manually analyzed by the SIEM system, which lacks visual means.

## The research direction of graph and security

These questions essentially look at each alarm and event independently, when in fact, the alarms and events associated with an attack should be related to each other. We can use graphs to correlate these original events and alarms for overall analysis. In fact, this is also a popular direction of current security research -- traceability graph, with the help of traceability graph we do the following security analysis and detection.

### 1.Graph detection

Traditional security solutions solve similar IOC detection problems, that is, the judgment of single-point, for the process, network connections, files and other entities is not safe. The actual problem faced by the attack, may not be detected at all points, but the actual behavior is dangerous. A single rule is not fully written, even if the rule can be written, such as hitting a command line, but it will trigger many alarms, and eventually the rule cannot be used. We need to comprehensively judge various combination relationships, which is what graph is good at, and also the problem of graph detection in security. Put all related events into one graph to comprehensively judge whether they are valid attacks.

**The research of graph detection started early, but faced with the dual challenges of computational volume and algorithm, the current industry applications are few, and the combined judgments are all in the way of sequence detection**. But the sequence rules can not write too many rules in the rule list, otherwise it will lead to the inability to apply the rules. In addition, if the rules are too detailed, and can not be well fuzzy matching rules, it is easy to be bypassed. Hackers can easily bypass these rules with the help of automation tools.

### 2.Graph association and tracing

Just mentioned alarms, which is another problem all the security products facing today, either the rules do not hit, or too many rule hits, we call this situation **alarm flooding**. Security staff processing capacity is limited, maybe 100 alarms a day is fine, if a day or even an hour 10000 alarms can not be handled, it is no different from no alarms.

This is really a problem of correlation classification and traceability in security:

1. An attack will generate a lot of alarms, such as brute force login, used malicious files, executed malicious commands, etc.. The correlation classification to correlate the related alarms in an attack together, which is something the graph is good at. These alarms also allow a comprehensive assessment of whether the current attack is valid or not.
2. Each alarm will only tell you is a process, file, or network problems, but how this problem occurred, how the hacker came in, where the file was downloaded from, what was done first and then what was done. Security products need to help customers complete this analysis process, the industry is currently with the help of SIEM, THP, SOC and other security products, all the original events are uploaded and saved, and then find security experts, starting from the alarm to check the original event logs, to see what happened before the alarm, which has a relationship may lead to an attack, the process is as short as a few minutes, long as several hours. **Security is a confrontation process, early is better**, the sooner it is found the sooner it is confronted, blocked or isolated, otherwise it is too late even if it is found. The relevant original event entities (process/network, etc.) can be imported into the graph, with the help of the graph to visually explore and trace the entire attack process, which is the process of graph traceability, academically called causal graph, traceability graph.

### 3.Graph knowledge mapping and prediction

We know that **current security is fundamentally based on rules or a priori knowledge**, each vulnerability, Trojan horse, attack tools, attack process, attack organizations have its characteristics, the first few rules are still relatively good description, the attack process and attack organizations are difficult to complete the description.

The current mainstream approach is to build a knowledge base based on security frameworks, the current mainstream has Kill Chain, ATTCK and other frameworks, which is the U.S. Department of Defense-led cyber attack war related to the two companies proposed a security analysis framework, equivalent to the mapping of the tactics and specific attack techniques that delineate the attack. Security academia, such as the University of Illinois and Purdue University have been working on similar issues in the last two years, which is the construction of a security knowledge base (security knowledge mapping). With a complete knowledge base, the ultimate vision of security can be accomplished, for example, if I know your attack process and attack organization, is it possible to stop an attack before it actually starts.

## Domestic and international status

At present, the intrusion detection system based on graph that is really taking the lead in the real world is Crowd Strike, a star security company in the U.S. It builds its security system entirely on graphs, which now amounts to doing both graph detection and graph correlation traceability, and is currently valued at 67 billion. Cloud computing giants AWS and Azure are following its approach. Domestic intrusion detection system based on graph, currently there are public information is ThreatBook and Sangfor, they are equivalent to do part of the work of graph correlation and traceability.

Qingteng Cloud Security's Wanxiang and Honeycomb have been working on intrusion detection for many years and have achieved industry-recognized security detection capabilities, so we chose to start with graph association and traceability, and develop a next-generation real-time intrusion detection system based on NebulaGraph combined with graph technology, focusing first on solving the pain point problems of alarm flooding and association traceability.

# Qingteng cloud security next-generation intrusion detection system

## Detection principle architecture diagram

As shown below, the focus is the reported attack alerts and some of the original events are unified and correlated in real time in the graph.

![Detection principle architecture diagram](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-2.png)

## Product effect

The output after correlation processing by the correlation engine is the attack event, an attack event may be associated with multiple attack alerts and visualized, the current product effect is as follows.

- The same attack using malicious files and trojans

  ![The same attack using malicious files and trojans](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-architecture-03.png)

- Mounting of sensitive containers

  ![Mounting of sensitive containers](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-architecture-04.png)

- Custom script detection

  ![Custom script detection](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-5.png)

## Advantages of NebulaGraph

NebulaGraph was chosen based on the following considerations.

1. The immediate need for graph queries, especially for multi-level process relationships during association and traceability. It is much easier to query a graph database with cypher than to query multi-level relationships in a relational database with SQL.
2. Large-scale storage, involving the storage of a large number of events and alarms.
3. High-performance query scenarios, associations need to ensure near real-time, the current relevant queries are in ms level.

Thanks to the good performance of NebulaGraph, the association engine enters the graph and calculates the association in a near real-time manner. The tool for importing graph data is the real-time graph entry component created by Qingteng based on Flink, which only needs to change the configuration file to complete the map of points and edges.

# Next research direction and plan

At present, the intrusion detection system of Qingteng cloud security mainly supports the association and traceability between single machine and some multi-machine scenarios. The next step is to support more scenarios of multi-machine association with the help of the graph, especially some typical security attack scenarios (rebound/horizontal movement), so as to further enable security with the graph and provide better service for customers.

There are many problems in applying NebulaGraph to graph security applications at present. Some suggestions are given for reference.

## 1.Push down optimization

In our process of use, the most important problem is the optimization of push down. Such as a diffuse process tree, while specifying the root node and leaf nodes, the order of statement writing may lead to query from the leaf node or the root node, query efficiency varies widely, and finally can only specify the leaf node conditions to force the query from the leaf node.

The performance of MATCH is also essentially a push-down optimization problem, this problem is also encountered a lot, our current approach is to maintain a cache pool for some of the statements have been queried, also note that the official enterprise version actually provides a partial cache function, this point is good if you can consider adding to the community version, is a practical application just need.

## 2.Some technical requirements for real-time scenarios

Since we are doing real-time association based on NebulaGraph, one of the major problems is how to achieve a balance between consistency and speed, because currently NebulaGraph cannot actually determine whether the graph entry is complete after writing to it. In practice, the association engine needs to poll downstream to determine whether the current vertices and edges is actually in the graph. This problem has some impact on the actual use and performance, and it is worth considering whether there is a better way.

We can see that some competing products, such as TigerGraph and TuGraph, provide the ability to customize the matching and traversal algorithms. In our multi-host association scenario, we have encountered such a need. At present, we can only achieve it by splitting multiple paths and splicing them together. Efficiency and speed are definitely compromised.

## 3.ToB deployment

Another pain point is the ToB deployment of NebulaGraph. In the security industry, there are still more on-site deployments at home. Especially, state-owned enterprises and government affairs are relatively sensitive to SaaS, so they will face the problem of efficiency and cost.

**The first is single-machine deployment**. The default production version is a distributed deployment with separate storage and computing designed for the cloud, which is still too cumbersome for ToB deployments and consumes relatively more resources than other databases.

**The second is HDD deployment**. Considering the cost, many companies, especially small and medium-sized companies, have limited security budgets, which makes it difficult to provide SSD deployment, not to mention the large capacity and high configuration of SSD machines required by the official website. Therefore, more optimization considerations should be made for HDD and low configuration machines.

Above, for Qingteng cloud security team engineer Zhou Wen to share.

**Thank you for reading this article.**

Want to experience the graph database up close? Now you can use NebulaGraph Cloud on AWS to build your own graph data system, come save a lot of deployment and installation time to get your business done [click the link](https://go.aws/3VZay2I) to use the graph database!

If you want to see the source code, you can go to GitHub to read, use, star it -> [GitHub](https://bit.ly/3f7XHex). Exchange graph database technology and application skills with other NebulaGraph users, and join ["Slack"](https://bit.ly/3SmvFu2) to play together!

Intrusion detection has always been a major direction of security research. In this topic, we will briefly introduce the intrusion detection system based on graph, hoping to have more people to join in the mining of graph and security application combination.

Next-generation Intrusion Detection System Based on Graph

Graph computing is precisely a technology that studies the relationship between data entities and portrays the data in the form of graphs.

Open Source Distributed Graph Computing Frameworks Comparison

# Dag Controller Introduction

Dag Controller is a component of NebulaGraph Explorer, which was released after several tests and tunings. It mainly solves the problem of integration of OLTP and OLAP, and the problem of graph algorithms in complex scenarios. To learn more, see [https://docs.nebula-graph.io/master/nebula-explorer/deploy-connect/ex-ug-deploy/#dag_controller](https://docs.nebula-graph.io/master/nebula-explorer/deploy-connect/ex-ug-deploy/#dag_controller).

Below is some information I would like to share about Dag Controller. Feel free to leave a comment and let's discuss it together.

I'm sure you're no stranger to OLTP and OLAP, but let me briefly introduce them again here: OLTP is a fast-response, real-time online data processing method. OLAP is an offline, data algorithm method for complex scenarios. For NebulaGraph, OLTP has a variety of query statements, such as GO, MATCH, etc. OLAP has a variety of graph algorithms, such as Pagerank, Louvain, WCC (Weakly Connected Components), K-Core, and Jaccard.

OLTP and OLAP do not exist independently. For example, we can use the subgraph returned by a MATCH statement as input to the PageRank algorithm, and the result of the PageRank can be written to NebulaGraph to continue executing a MATCH statement or other algorithms. Like building blocks, various [OLTP and OLAP](https://www.nebula-graph.io/posts/OLTP-vs-OLAP-what-are-the-differences) can be assembled to form a data processing method for complex scenarios.

The Dag Controller is a system that handles scenarios where OLTP and OLAP need to be linked and executed. 

![oltp-vs-olap](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/olap-and-oltp/1.png)

## Architecture

![dag-architecture](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/olap-and-oltp/2.png)

The responsibilities of Dag Controller:

1. It provides an HTTP interface for external communication and access.

It is used for job submission, stopping, deleting and other operations, as well as the configuration of the system environment.

2. Execution of DAG.

When Dag Controller executes DAG, the OLAP will call NebulaGraph Analytics, and the OLTP will call graphd to complete the execution of nGQL.

NebulaGraph Analytics is our graph computing system, supporting Pagerank, Louvain, WCC, Jaccard, and other graph algorithms, supporting HDFS and NebulaGraph data sources.

Graphd, metad, and storaged are components in NebulaGraph. Graphd is mainly responsible for nGQL parsing, storaged is responsible for data storage, and metad is responsible for metadata storage.

# Cases

## Case 1

![dag-use-case](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/olap-and-oltp/4.png)

The above figure shows a DAG model that runs the pagerank algorithm on a subgraph. First, we get a subgraph using an nGQL statement, and then we run the pagerank algorithm on this subgraph.

This method can be used when our graph size is particularly large and we only want to run the algorithm on part of the graph data.

## Case 2

![olap-and-oltp-use-case](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/olap-and-oltp/3.png)

The above figure shows a model to compute the shortest paths for two classes of vertices.

First, the vertex IDs of the two classes are obtained separately by using nGQL, and then the vertex IDs of the two classes are passed to the shortestpath algorithm, which computes the paths between the two classes in the full graph.

Each algorithm can be set up to run the algorithm based on the full graph or subgraph.

There are various DAG models, and different DAG models can be built according to different business scenarios.

# Technical implementation

## DAG model

The DAG refers to a directed graph without loops, and an instance of DAG is considered as a job, and a job has multiple task.

The task in Dag Controller can be an nGQL or a graph algorithm such as PageRank, Louvain, SSSP, etc.

When a job is executed, it needs to sort the task first, there is a lot of related code on the internet, so I won't repeat it here.

## Parallel execution

To ensure the execution efficiency of DAG, multiple DAGs need to be executed in parallel. At the same time, within a DAG, the task without upstream and downstream dependencies also needs to be executed in parallel.

How can multiple DAGs and tasks be executed in parallel? In simple terms, the DAGs and tasks are handled by two thread pools respectively.

The details are as follows:

1. When the system starts, allocate the job thread pool and task thread pool to handle the execution of job and task respectively.

2. Unexecuted jobs are periodically obtained from the database and transferred to the job thread pool.

3. When a job is executed, the task is sorted according to its upstream and downstream dependencies, and then it determines whether the execution of all upstreams of each task is completed, and then the task is handed over to the task thread pool for execution when the upstream execution is completed, or waits if the upstream execution is not completed.

4. If the job thread pool is full during job execution, you need to wait to obtain unexecuted jobs at regular intervals. The same is done when the task thread pool is full.

## Type check

Note that the data type verification problem exists between the data input and output of tasks. For example, task2 is the downstream of task1. The input of task2 must be of type int, and the output of task1 must be of type int.

## DAG stop

When you stop a job, you need to stop multiple task that are running in parallel. A task has a preparation phase, a running phase, and in the running phase there are cross-machine, multi-process situations. Stopping a job requires avoiding the orphan process problem.

## Custom algorithm support

We support treating a customer's algorithm as a task for DAG building. First, configure the algorithm-related parameter information in the system. When executing a job, the system is responsible for running the algorithm that corresponds to the task.

# How NebulaGraph Works
1. [NebulaGraph Architecture — A Bird’s Eye View](https://www.nebula-graph.io/posts/nebula-graph-architecture-overview)
2. [Benchmark: NebulaGraph vs Dgraph vs JanusGraph](https://www.nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph)
3. [Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph)
4. [Deploy the Graph Database on Kubernetes](https://www.nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes)


Dag controller mainly solves the problem of integration of OLTP and OLAP, and the problem of graph algorithms in complex scenarios. 

Integration  Practice of OLTP and OLAP

# Background

Traditional risk control mainly relies on manual review and empirical judgment. With the rise of digitalization in the past decade, leading companies began to introduce blacklisting systems to locate the risk points missed in the review stage through simple rules and post-facto audits, but this still cannot detect problems in advance. Since 2018, the anti-fraud system has gradually evolved to the stage of "intelligent decision engine" consisting of rule engines, real-time metric calculation, and machine learning technologies.

## Insurance Fraud Control Pain Points

The pain points of insurance fraud control are mainly reflected in the following aspects:

**Non-exchange information leads to poor risk control**: Data of insurance companies and the related industry is non-exchangeable. Even the data between various systems within the same insurance company is unshared. This can lead to asymmetry in business information and false claims, or the use of information asymmetry to increase claim amounts.

**Requirements of multi-dimensional computing**: Industry fraud diversity, specialization, and grouping will force the anti-fraud system to analyze data from multi-dimensional aspects, while the traditional analysis system based on relational models, dimensional models, or wide table models hardly analyzes data from dozens or even more dimensions.

**Accuracy requirements of algorithm recognition**: Business model tuning requires more feature data input during the training and serving stages. Generally, there are two methods to input data, one is writing more data, and the other is the supplementing of correlation data. However, if we go with the traditional method, it will be difficult to detect abnormal data with more than 4-hop queries, and the query latency is high.

## Underlying Storage Challenges


With the development of the anti-fraud system, the system faces challenges in the aspect of the underlying storage. 

The performance of traditional stand-alone databases has bottlenecks in capacity and query for linked data. In addition, business iterations and peaks and valleys have agility requirements for database scalability. Nowadays, most risk control systems use Hadoop ecosystem tools to co-accomplish system goals, and the corresponding development and operation and maintenance costs are high. In terms of real-time data analysis, it's difficult for traditional big data platforms to perform the analysis of multiple sources of data.


# Use Scenarios

NebulaGraph is used in many scenarios of insurance fraud detection. The following are 5 typical scenarios.


## Scenario 1: Relationship Detection


Customers may skip the family information in the application form. Such uncertain or missing family information are not well determined by the original method. While the graph data structure allows us to find relatives, which is not natively supported by other types of databases.

In addition, the regular form is filled with only immediate relatives, which does not reflect indirect relationships. In NebulaGraph's graph database, more hidden relationships can be discovered through the available data, including social and non-immediate family relationships. It allows for a comprehensive analysis of the family's insurance purchases.

## Scenario 2: Fraud Detection


Traditional fraud detection methods are mainly based on empirical rules and are unable to detect fraud in advance.

Taking car insurance as an example, there are relatively more entities involved in car insurance, including vehicles, owners, drivers, passengers, etc. It is very difficult to identify those entities among these data. However, in a graph database, it is possible to build a graph to see some outliers based on the number of claims or other business indicators, and it is also possible to classify groups with higher fraud rates through graph algorithms. In social relationship networks, the fraud possibility can be determined by the correlation with the blacklisted spots. In general, we can make full use of pattern matching and other graph database technologies to classify the major entities by using graph algorithms and then identify groups or individuals with high fraud rates based on business experience and existing fraud cases.


Fraud cases usually have certain characteristics. For example, a luxury used car with frequent accidents or multiple accident-related people driving a different vehicle frequently, or a high correlation between different vehicle groups and multiple accident-related owners. These are all suspicious characteristics that can be identified quickly and effectively by graph technology.

## Scenario 3: Precision Marketing 

In the scenarios of marketing, the data in a graph database can help identify target customers in depth. Based on the information left by customers, we can find social relationships, friends, and non-immediate family members other than immediate family members. Such relationship networks help us see a larger dimension of the customer's social circle. We can find more hidden relationships in the connections between customers and their friends. This can help us learn more about family insurance purchase behaviors.


In the private domain scenario, the marketing agents can recommend other insurance products based on the customer's purchase history. The groups with associated relationships may tend to prefer the same insurance products. From another perspective, such associated recommendations will be more convincing to customers. In the public domain scenario of getting target customers, graph databases can also help advertise to the target customers through the customer data system.

## Scenario 4: Insurance Underwriting and Claims Fraud Detection


At the underwriting and claim stages, we can find what insurance policies the insured has, the number of claims denied, and the association with known fraud cases by querying the in and out-degree of the insured. As for car insurance, we can detect fraud from "the case" to the "vehicle", to the "owner", to the "driver", and to the "vehicle", as shown in the above figure. In this way, we can find the fraud cases from the root cause. For example, in a DUI transfer case (the drunk driver A contacts his relative B to arrive at the scene and report to the insurance company in place of A to get insurance compensation after the DUI accident), we can analyze behaviors such as outgoing and incoming calls, and the location information of the involver. Through this information, multiple risk factors may be detected such as delayed reporting, non-frequently used cell phone numbers and distance from the scene.

## Scenario 5: Agents Management


A large number of domestic insurance agents and a high turnover rate may lead to difficulty in agent management. 

We can also build the agent's relationship network based on the graph model. The agent's professional habits and complaints received can be labeled in the graph to enhance the screening of agents from the source.

 For some underground insurance scenarios, such as forging the identity of a newcomer for commission. Fraud risks can be identified through cell phone numbers, electronic device numbers, IP associations, etc., all of which can effectively reduce customer complaints and insurance frauds caused by agents.



# Why NebulaGraph?

## Features of NebulaGraph

![NebulaGraph-Database](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-ecosystem.png)


To address the above anti-fraud pain points and the need for the system-level construct, NebulaGraph has the following features that are suitable for the assurance risk control needs:

1. **Multi-source data fusion**: The sources of insurance data are diverse, including bank data, social network data, government data, medical data, marketing data, and so on. The data fusion of these data sources is a key to the success of the fraud detection system that can establish a comprehensive analysis of customers to better help insurance companies achieve automatic and accurate underwriting.

2. **Support of graph data model**: The construct of an underlying data model is the key to the multi-dimensional analysis. NebulaGraph data models fit the storage and query presentation requirements of linked data and provide quicker analysis compared to the relational model while its schema is flexible.

3. **Cloud-native architecture**: A good storage system should enable upper-tier applications to compute and scale horizontally based on business needs. NebulaGraph is a distributed cloud native graph database. It can be easily scaled out and scaled in to meet the needs of business growth and shrinkage.

4. **Enterprise-level feature support**: NebulaGraph, as a basic software, provides the support of peripheral tools for a better exploration and visual operation and maintenance.

5. **Risk factor detection**: NebulaGraph's association analysis and graph computing engines can help better identify the risk factors with algorithms.

6. **Accuracy of auxiliary algorithms**: Machine learning relies on input data constructed by tuples, which tend to ignore the relationships between data. So the input of graph data can be used to enrich the contextual information and thus improve the accuracy of relevant models.

In a word, NebulaGraph connects dimensional data. It can be used in different scenarios to help insurance companies effectively identify and avoid hidden risks at different stages. For details about how to deploy and use NebulaGraph, please refer to [live demos](https://www.nebula-graph.io/demo/).  

NebulaGraph can be used in different scenarios to help insurance companies effectively identify and avoid hidden risks at different stages. 

Insurance Fraud Detection with Graph Technology | A Practice of NebulaGraph


![Overall Architecture - NebulaGraph](https://user-images.githubusercontent.com/57335825/87115631-a8852580-c228-11ea-877a-b48ba92fbb8b.png)

 NebulaGraph is an [open-source](https://en.wikipedia.org/wiki/Open-source_software) distributed graph database solution. As an open-source project, we would like those who are interested in graph databases to know as much as possible about it. This includes how NebulaGraph is designed and why it is a highly performant database.

As a result, we have developed a series of architecture articles. In them, we are going to cover three areas. First, we will provide a high-level overview of the architecture of NebulaGraph – the focus of this article. Next, we will cover how NebulaGraph’s storage engine works. And finally, we will illustrate how NebulaGraph’s query engine works, including how we have designed nGQL, our graph query language.

# NebulaGraph Architecture Diagram

![NebulaGraph Architecture Diagram](https://user-images.githubusercontent.com/38887077/76277232-1ba27d80-62c3-11ea-9389-cc1aca329c7b.png)

In the diagram above, there is a fully deployed NebulaGraph cluster that contains three services. They include a Meta Service, Query Service, and Storage Service. Each service has its own executable binary. These [binaries](https://github.com/vesoft-inc/nebula/releases) can be deployed either on the same set of hosts or on different hosts.

## Meta Service

In the below diagram, on the right side of the architecture diagram, it shows a cluster of Meta Service. The Meta Service follows a leader/follower architecture. The leader is elected by all Meta Service hosts in the cluster and serves all traffic. Followers stand by and take the updates replicated from the leader. Once the leader dies, one of the followers will be re-elected as the new leader.

![Meta Service](https://user-images.githubusercontent.com/38887077/76277238-1e9d6e00-62c3-11ea-9d10-8391c26b0b20.png)

The Meta Service does not only store and provide meta information about the graph, such as schema and graph partition information. It also takes the role of the coordinator to orchestrate the data movement and to force the leader shift.

## Separation of Query and Storage Layers

![Separation of Query and Storage Layers](https://user-images.githubusercontent.com/38887077/76277242-2230f500-62c3-11ea-8566-4b6be458adee.png)

In the above diagram, to the left of the Meta Service, is NebulaGraph’s main services. NebulaGraph divides the query execution and the graph storage into two separate services. Above the dotted line is the Query Service. Below the dotted line is the Storage Service.
This separation has several benefits. The most straightforward benefit is that the query layer and storage layer can be expanded or shrunk independently. The separation also makes the linear horizontal expansion possible. Last but not the least, the separation provides an opportunity to make it possible for the Storage Service to serve multiple computation layers.
The current Query Service can be considered a high-priority computation layer. Meanwhile an iterative computation framework could be another one.

## Stateless Query Layer

![Stateless Query Layer](https://user-images.githubusercontent.com/38887077/76277245-25c47c00-62c3-11ea-912b-5de3ce4e18fe.png)

As to the Query Service, each individual Query Service host runs a stateless query engine. The Query Service hosts will never communicate with each other. They only read the meta information from the Meta Service and interact with the Storage Service.

This design makes sure the Query Service cluster can be easily managed by Kubernetes (aka. k8s) or be deployed on the cloud. A cloud deployment is already available for NebulaGraph.
See [how to deploy NebulaGraph on Kubernetes](https://nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes/).

There are two ways to balance the Query Service load. The most common way is to place a [load balancer](https://nebula-graph.io/posts/nebula-graph-storage-banlancing-data-migration/) in front of the Query Service. The second way is to configure the client library with all Query Service hosts’ IP addresses. The client will randomly pick one to connect to.

Each query engine takes the request from the client, parses the statement, and generates an Abstract Syntax Tree (AST). Then the AST will be handed over to the execution planner and the optimizer. The final AST will be passed to the executors.

Below is a flow chart of how NebulaGraph’s query engine works. For a detailed explanation, please refer to [an introduction to the query engine](https://nebula-graph.io/posts/nebula-graph-query-engine-overview/).

![image](https://user-images.githubusercontent.com/38887077/78201029-28318480-74c3-11ea-90ed-03e9c2ca25d4.png)

## Shared-nothing Distributed Storage Layer

![Shared-nothing Distributed Storage Layer](https://user-images.githubusercontent.com/38887077/78201059-3a132780-74c3-11ea-95a0-61249c1566a4.png)

The Storage Service is designed to be a [shared-nothing distributed architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture). Each storage host has multiple local key/value stores as the physical data storage.

A quorum consensus protocol (we chose [RAFT](https://raft.github.io/) over paxos due to its simplicity) is built on top of the key/value stores. Together, they provide a distributed key/value store. On top of this distributed key/value store, a graph semantic layer is provided. It translates the graph operations into key/value operations. The graph data (vertices and edges) are hashed into multiple partitions by the rule (vertex_id % the_number_of_partitions).
A partition is a virtual set of data in NebulaGraph. These partitions are allocated to all storage hosts. The allocation information is stored in the Meta Service and can be accessed by storage hosts and query hosts.

# Conclusion

This bird’s eye view into NebulaGraph covers the architecture to provide a basic understanding of how the distributed graph database works. For a more detailed introduction to the major components of the system, please refer to:

- [An introduction to NebulaGraph's Storage Engine](https://nebula-graph.io/posts/nebula-graph-storage-engine-overview/)
- [An Introduction to NebulaGraph's Query Engine](https://nebula-graph.io/posts/nebula-graph-query-engine-overview/)

If you have anything to share about distributed systems or graph databases, please feel free to leave us a comment below. We look forward to hear from you.

# You might also like

1. [The Review on Graph Database](https://nebula-graph.io/posts/review-on-graph-databases/)
1. [Graph Query Language Comparison Series - Gremlin vs Cypher vs nGQL](https://nebula-graph.io/posts/graph-query-language-comparison-cypher-gremlin-ngql/)


NebulaGraph is an open-source, distributed, highly performant graph database. This article explains its technical architecture in detail.

Posts

Tag