We will now explore new techniques for pinpointing important content using common neighbors, pushing information flow based on friend relationships and geographic location, and using spatio-temporal relationship mapping to query the relationship between people. 

7 Fundamental Use Cases of Social Networks with NebulaGraph Database | EP 3

We have had a thorough introduction to graph data modeling and data import in the application of social network with graph technology. 

[**7 Fundamental Use Cases of Social Networks with NebulaGraph Database | Episode 1**](https://www.nebula-graph.io/posts/social-networks-with-graph-database-1)

Now, let me introduce more use cases in this episode.

# Identifying key people

Identifying influencers in social networks involves using a variety of metrics and methods to identify individuals who have a lot of influence in a given network. This is useful for many business scenarios, such as for marketing or researching the spread of information in a network.

There are many ways to identify them, and the specific methods and information, relationships, and perspectives considered also depend on the type of these key individuals, and the purpose of acquiring them.

Some common methods include looking at the number of followers a person has or the amount of content consumed, their reader engagement on their posts, videos, and the reach of their content (retweets, citations). These methods are also doable on the graph but are rather mundane, so I won't give examples. Here, we can try to derive these key people on the graph using graph algorithms that evaluate and calculate the importance of nodes.

## PageRank

PageRank is a very classic graph algorithm that iterates through the number of relationships between points on a graph to get a score (Rank) for each point. It was originally proposed by Google founders Larry Page and Sergey Brin and used in the early Google search engine to sort search results, where Page can be a pun on Larry Page's last name and Web Page.

PageRank has long been abandoned as too simple in modern, complex search engines, but it still shines in other graph-structured web scenarios, where we can roughly assume that all links are of similar importance and run the algorithm to find those key users in social networks.

In NebulaGraph, we can use NebulaGraph Algorithm, NebulaGraph Analytics to run PageRank on large full graphs, while in the daily analysis, validation, and design phases, we don't need to run results on full data, but on very small subgraphs (up to tens of thousands), we can easily run various graph algorithms in the browser to derive methods that can be used for production.

Today, we will use the built-in in-browser graph algorithm function of NebulaGraph Explorer to execute PageRank (the specific method is omitted here, you can refer to the [documentation](https://docs.nebula-graph.io/3.3.0/nebula-explorer/graph-explorer/graph-algorithm/ ), but it's really just a matter of mouse clicks).

![PageRank](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/PageRank.png)

We can see from the above that among all the green players (people) after PageRank calculation, "player.name: Tim Duncan" is the largest one, and the relationship associated with it does seem to be quite a lot, so we select him on the graph, then right-click to invert, select all the points except Tim Duncan, use the backspace key to delete all the other points, and then explore 1 to 5 steps in both directions with him as the starting point. In one step, we get Tim Duncan's subgraph.

![TimDuncan](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/TimDuncan.png)

As you can see from the subgraphs, Tim Duncan is associated with a very large number of other players, while some other very popular players have served with him in the very popular Spurs team, which confirms the way PageRank is evaluated.

Now let's see if the algorithm will come to the same conclusion for the other dimensions of determination.

## Betweenness Centrality

As you can see from the subgraphs, Tim Duncan is associated with a very large number of other players, while some other very popular players have served with him in the very popular Spurs team, which confirms the way PageRank is evaluated.

Now let's see if the algorithm will come to the same conclusion for the other dimensions of determination.

![Betweeness_centrality](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Betweeness_centrality.png)

From its five-hop subgraph, it can be seen that unlike the star-shape of the key figure Tim Duncan obtained from PageRank before, Dejounte Murray's subgraph shows clusters, where it is sensory, intuitive to imagine that Dejounte Murray is really on the necessary path of the minimal path between many nodes, while Tim Duncan seems to be associated with more important connecters.

![DejounteMurray](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/DejounteMurray.png)

In practical application scenarios, we usually have to understand the definitions in different ways, experiment with different execution results, and analyze to find the structural features that affect the key people we care about, and use them to choose different algorithms for different needs.

# Detect communities and clusters

Community detection in social networks is a technique to discover community structure by analyzing social relationships. A community structure is a set of nodes that are closely connected to each other in a social network, graph, and these nodes usually have similar characteristics or interests. For example, a community structure may manifest itself as a group of users who are clustered together based on common topics or interests.

The purpose of community detection is to identify the boundaries of different communities and determine the nodes in each community by analyzing the social network. This process can be done by using various algorithms such as label propagation algorithm, weakly connected component algorithm and Louvain algorithm. By discovering the community structure, we can better understand the structure and characteristics of social networks, and help social network service providers to better infer and predict behaviors in social networks, and help in good social network governance, advertisement placement, marketing, etc.

Since our dataset is fake-generated, the results I get under different algorithms do not show the real meaning, so this chapter just shows the results after community identification using several graph algorithms, in real-world cases, we should also use domain knowledge or other technical means on top of that to collaboratively give the portraits and labels of different groups and communities.

Effect of label propagation algorithm.

![LPA](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/LPA.png)



Louvain algorithm:

![Louvain](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Louvain.png)

WCC algorithm:

![WCC](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Louvain.png)

In later sections, we could in better chance verify these algorithms again on smaller and simpler subgraphs, with somewhat more interpretable results.

# Friend Closeness

With the community detection algorithm, it is actually possible to obtain friends with similar interests and close associations to some extent, in a global calculation. So how do we get the other close friends of a given user? We can get this information by counting the number of friends this user has in common with him in order to get this information!

Let's take "Tim Duncan" for example, we know that his two-degree friends (friends of friends: `(:player{name: "Tim Duncan"})-[:follow]-(f:player)-[:follow]-(fof:player)`) are also his friends: Mutual Friend, then it is reasonable to believe that those who have more friends in common with Tim Duncan may have a higher closeness to him.

```cypher
MATCH (start:`player`{name: "Tim Duncan"})-[:`follow`]-(f:`player`)-[:`follow`]-(fof:`player`),
(start:`player`)-[:`follow`]-(fof:`player`)
RETURN fof.`player`.name, count(DISTINCT f) AS NrOfMutualF ORDER BY NrOfMutualF DESC;
```

This query shows that "Tony Parker" and Tim have 5 friends in common and are the closest.

| fof.player.name   | NrOfMutualF |
| :---------------- | :---------- |
| Tony Parker       | 5           |
| Dejounte Murray   | 4           |
| Manu Ginobili     | 3           |
| Marco Belinelli   | 3           |
| Danny Green       | 2           |
| Boris Diaw        | 1           |
| LaMarcus Aldridge | 1           |
| Tiago Splitter    | 1           |

Here, let's verify this result through visualization!

First, let's see who the common friends (f:) are for each of the friends.

```cypher
MATCH (start:player{name: "Tim Duncan"})-[:`follow`]-(f:player)-[:`follow`]-(fof:player),
(start:player)-[:`follow`]-(fof:player)
RETURN fof.player.name, collect(DISTINCT f.player.name);
```

The result:

| fof.player.name   | collect(distinct f.player.name)                              |
| :---------------- | :----------------------------------------------------------- |
| Boris Diaw        | ["Tony Parker"]                                              |
| Manu Ginobili     | ["Dejounte Murray", "Tiago Splitter", "Tony Parker"]         |
| LaMarcus Aldridge | ["Tony Parker"]                                              |
| Tiago Splitter    | ["Manu Ginobili"]                                            |
| Tony Parker       | ["Dejounte Murray", "Boris Diaw", "Manu Ginobili", "Marco Belinelli", "LaMarcus Aldridge"] |
| Dejounte Murray   | ["Danny Green", "Tony Parker", "Manu Ginobili", "Marco Belinelli"] |
| Danny Green       | ["Dejounte Murray", "Marco Belinelli"]                       |
| Marco Belinelli   | ["Dejounte Murray", "Danny Green", "Tony Parker"]            |

 Then we visualize the result on Explorer.

- First, let's find out all of Tim's 2-degree friend paths

```cypher
MATCH p=(start:player{name: "Tim Duncan"})-[:`follow`]-(f:player)-[:follow]-(fof:player)
RETURN p
```

- Then we render the node size by degree in which we select Tim and Tony and find all paths between them for follow type edge, bidirectional, up to 2 hops:

We can see that they are the closest of friends to each other and that their mutual friends are also in the paths.

`["Dejounte Murray", "Boris Diaw", "Manu Ginobili", "Marco Belinelli", "LaMarcus Aldridge"]`

![closest_friend](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/closest_friend.png)

## Small groups in your friends

At this point, as mentioned earlier, the non-authenticity of this dataset itself makes the results of the community discovery algorithm unable to get the insightful connotation of it. Now we can follow this small subgraph to see how groups and communities can be distinguished among Tim's friends.

- Weakly connected components can split Tim's friends into two or three parts that are not connected to each other, which is very much in line with the intuitive understanding and definition of connected components.

![Tim_wcc](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_wcc.png)

- Label propagation, we can control the number of iterations on-demand to delineate different degrees of division by random propagation, which results in a certain degree of differentiation.

  20 iterations

  ![Tim_LPA](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_LPA_1000.png)

  1000 iterations

  ![Tim_LPA_1000](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_LPA_1000.png)

- Louvain, a more efficient and stable algorithm, basically under this subgraph we can get a very intuitive division with a very small number of iterations.

![Tim_Louvain](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_Louvain.png)

# New friend recommendation

Following the previous idea of 2-degree friends (friends of friends), we can easily add those 2-degree friends who are not yet friends as recommended friends, and the sorting rule is the number of friends they have in common with each other:

```cypher
MATCH (start:player{name: "Tim Duncan"})-[:`follow`]-(f:player)-[:`follow`]-(fof:player)
WHERE NOT (start:player)-[:`follow`]-(fof:player) AND fof != start
RETURN fof.player.name, count(DISTINCT f) AS NrOfMutualF ORDER BY NrOfMutualF DESC;
```

| fof.player.name   | NrOfMutualF |
| :---------------- | :---------- |
| LeBron James      | 2           |
| James Harden      | 1           |
| Chris Paul        | 1           |
| Yao Ming          | 1           |
| Damian Lillard    | 1           |
| JaVale McGee      | 1           |
| Kevin Durant      | 1           |
| Kyle Anderson     | 1           |
| Rudy Gay          | 1           |
| Russell Westbrook | 1           |

Obviously, LeBron is the most recommended! And look at who these mutual friends are.

| fof.player.name   | collect(distinct f.player.name)    |
| :---------------- | :--------------------------------- |
| James Harden      | ["Dejounte Murray"]                |
| LeBron James      | ["Danny Green", "Dejounte Murray"] |
| Chris Paul        | ["Dejounte Murray"]                |
| Yao Ming          | ["Shaquille O'Neal"]               |
| Damian Lillard    | ["LaMarcus Aldridge"]              |
| JaVale McGee      | ["Shaquille O'Neal"]               |
| Kevin Durant      | ["Dejounte Murray"]                |
| Kyle Anderson     | ["Dejounte Murray"]                |
| Rudy Gay          | ["LaMarcus Aldridge"]              |
| Russell Westbrook | ["Dejounte Murray"]                |


Again, let's look for LeBron James in the subgraph we just created! And find the two-step, two-way path between them, and sure enough, it only goes through `["Danny Green", "Dejounte Murray"]` and, without a direct connection.

![Tim_newFriend](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_newFriend.png)

Now, the system could send reminders to both sides: "HEY, maybe you two should make new friends!"

Identifying influencers in social networks involves using a variety of metrics and methods to identify individuals who have a lot of influence in a given network. This is useful for many business scenarios, such as for marketing or researching the spread of information in a network.

7 Fundamental Use Cases of Social Networks with NebulaGraph Database | EP 2

Two months ago, my colleague brought in a graph of Twitter interactions  and asked me if I could write a post on graph technology to explore social network analysis and see how those connections are recognized through graph technology.

![graph-database-on-social-network](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/graph-database-on-social-network-02.jpeg)

![graph-database-on-social-network-3](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/graph-database-on-social-network-03.jpeg)

So, how are typical social relationship graphs like the one above, or even more complex social network-based graphs, generated? In the following episodes, I will use the NebulaGraph database to conduct the social network analysis, and you can find step-by-step guides to try all the methods in Playground.

# Social Network Analysis on Graph Database

We can use a graph database in a social network system to represent users and their connections or relationships. Graph databases allow efficient querying of relationships between users, making various business implementations on social networks based on connection findings, statistics, and analysis feasible and efficient.

For example, graph databases can identify "influential users" in a network, recommend new connections (friendships, favorite content ) based on commonalities between users, or find different groups of people and communities to profile users. Graph databases are ideal for social networking systems where user relationships constantly change because they can support complex multi-hop queries and real-time writes and updates.

There have been quite a few examples of practical applications in social networks, including:

- Finding key people
- Identifying clusters of people and communities
- Determining the closeness between two users
- Recommending new friends
- Pinpointing important content using common neighbors
- Push information flow based on friend relationships and geographic location
- Use spatio-temporal relationship mapping to query the relationship between people, get the people who intersected in time and space, and the provinces visited

I will dive into each application with hands-on demonstrations to present graph databases' competence in a couple of episodes. Stay tuned!

# Graph Modeling

To showcase the SNS graph use cases, I'll build most of the examples on a typically small social network. I started by adding extra data on top of the NebulaGraph default dataset, basketballplayer:

Three new tags of vertices:

- address
- place
- post

Five new types of edges:

- created_post
- commented_at
- lived_in
- belong_to

It looks like this:

![schema_sketch](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/schema_sketch.png)

# Importing the data

## Load the default dataset

- In the Command Line Console, we could execute:play basketballplayer` to load the default dataset.

- Or, if we do so from NebulaGraph Studio/Explorer, click the Download from the Demos in the welcome page:

  ![baskertballplayer_studio_starter](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/baskertballplayer_studio_starter.png)

## Add the SNS Graph schema

First, the DDL for those new tags and edges:

```sql
CREATE TAG IF NOT EXISTS post(title string NOT NULL);
CREATE EDGE created_post(post_time timestamp);
CREATE EDGE commented_at(post_time timestamp);
CREATE TAG address(address string NOT NULL, `geo_point` geography(point));
CREATE TAG place(name string NOT NULL, `geo_point` geography(point));
CREATE EDGE belong_to();
CREATE EDGE lived_in();
```

## Load the data

Then we load the DML part to insert vertices and edges:

```sql
INSERT VERTEX post(title) values \
    "post1":("a beautify flower"), "post2":("my first bike"), "post3":("I can swim"), \
    "post4":("I love you, Dad"), "post5":("I hate coriander"), "post6":("my best friend, tom"), \
    "post7":("my best friend, jerry"), "post8":("Frank, the cat"), "post9":("sushi rocks"), \
    "post10":("I love you, Mom"), "post11":("Let's have a party!");

INSERT EDGE created_post(post_time) values \
    "player100"->"post1":(timestamp("2019-01-01 00:30:06")), \
    "player111"->"post2":(timestamp("2016-11-23 10:04:50")), \
    "player101"->"post3":(timestamp("2019-11-11 10:44:06")), \
    "player103"->"post4":(timestamp("2014-12-01 20:45:11")), \
    "player102"->"post5":(timestamp("2015-03-01 00:30:06")), \
    "player104"->"post6":(timestamp("2017-09-21 23:30:06")), \
    "player125"->"post7":(timestamp("2018-01-01 00:44:23")), \
    "player106"->"post8":(timestamp("2019-01-01 00:30:06")), \
    "player117"->"post9":(timestamp("2022-01-01 22:23:30")), \
    "player108"->"post10":(timestamp("2011-01-01 10:00:30")), \
    "player100"->"post11":(timestamp("2021-11-01 11:10:30"));

INSERT EDGE commented_at(post_time) values \
    "player105"->"post1":(timestamp("2019-01-02 00:30:06")), \
    "player109"->"post1":(timestamp("2016-11-24 10:04:50")), \
    "player113"->"post3":(timestamp("2019-11-13 10:44:06")), \
    "player101"->"post4":(timestamp("2014-12-04 20:45:11")), \
    "player102"->"post1":(timestamp("2015-03-03 00:30:06")), \
    "player103"->"post1":(timestamp("2017-09-23 23:30:06")), \
    "player102"->"post7":(timestamp("2018-01-04 00:44:23")), \
    "player101"->"post8":(timestamp("2019-01-04 00:30:06")), \
    "player106"->"post9":(timestamp("2022-01-02 22:23:30")), \
    "player105"->"post10":(timestamp("2011-01-11 10:00:30")), \
    "player130"->"post1":(timestamp("2019-01-02 00:30:06")), \
    "player131"->"post2":(timestamp("2016-11-24 10:04:50")), \
    "player131"->"post3":(timestamp("2019-11-13 10:44:06")), \
    "player133"->"post4":(timestamp("2014-12-04 20:45:11")), \
    "player132"->"post5":(timestamp("2015-03-03 00:30:06")), \
    "player134"->"post6":(timestamp("2017-09-23 23:30:06")), \
    "player135"->"post7":(timestamp("2018-01-04 00:44:23")), \
    "player136"->"post8":(timestamp("2019-01-04 00:30:06")), \
    "player137"->"post9":(timestamp("2022-01-02 22:23:30")), \
    "player138"->"post10":(timestamp("2011-01-11 10:00:30")), \
    "player141"->"post1":(timestamp("2019-01-03 00:30:06")), \
    "player142"->"post2":(timestamp("2016-11-25 10:04:50")), \
    "player143"->"post3":(timestamp("2019-11-14 10:44:06")), \
    "player144"->"post4":(timestamp("2014-12-05 20:45:11")), \
    "player145"->"post5":(timestamp("2015-03-04 00:30:06")), \
    "player146"->"post6":(timestamp("2017-09-24 23:30:06")), \
    "player147"->"post7":(timestamp("2018-01-05 00:44:23")), \
    "player148"->"post8":(timestamp("2019-01-05 00:30:06")), \
    "player139"->"post9":(timestamp("2022-01-03 22:23:30")), \
    "player140"->"post10":(timestamp("2011-01-12 10:01:30")), \
    "player141"->"post1":(timestamp("2019-01-04 00:34:06")), \
    "player102"->"post2":(timestamp("2016-11-26 10:06:50")), \
    "player103"->"post3":(timestamp("2019-11-15 10:45:06")), \
    "player104"->"post4":(timestamp("2014-12-06 20:47:11")), \
    "player105"->"post5":(timestamp("2015-03-05 00:32:06")), \
    "player106"->"post6":(timestamp("2017-09-25 23:31:06")), \
    "player107"->"post7":(timestamp("2018-01-06 00:46:23")), \
    "player118"->"post8":(timestamp("2019-01-06 00:35:06")), \
    "player119"->"post9":(timestamp("2022-01-04 22:26:30")), \
    "player110"->"post10":(timestamp("2011-01-15 10:00:30")), \
    "player111"->"post1":(timestamp("2019-01-06 00:30:06")), \
    "player104"->"post11":(timestamp("2022-01-15 10:00:30")), \
    "player125"->"post11":(timestamp("2022-02-15 10:00:30")), \
    "player113"->"post11":(timestamp("2022-03-15 10:00:30")), \
    "player102"->"post11":(timestamp("2022-04-15 10:00:30")), \
    "player108"->"post11":(timestamp("2022-05-15 10:00:30"));

INSERT VERTEX `address` (`address`, `geo_point`) VALUES \
    "addr_0":("Brittany Forge Apt. 718 East Eric  WV 97881", ST_Point(1,2)),\
    "addr_1":("Richard Curve Kingstad  AZ 05660", ST_Point(3,4)),\
    "addr_2":("Schmidt Key Lake Charles  AL 36174", ST_Point(13.13,-87.65)),\
    "addr_3":("5 Joanna Key Suite 704 Frankshire  OK 03035", ST_Point(5,6)),\
    "addr_4":("1 Payne Circle Mitchellfort  LA 73053", ST_Point(7,8)),\
    "addr_5":("2 Klein Mission New Annetteton  HI 05775", ST_Point(9,10)),\
    "addr_6":("1 Vanessa Stravenue Suite 184 Baileyville  NY 46381", ST_Point(11,12)),\
    "addr_7":("John Garden Port John  LA 54602", ST_Point(13,14)),\
    "addr_8":("11 Webb Groves Tiffanyside  MN 14566", ST_Point(15,16)),\
    "addr_9":("70 Robinson Locks Suite 113 East Veronica  ND 87845", ST_Point(17,18)),\
    "addr_10":("24 Mcknight Port Apt. 028 Sarahborough  MD 38195", ST_Point(19,20)),\
    "addr_11":("0337 Mason Corner Apt. 900 Toddmouth  FL 61464", ST_Point(21,22)),\
    "addr_12":("7 Davis Station Apt. 691 Pittmanfort  HI 29746", ST_Point(23,24)),\
    "addr_13":("1 Southport Street Apt. 098 Westport  KY 85907", ST_Point(120.12,30.16)),\
    "addr_14":("Weber Unions Eddieland  MT 64619", ST_Point(25,26)),\
    "addr_15":("1 Amanda Freeway Lisaland  NJ 94933", ST_Point(27,28)),\
    "addr_16":("2 Klein HI 05775", ST_Point(9,10)),\
    "addr_17":("Schmidt Key Lake Charles AL 13617", ST_Point(13.12, -87.60)),\
    "addr_18":("Rodriguez Track East Connorfort  NC 63144", ST_Point(29,30));

INSERT VERTEX `place` (`name`, `geo_point`) VALUES \
    "WV":("West Virginia", ST_Point(1,2.5)),\
    "AZ":("Arizona", ST_Point(3,4.5)),\
    "AL":("Alabama", ST_Point(13.13,-87)),\
    "OK":("Oklahoma", ST_Point(5,6.1)),\
    "LA":("Louisiana", ST_Point(7,8.1)),\
    "HI":("Hawaii", ST_Point(9,10.1)),\
    "NY":("New York", ST_Point(11,12.1)),\
    "MN":("Minnesota", ST_Point(15,16.1)),\
    "ND":("North Dakota", ST_Point(17,18.1)),\
    "FL":("Florida", ST_Point(21,22.1)),\
    "KY":("Kentucky", ST_Point(120.12,30)),\
    "MT":("Montana", ST_Point(25,26.1)),\
    "NJ":("New Jersey", ST_Point(27,28.1)),\
    "NC":("North Carolina", ST_Point(29,30.1));

INSERT EDGE `belong_to`() VALUES \
    "addr_0"->"WV":(),\
    "addr_1"->"AZ":(),\
    "addr_2"->"AL":(),\
    "addr_3"->"OK":(),\
    "addr_4"->"LA":(),\
    "addr_5"->"HI":(),\
    "addr_6"->"NY":(),\
    "addr_7"->"LA":(),\
    "addr_8"->"MN":(),\
    "addr_9"->"ND":(),\
    "addr_10"->"MD":(),\
    "addr_11"->"FL":(),\
    "addr_12"->"HI":(),\
    "addr_13"->"KY":(),\
    "addr_14"->"MT":(),\
    "addr_15"->"NJ":(),\
    "addr_16"->"HI":(),\
    "addr_17"->"AL":(),\
    "addr_18"->"NC":();

INSERT EDGE `lived_in`() VALUES \
    "player100"->"addr_4":(),\
    "player101"->"addr_7":(),\
    "player102"->"addr_2":(),\
    "player103"->"addr_3":(),\
    "player104"->"addr_0":(),\
    "player105"->"addr_5":(),\
    "player106"->"addr_6":(),\
    "player107"->"addr_1":(),\
    "player108"->"addr_8":(),\
    "player109"->"addr_9":(),\
    "player110"->"addr_10":(),\
    "player111"->"addr_11":(),\
    "player112"->"addr_12":(),\
    "player113"->"addr_13":(),\
    "player114"->"addr_14":(),\
    "player115"->"addr_15":(),\
    "player116"->"addr_16":(),\
    "player117"->"addr_17":(),\
    "player118"->"addr_18":();
```

## First glance at the data

Let's start with the stats of the data.

```sql
[basketballplayer]> SUBMIT JOB STATS;

+------------+
| New Job Id |
+------------+
| 10         |
+------------+
[basketballplayer]> SHOW STATS;
+---------+----------------+-------+
| Type    | Name           | Count |
+---------+----------------+-------+
| "Tag"   | "address"      | 19    |
| "Tag"   | "place"        | 14    |
| "Tag"   | "player"       | 51    |
| "Tag"   | "post"         | 10    |
| "Tag"   | "team"         | 30    |
| "Edge"  | "belong_to"    | 19    |
| "Edge"  | "commented_at" | 40    |
| "Edge"  | "created_post" | 10    |
| "Edge"  | "follow"       | 81    |
| "Edge"  | "lived_in"     | 19    |
| "Edge"  | "serve"        | 152   |
| "Space" | "vertices"     | 124   |
| "Space" | "edges"        | 321   |
+---------+----------------+-------+
Got 13 rows (time spent 1038/51372 us)
```

We could get all of the data:

```sql
MATCH ()-[e]->() RETURN e LIMIT 10000
```

As the data volume is quite small, we could render them all in the canvas of NebulaGraph Explorer:

![match_all](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/match_all.png)

Above are the two fundamental steps to embark on the journey of social network exploration. I'll introduce other applications in the next episode:
- Finding Key People
- Identifying clusters of people and communities
- Determining the closeness between two users

We can use a graph database in a social network system to represent users and their connections or relationships. Graph databases allow efficient querying of relationships between users, making various business implementations on social networks based on connection findings, statistics, and analysis feasible and efficient.

7 Fundamental Use Cases of Social Networks with NebulaGraph Database | EP 1

Qingteng Cloud Security is a host security unicorn company, which focuses on the security of cloud native applications. At present, its main business is the host product named Wanxiang and the container product named Honeycomb. It leads the industry and supports 8 million agents in total. The company's next-generation real-time intrusion detection system based on NebulaGraph and combined with graph technology has been initially put into the market and participated in the 2022 Network Protection, which has achieved good response.

In this topic, we will briefly introduce the intrusion detection system based on graph, hoping to have more people to join in the mining of graph and security application combination.

# The status and challenge of intrusion detection

## Mainstream intrusion detection system

Intrusion detection has always been a major direction of security research. The two sets of products of Qingteng, Wanxiang and Honeycomb, are host-based and container-based intrusion detection products, both of which have similar principles, as shown below.

![Intrusion detection system architecture diagram](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-1.png)

The agent is placed on the host and container side, receiving rules from the server side, combining the collected raw events (process/network connection/file read and write, etc.), and reporting them to the server side as alarms after hitting the rules written by security experts, such as the MD5 of the process file, execution command line, machine learning, and other features. The full amount of raw events are also reported to the server side log collection system and stored in the SIEM system.

Once the alarm is reported to the server, the security staff will take the information related to the alarm to SIEM system to check the events before and after the time of occurrence, log in to the relevant machine to check the relevant information if necessary, and make a comprehensive judgment on whether the current attack alarm is valid and do the relevant disposal.

This system is the current mainstream architecture of intrusion detection, but there are many problems:

1. For single-point detection based on single events and rules, it may result in false positives due to too loose rules or missed positives due to too strict rules.
2. Too many alarms may be triggered by the same attack, and security staff have a heavy workload in tracing the source.
3. The original events related to the same alarm need to be manually analyzed by the SIEM system, which lacks visual means.

## The research direction of graph and security

These questions essentially look at each alarm and event independently, when in fact, the alarms and events associated with an attack should be related to each other. We can use graphs to correlate these original events and alarms for overall analysis. In fact, this is also a popular direction of current security research -- traceability graph, with the help of traceability graph we do the following security analysis and detection.

### 1.Graph detection

Traditional security solutions solve similar IOC detection problems, that is, the judgment of single-point, for the process, network connections, files and other entities is not safe. The actual problem faced by the attack, may not be detected at all points, but the actual behavior is dangerous. A single rule is not fully written, even if the rule can be written, such as hitting a command line, but it will trigger many alarms, and eventually the rule cannot be used. We need to comprehensively judge various combination relationships, which is what graph is good at, and also the problem of graph detection in security. Put all related events into one graph to comprehensively judge whether they are valid attacks.

**The research of graph detection started early, but faced with the dual challenges of computational volume and algorithm, the current industry applications are few, and the combined judgments are all in the way of sequence detection**. But the sequence rules can not write too many rules in the rule list, otherwise it will lead to the inability to apply the rules. In addition, if the rules are too detailed, and can not be well fuzzy matching rules, it is easy to be bypassed. Hackers can easily bypass these rules with the help of automation tools.

### 2.Graph association and tracing

Just mentioned alarms, which is another problem all the security products facing today, either the rules do not hit, or too many rule hits, we call this situation **alarm flooding**. Security staff processing capacity is limited, maybe 100 alarms a day is fine, if a day or even an hour 10000 alarms can not be handled, it is no different from no alarms.

This is really a problem of correlation classification and traceability in security:

1. An attack will generate a lot of alarms, such as brute force login, used malicious files, executed malicious commands, etc.. The correlation classification to correlate the related alarms in an attack together, which is something the graph is good at. These alarms also allow a comprehensive assessment of whether the current attack is valid or not.
2. Each alarm will only tell you is a process, file, or network problems, but how this problem occurred, how the hacker came in, where the file was downloaded from, what was done first and then what was done. Security products need to help customers complete this analysis process, the industry is currently with the help of SIEM, THP, SOC and other security products, all the original events are uploaded and saved, and then find security experts, starting from the alarm to check the original event logs, to see what happened before the alarm, which has a relationship may lead to an attack, the process is as short as a few minutes, long as several hours. **Security is a confrontation process, early is better**, the sooner it is found the sooner it is confronted, blocked or isolated, otherwise it is too late even if it is found. The relevant original event entities (process/network, etc.) can be imported into the graph, with the help of the graph to visually explore and trace the entire attack process, which is the process of graph traceability, academically called causal graph, traceability graph.

### 3.Graph knowledge mapping and prediction

We know that **current security is fundamentally based on rules or a priori knowledge**, each vulnerability, Trojan horse, attack tools, attack process, attack organizations have its characteristics, the first few rules are still relatively good description, the attack process and attack organizations are difficult to complete the description.

The current mainstream approach is to build a knowledge base based on security frameworks, the current mainstream has Kill Chain, ATTCK and other frameworks, which is the U.S. Department of Defense-led cyber attack war related to the two companies proposed a security analysis framework, equivalent to the mapping of the tactics and specific attack techniques that delineate the attack. Security academia, such as the University of Illinois and Purdue University have been working on similar issues in the last two years, which is the construction of a security knowledge base (security knowledge mapping). With a complete knowledge base, the ultimate vision of security can be accomplished, for example, if I know your attack process and attack organization, is it possible to stop an attack before it actually starts.

## Domestic and international status

At present, the intrusion detection system based on graph that is really taking the lead in the real world is Crowd Strike, a star security company in the U.S. It builds its security system entirely on graphs, which now amounts to doing both graph detection and graph correlation traceability, and is currently valued at 67 billion. Cloud computing giants AWS and Azure are following its approach. Domestic intrusion detection system based on graph, currently there are public information is ThreatBook and Sangfor, they are equivalent to do part of the work of graph correlation and traceability.

Qingteng Cloud Security's Wanxiang and Honeycomb have been working on intrusion detection for many years and have achieved industry-recognized security detection capabilities, so we chose to start with graph association and traceability, and develop a next-generation real-time intrusion detection system based on NebulaGraph combined with graph technology, focusing first on solving the pain point problems of alarm flooding and association traceability.

# Qingteng cloud security next-generation intrusion detection system

## Detection principle architecture diagram

As shown below, the focus is the reported attack alerts and some of the original events are unified and correlated in real time in the graph.

![Detection principle architecture diagram](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-2.png)

## Product effect

The output after correlation processing by the correlation engine is the attack event, an attack event may be associated with multiple attack alerts and visualized, the current product effect is as follows.

- The same attack using malicious files and trojans

  ![The same attack using malicious files and trojans](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-architecture-03.png)

- Mounting of sensitive containers

  ![Mounting of sensitive containers](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-architecture-04.png)

- Custom script detection

  ![Custom script detection](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-5.png)

## Advantages of NebulaGraph

NebulaGraph was chosen based on the following considerations.

1. The immediate need for graph queries, especially for multi-level process relationships during association and traceability. It is much easier to query a graph database with cypher than to query multi-level relationships in a relational database with SQL.
2. Large-scale storage, involving the storage of a large number of events and alarms.
3. High-performance query scenarios, associations need to ensure near real-time, the current relevant queries are in ms level.

Thanks to the good performance of NebulaGraph, the association engine enters the graph and calculates the association in a near real-time manner. The tool for importing graph data is the real-time graph entry component created by Qingteng based on Flink, which only needs to change the configuration file to complete the map of points and edges.

# Next research direction and plan

At present, the intrusion detection system of Qingteng cloud security mainly supports the association and traceability between single machine and some multi-machine scenarios. The next step is to support more scenarios of multi-machine association with the help of the graph, especially some typical security attack scenarios (rebound/horizontal movement), so as to further enable security with the graph and provide better service for customers.

There are many problems in applying NebulaGraph to graph security applications at present. Some suggestions are given for reference.

## 1.Push down optimization

In our process of use, the most important problem is the optimization of push down. Such as a diffuse process tree, while specifying the root node and leaf nodes, the order of statement writing may lead to query from the leaf node or the root node, query efficiency varies widely, and finally can only specify the leaf node conditions to force the query from the leaf node.

The performance of MATCH is also essentially a push-down optimization problem, this problem is also encountered a lot, our current approach is to maintain a cache pool for some of the statements have been queried, also note that the official enterprise version actually provides a partial cache function, this point is good if you can consider adding to the community version, is a practical application just need.

## 2.Some technical requirements for real-time scenarios

Since we are doing real-time association based on NebulaGraph, one of the major problems is how to achieve a balance between consistency and speed, because currently NebulaGraph cannot actually determine whether the graph entry is complete after writing to it. In practice, the association engine needs to poll downstream to determine whether the current vertices and edges is actually in the graph. This problem has some impact on the actual use and performance, and it is worth considering whether there is a better way.

We can see that some competing products, such as TigerGraph and TuGraph, provide the ability to customize the matching and traversal algorithms. In our multi-host association scenario, we have encountered such a need. At present, we can only achieve it by splitting multiple paths and splicing them together. Efficiency and speed are definitely compromised.

## 3.ToB deployment

Another pain point is the ToB deployment of NebulaGraph. In the security industry, there are still more on-site deployments at home. Especially, state-owned enterprises and government affairs are relatively sensitive to SaaS, so they will face the problem of efficiency and cost.

**The first is single-machine deployment**. The default production version is a distributed deployment with separate storage and computing designed for the cloud, which is still too cumbersome for ToB deployments and consumes relatively more resources than other databases.

**The second is HDD deployment**. Considering the cost, many companies, especially small and medium-sized companies, have limited security budgets, which makes it difficult to provide SSD deployment, not to mention the large capacity and high configuration of SSD machines required by the official website. Therefore, more optimization considerations should be made for HDD and low configuration machines.

Above, for Qingteng cloud security team engineer Zhou Wen to share.

**Thank you for reading this article.**

Want to experience the graph database up close? Now you can use NebulaGraph Cloud on AWS to build your own graph data system, come save a lot of deployment and installation time to get your business done [click the link](https://go.aws/3VZay2I) to use the graph database!

If you want to see the source code, you can go to GitHub to read, use, star it -> [GitHub](https://bit.ly/3f7XHex). Exchange graph database technology and application skills with other NebulaGraph users, and join ["Slack"](https://bit.ly/3SmvFu2) to play together!

Intrusion detection has always been a major direction of security research. In this topic, we will briefly introduce the intrusion detection system based on graph, hoping to have more people to join in the mining of graph and security application combination.

Next-generation Intrusion Detection System Based on Graph

Graph computing is precisely a technology that studies the relationship between data entities and portrays the data in the form of graphs.

Open Source Distributed Graph Computing Frameworks Comparison

# Dag Controller Introduction

Dag Controller is a component of NebulaGraph Explorer, which was released after several tests and tunings. It mainly solves the problem of integration of OLTP and OLAP, and the problem of graph algorithms in complex scenarios. To learn more, see [https://docs.nebula-graph.io/master/nebula-explorer/deploy-connect/ex-ug-deploy/#dag_controller](https://docs.nebula-graph.io/master/nebula-explorer/deploy-connect/ex-ug-deploy/#dag_controller).

Below is some information I would like to share about Dag Controller. Feel free to leave a comment and let's discuss it together.

I'm sure you're no stranger to OLTP and OLAP, but let me briefly introduce them again here: OLTP is a fast-response, real-time online data processing method. OLAP is an offline, data algorithm method for complex scenarios. For NebulaGraph, OLTP has a variety of query statements, such as GO, MATCH, etc. OLAP has a variety of graph algorithms, such as Pagerank, Louvain, WCC (Weakly Connected Components), K-Core, and Jaccard.

OLTP and OLAP do not exist independently. For example, we can use the subgraph returned by a MATCH statement as input to the PageRank algorithm, and the result of the PageRank can be written to NebulaGraph to continue executing a MATCH statement or other algorithms. Like building blocks, various [OLTP and OLAP](https://www.nebula-graph.io/posts/OLTP-vs-OLAP-what-are-the-differences) can be assembled to form a data processing method for complex scenarios.

The Dag Controller is a system that handles scenarios where OLTP and OLAP need to be linked and executed. 

![oltp-vs-olap](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/olap-and-oltp/1.png)

## Architecture

![dag-architecture](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/olap-and-oltp/2.png)

The responsibilities of Dag Controller:

1. It provides an HTTP interface for external communication and access.

It is used for job submission, stopping, deleting and other operations, as well as the configuration of the system environment.

2. Execution of DAG.

When Dag Controller executes DAG, the OLAP will call NebulaGraph Analytics, and the OLTP will call graphd to complete the execution of nGQL.

NebulaGraph Analytics is our graph computing system, supporting Pagerank, Louvain, WCC, Jaccard, and other graph algorithms, supporting HDFS and NebulaGraph data sources.

Graphd, metad, and storaged are components in NebulaGraph. Graphd is mainly responsible for nGQL parsing, storaged is responsible for data storage, and metad is responsible for metadata storage.

# Cases

## Case 1

![dag-use-case](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/olap-and-oltp/4.png)

The above figure shows a DAG model that runs the pagerank algorithm on a subgraph. First, we get a subgraph using an nGQL statement, and then we run the pagerank algorithm on this subgraph.

This method can be used when our graph size is particularly large and we only want to run the algorithm on part of the graph data.

## Case 2

![olap-and-oltp-use-case](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/olap-and-oltp/3.png)

The above figure shows a model to compute the shortest paths for two classes of vertices.

First, the vertex IDs of the two classes are obtained separately by using nGQL, and then the vertex IDs of the two classes are passed to the shortestpath algorithm, which computes the paths between the two classes in the full graph.

Each algorithm can be set up to run the algorithm based on the full graph or subgraph.

There are various DAG models, and different DAG models can be built according to different business scenarios.

# Technical implementation

## DAG model

The DAG refers to a directed graph without loops, and an instance of DAG is considered as a job, and a job has multiple task.

The task in Dag Controller can be an nGQL or a graph algorithm such as PageRank, Louvain, SSSP, etc.

When a job is executed, it needs to sort the task first, there is a lot of related code on the internet, so I won't repeat it here.

## Parallel execution

To ensure the execution efficiency of DAG, multiple DAGs need to be executed in parallel. At the same time, within a DAG, the task without upstream and downstream dependencies also needs to be executed in parallel.

How can multiple DAGs and tasks be executed in parallel? In simple terms, the DAGs and tasks are handled by two thread pools respectively.

The details are as follows:

1. When the system starts, allocate the job thread pool and task thread pool to handle the execution of job and task respectively.

2. Unexecuted jobs are periodically obtained from the database and transferred to the job thread pool.

3. When a job is executed, the task is sorted according to its upstream and downstream dependencies, and then it determines whether the execution of all upstreams of each task is completed, and then the task is handed over to the task thread pool for execution when the upstream execution is completed, or waits if the upstream execution is not completed.

4. If the job thread pool is full during job execution, you need to wait to obtain unexecuted jobs at regular intervals. The same is done when the task thread pool is full.

## Type check

Note that the data type verification problem exists between the data input and output of tasks. For example, task2 is the downstream of task1. The input of task2 must be of type int, and the output of task1 must be of type int.

## DAG stop

When you stop a job, you need to stop multiple task that are running in parallel. A task has a preparation phase, a running phase, and in the running phase there are cross-machine, multi-process situations. Stopping a job requires avoiding the orphan process problem.

## Custom algorithm support

We support treating a customer's algorithm as a task for DAG building. First, configure the algorithm-related parameter information in the system. When executing a job, the system is responsible for running the algorithm that corresponds to the task.

# How NebulaGraph Works
1. [NebulaGraph Architecture — A Bird’s Eye View](https://www.nebula-graph.io/posts/nebula-graph-architecture-overview)
2. [Benchmark: NebulaGraph vs Dgraph vs JanusGraph](https://www.nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph)
3. [Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph)
4. [Deploy the Graph Database on Kubernetes](https://www.nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes)


Dag controller mainly solves the problem of integration of OLTP and OLAP, and the problem of graph algorithms in complex scenarios. 

Integration  Practice of OLTP and OLAP

# Background

Traditional risk control mainly relies on manual review and empirical judgment. With the rise of digitalization in the past decade, leading companies began to introduce blacklisting systems to locate the risk points missed in the review stage through simple rules and post-facto audits, but this still cannot detect problems in advance. Since 2018, the anti-fraud system has gradually evolved to the stage of "intelligent decision engine" consisting of rule engines, real-time metric calculation, and machine learning technologies.

## Insurance Fraud Control Pain Points

The pain points of insurance fraud control are mainly reflected in the following aspects:

**Non-exchange information leads to poor risk control**: Data of insurance companies and the related industry is non-exchangeable. Even the data between various systems within the same insurance company is unshared. This can lead to asymmetry in business information and false claims, or the use of information asymmetry to increase claim amounts.

**Requirements of multi-dimensional computing**: Industry fraud diversity, specialization, and grouping will force the anti-fraud system to analyze data from multi-dimensional aspects, while the traditional analysis system based on relational models, dimensional models, or wide table models hardly analyzes data from dozens or even more dimensions.

**Accuracy requirements of algorithm recognition**: Business model tuning requires more feature data input during the training and serving stages. Generally, there are two methods to input data, one is writing more data, and the other is the supplementing of correlation data. However, if we go with the traditional method, it will be difficult to detect abnormal data with more than 4-hop queries, and the query latency is high.

## Underlying Storage Challenges


With the development of the anti-fraud system, the system faces challenges in the aspect of the underlying storage. 

The performance of traditional stand-alone databases has bottlenecks in capacity and query for linked data. In addition, business iterations and peaks and valleys have agility requirements for database scalability. Nowadays, most risk control systems use Hadoop ecosystem tools to co-accomplish system goals, and the corresponding development and operation and maintenance costs are high. In terms of real-time data analysis, it's difficult for traditional big data platforms to perform the analysis of multiple sources of data.


# Use Scenarios

NebulaGraph is used in many scenarios of insurance fraud detection. The following are 5 typical scenarios.


## Scenario 1: Relationship Detection


Customers may skip the family information in the application form. Such uncertain or missing family information are not well determined by the original method. While the graph data structure allows us to find relatives, which is not natively supported by other types of databases.

In addition, the regular form is filled with only immediate relatives, which does not reflect indirect relationships. In NebulaGraph's graph database, more hidden relationships can be discovered through the available data, including social and non-immediate family relationships. It allows for a comprehensive analysis of the family's insurance purchases.

## Scenario 2: Fraud Detection


Traditional fraud detection methods are mainly based on empirical rules and are unable to detect fraud in advance.

Taking car insurance as an example, there are relatively more entities involved in car insurance, including vehicles, owners, drivers, passengers, etc. It is very difficult to identify those entities among these data. However, in a graph database, it is possible to build a graph to see some outliers based on the number of claims or other business indicators, and it is also possible to classify groups with higher fraud rates through graph algorithms. In social relationship networks, the fraud possibility can be determined by the correlation with the blacklisted spots. In general, we can make full use of pattern matching and other graph database technologies to classify the major entities by using graph algorithms and then identify groups or individuals with high fraud rates based on business experience and existing fraud cases.


Fraud cases usually have certain characteristics. For example, a luxury used car with frequent accidents or multiple accident-related people driving a different vehicle frequently, or a high correlation between different vehicle groups and multiple accident-related owners. These are all suspicious characteristics that can be identified quickly and effectively by graph technology.

## Scenario 3: Precision Marketing 

In the scenarios of marketing, the data in a graph database can help identify target customers in depth. Based on the information left by customers, we can find social relationships, friends, and non-immediate family members other than immediate family members. Such relationship networks help us see a larger dimension of the customer's social circle. We can find more hidden relationships in the connections between customers and their friends. This can help us learn more about family insurance purchase behaviors.


In the private domain scenario, the marketing agents can recommend other insurance products based on the customer's purchase history. The groups with associated relationships may tend to prefer the same insurance products. From another perspective, such associated recommendations will be more convincing to customers. In the public domain scenario of getting target customers, graph databases can also help advertise to the target customers through the customer data system.

## Scenario 4: Insurance Underwriting and Claims Fraud Detection


At the underwriting and claim stages, we can find what insurance policies the insured has, the number of claims denied, and the association with known fraud cases by querying the in and out-degree of the insured. As for car insurance, we can detect fraud from "the case" to the "vehicle", to the "owner", to the "driver", and to the "vehicle", as shown in the above figure. In this way, we can find the fraud cases from the root cause. For example, in a DUI transfer case (the drunk driver A contacts his relative B to arrive at the scene and report to the insurance company in place of A to get insurance compensation after the DUI accident), we can analyze behaviors such as outgoing and incoming calls, and the location information of the involver. Through this information, multiple risk factors may be detected such as delayed reporting, non-frequently used cell phone numbers and distance from the scene.

## Scenario 5: Agents Management


A large number of domestic insurance agents and a high turnover rate may lead to difficulty in agent management. 

We can also build the agent's relationship network based on the graph model. The agent's professional habits and complaints received can be labeled in the graph to enhance the screening of agents from the source.

 For some underground insurance scenarios, such as forging the identity of a newcomer for commission. Fraud risks can be identified through cell phone numbers, electronic device numbers, IP associations, etc., all of which can effectively reduce customer complaints and insurance frauds caused by agents.



# Why NebulaGraph?

## Features of NebulaGraph

![NebulaGraph-Database](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-ecosystem.png)


To address the above anti-fraud pain points and the need for the system-level construct, NebulaGraph has the following features that are suitable for the assurance risk control needs:

1. **Multi-source data fusion**: The sources of insurance data are diverse, including bank data, social network data, government data, medical data, marketing data, and so on. The data fusion of these data sources is a key to the success of the fraud detection system that can establish a comprehensive analysis of customers to better help insurance companies achieve automatic and accurate underwriting.

2. **Support of graph data model**: The construct of an underlying data model is the key to the multi-dimensional analysis. NebulaGraph data models fit the storage and query presentation requirements of linked data and provide quicker analysis compared to the relational model while its schema is flexible.

3. **Cloud-native architecture**: A good storage system should enable upper-tier applications to compute and scale horizontally based on business needs. NebulaGraph is a distributed cloud native graph database. It can be easily scaled out and scaled in to meet the needs of business growth and shrinkage.

4. **Enterprise-level feature support**: NebulaGraph, as a basic software, provides the support of peripheral tools for a better exploration and visual operation and maintenance.

5. **Risk factor detection**: NebulaGraph's association analysis and graph computing engines can help better identify the risk factors with algorithms.

6. **Accuracy of auxiliary algorithms**: Machine learning relies on input data constructed by tuples, which tend to ignore the relationships between data. So the input of graph data can be used to enrich the contextual information and thus improve the accuracy of relevant models.

In a word, NebulaGraph connects dimensional data. It can be used in different scenarios to help insurance companies effectively identify and avoid hidden risks at different stages. For details about how to deploy and use NebulaGraph, please refer to [live demos](https://www.nebula-graph.io/demo/).  

NebulaGraph can be used in different scenarios to help insurance companies effectively identify and avoid hidden risks at different stages. 

Insurance Fraud Detection with Graph Technology | A Practice of NebulaGraph


![Overall Architecture - NebulaGraph](https://user-images.githubusercontent.com/57335825/87115631-a8852580-c228-11ea-877a-b48ba92fbb8b.png)

 NebulaGraph is an [open-source](https://en.wikipedia.org/wiki/Open-source_software) distributed graph database solution. As an open-source project, we would like those who are interested in graph databases to know as much as possible about it. This includes how NebulaGraph is designed and why it is a highly performant database.

As a result, we have developed a series of architecture articles. In them, we are going to cover three areas. First, we will provide a high-level overview of the architecture of NebulaGraph – the focus of this article. Next, we will cover how NebulaGraph’s storage engine works. And finally, we will illustrate how NebulaGraph’s query engine works, including how we have designed nGQL, our graph query language.

# NebulaGraph Architecture Diagram

![NebulaGraph Architecture Diagram](https://user-images.githubusercontent.com/38887077/76277232-1ba27d80-62c3-11ea-9389-cc1aca329c7b.png)

In the diagram above, there is a fully deployed NebulaGraph cluster that contains three services. They include a Meta Service, Query Service, and Storage Service. Each service has its own executable binary. These [binaries](https://github.com/vesoft-inc/nebula/releases) can be deployed either on the same set of hosts or on different hosts.

## Meta Service

In the below diagram, on the right side of the architecture diagram, it shows a cluster of Meta Service. The Meta Service follows a leader/follower architecture. The leader is elected by all Meta Service hosts in the cluster and serves all traffic. Followers stand by and take the updates replicated from the leader. Once the leader dies, one of the followers will be re-elected as the new leader.

![Meta Service](https://user-images.githubusercontent.com/38887077/76277238-1e9d6e00-62c3-11ea-9d10-8391c26b0b20.png)

The Meta Service does not only store and provide meta information about the graph, such as schema and graph partition information. It also takes the role of the coordinator to orchestrate the data movement and to force the leader shift.

## Separation of Query and Storage Layers

![Separation of Query and Storage Layers](https://user-images.githubusercontent.com/38887077/76277242-2230f500-62c3-11ea-8566-4b6be458adee.png)

In the above diagram, to the left of the Meta Service, is NebulaGraph’s main services. NebulaGraph divides the query execution and the graph storage into two separate services. Above the dotted line is the Query Service. Below the dotted line is the Storage Service.
This separation has several benefits. The most straightforward benefit is that the query layer and storage layer can be expanded or shrunk independently. The separation also makes the linear horizontal expansion possible. Last but not the least, the separation provides an opportunity to make it possible for the Storage Service to serve multiple computation layers.
The current Query Service can be considered a high-priority computation layer. Meanwhile an iterative computation framework could be another one.

## Stateless Query Layer

![Stateless Query Layer](https://user-images.githubusercontent.com/38887077/76277245-25c47c00-62c3-11ea-912b-5de3ce4e18fe.png)

As to the Query Service, each individual Query Service host runs a stateless query engine. The Query Service hosts will never communicate with each other. They only read the meta information from the Meta Service and interact with the Storage Service.

This design makes sure the Query Service cluster can be easily managed by Kubernetes (aka. k8s) or be deployed on the cloud. A cloud deployment is already available for NebulaGraph.
See [how to deploy NebulaGraph on Kubernetes](https://nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes/).

There are two ways to balance the Query Service load. The most common way is to place a [load balancer](https://nebula-graph.io/posts/nebula-graph-storage-banlancing-data-migration/) in front of the Query Service. The second way is to configure the client library with all Query Service hosts’ IP addresses. The client will randomly pick one to connect to.

Each query engine takes the request from the client, parses the statement, and generates an Abstract Syntax Tree (AST). Then the AST will be handed over to the execution planner and the optimizer. The final AST will be passed to the executors.

Below is a flow chart of how NebulaGraph’s query engine works. For a detailed explanation, please refer to [an introduction to the query engine](https://nebula-graph.io/posts/nebula-graph-query-engine-overview/).

![image](https://user-images.githubusercontent.com/38887077/78201029-28318480-74c3-11ea-90ed-03e9c2ca25d4.png)

## Shared-nothing Distributed Storage Layer

![Shared-nothing Distributed Storage Layer](https://user-images.githubusercontent.com/38887077/78201059-3a132780-74c3-11ea-95a0-61249c1566a4.png)

The Storage Service is designed to be a [shared-nothing distributed architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture). Each storage host has multiple local key/value stores as the physical data storage.

A quorum consensus protocol (we chose [RAFT](https://raft.github.io/) over paxos due to its simplicity) is built on top of the key/value stores. Together, they provide a distributed key/value store. On top of this distributed key/value store, a graph semantic layer is provided. It translates the graph operations into key/value operations. The graph data (vertices and edges) are hashed into multiple partitions by the rule (vertex_id % the_number_of_partitions).
A partition is a virtual set of data in NebulaGraph. These partitions are allocated to all storage hosts. The allocation information is stored in the Meta Service and can be accessed by storage hosts and query hosts.

# Conclusion

This bird’s eye view into NebulaGraph covers the architecture to provide a basic understanding of how the distributed graph database works. For a more detailed introduction to the major components of the system, please refer to:

- [An introduction to NebulaGraph's Storage Engine](https://nebula-graph.io/posts/nebula-graph-storage-engine-overview/)
- [An Introduction to NebulaGraph's Query Engine](https://nebula-graph.io/posts/nebula-graph-query-engine-overview/)

If you have anything to share about distributed systems or graph databases, please feel free to leave us a comment below. We look forward to hear from you.

# You might also like

1. [The Review on Graph Database](https://nebula-graph.io/posts/review-on-graph-databases/)
1. [Graph Query Language Comparison Series - Gremlin vs Cypher vs nGQL](https://nebula-graph.io/posts/graph-query-language-comparison-cypher-gremlin-ngql/)


NebulaGraph is an open-source, distributed, highly performant graph database. This article explains its technical architecture in detail.

NebulaGraph Architecture — A Bird’s Eye View

SQL databases have been dominant in the market for a while now, helping companies organize, manage, and leverage their data to meet various goals. However, as the data landscape continues to evolve, more dynamic solutions are needed to tackle challenges that the traditional databases are unable to address. That's why NoSQL databases are gaining traction and the biggest businesses are already utilizing them to great success. The global NoSQL database market size passed US$ [7 billion](https://www.imarcgroup.com/nosql-market) in 2022 and is projected to grow to about US$ 35 billion in the next 5-6 years.


Graph databases in particular, a type of NoSQL database, have become particularly sought after. The market size for graph databases will grow by about 22% and pass US$ [8 billion](https://www.globenewswire.com/en/news-release/2022/07/14/2479487/28124/en/Insights-on-the-Graph-Database-Global-Market-to-2028-Rising-Demand-for-Solutions-With-the-Ability-to-Process-Low-Latency-Queries-is-Driving-Growth.html) by 2028. So let's look at what exactly is a NoSQL graph database, how they work and what benefits they offer.

# What is a NoSQL graph database?

A NoSQL [graph database](https://nebula-graph.io) is a type of non-relational, distributed database which employs a graph model. NoSQL stands for “Not only SQL” and refers to a new breed of databases that differ from traditional relational databases in their data model and performance. Graph databases are especially useful for data associated with relationships—everything from friendships on social netwo#rks to equipment supply chains or business processes. They can quickly traverse vast amounts of linked data points to discover insights and hidden connections between entities, making them ideal for network analysis– such as financial [fraud detection](https://www.nebula-graph.io/posts/graph-database-vs-relational-database), [recommendation engines](https://www.nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation) and many other use cases– all while performing at scale. 


Essentially, the natural structure of a NoSQL graph database encourages the creation of deep models that help businesses to uncover complex relationships across datasets; identify small changes in large amounts of related information; forecast outcomes; and create real-time dashboards with dynamic workflow automation.

**Also Read**: [Difference between graph database and relational database](https://www.nebula-graph.io/posts/graph-database-vs-relational-database)

## Essential components of a NoSQL graph database

A typical NoSQL graph database is made up of four essential components: nodes, edges, properties and labels.

* Nodes are the basic unit of data in a graph database. They can represent anything—a person, an organization, a product, or even a transaction. 
* Edges are the lines that connect nodes, and they represent the graph relationships between them. 
* Properties are the attributes of nodes and edges.
* Labels are used to categorize nodes and edges.

This [visual structure](https://www.nebula-graph.io/posts/visualization-in-nebula-graph) of data makes it easy to see how different pieces of data are related to each other, one of the reasons why NOSQL graph databases are very efficient at handling relationships between data. They don't have to rely on complex join operations, which can be slow and inefficient. Instead, they can simply follow the relationships between nodes to find the data they're looking for. This makes NOSQL graph databases perfect for applications that need to handle a lot of data relationships.

## Illustrating how a NoSQL graph database works

Think of it as a network of data points, all connected and able to communicate with each other. This makes it perfect for storing and managing complex data sets, such as social media networks or customer relationship data.

To illustrate this, let’s say you have a database of people and the relationships between them. In this database, each person would be represented by a node, and the relationships between them would be represented by edges.

So, if John is friends with Jane, there would be an edge between John’s node and Jane’s node. And if John is also friends with Bill, there would be an edge between John’s node and Bill’s node.

This might sound confusing, but it’s actually a very powerful way to store data. Because the relationships between nodes are represented by edges, it’s very easy to query the database to find out things like “who are all of the friends of John’s friends?” or “who are all of the friends of Jane’s friends?”

It’s also easy to add new data to a NoSQL graph database. For example, if John makes a new friend, you can just add a new edge between John’s node and the new friend’s node. There’s no need to restructure the entire database.

# Advantages of NoSQL graph databases

NoSQL graph databases are growing in popularity, and it's not hard to see why. They offer a number of advantages over traditional relational databases.
* Improved performance: Graph-based solutions greatly reduce complexity due to their connected nature. This dramatically reduces query times, allowing for faster access to valuable data points. 
* Flexibility: Instead of rigidly adhering to a predefined schema, NOSQL graph databases enable “schema-less” data models which can accommodate multiple use cases and types of business objects. 
* No redundancy: This makes it easier for data administrators to ensure accuracy and reduce space consumption on storage media. 
* Complex analysis: Due to their interconnected layers, graph databases provide invaluable insights by revealing hidden relationships among various elements within the structure that would have otherwise gone unnoticed. 
* Migration:  There is an inherent level of interoperability between various formats in this type of database structure; this simplifies migration processes and makes it easy for users to move large datasets without having to start from scratch.
* Large-scale data management: NoSQL graph databases can easily scale to store and manage billions of data points, while relational databases start to struggle at around the million-record mark.
* Speed: NOSQL graph databases are incredibly fast. They can return query results in milliseconds.

Check out this article for comprehensive [benefits of graph databases](https://www.nebula-graph.io/posts/why-use-graph-databases).

# Examples of NoSQL graph database applications

* Social media networks: Social media networks like Facebook and Twitter use graph databases to store information about users and their relationships. This allows them to recommend friends and content to users based on their interests and connections to other users.

* eCommerce platforms: Ecommerce platforms like Amazon and eBay use graph databases to store information about products, sellers, and buyers. This allows them to recommend products to buyers based on their purchase history and connections to other products.

* Location-based services: Location-based services like Foursquare and Yelp use graph databases to store information about businesses, users, and reviews. This allows them to recommend businesses to users based on their location and connections to other businesses.

A good example of a NoSQL graph database is Nebula Graph right here, a lightning fast open source graph database that can handle billions of nodes and trillions of edges without any performance issues. And because it's based on a distributed architecture, it's highly fault tolerant. It also offers an equally open source graph editing [front-end library](https://www.nebula-graph.io/posts/nebulagraph-veditor-introduction). Others include Neo4j, OrientDB, and ArangoDB among others.

# Choosing the right NoSQL graph database for your needs

Your decision should be based on the type of graph shapes and connections required for your project. Consider if you need directional or non-directional graph patterns, reachability between graph elements, graph maneuvering methods such as splitting or merging orphan nodes, etc. Make sure to explore what types of graph operations are supported by the database in question as not all databases may support certain graph operations or some may have better performance than others in certain scenarios. 

Equally think about your budget. NoSQL graph databases can range in price from free and open source options to expensive enterprise-level options. Choose the one that best fits your needs and budget.

# Conclusion
NoSQL graph databases are quickly becoming an integral part of managing the powerful amounts of data generated by today’s tech-heavy agile environment. With graph databases, data is organized into graph shapes, allowing for flexible and quick scaling of different types of information without having to go through strict pre-defined schemas. 

The architecture of a NoSQL database also provides the opportunity to have more sophisticated networking and navigation, allowing for easy retrieval of entire related data sets. This makes them a great choice for Agile settings where rapid processing and the ability to manage vast amounts of data with complex relationships are highly desired features.

# You may also like:

Has this article aroused your interest in graph databases? If yes, then don't hesitate and dive in. Explore more in-depth details of the concepts and practical applications of graph databases in this blog: 

[What is a graph database and what are its use cases - Definition, examples & trends](https://www.nebula-graph.io/posts/what-is-a-graph-database)

[Why use a graph database? Benefits of graph databases](https://www.nebula-graph.io/posts/why-use-graph-databases)


NoSQL graph databases are quickly becoming an integral part of managing the powerful amounts of data generated by today’s tech-heavy agile environment.

What is a NoSQL Graph Database?

From people's purchasing behavior to advanced medical treatments, virtually every industry now relies on data in some form or another. With the growth of technology and the access to and storage of large amounts of data, it can be argued that smart organizations have hit the jackpot. When used effectively, data can be an invaluable tool when making decisions, optimizing processes, pioneering new products, predicting outcomes, etc. Not surprisingly, many businesses have incorporated data-driven models into their operations, leading to improved efficiency, productivity and profitability. Data is indeed a goldmine for companies that are willing to use it strategically. But what is the place of analytics in all this? 

Analytics is the ultimate key to tapping into this immense power of data - any data. This means that understanding the tools that make this possible is essential. Marke-twise, the graph analytics market is forecasted to reach a value of [$2,522 million](https://www.marketsandmarkets.com/Market-Reports/graph-analytics-market-10738263.html) by 2024, representing a CAGR of 34.0% between 2019 and 2024. Route optimization is expected to be among the biggest growth drivers during this period.

In this article, we'll delve into graph analytics and graph databases, a combination that can lead to limitless opportunities through your data.

# What is Graph Analytics?

Graph analytics is a technology that uses algorithms to help analysts comprehend the relationships between data that is kept in a [graph database](https://nebula-graph.io). This is a rapidly growing field in data analysis that focuses on examining how certain entities relate to each other by understanding the connections between them. 

This approach uncovers meaningful patterns within complex datasets that would otherwise remain undetected due to their sheer size and complexity. 

## Why use graph analytics?

By uncovering previously unknown correlations, graph analytics provides a unique perspective that is not possible when using traditional data analysis methods. This way, organizations can answer complex questions without added complexity, providing them with an accurate and accessible approach to analyzing their data.

Graph analytics can be used extensively in fields such as network security, fraud detection, business intelligence, and social network analysis. Graphs allow for visual representations of connections between elements, which helps to gain insight into how these elements are related with each other. This gives organizations the ability to make more informed decisions based on real-time data mining, by recognizing trends or identifying clusters quickly and efficiently.

In addition, graph analytics provides organizations with increased accuracy levels when conducting predictive modeling and sentiment analysis of large datasets. 

Graph analytics can be used to find deep patterns and insights, and to answer questions about the data. For example, you could use graph analytics to find out which items are most strongly connected or to identify clusters of items.

## How graph analytics work

The typical graph structure is composed of nodes (also referred to as vertices) and edges. Nodes signify points in the data, like customers, organizations, or products. Edges indicate the connection or relation between nodes - they can be one-way or two-way and might have an associated weight to show how strong the relationship is.

Once a graph database is established, analytical work can be done. The graph algorithms from analytics tools can reveal values in the data or interesting details - like nodes with considerable activity. The data can also be rearranged for easier analysis – such as searching for nodes that meet specified criteria.

The result of graph analytics can provide a comprehensive understanding of the underlying structure of the analyzed data – from deriving patterns or anomalies to interpreting influence among groups while also discovering hidden information previously unknown.

Amongst the popular tools for graph analytics are NebulaGraph, IBM Graph, and Google Charts.

## Types of graph analytics
There are four main types of analytics that can be utilized in graph analytics: connectivity analysis, community analysis, path analysis, and centrality analysis.

**1. Connectivity analysis**

Connectivity analysis is used to determine how strong the relationships are between different data points (nodes). It can be utilized in graph analytics to detect vulnerabilities or outlying activity, such as excessively high or low activity.

**2. Community analysis**

Community analysis looks at how nodes interact and what elements affect their behavior. Nodes can be grouped together based on the similarities they have. From there, the data can be reorganized to come up with strategies to improve those interactions and create value out of them.

**3. Path analysis**

As the name suggests, path analysis focuses on the path from one node to the other. It’s particularly used to identify the shortest path between two nodes in a graph. The pathfinding algorithm takes into account the weights of the edges in the graph to find the most efficient path between two nodes.

**4. Centrality analysis**

Centrality analysis measures the importance of a node in a graph. There are several different measures of centrality, but all attempt to quantify how important a node is in the overall network.

Centrality can be applied in graph analytics to identify important nodes in a network and to understand the relationships between them. Centrality can also be used to help identify hubs and bottleneck nodes.

# Leading graph analytics use cases
Graph analytics can be leveraged for a range of uses, like:

**1. Route optimization**

Knowing the most efficient way to travel from Point A to Point B can save time, money, and effort. To get started with graph analytics for route optimization, one must first create a connected set of nodes outlining the route. Nodes can be airports, stops along a bus route, or any other points of interest related to the movement. 

After this initial step is complete, it is then possible to use graph analytics tools to analyze route time, cost of travel by factoring in fuel consumption and tolls, as well as look at alternative routes in order to find the quickest or most cost-effective way of reaching a destination.

**2. Social networks**

Graph analytics can be a powerful tool to help us better understand the complexities of social networks. By analyzing the interactions and relationships between users, graph analytics can reveal insights that would otherwise be impossible to uncover. This could include cases such as finding influential users, detecting communities that have formed within a network, and understanding user behavior over time. 

Perhaps more importantly, by identifying weak signals in user patterns, graph analytics could also be used to detect suspicious account activity or discover hidden malicious campaigns.

**3. Recommendation engines**

There are many ways that graph analytics can be used in recommendation engines. For example, graph analytics can be used to identify relationships between items in a dataset. This information can then be used to recommend related items to users. 

Additionally, graph analytics can be used to identify clusters of similar items within a dataset. This information can then be used to recommend similar items to users. Finally, graph analytics can be used to identify influencers within a dataset. This information can then be used to recommend influential items to users.

**Further Reading:** [Using graph databases in real-time recommendation](https://www.nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation) 

**4. Fraud detection**

Graph analytics can be used to detect fraud by identifying relationships between entities that may not be obvious. For example, if a company is trying to defraud the government of taxes by claiming false expenses, graph analytics can be used to identify relationships between employees, vendors, and suppliers that may not be immediately apparent.

Graph analytics can also be used to identify patterns of behavior that may indicate fraud. For example, if an employee always bills for a certain amount of expenses every month, but this month bills for twice as many expenses as usual, this could be an indication of fraud.

**Further Reading**: [How to visualize and detect fraud with knowledge graphs](https://www.nebula-graph.io/posts/fraud-detection-using-knowledge-and-graph-database)

**5. Journalism**

Graph analytics can help reporters and editors analyze complex relationships among people, places, things, and events.

For example, by analyzing data from social media posts, graph analytics can be used to identify the social networks of people who are talking about a certain topic. This information can then be used to help reporters and editors decide which topics to cover and which sources to contact for more information.

Graph analytics can also be used to map the relationships between different entities in a story. For example, if there is a stabbing incident that involves several people, a graph could be used to show the relationships between the people involved in the stabbing. This information could help reporters and editors understand the dynamics of the event and how it unfolded.

**6. Pandemic patterns**

Graph analytics offer a powerful tool that can be used to analyze patterns related to pandemics. It can be utilized to understand how connections between different elements of the system, such as person-to-person contact or patterns in viral spread, can be evaluated and synchronized. 

For example, if there is a new outbreak of a virus, graph analytics can be used to identify who may have been in contact with the person who contracted the virus. This information can then be used to help track and contain the spread of the virus.

**7. Analyzing communications networks**

Graph analytics can identify network communication patterns that would not be found using traditional methods. 

The deep analysis can provide insight into telecommunications service level agreements, customer trends, and security threats over time. This way, engineers can diagnose problems and create better solutions. 

As computing power improves, we can expect to see graph analytics being implemented for an ever broader range of use cases. 

# Graph analytics and graph databases

Graph analytics and graph databases work together by taking advantage of the structure of data in a graph database. A graph database is perfect for managing data that has complex relationships between entities, and this is where graph analytics come in. Graph analytics allows you to query and analyze these relationships to find patterns and insights that you wouldn't be able to see with other types of analysis.

For example, let's say you have a table of customers with information about each customer's name, address, and purchases. You could use a graph database to represent this data as a graph, with customers as nodes and their addresses and purchases as edges between them. 

This would allow you to query the data to find all customers who live in a certain city, for example. You can then use graph analytics to study their behavior and unearth powerful insights that can be used to create or recommend highly targeted products for this group of customers.

# Graph analytics with NebulaGraph

NebulaGraph is an open-source, distributed, and highly scalable graph database, designed to handle super large-scale graphs with latency in the milliseconds range. Major companies such as Tencent, Vivo, and WeChat have already incorporated NebulaGraph into their data management operations.

NebulaGraph provides a powerful [graph analytics](https://docs.nebula-graph.io/3.2.0/graph-computing/nebula-analytics/) framework aptly named NebulaGraph Analytics, an ultra-fast tool for performing graph analysis of data stored in the NebulaGraph database.

The NebulaGraph Analytics framework supports a variety of popular graph computing algorithms, including PageRank, APSP, KCore, SSSP,  DegreeWithTime, Louvain, TriangleCount, and ClosenessCentrality, among many others. 

# Conclusion

Data analysis is an emerging goldmine, a powerful tool for businesses seeking to unlock the potential of their data and uncover new opportunities for growth. With big data now commonplace, organizations have access to unprecedented amounts of information that can be used to gain insight into customer trends, identify risks, optimize processes, innovate new products and inform decision-making. Graph analytics has fundamentally changed what we can do with this data. 

This new way of leveraging data-driven analytics gives companies a unique opportunity to boost their competitive advantage and grow more sustainably. With its many applications embedded across almost every industry, graph analytics will certainly remain an integral tool in the modern era of data science.

By taking advantage of the potential afforded by graph analytics, developers are now able to create cutting-edge solutions that revolutionize how businesses look at large data sets. Rather than tediously sifting through loads of disorganized data, advanced solutions powered by graph analytics can produce actionable insights in a fraction of the time. This technology has incredible potential across all types of organizations - and developers should jump on the opportunity to be at the forefront of those changes. This way, they can begin to create innovative solutions that tap into the enormous potential of graph analytics to make sense of information with unprecedented speed and accuracy.

# Exploring Graph Databases?
Check out [**NebulaGraph Database**](https://nebula-graph.io) and join the open source community through [**Slack**](https://bit.ly/3SmvFu2) to uncover more practical use cases of graph databases.

By uncovering previously unknown correlations, graph analytics provides a unique perspective that is not possible when using traditional data analysis methods. 

Graph Analytics and Graph Databases


# Introduction
When a critical business asset such as an application, website or platform begins to experience heavy usage, it can become sluggish as the supporting database becomes overwhelmed. To keep the database running smoothly, it has to be scaled up.

One of the best ways to achieve this is through an approach known as database sharding, which has been enjoying a lot of use in recent years. Let's look at what database sharding is, the working, as well as the strengths that are attracting so many developers and organizations to this database scaling technique.

# What is database sharding?
Database sharding is a technique used to scale databases by breaking data in the original database into smaller portions which are then stored on multiple machines or database servers. Since all database servers usually carry the same foundational architecture, the shards can be processed together without any detriment to quality.

Sharding can be used with any type of database, but it is especially useful for databases that contain large amounts of data or that are accessed by many clients. In a sharded system, each server contains a portion of the total data and is responsible for servicing requests from clients that are associated with that portion of the data. 

# Benefits of sharding

The foremost advantage of database sharding is that it eliminates the issue of degraded application performance as usage increases. As an app grows, the number of users and the data it contains both grow. The database slows down as the volume of data and the number of concurrent users accessing the application increases. If this problem is left to persist, it'll eventually compromise customer experience. Sharding fixes this predicament by allowing parallel processing of smaller data sets.

With the growth of data-intensive businesses and popular applications, database sharding has emerged as a go-to solution for improving scalability and performance.

Here are the rest of the benefits that organizations can enjoy by embracing sharding.  

## 1. Increased performance

Database sharding helps to reduce latency by spreading out workloads across multiple servers. For example, if each application server had its own database instance instead of being part of one huge DBMS cluster, workloads would be expedited since individual app servers don’t have to compete for resources.

## 2. Improved security

Segmenting a database helps to maintain reasonable levels of access control and security. Instead of having all sensitive data stored under one database server,  you can segment it into multiple shards that deliver targeted access control in order to protect sensitive information from malicious attacks.

## 3. Scalability

Database sharding is an ideal solution for businesses experiencing rapid growth or sudden spikes in traffic. By simply adding more shards, your system will be able to scale flexibly with changing needs.  

Sharding also provides an excellent platform for future scalability as businesses and their datasets grow larger over time. As individual shards become more populated, they can be easily scaled up - or even split off - without any substantial disruption or downtime to the services consuming them.

## 4. Regional data distribution

Organizations can use database sharding to distribute large amounts of data across different regions in order to meet user requirements by increasing the speed of complex queries. Likewise, it allows applications to easily switch between databases based on the most relevant geographic region or usage pattern instead of relying on rigidly defined paths with complicated logic around them.

## 5. Eliminates total outage

In the past, when the application’s database computer failed, the application would also be down. But with database sharding, data is spread across multiple hosts so that if one fails, the application won't go offline.

## 6. Shorter response times

Over time, single databases typically take longer to access since the system requires looking through the entire big database. It takes a while to bring results based on user queries. However, when using sharding, there is less data to go through, so it drastically reduces the length of the search and effectively reduces the response time.

# How database sharding works

Basically, a dataset is broken up into smaller datasets, or 'shards'. Each shard has its own unique data, which is stored on separate computers or nodes. Even though these shards are stored and run separately from the nodes, they still share the same schema of the original database.

Developers use a special type of key known as a shard key to establish how to shard the dataset. In essence, the shard key functions as a kind of guidebook defining where each piece of data falls and how they are linked to one another.

Each shard works on its own based on the shared-nothing architecture - meaning it is unaware of the other shards and functions autonomously. Requests for data only trigger an individual shard to process the information that is stored in it.

# When to shard

To shard or not to shard depends on the size and complexity of the database. But generally speaking, you should do sharding when an application’s dataset becomes too large or too complex for a single database to handle.

Here are some general guidelines to help you decide if it’s time to do sharding.

* When the database has grown too large and needs to be split into multiple databases to improve performance. 
* When your database is experiencing contention issues caused by too many requests from simultaneous users, which leads to challenges such as timeouts.

Please note that sharding is not a substitute for proper database design, however. Even as you shard, the datasets should still make sense for how they will be used.

# Sharding methods

There are different sharding methods that apply distinct rules to the shard key in order to decide the correct node for each set of data. 

The common methods include range based sharding, directory based sharding, hashed sharding and geo sharding. 

## 1. Range based sharding

Range-based sharding, also known as dynamic sharding, splits databases according to a range with similar values. The database team will then assign a shard key for every range. For example, imagine dividing a product database according to size ranges or a customer database according to a range based on their age. Each range forms a shard which is then stored on a separate database server.

## 2. Directory based sharding

The shards are organized into a tree structure. The root of the tree is called the "directory", and each branch of the tree corresponds to a shard. When a client requests data from the database, the directory determines which branch of the tree to query in order to find the relevant shards. A lookup table is employed to associate the database data with its respective shard.

Directory-based sharding is typically used when there is a large number of shards involved. One disadvantage of directory-based sharding is that it can be more complicated to set up and manage than other types of sharding.

## 3. Hashed sharding

Hashed sharding uses hashes to divide the data into shards. Each shard is given a unique hash value. The data is then divided into chunks based on these hash values. The hash value is used to determine which shard will contain which value. This allows the data to be distributed evenly.  

## 4. Geo sharding

Geo sharding breaks and stores information according to geographical location. For example, a social networking platform might use different shards to save users according to country, city, etc. So when a user is located in, say, Los Angeles, their requests would be routed to the server that stores the shard for Los Angeles. This can improve performance because it reduces network latency between the client and the server.

# Challenges of database sharding

The benefits of sharding are certainly great for any business. But this technique also comes with a couple of challenges that you need to be aware of. 

## 1. Querying issues

Database sharding adds querying complications since the querying process may often need to go through multiple shards to obtain the info needed, and then piece together the info from those different shards to get the requested result. This happens when the data required for a specific query is spread across multiple shards. 

Fortunately, this can be resolved by using automated databases which will make managing and querying operations much simpler.

## 2. Implementation challenges

Many DBMS lack built-in features to enable sharding. As a result, database teams and developers will have to shard the databases manually, which can get quite overwhelming. In order to properly use sharding, an application developer must modify their code in order to split the queries and spread them across multiple shards. This means that more work is required on the development side before the shards can be utilized. 

You can overcome this challenge by using a [graph database](https://nebula-graph.io/) system like NebulaGraph that supports automated horizontal scaling.

## 3. Infrastructure expenses

If you are going to use physical shards, you can end up incurring higher costs to pay for the infrastructure. The more machines you add, the more you will spend. 


You can overcome this challenge by sharding on the cloud and taking advantage of the virtual infrastructure. The [NebulaGraph Cloud](https://www.nebula-graph.io/posts/nebula-graph-cloud-available-on-azure-marketplace) helps users conveniently deploy NebulaGraph clusters without having to take care of infrastructure.

## 4. Unbalanced distribution

When high volumes of requests or queries are sent to certain shards, the performance can suffer due to a bottleneck. Uneven data distribution is usually the cause of unbalanced shards, particularly in physical shards. For example, if there is a sudden surge in demand for a product whose data is stored on one shard, that shard will require more resources. 

To avoid this issue, optimizing the sharding process is key so that datasets likely to experience heavy usage receive priority.

## 5. Building back is complex

Once a database has been sharded, it’s tricky to rebuild it back to its previous unshaded status. By the time you want to rebuild, new data will have been written to the shards. This means the backups of the original database will have to be merged with the new data from the shards. This can consume a lot of developer time, not forgetting the added costs if you have to go for more developers and invest in new tools. 

# Getting the most out of database sharding: Beware of these scenarios

Many companies battle with ensuring that all shards are operating at the optimal level and that some shards are not underutilized while others are maxed. This slows down the access process, thus weakening the very main purpose of sharding.

Watch out for these scenarios and avoid them by taking the time to plan the sharding carefully especially when choosing the shard keys.

## 1. Attribute frequency

This refers to the amount of data with identical attributes that is stored in one shard as compared to others. For example, if the data team opts for 'color' as their shard key for a women's fashion website, and one color proves to be more popular than the rest, then it may cause data hotspots. Thus, it is essential to take frequency into account when selecting a shard key. Try as much as possible to choose a shard key that is unlikely to lead to an overloading of records in one shard while leaving the others underutilized.

## 2. Number of shards

The number of shards that can be made from the database is also dependent on the chosen shard key value. For instance, if the data team chooses YES/NO or HIGH/LOW as the shard key, then the total number of shards will only be two. Now, depending on your data volume and diversity, two shards could just be the perfect number or turn out to be too few and surprise you when the workload suddenly shoots up. 

## 3. Changes over time

As the database increases, the shard key should not change so considerably that a huge amount of one data type goes to one shard only. Let's say the shard key is based on FINISHED or UNFINISHED orders. Over time as you streamline your processes, the shard key for finished orders will increase steadily, which means the dataset for finished orders will outstrip its shard. 

# Are there alternatives to sharding?

Yes. Database sharding is basically one among several techniques that are used to scale databases. It’s based on the horizontal scaling concept, where additional computers are added to share an application’s workload as usage increases. By doing this, organizations can benefit from the advantage of fault tolerant architecture, meaning whenever one database fails, the application is not entirely disrupted. 

But besides sharding, there are more techniques that organizations can explore to scale their databases. Here are some of the more popular ones: 

## 1. Partitioning

Partitioning involves splitting up the data in a database into smaller, more manageable parts. This can be done in a number of ways, but the two common approaches are vertical and horizontal partitioning. Vertical partitioning uses columns to partition the database while horizontal splitting partitions the database by the rows.

For example, let's say you have a table that stores customer information and there is a column in there that arranges the customers according to countries. You could partition the table vertically by dividing it into different parts based on the customer's country of residence. This would create a separate table for each country, and would make it easier to manage and query the data.

## 2. Replication

Database replication makes copies of a database and keeps the copies in different machines. So when data is written to the primary database, it is also written to the secondary databases. This provides high availability because whenever the active database server fails, the next machine with the exact database copy swings into action. 

Some organizations use a combination of replication and sharding depending on the type of data they are handling as well as customer needs. 

## 4. Vertical scaling

Here, scaling is done by increasing the capacity of one database server instead of adding more servers as in the case of horizontal scaling.  More CPU, memory and disk capacity are added to a single server in order to handle increased loads.

The advantage of vertical scaling is that it's simpler and faster to implement than database sharding. The disadvantage is that once you reach the limits of a single server, you’ll need to add another server in order to continue scaling. 

# How can NebulaGraph help?
Organizations of all sizes are using NebulaGraph to build and manage modern databases for their applications. NebulaGraph offers high availability at scale with tools that support horizontal scaling. With NebulaGraph, you can build a distributed database that is resistant to failure and can handle large workloads. You can also create databases that are easy to manage and query with a graphical user interface. Simply start by [importing data into NebulaGraph.](https://www.nebula-graph.io/posts/importing-data-into-nebula-graph-using-nebula-importer)

# Conclusion

Modern businesses are driven by technology which requires a lot of data to power their platforms. It’s no wonder that [big data is projected to drive business growth](https://www.forbes.com/sites/forbesbusinesscouncil/2022/11/28/business-growth-through-big-data/?sh=7e9babb3784e) at a speed that may not have been possible before the age of data. This data is stored in machines that have limits on data storage, processing, and serving users. When this limit is reached as a result of growing use, then scaling becomes inevitable. 

The first step is to choose the scaling technique that will work best for your scenario. Database sharding is one such technique. And as we've just seen, it's a popular technology that many organizations are adopting. But be sure to do it in a logical way that will minimize latency and maximize performance. Otherwise, you may end up with even worse performance than before. Keep an eye on how you select the shard keys because this will have a huge impact on how the shards perform over time. The goal is to attain the kind of balance that ensures the workload is well balanced across all shards. 

If you don't feel like sharding is the right fit for your use case, you can always look into the other alternatives and pick whichever one best meets your current needs.

# See How NebulaGraph Database Shared-Nothing Structure Works
[An Introduction to NebulaGraph's Storage Engine](https://www.nebula-graph.io/posts/nebula-graph-storage-engine-overview)

Let's look at what database sharding is, the working, as well as the strengths that are attracting so many developers and organizations to this database scaling technique.

What is Database Sharding?

# Introduction

Graph databases are growing at an impressive pace, increasingly becoming the go-to DBMS for a growing number of large organizations. DB Engine, a website dedicated to the ranking of database management solutions, shows that [graph databases have experienced consistent popularity since 2013](https://db-engines.com/en/ranking_categories) when many companies started appreciating them. The growth actually surpasses all other forms of DBMS.
The worldwide graph database market is expected to increase to [11.25 Billion](https://www.prnewswire.com/news-releases/graph-database-market-size-to-reach-usd-11-25-billion-in-2030--increasing-demand-for-flexible-online-schema-environments-is-a-key-factor-driving-industry-demand-says-emergen-research-301478726.html#:~:text=Global%20graph%20database%20market%20is,is%20boosting%20market%20revenue%20growth.) by 2030, rising from 1.59 Billion in 2020.  This enormous surge is driven by the high need for elastic online schema environments.

![db-engines-ranking](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/what-is-a-graph-database/db-rankings.png)

# Why choose a graph database for your business

You should choose graph databases because they perform way better at scale compared to other DBMS. They make it possible to represent data in a much more intuitive way, a versatile way of visualizing data which enhances query performance and enables businesses to make valuable decisions like never before. They are also less intense in terms of computational demands.

This is important because for many businesses, the ability to store, visualize, query, and analyze data at a deeper level is key to success these days. Traditional databases can maintain large amounts of data and are capable of running complex queries, yet they rely on rigid and often complex schemas.  An alternative solution is a graph database that stores information as interconnected points. The interconnected structure helps to connect objects according to specific criteria, allowing data teams to find themes or patterns within the data more easily than with a traditional approach. 

Queries in a [graph database](https://nebula-graph.io/) run much faster, as they don't have to traverse entire tables or deal with foreign key constraints. In addition, since relationships between entities are stored, the data remains consistent even when it changes over time. 

![graph-query](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/what-is-a-graph-database/efficientquery.png)

# Benefits of graph databases
The key benefit of graph databases is the ability to handle big data that is deeply interconnected. As we are right in the middle of the data era, this one perk will prove invaluable to businesses that are focused on boosting their success through big data.

Here are the comprehensive benefits of graph databases:

## 1. High performance

Graph databases allow users to query linked data quickly and efficiently, making them much faster than traditional relational databases. With sophisticated algorithms and powerful search functions, graph databases can return complex query results across an interconnected network in milliseconds.

Querying in traditional databases can become slow very quickly when dealing with more complicated and larger data sets. Conversely, the performance of graph databases will remain consistent, no matter how large or complex the data is.

For example, [NebulaGraph](https://nebula-graph.io) can so easily store and process graphs with trillions of edges and vertices - the only open source graph database that is capable of doing this!

Also read: [NebulaGraph joins Data Benchmark Council](https://www.nebula-graph.io/posts/nebula-graph-joins-ldbc)

## 2. Data integrity

Graph databases are designed to maintain the integrity of data connections across a wide variety of data sources and formats. By connecting disparate pieces of information in a graph, organizations can ensure that their data remains consistent and accurate throughout their applications. 

Instead of simply storing individual pieces of information in a database, graph databases link related data together using connected nodes and edges. This allows users to easily visualize highly connected data and establish how any changes to the data will affect other areas, hence enhancing data integrity. 

![graph-database-benefits](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/what-is-a-graph-database/benefits-of-graph-databases.png)

## 3. Enhanced contextual awareness

Because graph databases store data in the form of relationships between objects, they enable users to uncover nuances in interactions that may not be evident at first glance with traditional techniques such as tabular relational models or hierarchical tree structures. 

This enhanced contextual awareness helps organizations make better decisions by leveraging high level insights from their data. This is a crucial benefit in the modern business landscape where there is far too much data that is just idle. Graph databases can mine this data and deliver transformative insights to spur organizational growth. 

## 4. Scalability

The schema and structure of the graph model is incredibly adaptable and can keep up with the high pace of ambitious data teams. There's no need to go through the long-winded process of modeling domains; just add to the current structure without disrupting its existing functionality.

Traditional database technologies often reach a scalability bottleneck when trying to bridge multiple sources of data. But graph technologies avoid this issue entirely with their distributed nature which allows more flexible scaling up or down as needed. 

NebulaGraph provides linear scalability, enabling users to expand the cluster with additional nodes or services without interfering with performance.        

## 5. Agility

Today's development landscape is rapidly evolving. It's all about flexibility and quick changes as needs are regularly shifting. Graph databases are the ideal solution for this kind of adaptive environment, where unified progress and management is becoming the standard. 

Check out the fundamental [differences between relational databases and non relational databases.](https://www.nebula-graph.io/posts/difference-between-relational-and-non-relational-database)

# Use cases for graph databases

Did you know that all the top tech giants such as Facebook and Google are already utilizing the revolutionary power of graph databases to build products that generate mega revenues? All these companies use graph databases in a broad range of products, harnessing the force of big data connections to build irresistible products.

The use cases can be as vast as the imagination of any ambitious business. Here are the most interesting ones.

## 1. Recommendation engines

[Real-time recommendation systems](https://www.nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation) are very popular these days, especially in retail e-commerce, travel and social networks . The purpose of a recommendation engine is to predict customer preferences and recommend products or services they might be interested in whenever they are using your platform. The application of recommendation engines is quite exciting and is already being used by many companies to achieve immense success. But it would have been impossible to get this done were it not for graph databases. 

![recommendation-engine-with-graph-database](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/what-is-a-graph-database/figure5%20(1).png)

With graph databases, you can easily discover connections and generate recommendations for users. These recommendations are based on powerful graph analytics that match  preferences  with relevant products or services. 

## 2. Fraud detection

Graph databases are being used to [detect fraud](https://www.nebula-graph.io/posts/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph) by tracking transactional relationships between entities over a period of time. 

By uncovering patterns of behavior across multiple financial transactions, it is possible to detect anomalies that could indicate fraudulent activities.

![fraud-detection-graph](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/what-is-a-graph-database/fraud-detection-graph-database.png)

## 3. Network Analysis

Graph databases can be used to trace connections between technology components such as wireless routers, cables and microwave links, providing administrators with useful insights such as the types of connections being used at any given time. 

This is important because network administrators have an ever-increasing responsibility when it comes to managing digital networks. They must be able to identify problems quickly and ensure that operational performance is maintained. They can use graphs to quickly find out which paths data packets take through the network by understanding IP addresses and traffic flow over large portions of the infrastructure. This way, they can know exactly where problematic areas are during troubleshooting or when planning upgrades or changes in configuration.

## 4. Power grid scheduling

Utility power grids are some of the most complex infrastructures in existence, requiring a fine balance between efficient usage and stability. Traditional models used for managing these grids relied heavily on tables and relational databases, leading to slow response times when working with high levels of data. However, graph databases can revolutionize how utility grids are scheduled. 

Graph databases store nodes and edges that represent real-world relationships, allowing more precise and detailed understanding of the power grid's dependencies. By organizing data into separate networks, utility providers can quickly make decisions regarding load shedding, rotational outages, or even unexpected problems like capacitor failures. 

In addition to providing flexibility in scheduling decisions, graph databases can also detect patterns of consumer behavior which can then be used to anticipate upcoming demand changes in order to deliver superior service while reducing costs.

## 5. Chatbot systems

Graph databases provide a flexible and easily navigable structure that allows bots to quickly access data and respond to queries in real time. Large datasets can be retrieved at high speeds, be used  to construct a useful inventory of customer preferences and generate meaningful dialogue based on the collected insights. 

Questions are organized in natural language and converted into knowledge graphs which are stored in graph databases. A semantic parser is used to resolve the questions in a  question-answer format. The answers are then retrieved from the knowledge graph and passed on to the user.

# When should I use graph databases?

You should use graph databases when you need to model huge and highly interconnected data. So if you find yourself facing complex tasks such as connecting data from multiple sources or analyzing intricate relationships between people, groups, events, or concepts, then turn to graph databases. 

They provide the flexibility to represent better inter-connectivity between elements and enable quick insights into connections between objects that are not easily modeled in other database systems. 

# Conclusion

Unlike traditional relational or document-based databases, graph databases provide powerful search capabilities and enable superior performance when performing complex queries on a large dataset. Additionally, graph databases make it much easier for developers to modify data structures without breaking existing applications or creating additional complexity due to changes in schema. Graph databases also boast greater scalability when dealing with high volumes of data whereas traditional systems typically suffer from poor responsiveness under such circumstances. 

With these unique advantages, it's no surprise that graph databases are experiencing tremendous growth and becoming the go-to choice for many organizations. They offer tremendous potential for improving how we organize, access, analyze and utilize our ever-growing connected datasets. 

Whether your organization handles large volumes of connected customer or product related data, graph databases could provide the crucial bridge you need to move from the isolated silos managed with traditional tools. 

# FAQ Section

## Should I use a graph database?
Yes, you should. Graph databases make it easy to explore relationships between data items, and are also very efficient at handling large quantities of data. They store data in a way that mirrors the structure of the real world, allowing them to quickly find the shortest path between two data points in the graph. 

## Why are graph databases better than relational databases?
Graph databases are better than relational databases because they are exceptionally good at handling complex data relationships. Relational databases are fundamentally a collection of tables, IDs and values. This approach is great for static data sets where relationships between the data are simple and well-defined. But it’s extremely difficult to use these databases for complex use cases. The more tables you have, the more relational databases will struggle as  performance is degraded.

Graph databases, on the other hand, store data in graphs, which makes it easy to represent complex relationships between the data. This means that graph databases can easily handle large amounts of data more efficiently than relational databases. With this in mind, you should not have difficulties [choosing between graph databases and relational databases.](https://www.nebula-graph.io/posts/graph-database-vs-relational-database)

## Why graph database is a good choice?
A graph database is a good choice because it represents relationships between entities in a simple graph structure. In other words, it's well-suited for representing data with complex relationships.

A graph database can quickly traverse these relationships to find connections between points in the graph. This makes them ideal for complex applications such as fraud detection, network security, and recommendation engines.

# Exploring Graph Databases?
Check out [**NebulaGraph Database**](https://nebula-graph.io) and join the **open source** community to uncover more practical use cases of graph databases.

# How NebulaGraph Works
1. [NebulaGraph Architecture — A Bird’s Eye View](https://www.nebula-graph.io/posts/nebula-graph-architecture-overview)
2. [Benchmark: NebulaGraph vs Dgraph vs JanusGraph](https://www.nebula-graph.io/posts/benchmarking-mainstraim-graph-databases-dgraph-nebula-graph-janusgraph)
3. [Comparison: Neo4j vs NebulaGraph vs JanusGraph](https://www.nebula-graph.io/posts/performance-comparison-neo4j-janusgraph-nebula-graph)
4. [Deploy the Graph Database on Kubernetes](https://www.nebula-graph.io/posts/how-to-deploy-nebula-graph-in-kubernetes)




Unlike traditional relational or document-based databases, graph databases provide powerful search capabilities and enable superior performance when performing complex queries on a large dataset. 