forked from ee08b397/graphframes.github.io
-
Notifications
You must be signed in to change notification settings - Fork 0
/
user-guide.html
1072 lines (770 loc) · 87.9 KB
/
user-guide.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>User Guide - GraphFrames 0.3 Documentation</title>
<meta name="description" content="GraphFrames 0.3 user guide">
<link rel="stylesheet" href="css/bootstrap.min.css">
<style>
body {
padding-top: 60px;
padding-bottom: 40px;
}
</style>
<meta name="viewport" content="width=device-width">
<link rel="stylesheet" href="css/bootstrap-responsive.min.css">
<link rel="stylesheet" href="css/main.css">
<script src="js/vendor/modernizr-2.6.1-respond-1.1.0.min.js"></script>
<link rel="stylesheet" href="css/pygments-default.css">
</head>
<body>
<!--[if lt IE 7]>
<p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/">Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true">install Google Chrome Frame</a> to better experience this site.</p>
<![endif]-->
<!-- This code is taken from http://twitter.github.com/bootstrap/examples/hero.html -->
<div class="navbar navbar-fixed-top" id="topbar">
<div class="navbar-inner">
<div class="container">
<div class="brand"><a href="index.html">
GraphFrames</a><span class="version">0.3</span>
</div>
<ul class="nav">
<li><a href="index.html">Overview</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">User Guides<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="quick-start.html">Quick Start</a></li>
<li><a href="user-guide.html">GraphFrames User Guide</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="api/scala/index.html#org.graphframes.package">Scala</a></li>
<li><a href="api/python/index.html">Python</a></li>
</ul>
</li>
</ul>
<!--<p class="navbar-text pull-right"><span class="version-text">v0.3</span></p>-->
</div>
</div>
</div>
<div class="container" id="content">
<h1 class="title">GraphFrames User Guide</h1>
<p>This page gives examples of how to use GraphFrames for basic queries, motif finding, and
general graph algorithms. This includes code examples in Scala and Python.</p>
<ul id="markdown-toc">
<li><a href="#creating-graphframes" id="markdown-toc-creating-graphframes">Creating GraphFrames</a></li>
<li><a href="#basic-graph-and-dataframe-queries" id="markdown-toc-basic-graph-and-dataframe-queries">Basic graph and DataFrame queries</a></li>
<li><a href="#motif-finding" id="markdown-toc-motif-finding">Motif finding</a></li>
<li><a href="#subgraphs" id="markdown-toc-subgraphs">Subgraphs</a></li>
<li><a href="#graph-algorithms" id="markdown-toc-graph-algorithms">Graph algorithms</a> <ul>
<li><a href="#breadth-first-search-bfs" id="markdown-toc-breadth-first-search-bfs">Breadth-first search (BFS)</a></li>
<li><a href="#connected-components" id="markdown-toc-connected-components">Connected components</a> <ul>
<li><a href="#strongly-connected-components" id="markdown-toc-strongly-connected-components">Strongly connected components</a></li>
</ul>
</li>
<li><a href="#label-propagation-algorithm-lpa" id="markdown-toc-label-propagation-algorithm-lpa">Label Propagation Algorithm (LPA)</a></li>
<li><a href="#pagerank" id="markdown-toc-pagerank">PageRank</a></li>
<li><a href="#shortest-paths" id="markdown-toc-shortest-paths">Shortest paths</a></li>
<li><a href="#triangle-count" id="markdown-toc-triangle-count">Triangle count</a></li>
</ul>
</li>
<li><a href="#saving-and-loading-graphframes" id="markdown-toc-saving-and-loading-graphframes">Saving and loading GraphFrames</a></li>
<li><a href="#message-passing-via-aggregatemessages" id="markdown-toc-message-passing-via-aggregatemessages">Message passing via AggregateMessages</a></li>
<li><a href="#graphx-graphframe-conversions" id="markdown-toc-graphx-graphframe-conversions">GraphX-GraphFrame conversions</a> <ul>
<li><a href="#graphframe-to-graphx" id="markdown-toc-graphframe-to-graphx">GraphFrame to GraphX</a></li>
<li><a href="#graphx-to-graphframe" id="markdown-toc-graphx-to-graphframe">GraphX to GraphFrame</a></li>
<li><a href="#example-conversions" id="markdown-toc-example-conversions">Example conversions</a></li>
</ul>
</li>
</ul>
<p><em>Note: Most examples use the GraphFrame from the first subsection:
<a href="user-guide.html#creating-graphframes">Creating GraphFrames</a>.</em></p>
<h1 id="creating-graphframes">Creating GraphFrames</h1>
<p>Users can create GraphFrames from vertex and edge DataFrames.</p>
<ul>
<li><em>Vertex DataFrame</em>: A vertex DataFrame should contain a special column named “id” which specifies
unique IDs for each vertex in the graph.</li>
<li><em>Edge DataFrame</em>: An edge DataFrame should contain two special columns: “src” (source vertex ID
of edge) and “dst” (destination vertex ID of edge).</li>
</ul>
<p>Both DataFrames can have arbitrary other columns. Those columns can represent vertex and edge
attributes.</p>
<p>A GraphFrame can also be constructed from a single DataFrame containing edge information.
The vertices will be inferred from the sources and destinations of the edges.</p>
<div class="codetabs">
The following example demonstrates how to create a GraphFrame from vertex and edge DataFrames.
<div data-lang="scala">
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="c1">// Vertex DataFrame</span>
<span class="k">val</span> <span class="n">v</span> <span class="k">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="o">(</span><span class="nc">List</span><span class="o">(</span>
<span class="o">(</span><span class="s">"a"</span><span class="o">,</span> <span class="s">"Alice"</span><span class="o">,</span> <span class="mi">34</span><span class="o">),</span>
<span class="o">(</span><span class="s">"b"</span><span class="o">,</span> <span class="s">"Bob"</span><span class="o">,</span> <span class="mi">36</span><span class="o">),</span>
<span class="o">(</span><span class="s">"c"</span><span class="o">,</span> <span class="s">"Charlie"</span><span class="o">,</span> <span class="mi">30</span><span class="o">),</span>
<span class="o">(</span><span class="s">"d"</span><span class="o">,</span> <span class="s">"David"</span><span class="o">,</span> <span class="mi">29</span><span class="o">),</span>
<span class="o">(</span><span class="s">"e"</span><span class="o">,</span> <span class="s">"Esther"</span><span class="o">,</span> <span class="mi">32</span><span class="o">),</span>
<span class="o">(</span><span class="s">"f"</span><span class="o">,</span> <span class="s">"Fanny"</span><span class="o">,</span> <span class="mi">36</span><span class="o">),</span>
<span class="o">(</span><span class="s">"g"</span><span class="o">,</span> <span class="s">"Gabby"</span><span class="o">,</span> <span class="mi">60</span><span class="o">)</span>
<span class="o">)).</span><span class="n">toDF</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"name"</span><span class="o">,</span> <span class="s">"age"</span><span class="o">)</span>
<span class="c1">// Edge DataFrame</span>
<span class="k">val</span> <span class="n">e</span> <span class="k">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="o">(</span><span class="nc">List</span><span class="o">(</span>
<span class="o">(</span><span class="s">"a"</span><span class="o">,</span> <span class="s">"b"</span><span class="o">,</span> <span class="s">"friend"</span><span class="o">),</span>
<span class="o">(</span><span class="s">"b"</span><span class="o">,</span> <span class="s">"c"</span><span class="o">,</span> <span class="s">"follow"</span><span class="o">),</span>
<span class="o">(</span><span class="s">"c"</span><span class="o">,</span> <span class="s">"b"</span><span class="o">,</span> <span class="s">"follow"</span><span class="o">),</span>
<span class="o">(</span><span class="s">"f"</span><span class="o">,</span> <span class="s">"c"</span><span class="o">,</span> <span class="s">"follow"</span><span class="o">),</span>
<span class="o">(</span><span class="s">"e"</span><span class="o">,</span> <span class="s">"f"</span><span class="o">,</span> <span class="s">"follow"</span><span class="o">),</span>
<span class="o">(</span><span class="s">"e"</span><span class="o">,</span> <span class="s">"d"</span><span class="o">,</span> <span class="s">"friend"</span><span class="o">),</span>
<span class="o">(</span><span class="s">"d"</span><span class="o">,</span> <span class="s">"a"</span><span class="o">,</span> <span class="s">"friend"</span><span class="o">),</span>
<span class="o">(</span><span class="s">"a"</span><span class="o">,</span> <span class="s">"e"</span><span class="o">,</span> <span class="s">"friend"</span><span class="o">)</span>
<span class="o">)).</span><span class="n">toDF</span><span class="o">(</span><span class="s">"src"</span><span class="o">,</span> <span class="s">"dst"</span><span class="o">,</span> <span class="s">"relationship"</span><span class="o">)</span>
<span class="c1">// Create a GraphFrame</span>
<span class="k">val</span> <span class="n">g</span> <span class="k">=</span> <span class="nc">GraphFrame</span><span class="o">(</span><span class="n">v</span><span class="o">,</span> <span class="n">e</span><span class="o">)</span></code></pre></figure>
<p>The GraphFrame constructed above is available in the GraphFrames package:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span></code></pre></figure>
</div>
<div data-lang="python">
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c"># Vertex DataFrame</span>
<span class="n">v</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="p">(</span><span class="s">"a"</span><span class="p">,</span> <span class="s">"Alice"</span><span class="p">,</span> <span class="mi">34</span><span class="p">),</span>
<span class="p">(</span><span class="s">"b"</span><span class="p">,</span> <span class="s">"Bob"</span><span class="p">,</span> <span class="mi">36</span><span class="p">),</span>
<span class="p">(</span><span class="s">"c"</span><span class="p">,</span> <span class="s">"Charlie"</span><span class="p">,</span> <span class="mi">30</span><span class="p">),</span>
<span class="p">(</span><span class="s">"d"</span><span class="p">,</span> <span class="s">"David"</span><span class="p">,</span> <span class="mi">29</span><span class="p">),</span>
<span class="p">(</span><span class="s">"e"</span><span class="p">,</span> <span class="s">"Esther"</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span>
<span class="p">(</span><span class="s">"f"</span><span class="p">,</span> <span class="s">"Fanny"</span><span class="p">,</span> <span class="mi">36</span><span class="p">),</span>
<span class="p">(</span><span class="s">"g"</span><span class="p">,</span> <span class="s">"Gabby"</span><span class="p">,</span> <span class="mi">60</span><span class="p">)</span>
<span class="p">],</span> <span class="p">[</span><span class="s">"id"</span><span class="p">,</span> <span class="s">"name"</span><span class="p">,</span> <span class="s">"age"</span><span class="p">])</span>
<span class="c"># Edge DataFrame</span>
<span class="n">e</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="p">(</span><span class="s">"a"</span><span class="p">,</span> <span class="s">"b"</span><span class="p">,</span> <span class="s">"friend"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"b"</span><span class="p">,</span> <span class="s">"c"</span><span class="p">,</span> <span class="s">"follow"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"c"</span><span class="p">,</span> <span class="s">"b"</span><span class="p">,</span> <span class="s">"follow"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"f"</span><span class="p">,</span> <span class="s">"c"</span><span class="p">,</span> <span class="s">"follow"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"e"</span><span class="p">,</span> <span class="s">"f"</span><span class="p">,</span> <span class="s">"follow"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"e"</span><span class="p">,</span> <span class="s">"d"</span><span class="p">,</span> <span class="s">"friend"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"d"</span><span class="p">,</span> <span class="s">"a"</span><span class="p">,</span> <span class="s">"friend"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"a"</span><span class="p">,</span> <span class="s">"e"</span><span class="p">,</span> <span class="s">"friend"</span><span class="p">)</span>
<span class="p">],</span> <span class="p">[</span><span class="s">"src"</span><span class="p">,</span> <span class="s">"dst"</span><span class="p">,</span> <span class="s">"relationship"</span><span class="p">])</span>
<span class="c"># Create a GraphFrame</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">GraphFrame</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span></code></pre></figure>
<p>The GraphFrame constructed above is available in the GraphFrames package:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span></code></pre></figure>
</div>
</div>
<h1 id="basic-graph-and-dataframe-queries">Basic graph and DataFrame queries</h1>
<p>GraphFrames provide several simple graph queries, such as node degree.</p>
<p>Also, since GraphFrames represent graphs as pairs of vertex and edge DataFrames, it is easy to make
powerful queries directly on the vertex and edge DataFrames. Those DataFrames are made available
as <code>vertices</code> and <code>edges</code> fields in the GraphFrame.</p>
<div class="codetabs">
<div data-lang="scala">
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="c1">// Display the vertex and edge DataFrames</span>
<span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">show</span><span class="o">()</span>
<span class="c1">// +--+-------+---+</span>
<span class="c1">// |id| name|age|</span>
<span class="c1">// +--+-------+---+</span>
<span class="c1">// | a| Alice| 34|</span>
<span class="c1">// | b| Bob| 36|</span>
<span class="c1">// | c|Charlie| 30|</span>
<span class="c1">// | d| David| 29|</span>
<span class="c1">// | e| Esther| 32|</span>
<span class="c1">// | f| Fanny| 36|</span>
<span class="c1">// | g| Gabby| 60|</span>
<span class="c1">// +--+-------+---+</span>
<span class="n">g</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">show</span><span class="o">()</span>
<span class="c1">// +---+---+------------+</span>
<span class="c1">// |src|dst|relationship|</span>
<span class="c1">// +---+---+------------+</span>
<span class="c1">// | a| b| friend|</span>
<span class="c1">// | b| c| follow|</span>
<span class="c1">// | c| b| follow|</span>
<span class="c1">// | f| c| follow|</span>
<span class="c1">// | e| f| follow|</span>
<span class="c1">// | e| d| friend|</span>
<span class="c1">// | d| a| friend|</span>
<span class="c1">// | a| e| friend|</span>
<span class="c1">// +---+---+------------+</span>
<span class="c1">// import Spark SQL package</span>
<span class="k">import</span> <span class="nn">org.apache.spark.sql.DataFrame</span>
<span class="c1">// Get a DataFrame with columns "id" and "inDeg" (in-degree)</span>
<span class="k">val</span> <span class="n">vertexInDegrees</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">inDegrees</span>
<span class="c1">// Find the youngest user's age in the graph.</span>
<span class="c1">// This queries the vertex DataFrame.</span>
<span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">groupBy</span><span class="o">().</span><span class="n">min</span><span class="o">(</span><span class="s">"age"</span><span class="o">).</span><span class="n">show</span><span class="o">()</span>
<span class="c1">// Count the number of "follows" in the graph.</span>
<span class="c1">// This queries the edge DataFrame.</span>
<span class="k">val</span> <span class="n">numFollows</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">filter</span><span class="o">(</span><span class="s">"relationship = 'follow'"</span><span class="o">).</span><span class="n">count</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="c"># Display the vertex and edge DataFrames</span>
<span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c"># +--+-------+---+</span>
<span class="c"># |id| name|age|</span>
<span class="c"># +--+-------+---+</span>
<span class="c"># | a| Alice| 34|</span>
<span class="c"># | b| Bob| 36|</span>
<span class="c"># | c|Charlie| 30|</span>
<span class="c"># | d| David| 29|</span>
<span class="c"># | e| Esther| 32|</span>
<span class="c"># | f| Fanny| 36|</span>
<span class="c"># | g| Gabby| 60|</span>
<span class="c"># +--+-------+---+</span>
<span class="n">g</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c"># +---+---+------------+</span>
<span class="c"># |src|dst|relationship|</span>
<span class="c"># +---+---+------------+</span>
<span class="c"># | a| b| friend|</span>
<span class="c"># | b| c| follow|</span>
<span class="c"># | c| b| follow|</span>
<span class="c"># | f| c| follow|</span>
<span class="c"># | e| f| follow|</span>
<span class="c"># | e| d| friend|</span>
<span class="c"># | d| a| friend|</span>
<span class="c"># | a| e| friend|</span>
<span class="c"># +---+---+------------+</span>
<span class="c"># Get a DataFrame with columns "id" and "inDegree" (in-degree)</span>
<span class="n">vertexInDegrees</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">inDegrees</span>
<span class="c"># Find the youngest user's age in the graph.</span>
<span class="c"># This queries the vertex DataFrame.</span>
<span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">groupBy</span><span class="p">()</span><span class="o">.</span><span class="n">min</span><span class="p">(</span><span class="s">"age"</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c"># Count the number of "follows" in the graph.</span>
<span class="c"># This queries the edge DataFrame.</span>
<span class="n">numFollows</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s">"relationship = 'follow'"</span><span class="p">)</span><span class="o">.</span><span class="n">count</span><span class="p">()</span></code></pre></figure>
</div>
</div>
<h1 id="motif-finding">Motif finding</h1>
<p>Motif finding refers to searching for structural patterns in a graph.</p>
<p>GraphFrame motif finding uses a simple Domain-Specific Language (DSL) for expressing structural
queries. For example, <code>graph.find("(a)-[e]->(b); (b)-[e2]->(a)")</code> will search for pairs of vertices
<code>a,b</code> connected by edges in both directions. It will return a <code>DataFrame</code> of all such
structures in the graph, with columns for each of the named elements (vertices or edges)
in the motif. In this case, the returned columns will be “a, b, e, e2.”</p>
<p>DSL for expressing structural patterns:</p>
<ul>
<li>The basic unit of a pattern is an edge.
For example, <code>"(a)-[e]->(b)"</code> expresses an edge <code>e</code> from vertex <code>a</code> to vertex <code>b</code>.
Note that vertices are denoted by parentheses <code>(a)</code>, while edges are denoted by
square brackets <code>[e]</code>.</li>
<li>A pattern is expressed as a union of edges. Edge patterns can be joined with semicolons.
Motif <code>"(a)-[e]->(b); (b)-[e2]->(c)"</code> specifies two edges from <code>a</code> to <code>b</code> to <code>c</code>.</li>
<li>Within a pattern, names can be assigned to vertices and edges. For example,
<code>"(a)-[e]->(b)"</code> has three named elements: vertices <code>a,b</code> and edge <code>e</code>.
These names serve two purposes:
<ul>
<li>The names can identify common elements among edges. For example,
<code>"(a)-[e]->(b); (b)-[e2]->(c)"</code> specifies that the same vertex <code>b</code> is the destination
of edge <code>e</code> and source of edge <code>e2</code>.</li>
<li>The names are used as column names in the result <code>DataFrame</code>. If a motif contains
named vertex <code>a</code>, then the result <code>DataFrame</code> will contain a column “a” which is a
<code>StructType</code> with sub-fields equivalent to the schema (columns) of
[[GraphFrame.vertices]]. Similarly, an edge <code>e</code> in a motif will produce a column “e”
in the result <code>DataFrame</code> with sub-fields equivalent to the schema (columns) of
[[GraphFrame.edges]].</li>
</ul>
</li>
<li>It is acceptable to omit names for vertices or edges in motifs when not needed.
E.g., <code>"(a)-[]->(b)"</code> expresses an edge between vertices <code>a,b</code> but does not assign a name
to the edge. There will be no column for the anonymous edge in the result <code>DataFrame</code>.
Similarly, <code>"(a)-[e]->()"</code> indicates an out-edge of vertex <code>a</code> but does not name
the destination vertex.</li>
<li>An edge can be negated to indicate that the edge should <em>not</em> be present in the graph.
E.g., <code>"(a)-[]->(b); !(b)-[]->(a)"</code> finds edges from <code>a</code> to <code>b</code> for which there is <em>no</em>
edge from <code>b</code> to <code>a</code>.</li>
</ul>
<p>More complex queries, such as queries which operate on vertex or edge attributes,
can be expressed by applying filters to the result <code>DataFrame</code>.</p>
<div class="codetabs">
<div data-lang="scala">
<p>For API details, refer to the <a href="api/scala/index.html#org.graphframes.GraphFrame">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="c1">// Search for pairs of vertices with edges in both directions between them.</span>
<span class="k">val</span> <span class="n">motifs</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">find</span><span class="o">(</span><span class="s">"(a)-[e]->(b); (b)-[e2]->(a)"</span><span class="o">)</span>
<span class="n">motifs</span><span class="o">.</span><span class="n">show</span><span class="o">()</span>
<span class="c1">// More complex queries can be expressed by applying filters.</span>
<span class="n">motifs</span><span class="o">.</span><span class="n">filter</span><span class="o">(</span><span class="s">"b.age > 30"</span><span class="o">).</span><span class="n">show</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<p>For API details, refer to the <a href="api/python/graphframes.html#graphframes.GraphFrame.find">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="c"># Search for pairs of vertices with edges in both directions between them.</span>
<span class="n">motifs</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s">"(a)-[e]->(b); (b)-[e2]->(a)"</span><span class="p">)</span>
<span class="n">motifs</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c"># More complex queries can be expressed by applying filters.</span>
<span class="n">motifs</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s">"b.age > 30"</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span></code></pre></figure>
</div>
</div>
<p>Many motif queries are stateless and simple to express, as in the examples above.
The next examples demonstrate more complex queries which carry state along a path in the motif.
These queries can be expressed by combining GraphFrame motif finding with filters on the result,
where the filters use sequence operations to construct a series of <code>DataFrame</code> <code>Column</code>s.</p>
<p>For example, suppose one wishes to identify a chain of 4 vertices with some property defined
by a sequence of functions. That is, among chains of 4 vertices <code>a->b->c->d</code>, identify the subset
of chains matching this complex filter:</p>
<ul>
<li>Initialize state on path.</li>
<li>Update state based on vertex <code>a</code>.</li>
<li>Update state based on vertex <code>b</code>.</li>
<li>Etc. for <code>c</code> and <code>d</code>.</li>
<li>If final state matches some condition, then the chain is accepted by the filter.</li>
</ul>
<p>The below code snippets demonstrate this process, where we identify chains of 4 vertices
such that at least 2 of the 3 edges are “friend” relationships.
In this example, the state is the current count of “friend” edges; in general, it could be any
<code>DataFrame Column</code>.</p>
<div class="codetabs">
<div data-lang="scala">
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.apache.spark.sql.Column</span>
<span class="k">import</span> <span class="nn">org.apache.spark.sql.functions.</span><span class="o">{</span><span class="n">col</span><span class="o">,</span> <span class="n">when</span><span class="o">}</span>
<span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span> <span class="k">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="c1">// Find chains of 4 vertices.</span>
<span class="k">val</span> <span class="n">chain4</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">find</span><span class="o">(</span><span class="s">"(a)-[ab]->(b); (b)-[bc]->(c); (c)-[cd]->(d)"</span><span class="o">)</span>
<span class="c1">// Query on sequence, with state (cnt)</span>
<span class="c1">// (a) Define method for updating state given the next element of the motif.</span>
<span class="k">def</span> <span class="n">sumFriends</span><span class="o">(</span><span class="n">cnt</span><span class="k">:</span> <span class="kt">Column</span><span class="o">,</span> <span class="n">relationship</span><span class="k">:</span> <span class="kt">Column</span><span class="o">)</span><span class="k">:</span> <span class="kt">Column</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">when</span><span class="o">(</span><span class="n">relationship</span> <span class="o">===</span> <span class="s">"friend"</span><span class="o">,</span> <span class="n">cnt</span> <span class="o">+</span> <span class="mi">1</span><span class="o">).</span><span class="n">otherwise</span><span class="o">(</span><span class="n">cnt</span><span class="o">)</span>
<span class="o">}</span>
<span class="c1">// (b) Use sequence operation to apply method to sequence of elements in motif.</span>
<span class="c1">// In this case, the elements are the 3 edges.</span>
<span class="k">val</span> <span class="n">condition</span> <span class="k">=</span> <span class="nc">Seq</span><span class="o">(</span><span class="s">"ab"</span><span class="o">,</span> <span class="s">"bc"</span><span class="o">,</span> <span class="s">"cd"</span><span class="o">).</span>
<span class="n">foldLeft</span><span class="o">(</span><span class="n">lit</span><span class="o">(</span><span class="mi">0</span><span class="o">))((</span><span class="n">cnt</span><span class="o">,</span> <span class="n">e</span><span class="o">)</span> <span class="k">=></span> <span class="n">sumFriends</span><span class="o">(</span><span class="n">cnt</span><span class="o">,</span> <span class="n">col</span><span class="o">(</span><span class="n">e</span><span class="o">)(</span><span class="s">"relationship"</span><span class="o">)))</span>
<span class="c1">// (c) Apply filter to DataFrame.</span>
<span class="k">val</span> <span class="n">chainWith2Friends2</span> <span class="k">=</span> <span class="n">chain4</span><span class="o">.</span><span class="n">where</span><span class="o">(</span><span class="n">condition</span> <span class="o">>=</span> <span class="mi">2</span><span class="o">)</span>
<span class="n">chainWith2Friends2</span><span class="o">.</span><span class="n">show</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">pyspark.sql.functions</span> <span class="kn">import</span> <span class="n">col</span><span class="p">,</span> <span class="n">lit</span><span class="p">,</span> <span class="n">udf</span><span class="p">,</span> <span class="n">when</span>
<span class="kn">from</span> <span class="nn">pyspark.sql.types</span> <span class="kn">import</span> <span class="n">IntegerType</span>
<span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="n">chain4</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s">"(a)-[ab]->(b); (b)-[bc]->(c); (c)-[cd]->(d)"</span><span class="p">)</span>
<span class="c"># Query on sequence, with state (cnt)</span>
<span class="c"># (a) Define method for updating state given the next element of the motif.</span>
<span class="n">sumFriends</span> <span class="o">=</span>\
<span class="k">lambda</span> <span class="n">cnt</span><span class="p">,</span><span class="n">relationship</span><span class="p">:</span> <span class="n">when</span><span class="p">(</span><span class="n">relationship</span> <span class="o">==</span> <span class="s">"friend"</span><span class="p">,</span> <span class="n">cnt</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">otherwise</span><span class="p">(</span><span class="n">cnt</span><span class="p">)</span>
<span class="c"># (b) Use sequence operation to apply method to sequence of elements in motif.</span>
<span class="c"># In this case, the elements are the 3 edges.</span>
<span class="n">condition</span> <span class="o">=</span>\
<span class="nb">reduce</span><span class="p">(</span><span class="k">lambda</span> <span class="n">cnt</span><span class="p">,</span><span class="n">e</span><span class="p">:</span> <span class="n">sumFriends</span><span class="p">(</span><span class="n">cnt</span><span class="p">,</span> <span class="n">col</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="o">.</span><span class="n">relationship</span><span class="p">),</span> <span class="p">[</span><span class="s">"ab"</span><span class="p">,</span> <span class="s">"bc"</span><span class="p">,</span> <span class="s">"cd"</span><span class="p">],</span> <span class="n">lit</span><span class="p">(</span><span class="mi">0</span><span class="p">))</span>
<span class="c"># (c) Apply filter to DataFrame.</span>
<span class="n">chainWith2Friends2</span> <span class="o">=</span> <span class="n">chain4</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">condition</span> <span class="o">>=</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">chainWith2Friends2</span><span class="o">.</span><span class="n">show</span><span class="p">()</span></code></pre></figure>
</div>
</div>
<p>The above example demonstrated a stateful motif for a fixed-length chain. Currently, in order to
search for variable-length motifs, users need to run one query for each possible length.
However, the above query patterns allow users to re-use the same code for each length, with the
only change being to update the sequence of motif elements (“ab”, “bc”, “cd” above).</p>
<h1 id="subgraphs">Subgraphs</h1>
<p>In GraphX, the <code>subgraph()</code> method takes an edge triplet (edge, src vertex, and dst vertex, plus
attributes) and allows the user to select a subgraph based on triplet and vertex filters.</p>
<p>GraphFrames provide an even more powerful way to select subgraphs based on a combination of
motif finding and DataFrame filters.</p>
<p><strong>Simple subgraph: vertex and edge filters</strong>:
The following example shows how to select a subgraph based upon vertex and edge filters.</p>
<div class="codetabs">
<div data-lang="scala">
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span>
<span class="c1">// Select subgraph of users older than 30, and edges of type "friend"</span>
<span class="k">val</span> <span class="n">v2</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">filter</span><span class="o">(</span><span class="s">"age > 30"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">e2</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">filter</span><span class="o">(</span><span class="s">"relationship = 'friend'"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">g2</span> <span class="k">=</span> <span class="nc">GraphFrame</span><span class="o">(</span><span class="n">v2</span><span class="o">,</span> <span class="n">e2</span><span class="o">)</span></code></pre></figure>
</div>
<div data-lang="python">
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="c"># Select subgraph of users older than 30, and edges of type "friend"</span>
<span class="n">v2</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s">"age > 30"</span><span class="p">)</span>
<span class="n">e2</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s">"relationship = 'friend'"</span><span class="p">)</span>
<span class="n">g2</span> <span class="o">=</span> <span class="n">GraphFrame</span><span class="p">(</span><span class="n">v2</span><span class="p">,</span> <span class="n">e2</span><span class="p">)</span></code></pre></figure>
</div>
</div>
<p><strong>Complex subgraph: triplet filters</strong>:
The following example shows how to select a subgraph based upon triplet filters which
operate on an edge and its src and dst vertices. This example could be extended to go beyond
triplets by using more complex motifs.</p>
<div class="codetabs">
<div data-lang="scala">
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="c1">// Select subgraph based on edges "e" of type "follow"</span>
<span class="c1">// pointing from a younger user "a" to an older user "b".</span>
<span class="k">val</span> <span class="n">paths</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">find</span><span class="o">(</span><span class="s">"(a)-[e]->(b)"</span><span class="o">)</span>
<span class="o">.</span><span class="n">filter</span><span class="o">(</span><span class="s">"e.relationship = 'follow'"</span><span class="o">)</span>
<span class="o">.</span><span class="n">filter</span><span class="o">(</span><span class="s">"a.age < b.age"</span><span class="o">)</span>
<span class="c1">// "paths" contains vertex info. Extract the edges.</span>
<span class="k">val</span> <span class="n">e2</span> <span class="k">=</span> <span class="n">paths</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="s">"e.src"</span><span class="o">,</span> <span class="s">"e.dst"</span><span class="o">,</span> <span class="s">"e.relationship"</span><span class="o">)</span>
<span class="c1">// In Spark 1.5+, the user may simplify this call:</span>
<span class="c1">// val e2 = paths.select("e.*")</span>
<span class="c1">// Construct the subgraph</span>
<span class="k">val</span> <span class="n">g2</span> <span class="k">=</span> <span class="nc">GraphFrame</span><span class="o">(</span><span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="o">,</span> <span class="n">e2</span><span class="o">)</span></code></pre></figure>
</div>
<div data-lang="python">
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="c"># Select subgraph based on edges "e" of type "follow"</span>
<span class="c"># pointing from a younger user "a" to an older user "b".</span>
<span class="n">paths</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s">"(a)-[e]->(b)"</span><span class="p">)</span>\
<span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s">"e.relationship = 'follow'"</span><span class="p">)</span>\
<span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s">"a.age < b.age"</span><span class="p">)</span>
<span class="c"># "paths" contains vertex info. Extract the edges.</span>
<span class="n">e2</span> <span class="o">=</span> <span class="n">paths</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">"e.src"</span><span class="p">,</span> <span class="s">"e.dst"</span><span class="p">,</span> <span class="s">"e.relationship"</span><span class="p">)</span>
<span class="c"># In Spark 1.5+, the user may simplify this call:</span>
<span class="c"># val e2 = paths.select("e.*")</span>
<span class="c"># Construct the subgraph</span>
<span class="n">g2</span> <span class="o">=</span> <span class="n">GraphFrame</span><span class="p">(</span><span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="p">,</span> <span class="n">e2</span><span class="p">)</span></code></pre></figure>
</div>
</div>
<h1 id="graph-algorithms">Graph algorithms</h1>
<p>GraphFrames provides the same suite of standard graph algorithms as GraphX, plus some new ones.
We provide brief descriptions and code snippets below.
See the <a href="api/scala/index.html#org.graphframes.lib">API docs</a> for more details.</p>
<p>Some of the algorithms are currently wrappers around GraphX implementations, so they may not be
more scalable than GraphX. More algorithms will be migrated to native GraphFrames implementations
in the future.</p>
<h2 id="breadth-first-search-bfs">Breadth-first search (BFS)</h2>
<p>Breadth-first search (BFS) finds the shortest path(s) from one vertex (or a set of vertices)
to another vertex (or a set of vertices). The beginning and end vertices are specified as
Spark DataFrame expressions.</p>
<p>See <a href="https://en.wikipedia.org/wiki/Breadth-first_search">Wikipedia on BFS</a> for more background.</p>
<div class="codetabs">
The following code snippets search for people connected to the user "Bob."
<div data-lang="scala">
<p>For API details, refer to the <a href="api/scala/index.html#org.graphframes.lib.BFS">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="c1">// Search from "Esther" for users of age <= 32.</span>
<span class="k">val</span> <span class="n">paths</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">bfs</span><span class="o">.</span><span class="n">fromExpr</span><span class="o">(</span><span class="s">"name = 'Esther'"</span><span class="o">).</span><span class="n">toExpr</span><span class="o">(</span><span class="s">"age < 32"</span><span class="o">).</span><span class="n">run</span><span class="o">()</span>
<span class="n">paths</span><span class="o">.</span><span class="n">show</span><span class="o">()</span>
<span class="c1">// Specify edge filters or max path lengths.</span>
<span class="n">g</span><span class="o">.</span><span class="n">bfs</span><span class="o">.</span><span class="n">fromExpr</span><span class="o">(</span><span class="s">"name = 'Esther'"</span><span class="o">).</span><span class="n">toExpr</span><span class="o">(</span><span class="s">"age < 32"</span><span class="o">)</span>
<span class="o">.</span><span class="n">edgeFilter</span><span class="o">(</span><span class="s">"relationship != 'friend'"</span><span class="o">)</span>
<span class="o">.</span><span class="n">maxPathLength</span><span class="o">(</span><span class="mi">3</span><span class="o">)</span>
<span class="o">.</span><span class="n">run</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<p>For API details, refer to the <a href="api/python/graphframes.html#graphframes.GraphFrame.bfs">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="c"># Search from "Esther" for users of age < 32.</span>
<span class="n">paths</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">bfs</span><span class="p">(</span><span class="s">"name = 'Esther'"</span><span class="p">,</span> <span class="s">"age < 32"</span><span class="p">)</span>
<span class="n">paths</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c"># Specify edge filters or max path lengths.</span>
<span class="n">g</span><span class="o">.</span><span class="n">bfs</span><span class="p">(</span><span class="s">"name = 'Esther'"</span><span class="p">,</span> <span class="s">"age < 32"</span><span class="p">,</span>\
<span class="n">edgeFilter</span><span class="o">=</span><span class="s">"relationship != 'friend'"</span><span class="p">,</span> <span class="n">maxPathLength</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span></code></pre></figure>
</div>
</div>
<h2 id="connected-components">Connected components</h2>
<p>Computes the connected component membership of each vertex and returns a graph with each vertex
assigned a component ID.</p>
<p>See <a href="https://en.wikipedia.org/wiki/Connected_component_(graph_theory)">Wikipedia</a> for background.</p>
<div class="codetabs">
<div data-lang="scala">
<p>For API details, refer to the <a href="api/scala/index.html#org.graphframes.lib.ConnectedComponents">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="k">val</span> <span class="n">result</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">connectedComponents</span><span class="o">.</span><span class="n">run</span><span class="o">()</span>
<span class="n">result</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"component"</span><span class="o">).</span><span class="n">orderBy</span><span class="o">(</span><span class="s">"component"</span><span class="o">).</span><span class="n">show</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<p>For API details, refer to the <a href="api/python/graphframes.html#graphframes.GraphFrame.connectedComponents">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">connectedComponents</span><span class="p">()</span>
<span class="n">result</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">"id"</span><span class="p">,</span> <span class="s">"component"</span><span class="p">)</span><span class="o">.</span><span class="n">orderBy</span><span class="p">(</span><span class="s">"component"</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span></code></pre></figure>
</div>
</div>
<h3 id="strongly-connected-components">Strongly connected components</h3>
<p>Compute the strongly connected component (SCC) of each vertex and return a graph with each vertex
assigned to the SCC containing that vertex.</p>
<p>See <a href="https://en.wikipedia.org/wiki/Strongly_connected_component">Wikipedia</a> for background.</p>
<div class="codetabs">
<div data-lang="scala">
<p>For API details, refer to the <a href="api/scala/index.html#org.graphframes.lib.StronglyConnectedComponents">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="k">val</span> <span class="n">result</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">stronglyConnectedComponents</span><span class="o">.</span><span class="n">maxIter</span><span class="o">(</span><span class="mi">10</span><span class="o">).</span><span class="n">run</span><span class="o">()</span>
<span class="n">result</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"component"</span><span class="o">).</span><span class="n">orderBy</span><span class="o">(</span><span class="s">"component"</span><span class="o">).</span><span class="n">show</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<p>For API details, refer to the <a href="api/python/graphframes.html#graphframes.GraphFrame.stronglyConnectedComponents">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">stronglyConnectedComponents</span><span class="p">(</span><span class="n">maxIter</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
<span class="n">result</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">"id"</span><span class="p">,</span> <span class="s">"component"</span><span class="p">)</span><span class="o">.</span><span class="n">orderBy</span><span class="p">(</span><span class="s">"component"</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span></code></pre></figure>
</div>
</div>
<h2 id="label-propagation-algorithm-lpa">Label Propagation Algorithm (LPA)</h2>
<p>Run static Label Propagation Algorithm for detecting communities in networks.</p>
<p>Each node in the network is initially assigned to its own community. At every superstep, nodes
send their community affiliation to all neighbors and update their state to the mode community
affiliation of incoming messages.</p>
<p>LPA is a standard community detection algorithm for graphs. It is very inexpensive
computationally, although (1) convergence is not guaranteed and (2) one can end up with
trivial solutions (all nodes are identified into a single community).</p>
<p>See <a href="https://en.wikipedia.org/wiki/Label_Propagation_Algorithm">Wikipedia</a> for background.</p>
<div class="codetabs">
<div data-lang="scala">
<p>For API details, refer to the <a href="api/scala/index.html#org.graphframes.lib.LabelPropagation">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="k">val</span> <span class="n">result</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">labelPropagation</span><span class="o">.</span><span class="n">maxIter</span><span class="o">(</span><span class="mi">5</span><span class="o">).</span><span class="n">run</span><span class="o">()</span>
<span class="n">result</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"label"</span><span class="o">).</span><span class="n">show</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<p>For API details, refer to the <a href="api/python/graphframes.html#graphframes.GraphFrame.labelPropagation">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">labelPropagation</span><span class="p">(</span><span class="n">maxIter</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="n">result</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">"id"</span><span class="p">,</span> <span class="s">"label"</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span></code></pre></figure>
</div>
</div>
<h2 id="pagerank">PageRank</h2>
<p>There are two implementations of PageRank.</p>
<ul>
<li>The first implementation uses the standalone [[GraphFrame]] interface and runs PageRank
for a fixed number of iterations. This can be run by setting <code>maxIter</code>.</li>
<li>The second implementation uses the <code>org.apache.spark.graphx.Pregel</code> interface and runs PageRank
until convergence. This can be run by setting <code>tol</code>.</li>
</ul>
<p>Both implementations support non-personalized and personalized PageRank, where setting a <code>sourceId</code>
personalizes the results for that vertex.</p>
<p>See <a href="https://en.wikipedia.org/wiki/PageRank">Wikipedia</a> for background.</p>
<div class="codetabs">
<div data-lang="scala">
<p>For API details, refer to the <a href="api/scala/index.html#org.graphframes.lib.PageRank">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="c1">// Run PageRank until convergence to tolerance "tol".</span>
<span class="k">val</span> <span class="n">results</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">pageRank</span><span class="o">.</span><span class="n">resetProbability</span><span class="o">(</span><span class="mf">0.15</span><span class="o">).</span><span class="n">tol</span><span class="o">(</span><span class="mf">0.01</span><span class="o">).</span><span class="n">run</span><span class="o">()</span>
<span class="c1">// Display resulting pageranks and final edge weights</span>
<span class="c1">// Note that the displayed pagerank may be truncated, e.g., missing the E notation.</span>
<span class="c1">// In Spark 1.5+, you can use show(truncate=false) to avoid truncation.</span>
<span class="n">results</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"pagerank"</span><span class="o">).</span><span class="n">show</span><span class="o">()</span>
<span class="n">results</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="s">"src"</span><span class="o">,</span> <span class="s">"dst"</span><span class="o">,</span> <span class="s">"weight"</span><span class="o">).</span><span class="n">show</span><span class="o">()</span>
<span class="c1">// Run PageRank for a fixed number of iterations.</span>
<span class="k">val</span> <span class="n">results2</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">pageRank</span><span class="o">.</span><span class="n">resetProbability</span><span class="o">(</span><span class="mf">0.15</span><span class="o">).</span><span class="n">maxIter</span><span class="o">(</span><span class="mi">10</span><span class="o">).</span><span class="n">run</span><span class="o">()</span>
<span class="c1">// Run PageRank personalized for vertex "a"</span>
<span class="k">val</span> <span class="n">results3</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">pageRank</span><span class="o">.</span><span class="n">resetProbability</span><span class="o">(</span><span class="mf">0.15</span><span class="o">).</span><span class="n">maxIter</span><span class="o">(</span><span class="mi">10</span><span class="o">).</span><span class="n">sourceId</span><span class="o">(</span><span class="s">"a"</span><span class="o">).</span><span class="n">run</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<p>For API details, refer to the <a href="api/python/graphframes.html#graphframes.GraphFrame.pageRank">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="c"># Run PageRank until convergence to tolerance "tol".</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">pageRank</span><span class="p">(</span><span class="n">resetProbability</span><span class="o">=</span><span class="mf">0.15</span><span class="p">,</span> <span class="n">tol</span><span class="o">=</span><span class="mf">0.01</span><span class="p">)</span>
<span class="c"># Display resulting pageranks and final edge weights</span>
<span class="c"># Note that the displayed pagerank may be truncated, e.g., missing the E notation.</span>
<span class="c"># In Spark 1.5+, you can use show(truncate=False) to avoid truncation.</span>
<span class="n">results</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">"id"</span><span class="p">,</span> <span class="s">"pagerank"</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">results</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">"src"</span><span class="p">,</span> <span class="s">"dst"</span><span class="p">,</span> <span class="s">"weight"</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c"># Run PageRank for a fixed number of iterations.</span>
<span class="n">results2</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">pageRank</span><span class="p">(</span><span class="n">resetProbability</span><span class="o">=</span><span class="mf">0.15</span><span class="p">,</span> <span class="n">maxIter</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
<span class="c"># Run PageRank personalized for vertex "a"</span>
<span class="n">results3</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">pageRank</span><span class="p">(</span><span class="n">resetProbability</span><span class="o">=</span><span class="mf">0.15</span><span class="p">,</span> <span class="n">maxIter</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">sourceId</span><span class="o">=</span><span class="s">"a"</span><span class="p">)</span></code></pre></figure>
</div>
</div>
<h2 id="shortest-paths">Shortest paths</h2>
<p>Computes shortest paths from each vertex to the given set of landmark vertices, where landmarks are
specified by vertex ID. Note that this takes edge direction into account.</p>
<p>See <a href="https://en.wikipedia.org/wiki/Shortest_path_problem">Wikipedia</a> for background.</p>
<div class="codetabs">
<div data-lang="scala">
<p>For API details, refer to the <a href="api/scala/index.html#org.graphframes.lib.ShortestPaths">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="k">val</span> <span class="n">results</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">shortestPaths</span><span class="o">.</span><span class="n">landmarks</span><span class="o">(</span><span class="nc">Seq</span><span class="o">(</span><span class="s">"a"</span><span class="o">,</span> <span class="s">"d"</span><span class="o">)).</span><span class="n">run</span><span class="o">()</span>
<span class="n">results</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"distances"</span><span class="o">).</span><span class="n">show</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<p>For API details, refer to the <a href="api/python/graphframes.html#graphframes.GraphFrame.shortestPaths">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">shortestPaths</span><span class="p">(</span><span class="n">landmarks</span><span class="o">=</span><span class="p">[</span><span class="s">"a"</span><span class="p">,</span> <span class="s">"d"</span><span class="p">])</span>
<span class="n">results</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">"id"</span><span class="p">,</span> <span class="s">"distances"</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span></code></pre></figure>
</div>
</div>
<h2 id="triangle-count">Triangle count</h2>
<p>Computes the number of triangles passing through each vertex.</p>
<div class="codetabs">
<div data-lang="scala">
<p>For API details, refer to the <a href="api/scala/index.html#org.graphframes.lib.TriangleCount">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="k">val</span> <span class="n">results</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">triangleCount</span><span class="o">.</span><span class="n">run</span><span class="o">()</span>
<span class="n">results</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"count"</span><span class="o">).</span><span class="n">show</span><span class="o">()</span></code></pre></figure>
</div>
<div data-lang="python">
<p>For API details, refer to the <a href="api/python/graphframes.html#graphframes.GraphFrame.triangleCount">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="n">triangleCount</span><span class="p">()</span>
<span class="n">results</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">"id"</span><span class="p">,</span> <span class="s">"count"</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span></code></pre></figure>
</div>
</div>
<h1 id="saving-and-loading-graphframes">Saving and loading GraphFrames</h1>
<p>Since GraphFrames are built around DataFrames, they automatically support saving and loading
to and from the same set of datasources.
Refer to the <a href="http://spark.apache.org/docs/latest/sql-programming-guide.html#data-sources">Spark SQL User Guide on datasources</a>
for more details.</p>
<p>The below example shows how to save and then load a graph.</p>
<div class="codetabs">
<div data-lang="scala">
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="c1">// Save vertices and edges as Parquet to some location.</span>
<span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">write</span><span class="o">.</span><span class="n">parquet</span><span class="o">(</span><span class="s">"hdfs://myLocation/vertices"</span><span class="o">)</span>
<span class="n">g</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">write</span><span class="o">.</span><span class="n">parquet</span><span class="o">(</span><span class="s">"hdfs://myLocation/edges"</span><span class="o">)</span>
<span class="c1">// Load the vertices and edges back.</span>
<span class="k">val</span> <span class="n">sameV</span> <span class="k">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">read</span><span class="o">.</span><span class="n">parquet</span><span class="o">(</span><span class="s">"hdfs://myLocation/vertices"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">sameE</span> <span class="k">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">read</span><span class="o">.</span><span class="n">parquet</span><span class="o">(</span><span class="s">"hdfs://myLocation/edges"</span><span class="o">)</span>
<span class="c1">// Create an identical GraphFrame.</span>
<span class="k">val</span> <span class="n">sameG</span> <span class="k">=</span> <span class="nc">GraphFrame</span><span class="o">(</span><span class="n">sameV</span><span class="o">,</span> <span class="n">sameE</span><span class="o">)</span></code></pre></figure>
</div>
<div data-lang="python">
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">graphframes.examples</span> <span class="kn">import</span> <span class="n">Graphs</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">Graphs</span><span class="p">(</span><span class="n">sqlContext</span><span class="p">)</span><span class="o">.</span><span class="n">friends</span><span class="p">()</span> <span class="c"># Get example graph</span>
<span class="c"># Save vertices and edges as Parquet to some location.</span>
<span class="n">g</span><span class="o">.</span><span class="n">vertices</span><span class="o">.</span><span class="n">write</span><span class="o">.</span><span class="n">parquet</span><span class="p">(</span><span class="s">"hdfs://myLocation/vertices"</span><span class="p">)</span>
<span class="n">g</span><span class="o">.</span><span class="n">edges</span><span class="o">.</span><span class="n">write</span><span class="o">.</span><span class="n">parquet</span><span class="p">(</span><span class="s">"hdfs://myLocation/edges"</span><span class="p">)</span>
<span class="c"># Load the vertices and edges back.</span>
<span class="n">sameV</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">read</span><span class="o">.</span><span class="n">parquet</span><span class="p">(</span><span class="s">"hdfs://myLocation/vertices"</span><span class="p">)</span>
<span class="n">sameE</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">read</span><span class="o">.</span><span class="n">parquet</span><span class="p">(</span><span class="s">"hdfs://myLocation/edges"</span><span class="p">)</span>
<span class="c"># Create an identical GraphFrame.</span>
<span class="n">sameG</span> <span class="o">=</span> <span class="n">GraphFrame</span><span class="p">(</span><span class="n">sameV</span><span class="p">,</span> <span class="n">sameE</span><span class="p">)</span></code></pre></figure>
</div>
</div>
<h1 id="message-passing-via-aggregatemessages">Message passing via AggregateMessages</h1>
<p>Like GraphX, GraphFrames provides primitives for developing graph algorithms.
The two key components are:</p>
<ul>
<li><code>aggregateMessages</code>: Send messages between vertices, and aggregate messages for each vertex.
GraphFrames provides a native <code>aggregateMessages</code> method implemented using DataFrame operations.
This may be used analogously to the GraphX API.</li>
<li>joins: Join message aggregates with the original graph.
GraphFrames rely on <code>DataFrame</code> joins, which provide the full functionality of GraphX joins.</li>
</ul>
<p>The below code snippets show how to use <code>aggregateMessages</code> to compute the sum of the ages
of adjacent users.</p>
<div class="codetabs">
<div data-lang="scala">
<p>For API details, refer to the <a href="api/scala/index.html#org.graphframes.lib.AggregateMessages">API docs</a>.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">import</span> <span class="nn">org.graphframes.examples</span>
<span class="k">import</span> <span class="nn">org.graphframes.lib.AggregateMessages</span>
<span class="k">val</span> <span class="n">g</span><span class="k">:</span> <span class="kt">GraphFrame</span> <span class="o">=</span> <span class="n">examples</span><span class="o">.</span><span class="nc">Graphs</span><span class="o">.</span><span class="n">friends</span> <span class="c1">// get example graph</span>
<span class="c1">// We will use AggregateMessages utilities later, so name it "AM" for short.</span>
<span class="k">val</span> <span class="nc">AM</span> <span class="k">=</span> <span class="nc">AggregateMessages</span>
<span class="c1">// For each user, sum the ages of the adjacent users.</span>
<span class="k">val</span> <span class="n">msgToSrc</span> <span class="k">=</span> <span class="nc">AM</span><span class="o">.</span><span class="n">dst</span><span class="o">(</span><span class="s">"age"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">msgToDst</span> <span class="k">=</span> <span class="nc">AM</span><span class="o">.</span><span class="n">src</span><span class="o">(</span><span class="s">"age"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">agg</span> <span class="k">=</span> <span class="n">g</span><span class="o">.</span><span class="n">aggregateMessages</span>
<span class="o">.</span><span class="n">sendToSrc</span><span class="o">(</span><span class="n">msgToSrc</span><span class="o">)</span> <span class="c1">// send destination user's age to source</span>
<span class="o">.</span><span class="n">sendToDst</span><span class="o">(</span><span class="n">msgToDst</span><span class="o">)</span> <span class="c1">// send source user's age to destination</span>
<span class="o">.</span><span class="n">agg</span><span class="o">(</span><span class="n">sum</span><span class="o">(</span><span class="nc">AM</span><span class="o">.</span><span class="n">msg</span><span class="o">).</span><span class="n">as</span><span class="o">(</span><span class="s">"summedAges"</span><span class="o">))</span> <span class="c1">// sum up ages, stored in AM.msg column</span>
<span class="n">agg</span><span class="o">.</span><span class="n">show</span><span class="o">()</span></code></pre></figure>
</div>
</div>
<p><code>AggregateMessages</code> does not yet have a Python API, but we expect to add one in the future.</p>
<p>For a more complex example, look at the code used to implement the
<a href="api/scala/index.html#org.graphframes.examples.BeliefPropagation$">Belief Propagation example</a>.</p>
<h1 id="graphx-graphframe-conversions">GraphX-GraphFrame conversions</h1>
<p>We provide utilities for converting between GraphFrame and GraphX graphs.
See the <a href="http://spark.apache.org/docs/latest/graphx-programming-guide.html">GraphX User Guide</a>
for details on GraphX.</p>
<h2 id="graphframe-to-graphx">GraphFrame to GraphX</h2>
<p>Conversion to GraphX creates a GraphX <code>Graph</code> which has <code>Long</code> vertex IDs and attributes
of type <code>Row</code>.</p>
<p>Vertex and edge attributes are the original rows in <code>vertices</code> and <code>edges</code>, respectively.</p>
<p>Note that vertex (and edge) attributes include vertex IDs (and source, destination IDs)
in order to support non-Long vertex IDs. If the vertex IDs are not convertible to Long values,
then the values are indexed in order to generate corresponding Long vertex IDs (which is an
expensive operation).</p>
<p>The column ordering of the returned <code>Graph</code> vertex and edge attributes are specified by
<code>GraphFrame.vertexColumns</code> and <code>GraphFrame.edgeColumns</code>, respectively.</p>
<h2 id="graphx-to-graphframe">GraphX to GraphFrame</h2>
<p>GraphFrame provides two conversions methods. The first takes any GraphX graph and converts
the vertex and edge <code>RDD</code>s into <code>DataFrame</code>s using schema inference. Those DataFrames
are then used to create a GraphFrame.</p>
<p>The second conversion method is more complex and is useful for users with existing GraphX code.
Its main purpose is to support workflows of the following form: (1) convert a GraphFrame to GraphX,
(2) run GraphX code to augment the GraphX graph with new vertex or edge attributes, and
(3) merge the new attributes back into the original GraphFrame.</p>
<p>For example, given:</p>
<ul>
<li>GraphFrame <code>originalGraph</code></li>
<li>GraphX <code>Graph[String, Int]</code> <code>graph</code> with a String vertex attribute we want to call “category”
and an Int edge attribute we want to call “count”</li>
</ul>
<p>We can call <code>fromGraphX(originalGraph, graph, Seq("category"), Seq("count"))</code> to produce
a new GraphFrame. The new GraphFrame will be an augmented version of <code>originalGraph</code>,
with new <code>GraphFrame.vertices</code> column “category” and new <code>GraphFrame.edges</code> column “count” added.</p>