along/secondo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
dfd1e745480fabd88139d2804acb90d5b6dc055e
secondo/Algebras/Distributed/ExampleDistributedAlgebra

836 lines
27 KiB
Plaintext
Raw Normal View History

firs commit
2026-01-23 17:03:45 +08:00
/*
//paragraph [10] title: [{\Large \bf ] [}]
//characters [1] formula: [$] [$]
//[ae] [\"{a}]
//[oe] [\"{o}]
//[ue] [\"{u}]
//[ss] [{\ss}]
//[Ae] [\"{A}]
//[Oe] [\"{O}]
//[Ue] [\"{U}]
//[**] [$**$]
//[toc] [\tableofcontents]
//[=>] [\verb+=>+]
//[:Section Translation] [\label{sec:translation}]
//[Section Translation] [Section~\ref{sec:translation}]
//[:Section 4.1.1] [\label{sec:4.1.1}]
//[Section 4.1.1] [Section~\ref{sec:4.1.1}]
//[Figure pog1] [Figure~\ref{fig:pog1.eps}]
//[Figure pog2] [Figure~\ref{fig:pog2.eps}]
//[newpage] [\newpage]
[10] Example: Creating a Distributed Spatial Database
Ralf Hartmut G[ue]ting, February 2015
[toc]
[newpage]
1 Introduction
In this example, we set up a small distributed spatial database, using the DistributedAlgebra of Secondo. We will use a cluster of six computers each with six cores and two disks. On each disk we use one Secondo server resulting in 12 nodes (called ~workers~). A controlling Secondo master sits on another computer.
On all systems the platform is Linux, more specifically Ubuntu 14.04.
2 Passphrase-less Connection
The first thing is to set up an ssh connection from the master to the workers so that the system on the master can log in to the workers without the user having to enter a password. This is done as follows.
(1) In the master's home directory, enter
---- ssh-keygen -t dsa
----
This generates a public key. The system asks for a password, just type return for an empty password. Also type return for the default location to store the public key.
(2) Configure ssh to know the worker computers. Into a file .ssh/config, we put the following:
----
Host *node1
HostName 132.176.69.181
User gueting
Host *node2
HostName 132.176.69.182
User gueting
Host *node3
HostName 132.176.69.183
User gueting
Host *node4
HostName 132.176.69.184
User gueting
Host *node5
HostName 132.176.69.185
User gueting
Host *node6
HostName 132.176.69.186
User gueting
----
This makes the worker computers available with names node1, node2, etc. and also sets the user name and home directory on these computers.
(3) Transmit the public key to the target computers. In this case, the home directory on node1 is mapped to all other nodes, so it suffices to do this on node1.
----
ssh-copy-id -i .ssh/id_dsa.pub <user>@server
ssh-copy-id -i .ssh/id_rsa.pub gueting@132.176.69.181
----
If the master system does not have the command ssh-copy-id, one can use instead:
----
cat ~/.ssh/*.pub | ssh <user>@<server> 'umask 077; cat >>.ssh/authorized_keys'
----
At this point the system on node1 asks once for a password.
Subsequently it is possible to log in on node1 by simply typing
---- ssh node1
----
One should log in once on each of the nodes because the system on the master needs to enter each node into its list of known hosts.
3 Getting Spatial Data to the Master
We use OpenStreetMap data about the German state of North-Rhine-Westphalia obtained in the form of shapefiles from GeoFabrik.
On http://download.geofabrik.de/ one can navigate a bit selecting Europe, then Germany. Then from the table Sub Regions in the row for Nordrhein-Westfalen, download
http://download.geofabrik.de/europe/germany/nordrhein-westfalen-latest.shp.zip
You can also use this link to download directly, if it is still available. Unpack the zip file into a directory nrw within secondo/bin.
The database has 8 kinds of objects, for example, Roads, Buildings, Waterways.
In directory secondo/bin start a Secondo system without transactions (for loading data) using the command
---- SecondoTTYNT
----
At the prompt, enter
----
Secondo => @Scripts/nrwImportShape.SEC
----
The contents of the file nrwImportShape are shown here:
----
# Importing NRW Data
close database
create database nrw
open database nrw
let Roads = dbimport2('../bin/nrw/roads.dbf') addcounter[No, 1]
shpimport2('../bin/nrw/roads.shp') namedtransformstream[GeoData]
addcounter[No2, 1] mergejoin[No, No2] remove[No, No2]
filter[isdefined(bbox(.GeoData))] validateAttr consume
let Waterways = dbimport2('../bin/nrw/waterways.dbf') addcounter[No, 1]
shpimport2('../bin/nrw/waterways.shp') namedtransformstream[GeoData]
addcounter[No2, 1] mergejoin[No, No2] remove[No, No2]
filter[isdefined(bbox(.GeoData))] validateAttr consume
let Railways = dbimport2('../bin/nrw/railways.dbf') addcounter[No, 1]
shpimport2('../bin/nrw/railways.shp') namedtransformstream[GeoData]
addcounter[No2, 1] mergejoin[No, No2] remove[No, No2]
filter[isdefined(bbox(.GeoData))] validateAttr consume
let Points = dbimport2('../bin/nrw/points.dbf') addcounter[No, 1]
shpimport2('../bin/nrw/points.shp') namedtransformstream[GeoData]
addcounter[No2, 1] mergejoin[No, No2] remove[No, No2]
filter[isdefined(bbox(.GeoData))] validateAttr consume
let Places = dbimport2('../bin/nrw/places.dbf') addcounter[No, 1]
shpimport2('../bin/nrw/places.shp') namedtransformstream[GeoData]
addcounter[No2, 1] mergejoin[No, No2] remove[No, No2]
filter[isdefined(bbox(.GeoData))] validateAttr consume
let Natural = dbimport2('../bin/nrw/natural.dbf') addcounter[No, 1]
shpimport2('../bin/nrw/natural.shp') namedtransformstream[GeoData]
addcounter[No2, 1] mergejoin[No, No2] remove[No, No2]
filter[isdefined(bbox(.GeoData))] validateAttr consume
let Buildings = dbimport2('../bin/nrw/buildings.dbf') addcounter[No, 1]
shpimport2('../bin/nrw/buildings.shp') namedtransformstream[GeoData]
addcounter[No2, 1] mergejoin[No, No2] remove[No, No2]
filter[isdefined(bbox(.GeoData))] validateAttr consume
let Landuse = dbimport2('../bin/nrw/landuse.dbf') addcounter[No, 1]
shpimport2('../bin/nrw/landuse.shp') namedtransformstream[GeoData]
addcounter[No2, 1] mergejoin[No, No2] remove[No, No2]
filter[isdefined(bbox(.GeoData))] validateAttr consume
----
This creates relations Roads, Waterways, etc. within a new database nrw.
Recently, this database is fairly large, for example, has 5.8 mio buildings. It took more than 3 hours to run this script.
4 Selecting a More Convenient Smaller Area
For our example, we do not want long waiting times and restrict to the area of the city of Dortmund. By some method (for example, loading some of the data into the GUI of Secondo and either reading coordinates or creating a rectangle there), we define a rectangle
----
let Dortmund = [const rect value (7.344992561477086 7.7006749345239855
51.42492667118238 51.610366440983924)]
----
The rectangle coordinates in parentheses follow the format (x1 x2 y1 y2).
We create a directory Dortmund located in the home directory of the user. We then run a script with contents
----
let BuildingsDo = Buildings feed filter[bbox(.GeoData) intersects Dortmund]
consume
query BuildingsDo count
save BuildingsDo to '../../Dortmund/Buildings'
let LanduseDo = Landuse feed filter[bbox(.GeoData) intersects Dortmund]
consume
query LanduseDo count
save LanduseDo to '../../Dortmund/Landuse'
let NaturalDo = Natural feed filter[bbox(.GeoData) intersects Dortmund]
consume
query NaturalDo count
save NaturalDo to '../../Dortmund/Natural'
let PlacesDo = Places feed filter[bbox(.GeoData) intersects Dortmund]
consume
query PlacesDo count
save PlacesDo to '../../Dortmund/Places'
let PointsDo = Points feed filter[bbox(.GeoData) intersects Dortmund]
consume
query PointsDo count
save PointsDo to '../../Dortmund/Points'
let RailwaysDo = Railways feed filter[bbox(.GeoData) intersects Dortmund]
consume
query RailwaysDo count
save RailwaysDo to '../../Dortmund/Railways'
let RoadsDo = Roads feed filter[bbox(.GeoData) intersects Dortmund]
consume
query RoadsDo count
save RoadsDo to '../../Dortmund/Roads'
let WaterwaysDo = Waterways feed filter[bbox(.GeoData) intersects Dortmund]
consume
query WaterwaysDo count
save WaterwaysDo to '../../Dortmund/Waterways'
----
Now subsets of all relations intersecting the area of rectangle Dortmund have been selected and saved in the directory Dortmund.
We close the database nrw and create a new database dortmund. This is done by a script with the following contents:
----
close database
create database dortmund
open database dortmund
restore Buildings from '../../Dortmund/Buildings'
restore Landuse from '../../Dortmund/Landuse'
restore Natural from '../../Dortmund/Natural'
restore Places from '../../Dortmund/Places'
restore Points from '../../Dortmund/Points'
restore Railways from '../../Dortmund/Railways'
restore Roads from '../../Dortmund/Roads'
restore Waterways from '../../Dortmund/Waterways'
# Runtime for ../../Scripts/buildDortmund.sec: Times (elapsed / cpu): 5:39min
# (339.269sec) /194.8sec = 1.74163
----
Of course, you may apply the same techniques to select an area you are interested in anywhere on the earth (as far as OSM and GeoFabrik data are available).
5 Preparations for the Distributed Database
In the cluster, we need to set up configuration files for the Secondo instances to be used as servers on the 12 disks. We assume, Secondo is already installed on node1 in the home directory of user gueting. The configuration files are slight variations of the standard SecondoConfig.ini and are to be located in directory secondo/bin as well. We call them SecondoConfig.ini.SM1 and SecondoConfig.ini.SM2. The essential entries in these configuration files that need to be changed are the path to the secondo-database directory to be used and the port number.
Here we use:
----
SecondoConfig.ini.SM1:
=====================
# Note: On windows you need absolute path names, e.g.
# C:\msys\1.0\home\user\secondo-dbtest
SecondoHome=/opt/psec/gueting-databases
# Port of the Secondo server (default: )
SecondoPort=31234
SecondoConfig.ini.SM2:
=====================
# Note: On windows you need absolute path names, e.g.
# C:\msys\1.0\home\user\secondo-dbtest
SecondoHome=/mnt/diskb/psec2/gueting-databases
# Port of the Secondo server (default: )
SecondoPort=34321
----
The files must also be put into the secondo/bin directory of the master and must be the same in the secondo/bin directory of the master and in the secondo/bin directory of node1 !!
In the database dortmund, we now set up two relations describing the cluster.
To be used for startup:
----
let StartCluster = [const rel(tuple([Server: string, Port: int, SecConf:string]))
value (
("node1" 31234 "SecondoConfig.ini.SM1")
("node2" 31234 "SecondoConfig.ini.SM1")
("node3" 31234 "SecondoConfig.ini.SM1")
("node4" 31234 "SecondoConfig.ini.SM1")
("node5" 31234 "SecondoConfig.ini.SM1")
("node6" 31234 "SecondoConfig.ini.SM1")
("node1" 34321 "SecondoConfig.ini.SM2")
("node2" 34321 "SecondoConfig.ini.SM2")
("node3" 34321 "SecondoConfig.ini.SM2")
("node4" 34321 "SecondoConfig.ini.SM2")
("node5" 34321 "SecondoConfig.ini.SM2")
("node6" 34321 "SecondoConfig.ini.SM2")
)]
----
For use in queries, we set up the relation:
----
let Cluster = [const rel(tuple([Server: string, Port: int])) value (
("132.176.69.181" 31234)
("132.176.69.182" 31234)
("132.176.69.183" 31234)
("132.176.69.184" 31234)
("132.176.69.185" 31234)
("132.176.69.186" 31234)
("132.176.69.181" 34321)
("132.176.69.182" 34321)
("132.176.69.183" 34321)
("132.176.69.184" 34321)
("132.176.69.185" 34321)
("132.176.69.186" 34321)
)]
----
6 Starting the Cluster
From now on, within all Secondo systems the DistributedAlgebra must have been activated. It is also important to ensure that the Secondo system on the master and the cluster node Secondos have the same algebras activated. Otherwise it may happen that a query that runs on the master does not work on the servers.
We start all the servers listed in relation StartCluster by the query:
----
query StartCluster feed extend[Started: startup(.Server, .Port, .SecConf, FALSE)]
consume
----
The query uses the startup operator of the DistributedAlgebra. The last parameter says whether the database Distributed should be created/restored to empty on the cluster. So it should be set to TRUE when using the cluster the first time and FALSE when restarting with an existing Distributed database.
[At the moment, automatic construction of the database Distributed does not work. It has to be constructed manually on each server. The query can only be used with parameter FALSE, or omitting this parameter.]
We can check whether all workers are ok:
----
query Cluster feed check_workers consume
----
It should have a result:
----
Server : 132.176.69.181
Port : 31234
Status : OK
Server : 132.176.69.182
Port : 31234
Status : OK
Server : 132.176.69.183
Port : 31234
Status : OK
Server : 132.176.69.184
Port : 31234
Status : OK
Server : 132.176.69.185
Port : 31234
Status : OK
Server : 132.176.69.186
Port : 31234
Status : OK
Server : 132.176.69.181
Port : 34321
Status : OK
Server : 132.176.69.182
Port : 34321
Status : OK
Server : 132.176.69.183
Port : 34321
Status : OK
Server : 132.176.69.184
Port : 34321
Status : OK
Server : 132.176.69.185
Port : 34321
Status : OK
Server : 132.176.69.186
Port : 34321
Status : OK
----
If this result appears, the cluster is ready for use.
If in the Status field we have an entry ERROR: bnl, it means that the database Distributed does not exist.
Finally, we can shut down all servers by
----
query StartCluster feed extend[Stop: shutdown(.Server, .Port)] consume
----
As an abbreviation, we can define functions:
----
let ClusterOn = fun() StartCluster feed
extend[Started: startup(.Server, .Port, .SecConf, FALSE)] consume;
let ClusterCheck = fun() Cluster feed check_workers consume;
let ClusterOff = fun() StartCluster feed
extend[Stop: shutdown(.Server, .Port)] consume;
----
We can then for example start the cluster simply by typing:
----
query ClusterOn()
----
We now assume the cluster is active with the result shown above for check\_workers.
7 Creating the Distributed Database dortmund
7.1 Random distribution
The most simple way to distribute relations is a random distribution.
----
let Buildings12r = Buildings feed extend[N: randint(12)] ddistribute[N, 12, Cluster]
----
Time: 1:36 min
This creates a distributed array of relations called Buildings12r. The naming convention says that it is the relation ~Buildings~, distributed into 12 array slots (one per worker in this case) and it is a random distribution (signified by ~r~).
Distribution is relatively slow, taking 1:36 minutes.
We can check how the random function has distributed elements into array slots:
----
query Buildings12r dloop[. count]
----
The dloop operator does not really loop but applies in parallel its parameter query ``. count'' to each element of the array, which is a relation with the same schema as ~Buildings~. The element of the array is referred to by ``.''.
Result of the dloop operation is a distributed array with fields of the result type of the query, hence an array of int in this case. The result is displayed as follows.
----
+++++++++++++++ BEGIN DARRAY +++++++++++++++
+++++++++++++++ Workers: +++++++++++++++
("132.176.69.181" 31234)
("132.176.69.182" 31234)
("132.176.69.183" 31234)
("132.176.69.184" 31234)
("132.176.69.185" 31234)
("132.176.69.186" 31234)
("132.176.69.181" 34321)
("132.176.69.182" 34321)
("132.176.69.183" 34321)
("132.176.69.184" 34321)
("132.176.69.185" 34321)
("132.176.69.186" 34321)
++++++++++++ DArray Index: 0 +++++++++++++++
23877
++++++++++++ DArray Index: 1 +++++++++++++++
23480
++++++++++++ DArray Index: 2 +++++++++++++++
23480
++++++++++++ DArray Index: 3 +++++++++++++++
23874
++++++++++++ DArray Index: 4 +++++++++++++++
23909
++++++++++++ DArray Index: 5 +++++++++++++++
23655
++++++++++++ DArray Index: 6 +++++++++++++++
23779
++++++++++++ DArray Index: 7 +++++++++++++++
23502
++++++++++++ DArray Index: 8 +++++++++++++++
23734
++++++++++++ DArray Index: 9 +++++++++++++++
23729
++++++++++++ DArray Index: 10 +++++++++++++++
23678
++++++++++++ DArray Index: 11 +++++++++++++++
23714
+++++++++++++++ END DARRAY +++++++++++++++
----
The total number of elements of the original ~Buildings~ relation is obtained by:
----
query Buildings count
----
The result is 284411.
We can check whether the sum of the numbers of all array fields is the same:
----
query Buildings12r dloop[. count] dtie[. + ..]
----
Result: 284411
Time: 32.35 seconds
The dtie operator applies an aggregate function to all fields of the distributed array to which it is applied. The aggregate function is specified by showing how it combines two arguments. These are refereed to by ``.'' and ``..'', respectively.
The running time for this query is about 32 seconds. As the time for executing the count query on a relation is practically 0, one can observe that there is considerable overhead in establishing connections to workers (to be improved).
As an example of executing a query on the distributed relation, suppose we want to find all Buildings of type "school".
----
query Buildings12r dloop[. feed filter[.Type contains "school"] consume]
dsummarize consume
----
Time: 36.65 seconds
Here in the dloop operator, we construct a distributed relation containing all school buildings. The dsummarize operator reads a distributed relation from all workers into a stream of tuples on the master. This stream is then collected into a relation by consume and returned.
As an example of a spatial query (a range query) we want to find all buildings in the suburb "Eichlinghofen". In the GUI, we create a rectangle around this suburb (as seen on the background map) and name it "Eichlinghofen".
A query on the master alone can the be written as:
----
query Buildings feed filter[bbox(.GeoData) inside Eichlinghofen] consume
----
Time: 2.56 seconds
It returns 2198 buildings.
The parallel query is:
----
query Buildings12r dloop[. feed filter[bbox(.GeoData) inside
[const rect value (7.39815473431277 7.420728205382089
51.473026681164775 51.48697825360396)]]
consume] dsummarize consume
----
Time: 29.95 seconds
It returns exactly the same 2198 buildings. The running time corresponds to the overhead of the parallel query in connecting with servers.
Note that we need to replace the database object Eichlinghofen by a constant in the distributed query, because Eichlinghofen is only known in the master database. By
----
query Eichlinghofen
----
we can get the values of the constant. A future version of the DistributedAlgebra should automatically provide this replacement.
7.2 Distribution by Standard Attribute
A random distribution is not suitable for performing joins between distributed relations as it would be necessary to consider in the join all pairs of partitions (144 in our case) with the need of moving data between nodes.
If we want to support joins by some standard attribute (i.e., int, real, string) we may distribute based on hash values of the attributes.
For example, we distribute Roads by their Osm\_id key value.
----
let Roads12_Osm_id =
Roads feed extend[N: hashvalue(.Osm_id, 999997) mod 12] ddistribute[N]
----
Time: 24.94 sec
A join on the Roads via Osm\_id can now be executed in parallel on the partitions, without losing any join tuples. We first perform the join locally on the master:
----
query Roads feed {r1} Roads feed {r2} itHashJoin[Osm_id_r1, Osm_id_r2]
count
----
Result: 67892
----
query Roads count
----
Result: 67892
The join combines each tuple exactly with itself because Osm\_id is a key. The parallel version is:
----
query Roads12_Osm_id Roads12_Osm_id dloopa[. feed {r1} .. feed {r2}
itHashJoin[Osm_id_r1, Osm_id_r2] count] dtie[. + ..]
----
Time: 53.91 seconds, 51.38
Result: 67892
The dloopa operator performs a binary operation on two distributed arrays, using slots with the same index on each worker. Indeed, the result number of tuples is the same as in the non-distributed join. - Why this query takes such a long time, is not clear.
7.3 Distribution by Spatial Attribute
Distribution by a spatial attribute is more complex but will enable us to perform spatial joins within partitions without losing join pairs.
First we need to determine a grid covering all spatial objects in the database. We could define a grid within the rectangle of nrw, but here we show how to determine the bounding box of just the elements of the Dortmund database.
----
query
Buildings feed projectextend[; Box: bbox(.GeoData)]
Landuse feed projectextend[; Box: bbox(.GeoData)] concat
Natural feed projectextend[; Box: bbox(.GeoData)] concat
Places feed projectextend[; Box: bbox(.GeoData)] concat
Points feed projectextend[; Box: bbox(.GeoData)] concat
Railways feed projectextend[; Box: bbox(.GeoData)] concat
Roads feed projectextend[; Box: bbox(.GeoData)] concat
Waterways feed projectextend[; Box: bbox(.GeoData)] concat
aggregateB[Box; fun(r1: rect, r2: rect) r1 union r2; [const rect value undef]]
----
The result is: [const rect value (7.29786 7.75405 51.3778 51.6712)]
Next we create a 20 x 20 grid covering this area. It is no problem if the grid is a bit larger than the enclosing rectangle.
----
let grid = [const cellgrid2d value (7.29 51.37 0.025 0.025 20)]
----
Here the five parameters for the grid are the left bottom corner (7.29 51.37), the cellwidth and cellheight (each 0.25) and the number of cells in a row (20).
With an operator called cellnumber we can now assign to each spatial object in the area of the grid the numbers of the grid cells which are covered by its bounding box. The cellnumber operator takes a rectangle and a grid definition and returns a stream of numbers of the cells covered by the rectangle.
----
let Buildings12s = Buildings feed
extendstream[Cell: cellnumber(bbox(.GeoData), grid)]
extend[N: .Cell mod 12] ddistribute[N, 12, Cluster]
----
Time: 2:15 min
Each node is in charge of the objects intersecting the grid cells that are mapped to its number between 0 and 11.
The query
----
query Buildings12s dloop[. count] dtie[. + ..]
----
returns 287754 whereas the original Buildings relation has 284411 objects. The reason is that some objects fall on cell boundaries and so are mapped to more than one node.
We use the spatial distribution as the main organisation of the database and therefore also distribute the other classes of objects in the same way:
----
let Landuse12s = Landuse feed
extendstream[Cell: cellnumber(bbox(.GeoData), grid)]
extend[N: .Cell mod 12] ddistribute[N, 12, Cluster]
let Natural12s = Natural feed
extendstream[Cell: cellnumber(bbox(.GeoData), grid)]
extend[N: .Cell mod 12] ddistribute[N, 12, Cluster]
let Places12s = Places feed
extendstream[Cell: cellnumber(bbox(.GeoData), grid)]
extend[N: .Cell mod 12] ddistribute[N, 12, Cluster]
let Points12s = Points feed
extendstream[Cell: cellnumber(bbox(.GeoData), grid)]
extend[N: .Cell mod 12] ddistribute[N, 12, Cluster]
let Railways12s = Railways feed
extendstream[Cell: cellnumber(bbox(.GeoData), grid)]
extend[N: .Cell mod 12] ddistribute[N, 12, Cluster]
let Roads12s = Roads feed
extendstream[Cell: cellnumber(bbox(.GeoData), grid)]
extend[N: .Cell mod 12] ddistribute[N, 12, Cluster]
let Waterways12s = Waterways feed
extendstream[Cell: cellnumber(bbox(.GeoData), grid)]
extend[N: .Cell mod 12] ddistribute[N, 12, Cluster]
----
We can see the objects that have been assigned to a single server using the get operation.
----
query get(Buildings12s, 8);
query get(Waterways12s, 8)
----
The get operation returns just one field of a distributed array, in this case field 8, containing a relation.
% If we use these commands in the Javagui, we get a figure like the following:
% Figure 1: Buildings and Waterways assigned to Server no. 8. [Server8.eps]
The spatial distribution supports spatial join which can now be done within each partition. For example, suppose we want to find intersections between Roads and Waterways. The query on the master alone is:
----
query Roads feed {r} Waterways feed {w}
itSpatialJoin[GeoData_r, GeoData_w]
filter[.GeoData_r intersects .GeoData_w]
extend[C: crossings(.GeoData_r, .GeoData_w)] count
Time: 9.6 sec
Result: 1328
----
The parallel query looks like this:
----
query Roads12s Waterways12s dloopa[. feed {r} .. feed {w}
itSpatialJoin[GeoData_r, GeoData_w]
filter[.GeoData_r intersects .GeoData_w]
extend[C: crossings(.GeoData_r, .GeoData_w)] count]
dtie[. + ..]
Time: 1:09 min
Result: 1408
----
We can observe that the resulting number is a bit higher than on the master. The reason is that some objects are duplicated at cell boundaries and we also get duplicate results in the spatial join (because the same two objects meet in different cells). However, there is a technique to avoid these duplicates.
----
query Roads12s Waterways12s dloopa[. feed {r} .. feed {w}
itSpatialJoin[GeoData_r, GeoData_w]
filter[(.Cell_r = .Cell_w) and
gridintersects([const cellgrid2d value (7.29 51.37 0.025 0.025 20)],
bbox(.GeoData_r), bbox(.GeoData_w), .Cell_r)]
filter[.GeoData_r intersects .GeoData_w]
extend[C: crossings(.GeoData_r, .GeoData_w)] count]
dtie[. + ..]
Time: 1:04 min
Result: 1328
----
First, we want to compute the spatial join only within cells. There may be objects assigned to one server within different cells that overlap. We remove these pairs by the condition
---- (.Cell_r = .Cell_w)
----
Second, the same objects may meet within different cells even on different servers. The operator gridintersects takes a grid definition, two bounding boxes, and a cell number. It computes the smallest intersection point of the two boxes. This point can lie only within one cell. If this is the cell whose cell number is given, the predicate returns TRUE, otherwise FALSE. That means that only for one cell and only one server will the intersection of these two objects be considered. In this way duplicates across servers are avoided.
We can see that the result number from the join is indeed the same as on a single machine.
7.4 Shuffling
Sometimes it is necessary to repartition a given distributed relation because a subsequent join requires another partitioning. This can be done by the dshuffle operator.
For example, suppose we are given the distributed relation Buildings12r (randomly distributed) of Section 7.1 but want to perform the join via Osm\_id of Section 7.2.
----
query
Buildings12r dshuffle1[hashvalue(.Osm_id, 999997) mod 12]
Buildings12r dshuffle1[hashvalue(.Osm_id, 999997) mod 12]
dloopa[. feed {r1} .. feed {r2}
itHashJoin[Osm_id_r1, Osm_id_r2] count] dtie[. + ..]
----
Unfortunately, this query does not finish on the cluster because the DistributedAlgebra creates 8 Secondo instances on a given node wheras only 6 cores are available. (To be improved.)
----
let Temp2 = Buildings12r dshuffle1[hashvalue(.Osm_id, 999997) mod 12]
query Temp2 Temp2 dloopa[. feed {r1} .. feed {r2}
itHashJoin[Osm_id_r1, Osm_id_r2] count] dtie[. + ..]
----
Result: 284411
The result shows that the join has been executed correctly on the partitioning of Temp2. The first query uses 4 cores, the second 6.
The dshuffle1 operator redistributes to the same workers and number of slots as the input distributed array has. There are variants to use a different number of slots or even another set of workers.
*/
Reference in New Issue Copy Permalink