HSR Texas Geo Database Benchmark

2010-02-26T08:19:14Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Get the scripts (Python 2.6+ required):
* Download the latest tar-archive: http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master (try reloading the url, the generation process takes some seconds)
* Or checkout using git: <code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for database specific import instructions.

Call the script generator in the <code>src</code> folder to get the available options: <code>python benchmark.py --help</code>. Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T15:03:15Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Get the scripts (Python 2.6+ required):
* Download the latest tar-archive: http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master
* Or checkout using git: <code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for database specific import instructions.

Call the script generator in the <code>src</code> folder to get the available options: <code>python benchmark.py --help</code>. Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T15:03:04Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Get the scripts (Python 2.6 required):
* Download the latest tar-archive: http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master
* Or checkout using git: <code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for database specific import instructions.

Call the script generator in the <code>src</code> folder to get the available options: <code>python benchmark.py --help</code>. Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T15:02:55Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Get the scripts (python 2.6 required):
* Download the latest tar-archive: http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master
* Or checkout using git: <code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for database specific import instructions.

Call the script generator in the <code>src</code> folder to get the available options: <code>python benchmark.py --help</code>. Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T14:54:43Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Get the scripts:
* Download the latest tar-archive: http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master
* Or checkout using git: <code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for database specific import instructions.

Call the script generator in the <code>src</code> folder to get the available options: <code>python benchmark.py --help</code>. Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T14:54:13Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Get the scripts:
* Download the latest tar-archive: http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master
* Or checkout using git: <code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for database specific import instructions.

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T14:49:09Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Get the scripts:
* Download the latest tar-archive: http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master
* Or checkout using git: <code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for import instructions.

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T14:48:58Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Get the scripts:
* Download the latest tar-archive: http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master
* Checkout using git: <code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for import instructions.

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T14:47:57Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Checkout the scripts using git: <code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code> or download the latest tar-archive: http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master

See <code>doc</code> folder for import instructions.

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T14:47:37Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Checkout the scripts using git:
<code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

Or download the latest tar-archive:
http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master

See <code>doc</code> folder for import instructions.

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T14:47:29Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Checkout the scripts using git:
<code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>
Or download the latest tar-archive:
http://gitorious.org/spatialbenchmark/spatialbenchmark/archive-tarball/master

See <code>doc</code> folder for import instructions.

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T09:09:06Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Checkout the scripts using git:
<code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for import instructions.

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> afterwards to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T09:08:43Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Checkout the scripts using git:
<code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for import instructions.

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1) or <code>-b</code> to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T09:08:25Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
Checkout the scripts using git:
<code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

See <code>doc</code> folder for import instructions.

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1), or <code>-b</code> to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-18T09:08:11Z

Dominik: /* Scripts for benchmark automation */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===
See <code>doc</code> folder for import instructions.

Checkout the scripts using git:
<code>git clone git://gitorious.org/spatialbenchmark/spatialbenchmark.git</code>

Call the script generator in the <code>src</code> folder to get the available options:
<code>python benchmark.py --help</code>

Use option <code>-l</code> first to load the datasubsets (see Query 1), or <code>-b</code> to run the benchmark. Specify the output filename with <code>-o FILE</code> or pipe the output directly to the sql interpreter (e.g. <code>python benchmark.py -d spatialite -l | spatialite texas.sqlite</code>).

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

Diskussion:HSR Texas Geo Database Benchmark

2010-02-18T07:39:17Z

Dominik: /* Review */

Feel free to discusss 'HSR Texas Spatial Database Benchmark' issues here! [[Benutzer:Stefan|Stefan]] 18:39, 22. Dez. 2009 (CET)

== To do ==
* Standardise data and input process (circumvent or minimize floating point problem)
* Define the variables (@bbox and @point)

== Observations ==

General:
* MS SQL Server representatives report that geography types are 20-30% slower than geometry types.
* About PostGIS 1.5 there was said, that geography types are somewhat faster (see http://blog.cleverelephant.ca/)
* Secondary index optimization makes performance about 50% faster (from: MS SQL Server discussion)
* The number of points of a query window is important for a secondary filter overhead.

Data and queries per se:
* These queries return quite large percent of the data overall (50%-100% for points and polygons)
* The Number of points for a query windows is very small.
* The query windows are "aligned bounding boxes" which may put in favor RTree (e.g. Postgres) or spatial grid (SQL Server) index, depending on a position of the window. If the window is aligned with the grid, performance can be much faster than when it's slightly off.

== Discussion Points ==

* Should we take an average of multiple runs with different values (@point, @bbox)?
* Should we vary the queries' result sizes? (see observations)
* Should we vary the shape and alignment of the query window? (see observations)
* Should we run the benchmark with parallelized queries?

== Review ==
* Query 1 actually shouldn't be included in the benchmark, data copying and preparation isn't really done often in a spatial database.

Diskussion:HSR Texas Geo Database Benchmark

2010-02-11T13:49:08Z

Dominik: /* Review */

Feel free to discusss 'HSR Texas Spatial Database Benchmark' issues here! [[Benutzer:Stefan|Stefan]] 18:39, 22. Dez. 2009 (CET)

== To do ==
* Standardise data and input process (circumvent or minimize floating point problem)
* Define the variables (@bbox and @point)

== Observations ==

General:
* MS SQL Server representatives report that geography types are 20-30% slower than geometry types.
* About PostGIS 1.5 there was said, that geography types are somewhat faster (see http://blog.cleverelephant.ca/)
* Secondary index optimization makes performance about 50% faster (from: MS SQL Server discussion)
* The number of points of a query window is important for a secondary filter overhead.

Data and queries per se:
* These queries return quite large percent of the data overall (50%-100% for points and polygons)
* The Number of points for a query windows is very small.
* The query windows are "aligned bounding boxes" which may put in favor RTree (e.g. Postgres) or spatial grid (SQL Server) index, depending on a position of the window. If the window is aligned with the grid, performance can be much faster than when it's slightly off.

== Discussion Points ==

* Should we take an average of multiple runs with different values (@point, @bbox)?
* Should we vary the queries' result sizes? (see observations)
* Should we vary the shape and alignment of the query window? (see observations)
* Should we run the benchmark with parallelized queries?

== Review ==
* Query 1 actually shouldn't be included in the benchmark, data copying and preparation isn't really done often in a spatial database.
* Query 2 is rather pointless without index on 'roadflg'. What should be proved with this query anyway?

HSR Texas Geo Database Benchmark

2010-02-11T08:32:49Z

Dominik: /* Query 2. A Non-spatial Selection: Count */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the 'roadflg' to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-11T08:32:24Z

Dominik: /* Query 5. Spatial Selection III: Intersect/Join Lines and Polygons */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the roadflg to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the 'railflg' to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-11T08:31:49Z

Dominik: /* Query 4. Spatial Selection II: Distance within */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the roadflg to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'. Make sure the index is used, which usually needs a vendor specific implementation.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the railflg to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-10T15:04:49Z

Dominik: /* Query 5. Spatial Selection III: Intersect/Join Lines and Polygons */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the roadflg to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area. We expect the railflg to have an index.

SELECT count(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-10T15:03:49Z

Dominik: /* Query 4. Spatial Selection II: Distance within */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the roadflg to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-10T15:03:30Z

Dominik: /* Query 3. Spatial Selection I: Intersect Point, Line and Polygons */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the roadflg to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'.

a)
SELECT COUNT(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT COUNT(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT COUNT(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-10T15:03:09Z

Dominik: /* Query 2. A Non-spatial Selection: Count */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads. We expect the roadflg to have an index.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT Count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT Count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT Count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'.

a)
SELECT COUNT(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT COUNT(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT COUNT(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-10T15:02:00Z

Dominik: /* Query 2. A Non-spatial Selection: Count */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads.

SELECT count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT Count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT Count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT Count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'.

a)
SELECT COUNT(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT COUNT(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT COUNT(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-10T13:25:22Z

Dominik: /* Query 4. Spatial Selection II: Distance within */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads.

SELECT Count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT Count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT Count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT Count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 20000 meters from a given point '@point'.

a)
SELECT COUNT(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 20000;

b)
SELECT COUNT(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 20000;

c)
SELECT COUNT(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 20000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

HSR Texas Geo Database Benchmark

2010-02-10T13:24:56Z

Dominik: /* Runtime variables for Queries 3 and 4 */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads.

SELECT Count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT Count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT Count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT Count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 1000 meters from a given point '@point'.

a)
SELECT COUNT(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 1000;

b)
SELECT COUNT(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 1000;

c)
SELECT COUNT(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 1000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: ST_MakeBox2D(ST_MakePoint(-102.311, 29.164), ST_MakePoint(-97.789, 32.836))
Points 60000: ST_MakeBox2D(ST_MakePoint(-102.7765, 28.786), ST_MakePoint(-97.3235, 33.214))
Points 80000: ST_MakeBox2D(ST_MakePoint(-103.1755, 28.462), ST_MakePoint(-96.9245, 33.538))
Points 100000: ST_MakeBox2D(ST_MakePoint(-103.508, 28.192), ST_MakePoint(-96.592, 33.808))
* Lines:
Line 500000: ST_MakeBox2D(ST_MakePoint(-101.743755, 29.62462), ST_MakePoint(-98.356245, 32.37538))
Line 1000000: ST_MakeBox2D(ST_MakePoint(-102.0982, 29.3368), ST_MakePoint(-98.0018, 32.6632))
Line 1500000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Line 2000000: ST_MakeBox2D(ST_MakePoint(-102.9893, 28.6132), ST_MakePoint(-97.1107, 33.3868))
Line 2500000: ST_MakeBox2D(ST_MakePoint(-103.242, 28.408), ST_MakePoint(-96.858, 33.592))
* Polygons:
Polygon 100000: ST_MakeBox2D(ST_MakePoint(-102.227875, 29.2315), ST_MakePoint(-97.872125, 32.7685))
Polygon 150000: ST_MakeBox2D(ST_MakePoint(-102.54375, 28.975), ST_MakePoint(-97.55625, 33.025))
Polygon 200000: ST_MakeBox2D(ST_MakePoint(-102.79645, 28.7698), ST_MakePoint(-97.30355, 33.2302))
Polygon 250000: ST_MakeBox2D(ST_MakePoint(-103.00925, 28.597), ST_MakePoint(-97.09075, 33.403))
Polygon 300000: ST_MakeBox2D(ST_MakePoint(-103.2686, 28.3864), ST_MakePoint(-96.8314, 33.6136))
Polygon 350000: ST_MakeBox2D(ST_MakePoint(-103.47475, 28.219), ST_MakePoint(-96.62525, 33.781))

Point data for Query 4
* Points:
Points 40000: ST_MakePoint(-102.311, 32.836)
Points 60000: ST_MakePoint(-102.7765, 33.214)
Points 80000: ST_MakePoint(-103.1755, 33.538)
Points 100000: ST_MakePoint(-103.508, 33.808)
* Lines:
Line 500000: ST_MakePoint(-101.743755, 30.37538)
Line 1000000: ST_MakePoint(-102.0982, 30.4632)
Line 1500000: ST_MakePoint(-102.54375, 31.025)
Line 2000000: ST_MakePoint(-102.9893, 29.3868)
Line 2500000: ST_MakePoint(-103.242, 30.592)
* Polygons:
Polygon 100000: ST_MakePoint(-101.227, 30.7685)
Polygon 150000: ST_MakePoint(-102.54375, 32.025)
Polygon 200000: ST_MakePoint(-102.79645, 32.2302)
Polygon 250000: ST_MakePoint(-102.009, 32.951)
Polygon 300000: ST_MakePoint(-102.2686, 32.1136)
Polygon 350000: ST_MakePoint(-102.47475, 33.781)

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark
* http://trac.osgeo.org/postgis/wiki/DevWikiGardenTest

Diskussion:HSR Texas Geo Database Benchmark

2010-02-09T08:03:24Z

Dominik: /* Review */

Feel free to discusss 'HSR Texas Spatial Database Benchmark' issues here! [[Benutzer:Stefan|Stefan]] 18:39, 22. Dez. 2009 (CET)

== To do ==
* Standardise data and input process (circumvent or minimize floating point problem)
* Define the variables (@bbox and @point)

== Observations ==

General:
* MS SQL Server representatives report that geography types are 20-30% slower than geometry types.
* About PostGIS 1.5 there was said, that geography types are somewhat faster (see http://blog.cleverelephant.ca/)
* Secondary index optimization makes performance about 50% faster (from: MS SQL Server discussion)
* The number of points of a query window is important for a secondary filter overhead.

Data and queries per se:
* These queries return quite large percent of the data overall (50%-100% for points and polygons)
* The Number of points for a query windows is very small.
* The query windows are "aligned bounding boxes" which may put in favor RTree (e.g. Postgres) or spatial grid (SQL Server) index, depending on a position of the window. If the window is aligned with the grid, performance can be much faster than when it's slightly off.

== Discussion Points ==

* Should we take an average of multiple runs with different values (@point, @bbox)?
* Should we vary the queries' result sizes? (see observations)
* Should we vary the shape and alignment of the query window? (see observations)
* Should we run the benchmark with parallelized queries?

== Review ==
* The limiting factor of Query 1 is mainly the performance of the harddisk.
* Query 1 is rather pointless without index on 'roadflg'. What should be proved with this query anyway?

Diskussion:HSR Texas Geo Database Benchmark

2010-02-09T08:02:10Z

Dominik: /* Review */

Feel free to discusss 'HSR Texas Spatial Database Benchmark' issues here! [[Benutzer:Stefan|Stefan]] 18:39, 22. Dez. 2009 (CET)

== To do ==
* Standardise data and input process (circumvent or minimize floating point problem)
* Define the variables (@bbox and @point)

== Observations ==

General:
* MS SQL Server representatives report that geography types are 20-30% slower than geometry types.
* About PostGIS 1.5 there was said, that geography types are somewhat faster (see http://blog.cleverelephant.ca/)
* Secondary index optimization makes performance about 50% faster (from: MS SQL Server discussion)
* The number of points of a query window is important for a secondary filter overhead.

Data and queries per se:
* These queries return quite large percent of the data overall (50%-100% for points and polygons)
* The Number of points for a query windows is very small.
* The query windows are "aligned bounding boxes" which may put in favor RTree (e.g. Postgres) or spatial grid (SQL Server) index, depending on a position of the window. If the window is aligned with the grid, performance can be much faster than when it's slightly off.

== Discussion Points ==

* Should we take an average of multiple runs with different values (@point, @bbox)?
* Should we vary the queries' result sizes? (see observations)
* Should we vary the shape and alignment of the query window? (see observations)
* Should we run the benchmark with parallelized queries?

== Review ==
* The limiting factor of Query 1 is mainly the performance of the harddisk.
* Query 1 is rather pointless without index on 'roadflg'

HSR Texas Geo Database Benchmark

2010-02-09T07:38:00Z

Dominik: /* Datasets */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''':
* 'Aligned with grid': e.g. <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.
* See chapter below.

Variable(s) '''@point''':
* E.g. <code>POINT(-101.3135 32.026)</code>.
* See chapter below.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads.

SELECT Count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT Count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT Count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT Count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 1000 meters from a given point '@point'.

a)
SELECT COUNT(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 1000;

b)
SELECT COUNT(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 1000;

c)
SELECT COUNT(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 1000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,002 || ST_MakeBox2D(ST_MakePoint(-103.208, 28.435), ST_MakePoint(-96.891, 33.460))
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,001 || ST_MakeBox2D(ST_MakePoint(-104.006, 27.787), ST_MakePoint(-96.093, 34.193))
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,001 || ST_MakeBox2D(ST_MakePoint(-104.904, 27.058), ST_MakePoint(-95.195, 34.830))
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,002 || ST_MakeBox2D(ST_MakePoint(-106.168, 26.032), ST_MakePoint(-93.932, 35.693))
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,161 || ST_MakeBox2D(ST_MakePoint(-102.169, 29.279), ST_MakePoint(-97.930, 30.515))
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,000,200 || ST_MakeBox2D(ST_MakePoint(-103.109, 28.516), ST_MakePoint(-96.991, 30.580))
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,500,680 || ST_MakeBox2D(ST_MakePoint(-103.518, 28.183), ST_MakePoint(-96.581, 31.255))
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,000,250 || ST_MakeBox2D(ST_MakePoint(-104.605, 27.301), ST_MakePoint(-95.494, 30.945))
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,500,450 || ST_MakeBox2D(ST_MakePoint(-106.041, 26.134), ST_MakePoint(-94.058, 30.754))
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,025 || ST_MakeBox2D(ST_MakePoint(-102.789, 28.775), ST_MakePoint(-97.3102, 33.174))
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,038 || ST_MakeBox2D(ST_MakePoint(-103.261, 28.391), ST_MakePoint(-96.8380, 33.556))
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,019 || ST_MakeBox2D(ST_MakePoint(-103.674, 28.057), ST_MakePoint(-96.4257, 33.814))
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 250,005 || ST_MakeBox2D(ST_MakePoint(-104.272, 27.571), ST_MakePoint(-95.8272, 34.041))
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 300,014 || ST_MakeBox2D(ST_MakePoint(-104.691, 27.230), ST_MakePoint(-95.4083, 34.451))
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,002 || ST_MakeBox2D(ST_MakePoint(-105.536, 26.545), ST_MakePoint(-94.5637, 35.042))
|-
|}

=== Runtime variables for Queries 3 and 4 ===

Bounding Box data for Query 3 (correspond to an area of about 50% of the elements from the respective subarea)
* Points:
Points 40000: POLYGON((-102.311 32.836, -102.311 29.164, -97.789 29.164 ,-97.789 32.836, -102.311 32.836))
Points 60000: POLYGON((-102.7765 33.214, -102.7765 28.786, -97.3235 28.786 ,-97.3235 33.214, -102.7765 33.214))')
Points 80000: POLYGON((-103.1755 33.538, -103.1755 28.462, -96.9245 28.462 ,-96.9245 33.538, -103.1755 33.538))')
Points 100000: POLYGON((-103.508 33.808, -103.508 28.192, -96.592 28.192 ,-96.592 33.808, -103.508 33.808))')
* Lines:
Line 500000: POLYGON((-101.743755 32.37538, -101.743755 29.62462, -98.356245 29.62462 ,-98.356245 32.37538, -101.743755 32.37538))')
Line 1000000: POLYGON((-102.0982 32.6632, -102.0982 29.3368, -98.0018 29.3368 ,-98.0018 32.6632, -102.0982 32.6632))')
Line 1500000: POLYGON((-102.54375 33.025, -102.54375 28.975, -97.55625 28.975 ,-97.55625 33.025, -102.54375 33.025))')
Line 2000000: POLYGON((-102.9893 33.3868, -102.9893 28.6132, -97.1107 28.6132 ,-97.1107 33.3868, -102.9893 33.3868))')
Line 2500000: POLYGON((-103.242 33.592, -103.242 28.408, -96.858 28.408 ,-96.858 33.592, -103.242 33.592))')
* Polygons:
Polygon 100000: POLYGON((-102.227875 32.7685, -102.227875 29.2315, -97.872125 29.2315 ,-97.872125 32.7685, -102.227875 32.7685))')
Polygon 150000: POLYGON((-102.54375 33.025, -102.54375 28.975, -97.55625 28.975 ,-97.55625 33.025, -102.54375 33.025))')
Polygon 200000: POLYGON((-102.79645 33.2302, -102.79645 28.7698, -97.30355 28.7698 ,-97.30355 33.2302, -102.79645 33.2302))')
Polygon 250000: POLYGON((-103.00925 33.403, -103.00925 28.597, -97.09075 28.597 ,-97.09075 33.403, -103.00925 33.403))')
Polygon 300000: POLYGON((-103.2686 33.6136, -103.2686 28.3864, -96.8314 28.3864 ,-96.8314 33.6136, -103.2686 33.6136))')
Polygon 350000: POLYGON((-103.47475 33.781, -103.47475 28.219, -96.62525 28.219 ,-96.62525 33.781, -103.47475 33.781))')

Point data for Query 4
* Points:
Points 40000: POINT((-102.311 32.836))
Points 60000: POINT((-102.7765 33.214))
Points 80000: POINT((-103.1755 33.538))
Points 100000: POINT((-103.508 33.808))
* Lines:
Line 500000: POINT((-101.743755 32.37538))
Line 1000000: POINT((-102.0982 32.6632))
Line 1500000: POINT((-102.54375 33.025))
Line 2000000: POINT((-102.9893 33.3868))
Line 2500000: POINT((-103.242 33.592))
* Polygons:
Polygon 100000: POINT((-102.227875 32.7685))
Polygon 150000: POINT((-102.54375 33.025))
Polygon 200000: POINT((-102.79645 33.2302))
Polygon 250000: POINT((-103.00925 33.403))
Polygon 300000: POINT((-103.2686 33.6136))
Polygon 350000: POINT((-103.47475 33.781))

=== Scripts for benchmark automation ===

tbd.

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark

Diskussion:HSR Texas Geo Database Benchmark

2010-02-08T07:42:53Z

Dominik:

Feel free to discusss 'HSR Texas Spatial Database Benchmark' issues here! [[Benutzer:Stefan|Stefan]] 18:39, 22. Dez. 2009 (CET)

== To do ==
* Standardise data and input process (circumvent or minimize floating point problem)
* Define the variables (@bbox and @point)

== Observations ==

General:
* MS SQL Server representatives report that geography types are 20-30% slower than geometry types.
* About PostGIS 1.5 there was said, that geography types are somewhat faster (see http://blog.cleverelephant.ca/)
* Secondary index optimization makes performance about 50% faster (from: MS SQL Server discussion)
* The number of points of a query window is important for a secondary filter overhead.

Data and queries per se:
* These queries return quite large percent of the data overall (50%-100% for points and polygons)
* The Number of points for a query windows is very small.
* The query windows are "aligned bounding boxes" which may put in favor RTree (e.g. Postgres) or spatial grid (SQL Server) index, depending on a position of the window. If the window is aligned with the grid, performance can be much faster than when it's slightly off.

== Discussion Points ==

* Should we take an average of multiple runs with different values (@point, @bbox)?
* Should we vary the queries' result sizes? (see observations)
* Should we vary the shape and alignment of the query window? (see observations)
* Should we run the benchmark with parallelized queries?

== Review ==
* The limiting factor of Query 1 is mainly the performance of the harddisk.

Diskussion:HSR Texas Geo Database Benchmark

2010-01-22T15:05:29Z

Dominik:

Feel free to discusss 'HSR Texas Spatial Database Benchmark' issues here! [[Benutzer:Stefan|Stefan]] 18:39, 22. Dez. 2009 (CET)

== To do ==
* Standardise data and input process (circumvent or minimize floating point problem)
* Define the variables (@bbox and @point)

== Observations ==

General:
* MS SQL Server representatives report that geography types are 20-30% slower than geometry types.
* About PostGIS 1.5 there was said, that geography types are somewhat faster (see http://blog.cleverelephant.ca/)
* Secondary index optimization makes performance about 50% faster (from: MS SQL Server discussion)
* The number of points of a query window is important for a secondary filter overhead.

Data and queries per se:
* These queries return quite large percent of the data overall (50%-100% for points and polygons)
* The Number of points for a query windows is very small.
* The query windows are "aligned bounding boxes" which may put in favor RTree (e.g. Postgres) or spatial grid (SQL Server) index, depending on a position of the window. If the window is aligned with the grid, performance can be much faster than when it's slightly off.

== Discussion Points ==

* Should we take an average of multiple runs with different values (@point, @bbox)?
* Should we vary the queries' result sizes? (see observations)
* Should we vary the shape and alignment of the query window? (see observations)
* Should we run the benchmark with parallelized queries?

Diskussion:HSR Texas Geo Database Benchmark

2010-01-22T14:59:34Z

Dominik:

Feel free to discusss 'HSR Texas Spatial Database Benchmark' issues here! [[Benutzer:Stefan|Stefan]] 18:39, 22. Dez. 2009 (CET)

== To do ==
* Standardise data and input process (circumvent or minimize floating point problem)
* Define the variables (@bbox and @point)

== Observations ==

General:
* MS SQL Server representatives report that geography types are 20-30% slower than geometry types.
* About PostGIS 1.5 there was said, that geography types are somewhat faster (see http://blog.cleverelephant.ca/)
* Secondary index optimization makes performance about 50% faster (from: MS SQL Server discussion)
* The number of points of a query window is important for a secondary filter overhead.

Data and queries per se:
* These queries return quite large percent of the data overall (50%-100% for points and polygons)
* The Number of points for a query windows is very small.
* The query windows are "aligned bounding boxes" which may put in favor RTree (e.g. Postgres) or spatial grid (SQL Server) index, depending on a position of the window. If the window is aligned with the grid, performance can be much faster than when it's slightly off.

== Discussion Points ==

* Should we take an average of multiple runs with different values (@point, @bbox)?
* Should we vary the queries' result sizes? (see observations below)
* Should we vary the shape and alignment of the query window? (see observations)

MediaWiki

2010-01-21T10:16:08Z

Dominik: /* Manuelle Anpassungen */

Das GISpunkt Wiki basiert auf der MediaWiki-Software. Nachfolgend u.a. Hinweise zur Konfiguration auf dem GISpunkt-Rechner (gis.hsr.ch). Ganz unten einige allgemeine Weblinks zu MediaWiki.

Siehe auch:
* [[Joomla]]

== How To ==

* Exportieren:
** Beispiel: http://gis.hsr.ch//wiki/Spezial:Exportieren/HSR

== MediaWiki-Installation GISpunkt Wiki ==

=== Letzte Aktualisierung ===
* Update auf Version 1.15.1
** Extension [http://www.mediawiki.org/wiki/Extension:Newuserlog Newuserlog] ist seit 1.14.0 im Core und wurde entfernt
* Update auf Version 1.11
** Dabei ist folgender Fehler aufgetreten:
*** http://www.mediawiki.org/wiki/Manual:Short_URL/Page_title_--_Working_method_with_mediawiki-1.11
*** http://www.organicdesign.co.nz/MediaWiki_1.11_title_extraction_bug

=== Angepasste Hauptseite (Startseite) ===

Mit [[MediaWiki:Sidebar]] (Navigation) angepasst (als WikiSysop). Original siehe "Diskussion:Hauptseite".

Test: Siehe [[Hauptseite]] (= Startseite).

=== LocalSettings.php ===

* Erlaubte File Upload (Hochladen) 'File Extensions': pdf, wav, mp3, ogg, zip.
* Das Verzeichnis $wgUploadDirectory und alle Subelemente müssen im Filesystem für den Webserver-User beschreibbar sein

=== Manuelle Anpassungen ===

* Der benutzte Skin (Monobook) wurde angepasst, damit auf Mobiltelefonen das Handheld-CSS erzwungen wird. Dazu ist die Erkennung über den User-Agent-String in MonoBook.php implementiert.
* Das handheld.css wurde angepasst: #siteSub { display: none; }

=== WikiPointMap ===

Zur Anleitung der [[WikiPointMap]], die zu den AddPageFromMap- und KML Export-Extensions passt.

=== KML Export Extension ===

KML Export ist eine Extension, welche KML-Dateien vom Inhalt von Artikelseiten für Google Earth generiert.

Weitere Informationen siehe [[KML Export Extension]].

=== AddPageFromMap Extension ===

Um die Webapplikation zusammen mit WikiPointMap nutzen zu können, muss diese AddPageFromMap-Extension installiert werden.

Siehe originale Startseite der [[AddPageFromMap Extension]].

=== DataMatrix-Extensions ===

Diese [http://www.haelfi.ch/sa1/Information_MediaWiki_Extensions DataMatrix-Extensions] sind Bestandteil der Studienarbeit [http://www.haelfi.ch/sa1 MobileTags4Wikis].

Download: http://www.haelfi.ch/sa1/Installation_Extensions

=== VideoFlash-Extension ===

VideoFlash is an extension that displays videos from Youtube, GoogleVideo, Dailymotion, Sevenload, Revver and similar services based on FLV video format.

Siehe [http://www.mediawiki.org/wiki/Extension:VideoFlash Extension:VideoFlash]

=== SlideShare-Extension ===

Parser function for embedding SlideShare presentations into wiki pages.

Siehe [http://www.mediawiki.org/wiki/Extension:SlideShare Extension:SlideShare]

=== OggHandler-Extension ===

OggHandler is an extension which allows you to display Ogg audio and video files in wiki pages, using the same syntax as for image files.

Siehe [http://www.mediawiki.org/wiki/Extension:OggHandler Extension:OggHandler]

=== FlashMP3-Extension ===

FlashMP3 embeds a simple Flash-player for playback of mp3-files.
Minimale Änderung am Code, der ID-Parameter muss für mehrere Instanzen auf einer Seite nicht mehr angegeben werden.

Siehe [http://www.mediawiki.org/wiki/Extension:FlashMP3 Extension:FlashMP3]

=== LanguageSelector-Extension ===

Lässt Auswahl der UI-Sprache zu. Auswahl wird für anonyme User aufgrund der im Header mitgeschickten "Accept-Language" getroffen. Über Parameter uselang (nur Request) resp. setlang (in Cookie gespeichert) kann die Sprachauswahl auch manuell beeinflusst werden.

Siehe [http://www.mediawiki.org/wiki/Extension:LanguageSelector Extension:LanguageSelector]

=== reCAPTCHA-Extension ===

Platziert ein Captcha bei der Neuregistrierung von Benutzern. Zudem wird ein Captcha platziert, wenn eine Seite editiert wurde und dabei externe Links hinzugefügt wurden.

Siehe [http://www.mediawiki.org/wiki/Extension:ReCAPTCHA Extension:ReCAPTCHA]

=== Lucene-Search ===

Lucene-Search wird für verbesserte Volltextsuchresultate eingesetzt.

Siehe [http://www.mediawiki.org/wiki/Extension:Lucene-search Extension:Lucene-search] und [http://www.mediawiki.org/wiki/Extension:MWSearch Extension:MWSearch]

Die cron-job-Konfig. startet 1x pro Stunde.

=== MediaWikiAuthTool ===

Service, der (externen) Applikationen eine User-Authentisierung auf der MediaWiki-Datenbank erlaubt (Remote via XML).

Siehe [[MediaWikiAuthTool]].

== MediaWiki-Erweiterungen für GIS ==

Siehe auch [[WikiGIS]] und die MediaWiki-Projektseite.

Wiki-Karte und Geo-Tagging von Artikel:
* http://www.mediawiki.org/wiki/Extension:GeoRSS - WikiMedia extension for GeoRSS. Easy interface for geotagging MediaWiki articles, and publishes a GeoRSS feed of all mapped articles in the Wiki, for mapping and syndication. By Mikel Maron.
* http://hexten.net/wiki/index.php/Wiki_Widgets - Hexten - Wiki Widgets are extensions for MediaWiki that provide simple ways to embed information from other sources within a wiki page.
* http://www.mediawiki.org/wiki/Extension:Google_Maps - WikiMedia extension for Google Maps
* http://www.mediawiki.org/wiki/Extension:KML_Export - WikiMedia extension for KML Export
* http://wikimap.sourceforge.net/Main_Page - WikiMap is a dynamic ('web 2.0') map that allows you to easily add locations to it. These locations show up on the map as markers. When you click a marker a little infowindow pops up. This infowindow can contain text, images, video, you name it. WikiMap is based on Openlayers and WordPress. Author: Robert Buzink.

Google Maps Extensions für MediaWiki:
* Listen:
** http://de.wikipedia.org/wiki/Benutzer:Arcy#MediaWiki_anpassen
** http://en.giswiki.org/wiki/MediaWiki_GIS_Extensions
** http://en.giswiki.org/wiki/Google_Maps_Extensions/Probleme
* Tutorials für diverse kleinere Erweiterungen des MediaWikis
** http://www.giswiki.org/index.php/MediaWiki_GIS_Extensions#Infofenster-Extension
** http://www.giswiki.org/index.php/MediaWiki_GIS_Extensions#Marker_Extension
* Google Map Extension, die es erlaubt, Geoinformationsdaten aus RSS Feeds in eine Karte einzubinden [http://meta.wikimedia.org/wiki/User:Mafs/GoogleMapExtension].
* [http://meta.wikimedia.org/wiki/Gis Manske's Gis-Extension auf WikiMedia]
** Erweiterung der Datenbank http://en.giswiki.org/wiki/GIS-Extension
* Eine Erweiterung für das Einbinden einer Google Map in MediaWiki http://en.giswiki.org/wiki/Google_Maps_Extensions
* Beispielcode für die Erstellung einer Google Map im GISWiki http://en.giswiki.org/wiki/Google_Maps_Extensions/GISWiki
* Tutorial und Code für das Anzeigen einer georeferenzierte Karte mit eigenen Punktkoordinaten http://en.giswiki.org/wiki/Point-Mapping_Extension

== MediaWiki-Erweiterungen allgemein ==

* [http://www.mediawiki.org/wiki/Extension:Pdf_Book Pdf_Book] - PDF erzeugen aus Artikel, die in Kategorien oder Bullet Listen organisiert sind.
* Verbesserte Suche
* User Management:
** http://www.mediawiki.org/wiki/Extension:Newuserlog
** http://www.mediawiki.org/wiki/Extension:New_User_Email_Notification
* [http://www.mediawiki.org/wiki/Extension:ScuttleBookmarks ScuttleBookmarks] - Displays Scuttle bookmarks in a blogroll type view
* Wiki-Datbase: In sep. Wiki testen?

HSR Texas Geo Database Benchmark

2010-01-19T12:35:06Z

Dominik: /* Datasets */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''' (tbd.):
* 'Aligned with grid': <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.

Variable(s) '''@point''' (tbd.):
* <code>POINT(-101.3135 32.026)</code>.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads.

SELECT Count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT Count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT Count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT Count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 1000 meters from a given point '@point'.

a)
SELECT COUNT(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 1000;

b)
SELECT COUNT(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 1000;

c)
SELECT COUNT(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 1000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All locations and Point of interest for the state of Texas. || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,744 || -103.208 33.565,-103.208 28.435,-96.891 28.435,-96.891 33.565
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,051 || -104.006 34.213,-104.006 27.787,-96.093 27.787,-96.093 34.213
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,174 || -104.904 34.942,-104.904 27.058,-95.195 27.058,-95.195 34.942
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,498 || -106.168 35.968,-106.168 26.032,-93.932 26.032,-93.932 35.968
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,282 || -102.169 32.720,-102.169 29.279,-97.930 29.279,-97.930 32.720
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,001,665 || -103.109 33.484,-103.109 28.516,-96.991 28.516,-96.991 33.484
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,498,936 || -103.518 33.816,-103.518 28.183,-96.581 28.183,-96.581 33.816
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,007,473 || -104.605 34.699,-104.605 27.301,-95.494 27.301,-95.494 34.699
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,498,941 || -106.041 35.865,-106.041 26.134,-94.058 26.134,-94.058 35.865
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,860 || -102.789 33.224,-102.789 28.775,-97.310 28.775,-97.3102 33.224
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,655 || -103.261 33.608,-103.261 28.391,-96.838 28.391,-96.8380 33.608
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,814 || -103.674 33.943,-103.674 28.057,-96.425 28.057,-96.4257 33.943
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 249,685 || -104.272 34.429,-104.272 27.571,-95.827 27.571,-95.8272 34.429
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 299,965 || -104.691 34.769,-104.691 27.230,-95.408 27.230,-95.4083 34.769
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,449 || -105.536 35.455,-105.536 26.545,-94.563 26.545,-94.5637 35.455
|-
|}

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark

HSR Texas Geo Database Benchmark

2010-01-18T12:14:23Z

Dominik: /* Datasets */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''' (tbd.):
* 'Aligned with grid': <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.

Variable(s) '''@point''' (tbd.):
* <code>POINT(-101.3135 32.026)</code>.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads.

SELECT Count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT Count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT Count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT Count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 1000 meters from a given point '@point'.

a)
SELECT COUNT(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 1000;

b)
SELECT COUNT(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 1000;

c)
SELECT COUNT(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 1000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

=== Download ===

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || All locations and Point of interest for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,744 || -103.208 33.565,-103.208 28.435,-96.891 28.435,-96.891 33.565
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,051 || -104.006 34.213,-104.006 27.787,-96.093 27.787,-96.093 34.213
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,174 || -104.904 34.942,-104.904 27.058,-95.195 27.058,-95.195 34.942
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,498 || -106.168 35.968,-106.168 26.032,-93.932 26.032,-93.932 35.968
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,282 || -102.169 32.720,-102.169 29.279,-97.930 29.279,-97.930 32.720
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,001,665 || -103.109 33.484,-103.109 28.516,-96.991 28.516,-96.991 33.484
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,498,936 || -103.518 33.816,-103.518 28.183,-96.581 28.183,-96.581 33.816
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,007,473 || -104.605 34.699,-104.605 27.301,-95.494 27.301,-95.494 34.699
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,498,941 || -106.041 35.865,-106.041 26.134,-94.058 26.134,-94.058 35.865
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,860 || -102.789 33.224,-102.789 28.775,-97.310 28.775,-97.3102 33.224
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,655 || -103.261 33.608,-103.261 28.391,-96.838 28.391,-96.8380 33.608
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,814 || -103.674 33.943,-103.674 28.057,-96.425 28.057,-96.4257 33.943
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 249,685 || -104.272 34.429,-104.272 27.571,-95.827 27.571,-95.8272 34.429
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 299,965 || -104.691 34.769,-104.691 27.230,-95.408 27.230,-95.4083 34.769
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,449 || -105.536 35.455,-105.536 26.545,-94.563 26.545,-94.5637 35.455
|-
|}

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark

HSR Texas Geo Database Benchmark

2010-01-18T12:10:17Z

Dominik: /* Datasets */

'''The HSR Texas Spatial Database Benchmark''' - A Proposal

Date of first proposal: December 21, 2009.
Status: Call for Comments and Call for Participation.

== Introduction ==

[[Bild:HSR_Spatial_Benchmark_Fig1.png|thumb|200px|right|Draft of the benchmark.]]

This is a proposal for a spatial database benchmark from the University of Applied Sciences Rapperswil (HSR). A study of existing database benchmarks revealed that there exists no information publicly available which compares spatial database systems regarding their performance. Spatial 'database management systems' (DMBS) typically form the persistence layer of a geographic information system (GIS).

Thus the Institute for Software at the University of Applied Sciences Rapperswil (HSR) decided to propose such a benchmark. This benchmark is being called 'The HSR Texas Spatial Database Benchmark' because it was defined from a HSR institute and because data comes from Texas USA.

The benchmark is based on a predefined set of queries. These queries consist of simple spatial queries, defined in the OpenGIS(tm) 'Simple Features Interface Standard (SFS)'. The queries are performed on different-sized data sets for monitoring the behavior on various loads as well as on different hardware systems.

In the following sections the methodology, the queries are explained and the used datasets are defined. This proposal concludes with a Call for Comments about the benchmark as well as a Call for Participation to apply and test this benchmark on existing DBMS software.

== Methodology ==

The benchmark follows the below defined rules of engagement.
* Each test runs three times in a row, the results of the third run are used for the comparison: this benchmark assumes full system caches ('hot' benchmark).
* Each test takes place on the same machine.
* All other DBMS are shut down while the tests are running.
* Each DBMS has the same data which comes from real world data sets (as indicated below).
* The coordinate reference system used is spherical (geographical).
* Each test uses bounding box or point variables which are the same for the respective systems in the test. These variables are chosen at random from within the subset space.

Any hardware used needs to be specified according to following points:
* System Type
* Model
* Processor
* RAM
* Hard drive
* Operating system

There are two variants of the test: "No tuning/initial" and "Tuned".
# The variant "No tuning/initial" is mandatory in order to make benchmarks more widely applicable. The respective default installation of one DBMS is used.
# The variant "Tuned" is optional and needs proper documentation of all tuned parameters and activities.

== Queries ==

The following queries are selected from the SQL functions defined by the SFS. The placeholder '{dataset}' will be substituted by the respective table names. The variable @bbox complies with randomly selected rectangular polygons in the geographical range of each data set, but which are the same for every system. The variable @point corresponds to a random point which is located in the area of the subsets. "geo" identifies the table column with geographical data. The following statements are given in pseudo SQL code.

Variable(s) '''@bbox''' (tbd.):
* 'Aligned with grid': <code>POLYGON ((-101.3135 32.026, -101.3135 29.974, -98.7865 29.974, -98.7865 32.026, -101.3135 32.026))</code>.

Variable(s) '''@point''' (tbd.):
* <code>POINT(-101.3135 32.026)</code>.

=== Query 1. Loading the data ===
Create tiered data subsets by sub-dividing with a given bounding box '@bbox', creating the indices.

SELECT * INTO {dataset}
FROM {original dataset}
WHERE ST_Intersects(@bbox, geo);
CREATE SPATIAL INDEX idx ON {dataset} ([geo]);

=== Query 2. A Non-spatial Selection: Count ===
Count all railroads.

SELECT Count(*)
FROM {dataset lines} l
WHERE l.roadflg='Y';

=== Query 3. Spatial Selection I: Intersect Point, Line and Polygons ===
Count all a) points, b) lines, c) polygons that intersect with a given bounding box '@bbox'.

a)
SELECT Count(*)
FROM {dataset points} p
WHERE ST_Intersects(@bbox, p.geo);

b)
SELECT Count(*)
FROM {dataset lines} l
WHERE ST_Intersects(@bbox, l.geo);

c)
SELECT Count(*)
FROM {dataset polygons} pg
WHERE ST_Intersects(@bbox, pg.geo);

=== Query 4. Spatial Selection II: Distance within ===
Count all a) points, b) lines, c) polygons that are within 1000 meters from a given point '@point'.

a)
SELECT COUNT(*)
FROM {dataset points} p
WHERE ST_Distance(@point, geo) <= 1000;

b)
SELECT COUNT(*)
FROM {dataset lines} l
WHERE ST_Distance(@point, geo) <= 1000;

c)
SELECT COUNT(*)
FROM {dataset polygons} pg
WHERE ST_Distance(@point, geo) <= 1000;

=== Query 5. Spatial Selection III: Intersect/Join Lines and Polygons ===
Count all railroads that intersect with a water area.

SELECT COUNT(*)
FROM {dataset lines} l, areawater_full pg
WHERE ST_Intersects(pg.geom, l.geom) = 1 AND l.railflg = 'Y';

== Datasets ==

[[Bild:HSR_Spatial_Benchmark_Fig2.png|thumb|400px|right|Visualization of the Point (left) and polygon (right) data set.]]

At the Free and Open Source Software for Geospatial Conference (FOSS4G) 2009, a 'Web Mapping Performance Shoot-out' was performed which compared the open source GIS software products GeoServer and MapServer ([http://2009.foss4g.org/presentations/#presentation_109]). There, PostgreSQL/PostGIS and Oracle have been tested too. Several sets from the TIGER shapefiles of Texas that can be downloaded from the U.S. Census Bureau where used as data basis.

* http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g)
** vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
** GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

This data is proposed to use within this benchmark containing rivers, roads, railroads, Points-of-interest and water areas. The data originate from the following shape files:

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Data sets from the TIGER shapefiles'''
|- bgcolor="#e0e0e0"
! Shape file name || gnis_names09 || edges_merge || areawater_merge
|- bgcolor="#eeeeee"
| Type || Point || Multilinestring || Multipolygon
|- bgcolor="#eeeeee"
| Number of records || 103,000 || Over 5 M || 380,000
|- bgcolor="#eeeeee"
| SRID || EPSG:4326 || EPSG:4326 || EPSG:4326
|- bgcolor="#eeeeee"
| Source || GNIS database || TIGER 2008 || TIGER 2008
|- bgcolor="#eeeeee"
| Description || All line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. || All locations and Point of interest for the state of Texas. || The TIGER set of polygons describing water surface for the state of Texas.
|-
|}

These data sets are divided by a bounding-box procedure to subsets. Using these subsets the behavior of the respective DBMS can be observed in different-sized data sets. The figure to the right shows the gnis_names09 data set (left) and the areawater_merge data set (right).

It follows the definition of the (sub-)datasets (coordinates in [[WGS94]]):

=== Point data gnis_names09 ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile gnis_names09'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| gnis_names_40000 ||style="text-align:right"| 40,744 || -103.208 33.565,-103.208 28.435,-96.891 28.435,-96.891 33.565
|- bgcolor="#eeeeee"
| gnis_names_60000 ||style="text-align:right"| 60,051 || -104.006 34.213,-104.006 27.787,-96.093 27.787,-96.093 34.213
|- bgcolor="#eeeeee"
| gnis_names_80000 ||style="text-align:right"| 80,174 || -104.904 34.942,-104.904 27.058,-95.195 27.058,-95.195 34.942
|- bgcolor="#eeeeee"
| gnis_names_100000 ||style="text-align:right"| 100,498 || -106.168 35.968,-106.168 26.032,-93.932 26.032,-93.932 35.968
|-
|}

=== Multilinestring data edges_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile edges_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| edges_500000 ||style="text-align:right"| 500,282 || -102.169 32.720,-102.169 29.279,-97.930 29.279,-97.930 32.720
|- bgcolor="#eeeeee"
| edges_1000000 ||style="text-align:right"| 1,001,665 || -103.109 33.484,-103.109 28.516,-96.991 28.516,-96.991 33.484
|- bgcolor="#eeeeee"
| edges_1500000 ||style="text-align:right"| 1,498,936 || -103.518 33.816,-103.518 28.183,-96.581 28.183,-96.581 33.816
|- bgcolor="#eeeeee"
| edges_2000000 ||style="text-align:right"| 2,007,473 || -104.605 34.699,-104.605 27.301,-95.494 27.301,-95.494 34.699
|- bgcolor="#eeeeee"
| edges_2500000 ||style="text-align:right"| 2,498,941 || -106.041 35.865,-106.041 26.134,-94.058 26.134,-94.058 35.865
|-
|}

=== Multipolygon data areawater_merge ===

{| class="prettytable"
|+ style="padding-bottom:1em;" | '''Table: Shapefile areawater_merge'''
|- bgcolor="#e0e0e0"
! style="width:12em"| Subset || Number of records || Bounding box
|- bgcolor="#eeeeee"
| areawater_100000 ||style="text-align:right"| 100,860 || -102.789 33.224,-102.789 28.775,-97.310 28.775,-97.3102 33.224
|- bgcolor="#eeeeee"
| areawater_150000 ||style="text-align:right"| 150,655 || -103.261 33.608,-103.261 28.391,-96.838 28.391,-96.8380 33.608
|- bgcolor="#eeeeee"
| areawater_200000 ||style="text-align:right"| 200,814 || -103.674 33.943,-103.674 28.057,-96.425 28.057,-96.4257 33.943
|- bgcolor="#eeeeee"
| areawater_250000 ||style="text-align:right"| 249,685 || -104.272 34.429,-104.272 27.571,-95.827 27.571,-95.8272 34.429
|- bgcolor="#eeeeee"
| areawater_300000 ||style="text-align:right"| 299,965 || -104.691 34.769,-104.691 27.230,-95.408 27.230,-95.4083 34.769
|- bgcolor="#eeeeee"
| areawater_350000 ||style="text-align:right"| 350,449 || -105.536 35.455,-105.536 26.545,-94.563 26.545,-94.5637 35.455
|-
|}

== Call for Comments and Call for Participation ==

The 'HSR Texas Spatial Database Benchmark' was conducted and verified by a first time in December 2009 during a Master Seminar by the Institute for Software at HSR. Two database management systems software (DBMS), one commercially available (Microsoft SQL Server 2008 Spatial) an one under an Open Source license (PostgreSQL 8.4.1/PostGIS 1.4.0), have been chosen. The results are currently evaluated and will probably be published in 2010.

Two actions from researchers and volunteers may complement this research:
* Submit comments on the 'HSR Texas Spatial Database Benchmark' itself.
* Do further experiments on existing DBMS.

We are looking forward for a fruitful discussion!

== Feedback, Discussion and Contact ==

Please direct any comments either to the discussion page, to the [http://lists.osgeo.org/mailman/listinfo/benchmarking OSGEOs benchmarking mailing list] - or directly to [[Benutzer:Stefan|Prof. S. Keller from Institute for Software at HSR]].

== Weblinks ==
* [http://wiki.osgeo.org/wiki/Benchmarking_2009#Data GeoServer vs. MapServer Benchmark 2009]
* List of benchmarks and performance tests: http://delicious.com/sfkeller/database+benchmark

Diskussion:HSR Texas Geo Database Benchmark

2010-01-15T12:34:00Z

Dominik:

Feel free to discusss 'HSR Texas Spatial Database Benchmark' issues here! [[Benutzer:Stefan|Stefan]] 18:39, 22. Dez. 2009 (CET)

== ToDo ==
* standardise data and input process (circumvent or minimize floating point problem)
* define the variables (@bbox and @point)
** should we take an average of multiple runs with different values?
* should we vary the queries' result sizes? (see observations below)
* should we vary the shape and alignment of the query window? (see observations below)

== Observations ==

General:
* MS SQL Server representatives report that geography types are 20-30% slower than geometry types.
* About PostGIS 1.5 there was said, that geography types are somewhat faster (see http://blog.cleverelephant.ca/)
* Secondary index optimization makes performance about 50% faster (from: MS SQL Server discussion)
* The number of points of a query window is important for a secondary filter overhead.

Data and queries per se:
* These queries return quite large percent of the data overall (50%-100% for points and polygons)
* The Number of points for a query windows is very small.
* The query windows are "aligned bounding boxes" which may put in favor RTree (e.g. Postgres) or spatial grid (SQL Server) index, depending on a position of the window. If the window is aligned with the grid, performance can be much faster than when it's slightly off.

== Discussion Points ==

Typically the bbox variable is "grid aligned" - What's the impact? Is this typical?

tdb.