23 August, 2009 17:29
GIS Technology Stacks
Posted by mimo under [ Systems Architechure ][ (0) Comment ] | [ (0) Trackbacks ]
In order to produce a single view of GIS data derived from multiple sources, it is necessary to abstract the rendering of the data from the actual datasources. This is similar to the concept of a Services Oriented Architecture in which business data are rendered from their system domain to a domain specific language that suits the needs of the enterprise.
The term that is given to the systems required to perform the rendering function are called ?stacks? in the GIS world.
The Stack typically is composed of: Spatial Database, Internet Mapping Server (map services), Mapping Client, and Web Mapping Client.
There are a few different technology streams available for provisioning stacks. I have been doing research into the different stacks available for service enabling GIS data.
Database:
Used for storing, indexing and querying spatial data. The most common of these is the ShapeFile or File Geodatabase. This is simply a set of specially configured files that combined together make a -database-. In the world of -real- databases, there are several common options
- Oracle Spatial - supposed to be the best in terms of accuracy and speed. It is expensive, and requires specially trained resources to keep it running
- PostGres/PostGIS - best open source spatial database. It is fast, reliable and widely supported. There are good toolsets for it on most platforms
- Sql Server Spatial - New spatial database on the block already has decent driver support, work is still required on optimization. The Geometry (planar) data type has support now on Mapserver and Geoserver, but not the Geography datatype. ESRI requires the use of the ESRI server for full connectivity.
- ESRI SDE - ESRI spatial extension for Oracle and SQL server - expensive, accurate, requires ESRI specific expertise. This product was moved into the ESRI Server offering as of version 9.3
Currently at Matrix (where I work) we are using:
- SQL server for base and field data
- SHP (ESRI Shape files) for project data used for spatial analysis and other core GIS functions
- PostGres/PostGIS for hydrogeology data, and possibly base data.
Internet Mapping Server (IMS):
Internet Mapping Servers provides mapping services. These appear to be the key piece that we are missing to make this whole thing work on an enterprise scale.
IMS typically are subdivided into WMS (Web Mapping) and WFS (Web Feature Mapping) servers (Article HERE describing the differences) . The Web Mapping Services are typically image based tiles that are rendered by the server through a map template from the datasources defined for that template, and output to image tiles. This is the most commonly used approach, and is what Google Maps, Virtual Earth etc are doing in the backend on a massive scale. The issue here is that while tiles are the fastest way of returning properly formatted spatial data, the objects (or Features in GIS speak) represented on the images cannot be directly manipulated. Additional query to database directly type techniques can be used to get around this if necessary. A better approach though is to use Tiled Map Services for base data or displaying dense datasets that do not change often, and use Features (vector/object data) for dynamic datasets or in places where the object density is reduced (higher zoom levels)
The Web Feature Service returns actual object or in GIS speak Feature definitions. A typical IMS server will support returning data in both WMS and WFS type services. The WFS data return is typically an XML stream formatted in KML, GML, or GEORSS format. There are probably some proprietary ones as well for ESRI specific data.
The players out there on this front are:
- ESRI Server - full featured, very expensive, but does it all. It also has the ability to do spatial analysis through a web browser.
- Mapserver - older open source IMS system we have tried it at Matrix. It works well, but has a more 'command line' approach to layer provisioning and requires a higher understanding of specific mapping techniques to make it work.
- FME Server - new this year. It is very useful for doing spatial transforms, analysis and providing data from a variety of sources, but is not an IMS server in production. It has a complicated licensing scheme based on engines that makes it difficult to get sense of what the cost of the system is. However, FME tries to mitigate this by giving users a long evaluation period to asses what they might need.
- Manifold IMS - very inexpensive, fast, but I was not crazy about the implementation. It is built around the idea of using the Manifold user interface to generate code for a website. The downside is the code that is generated is an older style postback server approach instead of a more modern Ajax style user interface. It is likely possible to customize this to give a better user experience, but it seems like a lot of work. However, for the cost its an excellent alternative.
- MapGuide - open source and commercial versions available. The open source version is the prior year's commercial offering. Seems to be good, but has spotty support. Similar to Manifold, it Is built more around a software factory approach, so is quick to get up and running without programming.
- GeoServer - open source, widely supported, has a very nice administration interface, and decent driver support. Supported by a very active community. It also has an associated commercial support style model called OpenGeo, which packages the various open source technologies together into one offering.
It has a wide support base, uses XML for its layer definitions, has a very nice looking administration console, and excellent documentation.
Geoserver also has several add-ons that look really good as well. One of immediate interest is GeoWebCache, which is a WMS tile caching and optimization server in effect a spatial proxy server.
Another of interest, although very new on the scene is a WFS based versioning module, of which one branch appears to be looking into using SVN as the backing for spatial feature version control (http://geoserver.org/display/GEOS/Versioning+WFS)
Map Editing Clients:
These are clients that allow GIS experts to create, manage and manipulate the spatial data directly. They can derive spatial data from a variety of sources to produce an overall map.
Typical clients are:
- ArcMap - which is what ESRI collectively calls its mapping clients, and it is broken up into three levels in terms of increasing cost/features: ArcView,ArcEditor and ArcInfo
- ArcView - Basic ESRI spatial data editing tool - designed for single user single datasource
- ArcEditor - Encompasses ArcView plus adds multiple user editing
- ArcInfo - All the features of the other Arcs plus spatial analysis
- Manifold - Low cost equivalent to ArcMap. Not as powerful as the ESRI offerings, but does some things better/faster than ArcMap. It is very useful as a general non-GIS client for editing spatial data.
- Quantum GIS - open source GIS editor - pretty good equivalent to the ArcMap in terms of features etc.
Udig - open source GIS client by refractions research who build and maintain PostGis. This is an excellent well thought out easy to use client, and I would recommend it
At Matrix, we will likely continue to use the core ESRI ArcMap series for the main GIS team. Manifold will likely be used for people who need to do desktop editing, but are doing so on a more limited basis. Quantum GIS has not been experimented with enough to know if it will be useful at Matrix.
Web Mapping Clients
There are a number of web mapping clients available. They are all roughly feature equivalent, and offer slightly different advantages. If the underlying data provisioning is done correctly, supporting more than one client is not a huge outlay of time
Typical Web Mapping Clients are:
Google Maps - the most common client. Base data provided by Google must be paid for on a $10,000 per year license for commercial use. Javascript based, relatively easy to work with. The APIs however are designed more towards support consumer mashup style development, and can be confusing to work with for developers used to more enterprise GIS style APIs
Virtual Earth (now bing maps) - comes in two versions now: javascript and Silverlight. We have successfully used the javascript version to build basic demonstration map tools. However, the Silverlight one offers a lot more promise for corporate mapping applications. It is much faster, supports c# based development and can leverage WCF style services. For Matrix Rapid development, we think this will be the best client to focus on building a framework around. Virtual Earth also provisions base maps at a cost roughly equivalent to the Google Maps cost ~10,000 US per year.
Open Layers - open source equivalent to Google Earth and Virtual Earth javascript mapping clients. It offers similar functionality, but does not come with a base maps subscription. OpenStreetMaps provides basic street level mapping data, but will not be of much use for Matrix given that most of the work focuses in areas where the OpenStreetMap approach will not work. There are a number of sources Matrix GIS team has studied on this front for base map provisioning
The term that is given to the systems required to perform the rendering function are called ?stacks? in the GIS world.
The Stack typically is composed of: Spatial Database, Internet Mapping Server (map services), Mapping Client, and Web Mapping Client.
There are a few different technology streams available for provisioning stacks. I have been doing research into the different stacks available for service enabling GIS data.
Database:
Used for storing, indexing and querying spatial data. The most common of these is the ShapeFile or File Geodatabase. This is simply a set of specially configured files that combined together make a -database-. In the world of -real- databases, there are several common options
- Oracle Spatial - supposed to be the best in terms of accuracy and speed. It is expensive, and requires specially trained resources to keep it running
- PostGres/PostGIS - best open source spatial database. It is fast, reliable and widely supported. There are good toolsets for it on most platforms
- Sql Server Spatial - New spatial database on the block already has decent driver support, work is still required on optimization. The Geometry (planar) data type has support now on Mapserver and Geoserver, but not the Geography datatype. ESRI requires the use of the ESRI server for full connectivity.
- ESRI SDE - ESRI spatial extension for Oracle and SQL server - expensive, accurate, requires ESRI specific expertise. This product was moved into the ESRI Server offering as of version 9.3
Currently at Matrix (where I work) we are using:
- SQL server for base and field data
- SHP (ESRI Shape files) for project data used for spatial analysis and other core GIS functions
- PostGres/PostGIS for hydrogeology data, and possibly base data.
Internet Mapping Server (IMS):
Internet Mapping Servers provides mapping services. These appear to be the key piece that we are missing to make this whole thing work on an enterprise scale.
IMS typically are subdivided into WMS (Web Mapping) and WFS (Web Feature Mapping) servers (Article HERE describing the differences) . The Web Mapping Services are typically image based tiles that are rendered by the server through a map template from the datasources defined for that template, and output to image tiles. This is the most commonly used approach, and is what Google Maps, Virtual Earth etc are doing in the backend on a massive scale. The issue here is that while tiles are the fastest way of returning properly formatted spatial data, the objects (or Features in GIS speak) represented on the images cannot be directly manipulated. Additional query to database directly type techniques can be used to get around this if necessary. A better approach though is to use Tiled Map Services for base data or displaying dense datasets that do not change often, and use Features (vector/object data) for dynamic datasets or in places where the object density is reduced (higher zoom levels)
The Web Feature Service returns actual object or in GIS speak Feature definitions. A typical IMS server will support returning data in both WMS and WFS type services. The WFS data return is typically an XML stream formatted in KML, GML, or GEORSS format. There are probably some proprietary ones as well for ESRI specific data.
The players out there on this front are:
- ESRI Server - full featured, very expensive, but does it all. It also has the ability to do spatial analysis through a web browser.
- Mapserver - older open source IMS system we have tried it at Matrix. It works well, but has a more 'command line' approach to layer provisioning and requires a higher understanding of specific mapping techniques to make it work.
- FME Server - new this year. It is very useful for doing spatial transforms, analysis and providing data from a variety of sources, but is not an IMS server in production. It has a complicated licensing scheme based on engines that makes it difficult to get sense of what the cost of the system is. However, FME tries to mitigate this by giving users a long evaluation period to asses what they might need.
- Manifold IMS - very inexpensive, fast, but I was not crazy about the implementation. It is built around the idea of using the Manifold user interface to generate code for a website. The downside is the code that is generated is an older style postback server approach instead of a more modern Ajax style user interface. It is likely possible to customize this to give a better user experience, but it seems like a lot of work. However, for the cost its an excellent alternative.
- MapGuide - open source and commercial versions available. The open source version is the prior year's commercial offering. Seems to be good, but has spotty support. Similar to Manifold, it Is built more around a software factory approach, so is quick to get up and running without programming.
- GeoServer - open source, widely supported, has a very nice administration interface, and decent driver support. Supported by a very active community. It also has an associated commercial support style model called OpenGeo, which packages the various open source technologies together into one offering.
It has a wide support base, uses XML for its layer definitions, has a very nice looking administration console, and excellent documentation.
Geoserver also has several add-ons that look really good as well. One of immediate interest is GeoWebCache, which is a WMS tile caching and optimization server in effect a spatial proxy server.
Another of interest, although very new on the scene is a WFS based versioning module, of which one branch appears to be looking into using SVN as the backing for spatial feature version control (http://geoserver.org/display/GEOS/Versioning+WFS)
Map Editing Clients:
These are clients that allow GIS experts to create, manage and manipulate the spatial data directly. They can derive spatial data from a variety of sources to produce an overall map.
Typical clients are:
- ArcMap - which is what ESRI collectively calls its mapping clients, and it is broken up into three levels in terms of increasing cost/features: ArcView,ArcEditor and ArcInfo
- ArcView - Basic ESRI spatial data editing tool - designed for single user single datasource
- ArcEditor - Encompasses ArcView plus adds multiple user editing
- ArcInfo - All the features of the other Arcs plus spatial analysis
- Manifold - Low cost equivalent to ArcMap. Not as powerful as the ESRI offerings, but does some things better/faster than ArcMap. It is very useful as a general non-GIS client for editing spatial data.
- Quantum GIS - open source GIS editor - pretty good equivalent to the ArcMap in terms of features etc.
Udig - open source GIS client by refractions research who build and maintain PostGis. This is an excellent well thought out easy to use client, and I would recommend it
At Matrix, we will likely continue to use the core ESRI ArcMap series for the main GIS team. Manifold will likely be used for people who need to do desktop editing, but are doing so on a more limited basis. Quantum GIS has not been experimented with enough to know if it will be useful at Matrix.
Web Mapping Clients
There are a number of web mapping clients available. They are all roughly feature equivalent, and offer slightly different advantages. If the underlying data provisioning is done correctly, supporting more than one client is not a huge outlay of time
Typical Web Mapping Clients are:
Google Maps - the most common client. Base data provided by Google must be paid for on a $10,000 per year license for commercial use. Javascript based, relatively easy to work with. The APIs however are designed more towards support consumer mashup style development, and can be confusing to work with for developers used to more enterprise GIS style APIs
Virtual Earth (now bing maps) - comes in two versions now: javascript and Silverlight. We have successfully used the javascript version to build basic demonstration map tools. However, the Silverlight one offers a lot more promise for corporate mapping applications. It is much faster, supports c# based development and can leverage WCF style services. For Matrix Rapid development, we think this will be the best client to focus on building a framework around. Virtual Earth also provisions base maps at a cost roughly equivalent to the Google Maps cost ~10,000 US per year.
Open Layers - open source equivalent to Google Earth and Virtual Earth javascript mapping clients. It offers similar functionality, but does not come with a base maps subscription. OpenStreetMaps provides basic street level mapping data, but will not be of much use for Matrix given that most of the work focuses in areas where the OpenStreetMap approach will not work. There are a number of sources Matrix GIS team has studied on this front for base map provisioning
