Philip Sargent
26 February 1998 13:23 CET
(minor corrections Septermber 19, 1999)
This paper reviews enterprise-scale GIS middleware that holds spatial data in relational databases (RDBMs) or in "universal servers" (object-relational databases) and which present an SQL function interface to that spatial data.
ESRI's Spatial Database Engine (SDE), now to be renamed Spatial Server, MapInfo's SpatialWare (SW), and Oracle's Spatial Data Option (SDO), now renamed Oracle 8 Spatial Cartridge, and Autodesk's VISION* have different roles in future componentized OpenGIS architectures. There are currently no announced GIS middleware products from Siemens-Nixdorf or Intergraph.
(MapInfo's SpatialWare (SW), originally developed by Unisys, is a cross-platform and standards-based alternative to O8SC.)
Pragmatic architecture design has as much to do with available technology and purchasable products, as it has to do with core technology or function-call interfaces. Thus it is useful to understand the 3 or 4 distinct markets for GIS products as well as understanding the underlying software. Market focus also tells us which product features are likely to be well supported and which will never work properly.
Oracle uses the term "SDO" only with Oracle 7. Essentially the same product for Oracle 8 is called the Spatial Cartridge (O8SC). This paper does not explore any technology differences between them. Verbal statements from Oracle representatives in 1997 stated that both SDO and O8SC used four tables, required some SQL and populated temporary tables in order to do spatial queries.
In order to reduce confusion, this paper will always use the term O8SC instead of SDO. ESRI' is apparently renaming SDE as Spatial Server.
GIS markets can be roughly categorised as:
There is also a distinction between transaction types:
None of these middleware packages offer long transactions and version control - it is antithetical to how commercial RDBMSs work. This is the province of Laser-Scan and Smallworld. In practice, most customers who could benefit from long transactions are using whole-dataset update GIS (Intergraph, ESRI) plus organisational rules.
It not entirely a coincidence that the most modern commercial object-repository technologies are european (Laser-Scan, Smallworld, O2) whereas those based on older RDBMS technology are american: the reasons are to do with market history. This paper, however, concentrates on available middleware offering SQL interfaces. These will be OpenGIS Consortium SQL interfaces in future and will be supported by object-repositories, but currently only RDBM-based systems are available for this middleware market.
This is primarily sold by Oracle as a way of expanding the number of people who use geo-referenced data and to make it possible for them to do it without buying a GIS system at all. The primary market for this product is the same as Oracle's existing bespoke market: large corporations and organisations who develop their own front-end forms and user-interfaces (preferably using Oracle software tools) to organisational data with some georeferencing.
O8SC targets one kind of user:
The primary application type is one where the geodata forms a small part of the whole. The target users are software developers. In this market O8SC competes directly with, and substitutes for, GIS systems.
This market does not require leading-edge GIS technology. In particular, Oracle seem happy with vectors between vertices and show little sign of trying to use arcs in future.
The O8SC function interface is SQL extended with 3 new datatypes (e.g. complex polygons) and 10 predicates for relationship tests (e.g. ANYINTERSECT).
SDE targets two kinds of user:
In existing releases of SDE the SQL interface is not available to clients and programmers must use a function interface SDEIF.
Although some implementations of SDE use much the same technology as O8SC, they are not used for the same purpose. This is most easily seen by looking at who they are sold to. Each has a well-defined SQL function call interface but the situation is confused because each is often used together with other software.
Although the SDE function interface uses SQL, it is larger (hundreds of functions), more complex and more "high level" than O8SC. SDE encapsulates many of the functions of a complete GIS system, not just the basic features supported by O8SC, i.e. SDE supports co-ordinate transformations, layer handling etc. MapObjects also presents a set of functions for handling (US) postal addresses. Thus for programmer-users, SDE is more appropriate for complex GIS tasks. Using O8SC the programmers would have to write it all themselves.
MapInfo's SW targets one kind of user primarily:
There is also a tcl/tk and C toolkit for Unix developers but the MapInfo company is primarily focussed on Windows, not Unix. This toolkit is probably a legacy from the Unisys development team.
SW is a direct and comparable competitor to O8SC. It has 10 new SQL functions and 7 new predicates, but they are all compliant with the proposed SQL3 MM/Spatial draft standard. SW also runs on Informix and (soon) DB2 as well as Oracle.
SHL's VISION* targets one kind of user primarily: software developers who will create
SHLVISION* is a direct and comparable competitor to O8SC and SW. It stores spatial data only in an Oracle database. SHL will be offering a long-transaction Oracle Cartridge in 1998.
SDE, SW and O8SC all implement their own spatial indexing on all platforms, with one exception: when SDE is implemented on top of O8SC then it uses O8SC's spatial indexing.
SDE and SW store spatial data in relational tables when implemented on ordinary relational databases (so did Oracle with SDO).
SDE, SW and O8SC all appear store spatial data in relational tables when implemented on object-relational Universal Servers (Oracle8, Informix, or DB2) but also use some of the extra functionality provided by the database system.
All products are evolving fast.
MapInfo/SW will continue to attempt to climb from simple desktop users towards enterprise deployment and thus can be expected to follow SDE in functions, but always as a cheaper, simpler alternative. It could be a severe competitor to Oracle because (a) MapInfo "owns" the mapping desktop in Microsoft Office, (b) SW runs on several databases, not just one.
Many file-based OCX tools will introduce SW, O8SC, SDE and/or OGC interfaces in 1998. Currently only Enigma and GeoMedia appear to do this (Enigma with SDO since November 1996and GeoMedia with SDO ) and Sylvan has announced OGC/SQL capability in January 1998; but we should expect that Blue Marble's GeoView, OpenMaps' DBMap/DBServer (which already works on plain Oracle) and possibly EMap and GeoConcept will follow.
(Note that Intergraph's GeoMedia is an application that can be driven by OLE Automation but it is not an OCX component and support for O8SC rather than SDO has not been announced yet.)
Thus although Oracle competes with ESRI in one market, O8SC makes life easier for ESRI because it has less software to develop and maintain within SDE when it runs on O8SC. ESRI can concentrate on the higher-value provision of sophisticated GIS functions rather than low-level spatial database support. Eventually we can expect that ESRI will only support SDE on top of SQL3-capable RDBMSs. However Oracle will no doubt extend the GIS functionality of O8SC thus squeezing ESRI further "up" the market to much more specialised and less profitable users.
The other commercial danger for ESRI is that once the data is in O8SC, OGC or SQL3-MM/Spatial databases, many, many other server-compatible software tools will be able to access it. This is mitigated by the fact that SDE uses a complex schema of extra tables and attributes to provide the extra functionality above that required to support simple features,. The competing server-compatible software tools will not be able to use these extra schema elements until they reverse-engineer the SDE schema.
MapInfo's SW technically competes directly with Oracle's O8SC, but in practice this can only occur where a potential customer is already committed to Oracle. (We can probably assume that most customers' decision to buy a particular RDBMS is only partially influenced by the spatial software.) On all other RDBMSs it competes with ESRI only; with the distinction that it targets a less sophisticated, and presumably cheaper, GIS interface.
"On price per seat, the relational market is being eroded at 20% to 30% per year", from recent reports by Spikes Caell and Bloor Research: "Traditional relational databases don't play a role except as high-volume feeder systems." Manipulating complex data is now what customers are prepared to pay for as it is where they earn most money, and the big RDBMS companies will not be able to make enough money from plain RDBMSs with the prices dropping so fast. [Computing Newspaper 22 Jan.1998]. Oracle says that their database sales are up by 25% this year, but they also say that the total growth in the DRBMS market is such that they cannot maintain that unless they increase market share - which is very difficult as they already have a big chunk of it. As a result their share value dropped by a third.
The reason the prices are dropping is:
The RDBMS companies are getting into other businesses: Oracle has moved into applications, e.g. Oracle Financials accounting package, and services. Large customer organisations are putting their new money into (a) Internet applications and (b) year 2000 problems and not databases.
IDC expects the Spatial Infomation Management (SIM) market to grow from $58M in 1996 to $1.4B in 1999 which represents a growth rate of ~200% over a 4 year period. About ~60% of the SIM market is located within the U.S. All of which makes this SIM (GIS middleware) market look very attractive to large, mean, hungry, desparate, ruthless American RDBMS companies.
ESRI, MapInfo and Oracle are all members of the OpenGIS Consortium. ESRI has published plans on the web to implement OpenGIS interfaces. All are believed to have working prototypes of the "simple features" interface but there is no public committment to sell software using it before the end of 1998.
The OGC "Simple Features" proposal is defined for 3 platforms: SQL, CORBA and COM. ESRI participated in all three, MapInfo, Oracle, IBM and Informix participated in the SQL proposal, MapInfo participated in the COM proposal and Oracle participated in the CORBA proposal. (Others also participated, but that is not relevant here.) The OGC "Well Known ..." representations are the same on all platforms.
The SQL platform of OGC "Simple Features" mandates an ODBC 2.0-campatible database. The SQL flavour can itself be implemented in 3 flavours and OGC mandates a schema for each:
There are 37 new functions and predicates defined, e.g. touch(), within(), overlap(), contains(), distance().
All three flavours mandate a description of the spatial reference system (projection) using a Well Known Text representation. This is a sophisticated grammar that encodes all known projections and their various parameterisations into an ASCII text string.
ESRI announced their intention of supporting all three flavours of OGC/SQL in Spatial Server (future releases of SDE) at Interop'97 conference in Santa Barbara (December 1997); due for release sometime in Q1, 1998:
ESRI also announced two further important aspects:
This is important because it will allow any OpenGIS client to access data held in legacy ESRI file formats, and to be largely unaware of when that data may be migrated to a database, or any OpenGIS-compatible non-ESRI system, in future. However, OGC intentionally does not have any conformance level certification process for SQL simple features: any subset is allowable, so ESRI is officially allowed expose only restricted interfaces while using rich interfaces within its own software as a way of keeping customers loyal to its own software.
If everything goes perfectly, the OGC and ISO standards will be at best sub-sets of each other, but will not be identical.
The draft SQL3/MM-spatial will be edited at the ISO meeting in Brazil in Arpil 1998. The OGC has made change suggestions (decided in Munchen in February 1998), but whatever happens in Brazil, the OGC will very likely alter their "Simple Features" specification to be entirely compatible with whatever emerges from ISO. Compatible does not mean identical. "Simple Features", after intense lobbying in committee in 1997, does straight lines but not curved arcs, whereas the SQL3/MM-spatial draft does circular and elliptical arcs.
The next OGC proposal "Complex Features" will do arcs, but it will also do a lot more than ISO: relationships and topology.
All these technologies: SDE, O8SC, SW, omit once crucial fact: GIS updates often require "long transactions" on a versioned-database, e.g. checkin/checkout semantics, not the short transactions supported by RDBMSs. Review articles which explain this issue in some detail are available at Laser-Scan and Smallworld (both members of OGC).
The document you are reading reviews three products which provide a spatial function interface to spatial data held in a commercial, general-purpose RDBMS (we are not here considering proper object-oriented repositories). In order to support long transactions, the spatial layer must largely ignore (or use for its own purposes) the RDBMS transactions, and instead provide a completely different long-transaction function interface. This is what the Smallworld product does: its "Version Managed Data Store" (VMDS) stores the spatial data, but it does not have an SQL or extended-SQL interface to that spatial data, only the attribute data. Laser-Scan's product does the same.
The reason that the object-repository products are not covered in more detail in this review is the current lack of an SQL interface for the spatial data. Object GIS repositories do not store the spatial data in a commercial RDBMS (except in a development project which rehosted VMDS on Oracle SDO). Many users are interested in an ESRI-proprietary interface because lots of spatial data is stored in ESRI systems. Far less spatial data is stored in Smallworld systems so their OO function interface ("Magik") is of less commercial interest.
Other development projects include "InSync" which merges the two types of transaction for the two types of data (Oracle RDBMS and VMDS) by symmetric replication.
Note that OGC Simple Features does not say anything about topology maintenance. Some members of OGC expect that topology will be maintained by the client application which uses the Simple Features interface to access a geodata repository. This is may be a mistaken view: others are adamant that it is the responsibility of the database to maintain integrity. "It may be a requirement of of the client application to keep [the topology] correct, but it is the database's responsibility to ensure that an out-of-sync database is not allowed", (Adrian Cuthbert, Laser-Scan).
Note that it is quite permissible for the geodata repository to maintain topology constraints and to return error codes if topology is broken or rollback transactions to prevent any topology violation at all. GeoOODBMS repositories can do this easily, and ESRI's Spatial Server may also be theoretically capable. Clearly for many GIS applications, e.g. land parcel records, it is crucial to maintain topology so topology maintenance will be a key feature distinguishing competing OpenGIS-compatible products and also a key feature to be resolved in designing any new GIS software architecture.
After topology, the next step is user-defined constraint maintenance within the geodata repository. Again, GeoOODBMSs with methods can do this easily today. Topic 4 of OGC's Abstract Specification is "Stored Functions and Interpolation"; but not only is this probably several years away from implementation (given that other topics are much more urgent), but also it does not mention constraint maintenance or topology.
This paper is about SQL/RDBMS spatial middleware, but it would be misleading not to list non-RDBMS systems which will support OpenGIS interfaces in the near future. Laser-Scan and Smallworld have already been mentioned and these use specifically spatial versioned databases, but there are also several GIS products based on the French O2 general purpose OODBMS which adheres to ODMG standards. (O2 has recently been bought by Unidata). Most of these products developed from the GEO2DIS ESPRIT project and some appear to have no plans for OpenGIS interfaces yet, e.g. Fleximage's O2Géo. AIS-SAI has a copy of the Matra/O2 product "IDA" installed at Ispra.
A common reason for analysts not considering GeoOODBMSs is because they have a shorter history of managing vast amounts of mission-critical data in operational installations. However, Laser-Scan has installations with NIMA, NOAA, USGS, US Census and many national mapping agencies, and Smallworld has a 3000-seat installation with Deutsche Telekom. Thus these modern systems should certainly be considered as realistic options to purchasers.
A technically interesting development is by Objective Insight Inc, a recent OGC member, and Universal Spatial Limited (USL, Canada) who are re-engineering their CARIS++ "standard spatial objects" component library (written in C++) to work with the Unidata/O2 spatial server using OpenGIS interfaces. However, there does not seem to be a product called Unidata/O2 spatial server yet, what is offered is an integration of CARIS++ with the generic O2 product. Where they can use OpenGIS interfaces in such an architecture is not clear.
The drawback of using an ODMG-compliant OODBMS is that none of the O2 spatial products appear to deliver long-transactions or versioned spatial databases. ODMG defines a flexible transaction scheme, but does not standardise a long-transaction protocol or versioning. This is an area where the proprietary products are in advance of the standards and we cannot look to OpenGIS to provide a solution within the next few years.
In NIMA's (now outdated) "GEOSPATIAL INFORMATION INFRASTRUCTURE TECHNOLOGY FORECAST" Sept 19, 1997, Harperg ( ) says:
"Looking specifically at spatial data one can see a number of specific needs. Firstly, there is a need for robust spatial joins. Secondly, the existing special data engine (SDE) and special data option (SDO - O8SC) technology approaches by ESRI and Oracle, respectively, needs reconsideration. Each is a highly proprietary system that makes integration very difficult in today's environment. They also require an undue amount of work on behalf of the requester in formulating queries. As the data becomes more complex the query formulation will become unmanageable. What is needed is a technology that allows reasonably straightforward queries to be formulated to interact with both spatial and the attribution data with optimal performance. This is a significant challenge but data complexity and volume demands that it be met one way or another."
Note that Harperg may have been assuming the use of pure RDBMS here when he talked about "spatial joins" (although elsewhere in his forecast he explored OODBMSs). He also seemed to assume that the user would be dealing with a "naked" system and not using user-friendly GIS software libraries that just access O8SC or SDE as a back-end repository. That is unrealistic today. He did not appear to have heard of SW.
Note that "spatial joins" are not exclusively a relational construct: a spatial join selects combinations on the basis of spatial criteria irrespective of the underlying technology. However, most people only use the word "join" in connection with RDBMSs.
A system only needs spatial joins if the user sees a low-level SQL interface. If the server only presents an OGC interface, it can use non-relational technology to implement the spatial operations.
This technology forecast made some unrealistic predictions. It has been superceeded by the OpenGIS Consortium's "Abstract Specification" documents which go into the various elements with more thought. Note that the publicly available OGC Abstract Specification is always some months out of date compared with the current draft (which is available only to OGC members).