I’ve already submitted some patches for fTools, now it’s time to look at GdalTools. I have already added an “Info” tool to display information about the raster, implemented internationalisation support, added several new options to the “Merge” and “Warp” tools, and now I am working on a batch mode. There are also plans to add more tools.
Hopefully I will be able to get most of the work done before the New Year, and then I will start improving Statist and developing another plugin.
Less than a month has passed since the release of QGIS 1.2.0 and now… 1.3.0 is available. Pretty fast, huh?
There are not that many changes, mostly bug fixes and minor improvements. There is a tendency to switch to native analysis tools (i.e. not related to GRASS), for example this version includes the “Raster terrain analysis” plugin for terrain analysis.
The announcement, as usual, is available on the official blog.
Release. There are a lot of changes, especially in the digitising tools: undo/redo support, history of edits, feature simplification and merging, ability to remove holes in polygons, and many other useful features. And also added support for attribute table field aliases, support for keyboard shortcut customisation, plugin and provider for working with OpenStreetMap… going through all the new features will take a long time.
The so-called visual changelog can be found on the developers’ blog. I haven’t translated the announcements in the wiki into Ukrainian and Russian yet because the official version isn’t ready yet. I think it will be cleaned up by tonight, and then I can start translating.
A new version of matplotlib 0.99 has been released recently, the list of changes can be found here. I have rebuilt the package for OSGeo4W, updated the corresponding wiki page and uploaded the new package to the server.
I continue to work on the Statist plugin. It’s been a while since I updated the public repository - I’ve been working with my local copy. But today I uploaded a pretty big update.
Among the most notable changes:
ability to export frequency distribution histogram in PNG, PS, EPS, SVG and PDF formats
added toolbar with interactive plot navigation tools such as zoom and pan
the frequency distribution can be displayed as a histogram or a line graph.
ability to manually set range for both axes
improved and more polished UI
extended number of statistical values calculated by the plugin
fixed the issue with the broken plot resizing when changing the plugin window size.
There is only one major bug left (there are probably others, but they have not shown up yet) - in some cases, the histogram is displayed in a rather strange, I would say suboptimal, way. I have some ideas about how to fix this and will have to test them.
Enjoy the plugin. If you have any problems and/or feature requests, do not hesitate to email the author :-).
It is used to obtain statistical information on the specified field of the vector layer attribute table. Both numeric (integer, real, and date) and text (string) fields are supported. The plugin can work on the whole attribute table as well as on selected features. In addition to displaying basic statistical values, Statist also displays a frequency distribution histogram of the field values.
Statist plugin dialog
To use Statist, it is necessary to have matplotlib installed (it can be installed via OSGeo4W or downloaded from the project page, as it is used to display the frequency distribution histogram.
The plugin is available from my QGIS plugins repository. Comments, feature requests, and bug reports are welcome. It is best to post them in the bugtracker, but email is fine too.
If someone does not need a frequency distribution histogram and unnecessary dependencies, they can use the “Basic Statistics” tool from fTools (now included in core). After my patch, it has the same functionality as Statist except for the frequency distribution histogram.
Let’s summarise what was said in the posts about PostGIS and ArcSDE. The image loading speed tests showed that PostGIS was much slower than ArcSDE.
However, do not forget that WKTRatser is still in an early stage of development (version 0.1.6 at the time of testing) whereas ArcSDE has been around for years. Also, rasters are usually loaded into the database once, so the load speed test is of little practical value. It would be much more interesting to compare the performance of these products when processing rasters. Unfortunately, this is not possible for a number of reasons.
ArcSDE was tested on the same machine using the same dataset (see PostGIS test). The discs were formatted before the test, and the system was restored from the snapshot. As I could not get ArcSDE to work with my self-compiled PostgreSQL 8.3.7, I tested it on its bundled PostgreSQL 8.3.0 + ArcGIS 9.3 SP1 (build 1850) + ArcSDE 9.3.
As in the PostGIS test, the database cluster was located on a separate 80 GB disc. For the purity of the experiment, the original image in MrSID format was converted to ERDAS IMAGINE using ArcGIS tools. Loading and all other operations were done using ArcGIS Python scripts, not ArcCatalog.
The conversion to the ERDAS IMAGINE format has been carried out with the following script:
This operation took exactly 2 seconds :-). The time is so small that it can be neglected. Now we can load the raster into the created dataset:
import arcgisscripting
gp = arcgisscritpting.create()
gp.toolbox ="management"gp.workspace ="d:\raster"gp.CreateRasterDataset_management("N-38-45.img","Database Connections/raster.sde","LAST","FIRST","0","#","NONE","0","NONE")
The raster loading process was quite fast, taking only 1337 s (~22 min). After loading the image, the database cluster grew from 42,495,444 bytes (~40.5 MB) to 6,934,776,988 bytes (~6.45 GB).
Hitachi DeskStar HDT725032VLA360 (320 Gb). Divided into two partitions: 10 Gb for the operating system and the rest for data
Hitachi DeskStar HDS721680PLA380 (80 Gb). Single partition for the entire disk.
OS: Windows XP Pro SP3
The test itself was carried out according to Mateusz’s instructions. When the SQL representation of the raster was created, the resulting SQL file was written to the second disc, and before loading it into the database, the file was moved to the data partition of the first disc. After each stage of testing, all disks were defragmented using OS tools, and the machine rebooted.
This Landsat scene was used as a test image. Quite detailed information about the image can be found in the file N-38-45-45.met, which is located next to it, and I give a partially reduced output of gdalinfo below:
Before starting the test, a snapshot of the system was taken, and all disks were defragmented using OS tools. The source data (i.e., the raster) is located on the second partition of the 320 Gb disc.
Unfortunately, both GDAL and WKTRaster do not yet fully support the MrSID format, so the raster was converted to the ERDAS IMAGINE format (*.img) using the following command:
The conversion took 1160 s (~19 min), and the resulting file occupied ~4.7 Gb of disk space. This file was used in all subsequent operations. We have the image in the supported format, and now we need to build pyramids (overviews), which are not present in our file:
Now we can prepare the image for loading into the database. The preparation consists of using the Python script gdal2wktraster.py (shipped with WKTRaster) to generate an SQL file with the image dump.
This is quite a long process, so make sure you have some tea/coffee. On my test platform, it took 2904 s (~48 min). At the end, the script generates a report of the work done, showing how many tables will be created in the database after the script is loaded and how many tiles (blocks) will be in each table:
------------------------------------------------------------
Summary of GDAL to WKT Raster processing:
------------------------------------------------------------
Number of processed raster files: 1
List of generated tables (number of tiles):
1 N_38_45_img (103152)
2 o_2_N_38_45_img (25872)
3 o_4_N_38_45_img (6468)
4 o_8_N_38_45_img (1638)
5 o_16_N_38_45_img (420)
6 o_32_N_38_45_img (110)
7 o_64_N_38_45_img (30)
8 o_128_N_38_45_img (9)
The gdal2wktraster.py script produced the file N_38_45_img_rb_128.sql, which took up — hold on tight :-) — 13,564,596,564 bytes (~12.6 GB). There you go.
Since all the necessary preparations were made at the PostGIS configuration stage, the only remaining step is to load this monstrous script into the database.:
Before moving on to the actual testing, I will describe the process of configuring PostgreSQL + PostGIS + WKTRaster. Since PostgreSQL has been built from source, all the work normally performed by the installer (creating users, initialising database clusters, etc.) has to be done manually.
The compilation process was described in the previous post. Here I will assume that you are using Windows XP Pro; PostgreSQL, GEOS, Proj, PostGIS and WKTRaster are already compiled, and everything is in the c:\postgres directory.
