Preferred offline raster format, five months from now?

mark03 · January 10, 2012, 00:18:30

Hi everyone,

New user here. Very impressed with Locus! I hope to use it for hiking in the mountains near Seattle. To that end, I've started to research the best way to get US Geological Survey topo maps into Locus (offline).

USGS is transitioning from GeoTIFF to GeoPDF. A new database of the latest-revised topo maps is now online for free at nationalmap.gov in their "historical topos" collection. The quality is much better than the "DRG" GeoTIFFs that we have been using for years. Fortunately, although they are in GeoPDF, the GDAL library (gdal.org) can extract the raster contents and save in a variety of formats. So GeoPDF is not a problem.

That brings us to the mess that is the mobile map-format space... Appropriate formats already existed (e.g. tiled geotiff, and a free/open sqlite format---Spatialite and Rasterlite, http://www.gaia-gis.it/gaia-sins/ ---complete with an actual SPECIFICATION

). But, the original players in this space chose poorly (IMO) and so Locus was stuck with a choice: use a "good" format, that no one else is using, or use formats that actually work with existing software.

Anyway, my goal is to write something in Python to take the GeoPDF rasters and put them in a Locus-compatible format. I don't want to depend on proprietary solutions like MAPC2MAPC, TTQV4, etc., partly because I am on Linux, and partly because I think it just encourages the proliferation of more useless formats. GDAL can do most of the hard work here, including the import from GeoPDF, tiling/pyramid creation, and any needed warping / re-projection. I just need to settle on a target output format, and then I need a spec for that.

Locus sqlite is simple enough, but from what I've been able to understand of some older discussions here, not the fastest? Then there is RMAP format, and GEMF format, but I can't find documentation or code for either of them. I guess it is also possible, that by the time I finish a tool, there will be another new format supported! So I am looking for advice. What should I target if I want to get this working by summertime?

I was also wondering...
1) Does Locus support multiple image types in its tiled formats? I exported a simple sqlite and saw that the blobs are PNG. Can the blobs also be JPEG? This could save a lot of space, without too much quality loss.
2) USGS topos are [almost] all in NAD27 datum. Is NAD27 (EPSG:4267) supported in Locus? Or should I plan on re-projecting into something like WGS84? I get the idea WGS84 is just "assumed" in some of the simple tile formats... is that right?

Thanks,

Mark

stebu · January 10, 2012, 09:13:29

Hi!

The GEMF definition can be found at http://www.cgtk.co.uk/gemf
GEMF is fast and light to use. And it supports huge (> 2GB) maps, which are efficiently split into several files.
There're some example Python scripts in that page.

My sqlite-->GEMF converter can be found via a link. It is written in C, but I wrote testing versions of some algorithms in Python.
I guess my input is mostly in the way of handling non-rectangular, skattered maps. If you wish, I can share the code.

mark03 · January 10, 2012, 20:12:02

Thanks stebu,

Somehow I missed that page. Very nice! From your username, I assume you are the author?

Anyway, GEMF looks like a good solution. What would be the pros/cons vs RMAP from a technical point of view? (GEMF probably wins anyway, because it is documented, and RMAP is not.)

It would give me more confidence if there were an emerging standard for mobile tile stores---for example, if Oruxmaps would also support GEMF. Maybe if I can get the topos nicely encoded into ready-to-use GEMF and put them up as a torrent, that will spur demand 8-)

I assume menion plans to support GEMF in Locus indefinitely?

Mark

PS: I may not have time to work on this for a while, but if you wouldn't mind sharing your Python code that would be great--thanks!
PPS: Are the tiles assumed to be in "web mercator" format? meaning that if my source maps are NAD27 I need to warp them into WGS84 before tiling?

stebu · January 10, 2012, 23:44:34

Hi Mark

Well, actually Al Budden is the author for GEMF. I just made the conversion software.

There are possibilities to define various stuff with GEMF maps:
-The tile size has its specific field in the header.
-The provider info (e.g. the first provider) could hold a code, which defines the projection
-and maybe some other definitions for the maps presentation.
But these are matters for Al to decide. And after that Menion needs to implement them into Locus.

I'm fairly positive that currently Locus only supports GEMF maps with WGS84 and the ellipsoidal projection (as in Google maps).

I guess the beast has been let free long ago, it will be extremly difficult to standardise a single map format for raster maps.
GEMF has the advantage of being well documented and simple to implement. If it is not adopted as the standard, it could still be useful as a conversion tool between different formats.

I'll have to check the state of my software. I'll PM you within a couple of days.

stebu

joeloc · January 11, 2012, 18:40:22

Basically, i hate the idea of any conversion. All those custom mobile formats might have been necessary ten years ago, when we had no horsepower to speak of. Nowadays, with at least 2x1ghz in our pockets... whats the point?

If locus doesnt want to support geotiff/geopdf directly, a local wms server on android easily could. No reason not to use the same map files on desktop and on mobile... to hell with all conversions

Menion · January 11, 2012, 18:41:47

do you think that 2 x 1GHz on mobile is same as 2 x 1GHz on desktop PC? This is not correct and difference is huuuuge!

mark03 · January 11, 2012, 18:54:22

Well, now we see that GeoTIFF is soooo last decade. The future is GeoPDF! And guess what? Dragging a map view around in Adobe Reader feels a big laggy on my desktop, a quad-core i7!! The thing to keep in mind is that no matter how fast and capable the mobile devices get,

1) We will always be able to measure battery life in units of CPU cycles. Fewer cycles => longer battery life. I like that.
2) For any given map size/complexity, if a given mobile device X can render it well in a non-tiled format, then the same mobile device will be able to render a much larger map well, when packed efficiently. There will always be that differential.

But, now that we have Menion's attention... Would you agree that GEMF is a good target for an offline map conversion project?

joeloc · January 11, 2012, 18:55:08

Of course its not the same. But does it matter? A local tile server could easily do intelligent caching if it thinks the source format is too cpu intensive to read. This could even go as far as converting whole regions of a map into something else for quicker reads. But whatever it does for speed reasons, it should not ever bother the user with it. I just copy over my geotiff and am done.

I really think the time is ripe to kill custom formats alltogether. Those silly conversion things have messed up the geo world long enough and i hate them with a passion. A bit radical, but true :-).

Of course writing that is quite an undertaking. It doesnt necessarily have to be included in locus or be written by menion. An external tile server app still seems like a good idea to me. If only i was more into java coding and less into mountainbike travelling... i would give it a try. And then shut up forever when i find out its not doable...

joeloc · January 11, 2012, 18:59:05

Note that the cpu and battery intensive conversions would only have to be done ONCE on the mobile... or whenever the tile server cache is cleared. So its not a battery nor a speed issue really. The important thing here is that everything happens seemlessly on android and the user is never ever bothered with desktop tools. Hell... desktops are dying anyway, mobile horsepower is increasing by the minute, so why not think into this direction?

Menion · January 11, 2012, 19:01:37

agree Mark. Create as fast and mainly as simple as possible format, increase battery life and also allow to use feature also on less powered devices. So Joe, I personally do not think that problem is converting to another format. Why not? If there will be any simple and useful tool and final format will be well documented and widely used in many apps ... than by me, why not. It's similar with video format. On desktop, you'll always be one step far compare to mobile device. Ah sorry, I forget, you note can play Blue-ray disks

and about format ... I'm not expert on this, here on forum is many more specialists than me, Joe for example

(this is not a joke) anyway I don't think that current GEMF format is best format for wide use. Mainly because it currently support only Spherical mercator (and in case of Locus also some other Mercator modification). For example RMAP format, is pretty simple and usable. Anyway from my experience, I discovered around ten modifications and because of lack of documentation, not every RMAP is working in Locus. It's really mystery for me, why not yet exist any simple and fast, widely used and mainly general, format for tiled maps ... weird

Menion · January 11, 2012, 19:04:13

Quote from: "joeloc"Note that the cpu and battery intensive conversions would only have to be done ONCE on the mobile... or whenever the tile server cache is cleared. So its not a battery nor a speed issue really. The important thing here is that everything happens seemlessly on android and the user is never ever bothered with desktop tools. Hell... desktops are dying anyway, mobile horsepower is increasing by the minute, so why not think into this direction?

so you want to have converted copy of map in mobile?

Joe, I really do not think that mobiles have such power ... isn't anywhere available some compare of mobile and desktop processors at one graph?

joeloc · January 11, 2012, 19:07:34

Two years from now, youll have to do withouth desktop conversion tools, because fewer and fewer people will actually own a desktop computer. So in my opinion, time & work spent on desktop tools is simply wasted in the long run.

joeloc · January 11, 2012, 19:12:30

Not a complete copy obviously. The tile server just reads whatever part is wanted from whatever format it supports and sends it over to locus. It also caches it into something quicker if it thinks reading the original data was too slow/too complex.

I would gladly give up a few milliseconds when looking at a map region for the very first time for the benefit of getting rid of desktop tools and custom formats.

Menion · January 11, 2012, 19:20:31

I know and understand, it was just "joke" with copy of data.

Anyway, I'm don't share idea with future after two years. But ... we'll see for sure

When any format, well documented, simply to use and to create occur, you should be sure, that Locus will have it. Unfortunately to write own GeoPDF or GeoTIFF readed ... I'm worried that I'm skilled enough for this ...

mark03 · January 11, 2012, 19:36:25

I dunno, Joe, I still think we're going to pay for that concept in battery life. That always ends up being the limiting factor. Maybe if we get another order-of-magnitude improvement in energy density... It will happen, eventually, although I do worry about the safety implications. Imagine 50 watt-hours (that's 180 kW-seconds--imagine it) of energy sitting in your pocket. It's a bit frightening.

Re: GEMF vs RMAP, it seems like GEMF could be extended to support other projections. Until someone documents and standardizes "open RMAP" I am too nervous to use it.

If you want a third possibility, I would suggest taking a look at spatialite and rasterlite (the tiled-raster extension of the standard). These are at least supported in GDAL, which instantly elevates them into standards, because the GDAL library is kind of universal in GIS software. They are sqlite formats of higher complexity than the simple sqlite Locus uses today. I don't know enough about it to say if it solves the speed problems, though.