Hi everyone,
New user here. Very impressed with Locus! I hope to use it for hiking in the mountains near Seattle. To that end, I've started to research the best way to get US Geological Survey topo maps into Locus (offline).
USGS is transitioning from GeoTIFF to GeoPDF. A new database of the latest-revised topo maps is now online for free at nationalmap.gov in their "historical topos" collection. The quality is much better than the "DRG" GeoTIFFs that we have been using for years. Fortunately, although they are in GeoPDF, the GDAL library (gdal.org) can extract the raster contents and save in a variety of formats. So GeoPDF is not a problem.
That brings us to the mess that is the mobile map-format space... Appropriate formats already existed (e.g. tiled geotiff, and a free/open sqlite format---Spatialite and Rasterlite, http://www.gaia-gis.it/gaia-sins/ (http://www.gaia-gis.it/gaia-sins/) ---complete with an actual SPECIFICATION :o). But, the original players in this space chose poorly (IMO) and so Locus was stuck with a choice: use a "good" format, that no one else is using, or use formats that actually work with existing software.
Anyway, my goal is to write something in Python to take the GeoPDF rasters and put them in a Locus-compatible format. I don't want to depend on proprietary solutions like MAPC2MAPC, TTQV4, etc., partly because I am on Linux, and partly because I think it just encourages the proliferation of more useless formats. GDAL can do most of the hard work here, including the import from GeoPDF, tiling/pyramid creation, and any needed warping / re-projection. I just need to settle on a target output format, and then I need a spec for that.
Locus sqlite is simple enough, but from what I've been able to understand of some older discussions here, not the fastest? Then there is RMAP format, and GEMF format, but I can't find documentation or code for either of them. I guess it is also possible, that by the time I finish a tool, there will be another new format supported! So I am looking for advice. What should I target if I want to get this working by summertime?
I was also wondering...
1) Does Locus support multiple image types in its tiled formats? I exported a simple sqlite and saw that the blobs are PNG. Can the blobs also be JPEG? This could save a lot of space, without too much quality loss.
2) USGS topos are [almost] all in NAD27 datum. Is NAD27 (EPSG:4267) supported in Locus? Or should I plan on re-projecting into something like WGS84? I get the idea WGS84 is just "assumed" in some of the simple tile formats... is that right?
Thanks,
Mark
Hi!
The GEMF definition can be found at http://www.cgtk.co.uk/gemf (http://www.cgtk.co.uk/gemf)
GEMF is fast and light to use. And it supports huge (> 2GB) maps, which are efficiently split into several files.
There're some example Python scripts in that page.
My sqlite-->GEMF converter can be found via a link. It is written in C, but I wrote testing versions of some algorithms in Python.
I guess my input is mostly in the way of handling non-rectangular, skattered maps. If you wish, I can share the code.
Thanks stebu,
Somehow I missed that page. Very nice! From your username, I assume you are the author?
Anyway, GEMF looks like a good solution. What would be the pros/cons vs RMAP from a technical point of view? (GEMF probably wins anyway, because it is documented, and RMAP is not.)
It would give me more confidence if there were an emerging standard for mobile tile stores---for example, if Oruxmaps would also support GEMF. Maybe if I can get the topos nicely encoded into ready-to-use GEMF and put them up as a torrent, that will spur demand 8-)
I assume menion plans to support GEMF in Locus indefinitely?
Mark
PS: I may not have time to work on this for a while, but if you wouldn't mind sharing your Python code that would be great--thanks!
PPS: Are the tiles assumed to be in "web mercator" format? meaning that if my source maps are NAD27 I need to warp them into WGS84 before tiling?
Hi Mark
Well, actually Al Budden is the author for GEMF. I just made the conversion software.
There are possibilities to define various stuff with GEMF maps:
-The tile size has its specific field in the header.
-The provider info (e.g. the first provider) could hold a code, which defines the projection
-and maybe some other definitions for the maps presentation.
But these are matters for Al to decide. And after that Menion needs to implement them into Locus.
I'm fairly positive that currently Locus only supports GEMF maps with WGS84 and the ellipsoidal projection (as in Google maps).
I guess the beast has been let free long ago, it will be extremly difficult to standardise a single map format for raster maps.
GEMF has the advantage of being well documented and simple to implement. If it is not adopted as the standard, it could still be useful as a conversion tool between different formats.
I'll have to check the state of my software. I'll PM you within a couple of days.
stebu
Basically, i hate the idea of any conversion. All those custom mobile formats might have been necessary ten years ago, when we had no horsepower to speak of. Nowadays, with at least 2x1ghz in our pockets... whats the point?
If locus doesnt want to support geotiff/geopdf directly, a local wms server on android easily could. No reason not to use the same map files on desktop and on mobile... to hell with all conversions :)
do you think that 2 x 1GHz on mobile is same as 2 x 1GHz on desktop PC? This is not correct and difference is huuuuge!
Well, now we see that GeoTIFF is soooo last decade. The future is GeoPDF! And guess what? Dragging a map view around in Adobe Reader feels a big laggy on my desktop, a quad-core i7!! The thing to keep in mind is that no matter how fast and capable the mobile devices get,
1) We will always be able to measure battery life in units of CPU cycles. Fewer cycles => longer battery life. I like that.
2) For any given map size/complexity, if a given mobile device X can render it well in a non-tiled format, then the same mobile device will be able to render a much larger map well, when packed efficiently. There will always be that differential.
But, now that we have Menion's attention... Would you agree that GEMF is a good target for an offline map conversion project?
Of course its not the same. But does it matter? A local tile server could easily do intelligent caching if it thinks the source format is too cpu intensive to read. This could even go as far as converting whole regions of a map into something else for quicker reads. But whatever it does for speed reasons, it should not ever bother the user with it. I just copy over my geotiff and am done.
I really think the time is ripe to kill custom formats alltogether. Those silly conversion things have messed up the geo world long enough and i hate them with a passion. A bit radical, but true :-).
Of course writing that is quite an undertaking. It doesnt necessarily have to be included in locus or be written by menion. An external tile server app still seems like a good idea to me. If only i was more into java coding and less into mountainbike travelling... i would give it a try. And then shut up forever when i find out its not doable...
Note that the cpu and battery intensive conversions would only have to be done ONCE on the mobile... or whenever the tile server cache is cleared. So its not a battery nor a speed issue really. The important thing here is that everything happens seemlessly on android and the user is never ever bothered with desktop tools. Hell... desktops are dying anyway, mobile horsepower is increasing by the minute, so why not think into this direction?
agree Mark. Create as fast and mainly as simple as possible format, increase battery life and also allow to use feature also on less powered devices. So Joe, I personally do not think that problem is converting to another format. Why not? If there will be any simple and useful tool and final format will be well documented and widely used in many apps ... than by me, why not. It's similar with video format. On desktop, you'll always be one step far compare to mobile device. Ah sorry, I forget, you note can play Blue-ray disks ;)
and about format ... I'm not expert on this, here on forum is many more specialists than me, Joe for example :) (this is not a joke) anyway I don't think that current GEMF format is best format for wide use. Mainly because it currently support only Spherical mercator (and in case of Locus also some other Mercator modification). For example RMAP format, is pretty simple and usable. Anyway from my experience, I discovered around ten modifications and because of lack of documentation, not every RMAP is working in Locus. It's really mystery for me, why not yet exist any simple and fast, widely used and mainly general, format for tiled maps ... weird
Quote from: "joeloc"Note that the cpu and battery intensive conversions would only have to be done ONCE on the mobile... or whenever the tile server cache is cleared. So its not a battery nor a speed issue really. The important thing here is that everything happens seemlessly on android and the user is never ever bothered with desktop tools. Hell... desktops are dying anyway, mobile horsepower is increasing by the minute, so why not think into this direction?
so you want to have converted copy of map in mobile? :) Joe, I really do not think that mobiles have such power ... isn't anywhere available some compare of mobile and desktop processors at one graph?
Two years from now, youll have to do withouth desktop conversion tools, because fewer and fewer people will actually own a desktop computer. So in my opinion, time & work spent on desktop tools is simply wasted in the long run.
Not a complete copy obviously. The tile server just reads whatever part is wanted from whatever format it supports and sends it over to locus. It also caches it into something quicker if it thinks reading the original data was too slow/too complex.
I would gladly give up a few milliseconds when looking at a map region for the very first time for the benefit of getting rid of desktop tools and custom formats.
I know and understand, it was just "joke" with copy of data.
Anyway, I'm don't share idea with future after two years. But ... we'll see for sure ;)
When any format, well documented, simply to use and to create occur, you should be sure, that Locus will have it. Unfortunately to write own GeoPDF or GeoTIFF readed ... I'm worried that I'm skilled enough for this ...
I dunno, Joe, I still think we're going to pay for that concept in battery life. That always ends up being the limiting factor. Maybe if we get another order-of-magnitude improvement in energy density... It will happen, eventually, although I do worry about the safety implications. Imagine 50 watt-hours (that's 180 kW-seconds--imagine it) of energy sitting in your pocket. It's a bit frightening.
Re: GEMF vs RMAP, it seems like GEMF could be extended to support other projections. Until someone documents and standardizes "open RMAP" I am too nervous to use it.
If you want a third possibility, I would suggest taking a look at spatialite and rasterlite (the tiled-raster extension of the standard). These are at least supported in GDAL, which instantly elevates them into standards, because the GDAL library is kind of universal in GIS software. They are sqlite formats of higher complexity than the simple sqlite Locus uses today. I don't know enough about it to say if it solves the speed problems, though.
How much longer do you think desktops will live? The sooner you cut off all dependencies, the better you'll be prepared...
But anyway, all this geotiff/python/tileserver/conversion things is mainly for power users. I think the "normal guy" would benefit most if you could strike deals with mapping companies and offer in-app download purchases or whatever. Sad but true...
Talking formats, The advantage of rmap was mainly that it is supported for export by ttqv and compeland, who in turn have deals with many commercial map providers. So in theory, users dont need to dive down to command line level to create locus mapa.
Btw... i still cannot believe that todays devices would have any issues whatsoever with decoding geotif. Whats all the scaryness about? Its just a few tiles with a few coordinates and some raster data and overviews or not. Why would that be any problem at all, cpu or battery wise? What hyper complex tiff implementation details am i missing?
Of course it needs a bit more code than something as simple as rmap. But once the code is there, wouldnt it run just as efficient?
another idea: in one years time, osm will beat all commercialy available maps both in accuracy and rendering quality everywhere on this planet. so all thats really needed is advancements in mapsforge, custom raster data is a dying thing of the past :)
just kidding...
Quote from: "joeloc"How much longer do you think desktops will live? The sooner you cut off all dependencies, the better you'll be prepared...
I'm very much a "live free or die" kind of person, so personally I hope the desktop, or something like it, lives forever. I like my phone, and I'm impressed with my parents' iPad, but if the walled-garden / app-store model of computing is our only future, we will have lost something profound.
(Please note, when I say "free" I mean free in the sense of freedom, not in the sense of price. I have no problem paying for quality software like Locus Pro, but I don't like proprietary interchange formats, so I don't care what TTQV4 or Compeland happen to use, unless those formats are open, like geotiff. And, top priority should go to supporting those formats used in open-source software like GDAL, because that ultimately makes the world a better place. BTW, geopdf seems to be going the way of pdf, i.e. an initially proprietary format shoved down our throats but eventually standardized and open. All of the $$$ for one company while sort-of keeping the moral high ground :roll: )
If you're interested in some radical speculation on the future of mobile vs desktop computing, I highly recommend Cory Doctorow's recent keynote at the 28th Chaos Computer Conference in Berlin, "The Coming War on General Purpose Computation":
http://www.youtube.com/watch?v=HUEvRyem ... r_embedded (http://www.youtube.com/watch?v=HUEvRyemKSg&feature=player_embedded)
Sorry for the tangent.
I totally agree with free and open formats. Unfortunately, thats not how the mapping world works today. In the end, the user buys a dvd from kompass or ign with a pretty map and then wants to use his map in Locus. How to make this as easy and painless as possible?
Right now this involves hunting down OziMapTrans in the dark web on some shady download sites, convert tne proprietary dvd to geotiff, then download free ttqv demo version, import the geotif, export to rmap.
Not impossible, but still a major hassle. How to make it simpler? I dont know. But another custom format unfortunately wont help a lot.
Ps, sorry for posting so much. Sitting on a beach on gran canaria waiting for the night... a bit boring :)
Astronomer, eh? That's my hobby too. Lucky you, in Gran Canaria!
The way it seems to work here in the US, at least for topographic maps, is
1) The data are free, but sometimes inconvenient to obtain, as the govt doesn't like spending money on fast pipes.
2) Commercial entities take the free data, and convert to a proprietary format that only works on their GPS, or in their software (e.g. National Geographic TOPO!). They also make sure their HW and SW won't read the free formats.
3) Profit!!
(As you can probably tell, I don't have respect for this business model. I think it is fundamentally immoral.)
My starting premise is usually that the data are free to begin with. When this is not true, then the rules change... e.g. it used to be like that for roadmaps, but as you noted OSM is dragging nav data into the free world too. Good times!
Hi, I am using Locus with my own GIS data layers, using Maptiler and MOBAC to create the RMAP SQLite maps. These are both free (open source) tools which work on both Linux and Windows. See my blog http://pvanb.wordpress.com/2012/03/19/c ... bile-maps/ (http://pvanb.wordpress.com/2012/03/19/create-mobile-maps/) for a short tutorial. It involves a few steps, but once you get the hang of it it is simple enough
on the topic of the multiple formats, it certainly is a pain, but MOBAC seems to be able to convert quite a few mobile map formats back and forth, with the added advantage that you can create the same map for different mapping apps with virtually the same effort.
Edit; of course, Maptiler and MOBAC are desktop solutions, so not everybody in this thread will be happy with it, but I am not sure how soon you will see mobile apps that can do the same, not for very large maps anyway.
Hi
I'm a very happy user of Locus and I use it with topographic maps from France generated with GDAL/MapTiler and Mobac (a tool chain very similar to the one used by pvb.
Planning a trip to the US, I'd like to do the same with USGS topo maps. So far, the TIFF or PDF maps of the 7.5 series I've found all include a border around the real map containing the legend. This border is annoying if you want to stitch several maps together before generating the Locus maps (for example, to cover the whole Yosemite valley in a single database) as the maps overlap. Does anyone know a way to automate (with Python for example) the stitching process? In theory, one should be able to calculate the georeferences of the 4 topo map corners within the larger map.
Alternatively, does anyone know a tile server for this kind of map?
GlobalMapper has an option to auto-crop these types of borders when loading tiles. It sometimes works nicely and sometimes not at all. Not sure if this is available in the demo version, you d have to try it out.
After some searching on the web, I found a small open source program Quadjoin (//http://quadjoin.sourceforge.net/) written by Frank Glandorf that does exactly that I wanted, stitching 7.5 topo maps. When tested on the Grand Canyon area after some Maptiler processing, I found a little offset of a few meters on longitude but it is also present in the original geotif files from the USGS (using the projection description included in the file).
Vive le web et l'open source (as we say in French)!
It would be very powerful of locus could read or import geotiffs directly. It is an old format but very universal. I haven't tried importing custom rasters yet but I will be looking at this in the coming weeks.
While the open SQL raster is berry efficient and powerful if locus could read geo jp2k files directly it would be very cool. Jpg 2000 is an efficient wavelet compression and I believe an open format. I have been horsing around with converting raster formats for mobile platforms.for 15 years and having a mobile mapping program that you could just drop your own custom images on would be a game changer.
Hey, I'm back with an update on my USGS topo-mosaicking project.
I ended up going with the GEMF format, just because it is documented, and RMAP really isn't. Also, there was python code provided to generate a GEMF file from a directory structure containing the raster-tile "pyramid."
I am starting out with the "historic" USGS GeoPDFs, which are the same maps as digital raster graphic (DRGs), but re-scanned at higher resolution and archived in one place for the whole country. I had to get the complete list for the state of Washington, sort it in a spreadsheet, then write a script which attempts to pick out and download only the most recent version of each 7.5' quad. This is an error-prone process, because many of the names are not perfectly consistent over time; for example, "Mountain" becomes "Mtn" or "Mt" and so on. So it requires some manual intervention...
Next, GDAL tools are used to extract the rasters from each GeoPDF at 400 dpi. At the same time, the "neatline" from the GeoPDF metadata is used to remove the map collars, and the maps are warped from whatever projection/datum they had in the GeoPDF, to the google-maps standard EPSG:900913, with the result saved as GeoTIFF.
Then, I use a modified version of the gdal2tiles.py included in the GDAL distribution to simultaneously mosaic a bunch of these GeoTIFFs and save the result as a tile pyramid at zoom levels 10-16. (Modified, because out of the box it only supports PNG tiles, and I want JPG--much smaller with negligible loss in quality.) This is the interesting part, as I have only tried a set of four quads; my goal is to mosaic ~1000 of them (the whole state).
Finally, a modified version of the python code on the GEMF web page takes the tile-pyramid directory structure and packs it into a GEMF file. Transfer to my phone, and voila! It performs great in Locus!
So the whole thing is done using open-source tools, and best of all, it's entirely at the command line--no GUIs. Scriptable FTW :D
I hope to have some larger GEMFs to play with in another week, and will report back here.
Does anyone know the file-size limit on an Android-formatted SD card? Is it 2 GB, 4 GB, or something else?
Mark
Quote from: "mark03"Does anyone know the file-size limit on an Android-formatted SD card? Is it 2 GB, 4 GB, or something else?
Mark
We had discussions here in the forum: 2GB for Android 2.x, 4GB for Android >=3.
However taking into account that most phones use 2.x one should consider that sharing maps >2GB will restrict the usage to few phones/users.
Quote from: "tommi62"We had discussions here in the forum: 2GB for Android 2.x, 4GB for Android >=3.
Thanks. The whole state of Washington will probably come in near 32 GB, so I may split into west/east. GEMF has support for splitting a single map across multiple files. I hope Locus' GEMF implementation supports that...?
Tommi are you sure with 4GB limit on all 3+ devices? I did not tested it, but if this is true, I can change all possible testing I do in Locus to prevent using maps bigger then 2GB on all devices. May anyone other confirm that such map work in Locus or any other program?
Hi Menion,
my statement is derived from these posts:
viewtopic.php?f=27&t=2031&p=13044&hilit=4GB#p13044 (http://forum.asamm.cz/viewtopic.php?f=27&t=2031&p=13044&hilit=4GB#p13044)
viewtopic.php?f=25&t=261&p=965&hilit=4GB#p965 (http://forum.asamm.cz/viewtopic.php?f=25&t=261&p=965&hilit=4GB#p965)
However I find it not so useful to use maps >2GB as this will split users/phones which can use them and those who can't.
Hi Menion,
From my perspective the actual limit should not matter that much. I will be making maps much bigger than 2 or 4 GB anyway. But the GEMF spec defines how to split one map across files like foo.gemf, foo.gemf-1, foo.gemf-2, and so on. Does Locus support that part of the spec?
Quote from: "mark03"Hi Menion,
From my perspective the actual limit should not matter that much. I will be making maps much bigger than 2 or 4 GB anyway. But the GEMF spec defines how to split one map across files like foo.gemf, foo.gemf-1, foo.gemf-2, and so on. Does Locus support that part of the spec?
I tested my GEMF converter with split files, but each file was much smaller than 2 GB.
I'm
almost positive, that I tested that map on Locus.
And I'm positive, that each map I that tested in Locus worked fine.
Success!
Over the weekend I generated topo mosaics covering half the state (about 800 source GeoPDFs), tiled at zoom levels 9 to 16. On my phone's screen (Samsung Vibrant), zoom level 16 is somewhat zoomed-in compared to the paper maps, but still sharp thanks to the new GeoPDF scans which are typically at 500 dpi--much better than the old DRG GeoTIFFs. Panning and zooming is still snappy, even with the map split across several 2-GByte GEMF files.
This is a good way to find errors in the USGS georeferencing. I've found at least two quads with the wrong coordinates, and sent feedback to the people at the historical maps collection. Their database is also missing ~30 quads here and there across the state, which I hope will be filled in this summer.
I think I can probably fit the entire state in about 20 GBytes. I am using JPEG tiles for considerable space savings over PNG, with very little loss in quality.
Quote from: "mark03"So the whole thing is done using open-source tools, and best of all, it's entirely at the command line--no GUIs. Scriptable FTW :D
I'm trying to process the tracks from my last summer vacation in Soutwest USA and I need to re-process the USGS maps I had stichted using various tools (including Maptiler and Mobac). As I'm also a Python user, I'm interested in the scripts that seems to automate this process. Is there any chance that they could be shared, in an open source spirit?
Quote from: "kerenoc"I'm trying to process the tracks from my last summer vacation in Soutwest USA and I need to re-process the USGS maps I had stichted using various tools (including Maptiler and Mobac). As I'm also a Python user, I'm interested in the scripts that seems to automate this process. Is there any chance that they could be shared, in an open source spirit?
Check the GEMF site //http://www.cgtk.co.uk/gemf , it has a link to some Python scripts.
Thanks for the link. Still searching for the code specific to USGS maps (especially to remove maps collars).