Author Topic: sqlite to GEMF map conversion tool  (Read 7529 times)

Offline stebu

  • Premium
  • Padavan of Locus
  • ***
  • Posts: 252
  • Thanked: 1 times
    • View Profile
  • Device: Xcover 2
sqlite to GEMF map conversion tool
« on: October 03, 2011, 17:43:11 »
This is a small PC utility that enables to convert and combine several sqlitedb maps (downloaded from your Android /Locus/maps into PC) and other GEMF maps into one (possibly HUGE) GEMF map.

GemfTool information and download

Enjoy!

Nice comments are wellcome on this board :roll:
-stebu
(edit: 30.10.2011 Additional functionality, ne name)
« Last Edit: October 30, 2011, 15:02:35 by stebu »
 

Offline stebu

  • Premium
  • Padavan of Locus
  • ***
  • Posts: 252
  • Thanked: 1 times
    • View Profile
  • Device: Xcover 2
Re: sqlite to GEMF map conversion tool
« Reply #1 on: October 05, 2011, 10:17:03 »
Version beta.002 released at the same link.
Added some string length checking for crazy inputs.
Added check that the header table fits completely inside a maximum (2Gb) file.

I tested by creating a combined map from Amsterdam and Helsinki at zoom level 14. It's a foolish effort, header contains over 600000 empty tiles (7.2Mb)! Took a couple of minutes to create. Not recommended practice, but it worked.

  zoom, min x, max x, min y, max y, (act/area)
    14,  8410,  9323,  4701,  5391,     (157/631574)
« Last Edit: January 01, 1970, 01:00:00 by Guest »
 

Offline stebu

  • Premium
  • Padavan of Locus
  • ***
  • Posts: 252
  • Thanked: 1 times
    • View Profile
  • Device: Xcover 2
Re: sqlite to GEMF map conversion tool
« Reply #2 on: October 29, 2011, 01:23:43 »
I skipped a version, it's now 004.

The tool has been renamed to GemfTool(.exe). It can now combine Locus' sqlite maps and gemf maps (and a mixture of them). The empty tiles (e.g. between distant maps) can be removed by the map combination process. It can read and write huge maps in 2Gb chunks.

Enjoy!
« Last Edit: January 01, 1970, 01:00:00 by Guest »
 

Offline stebu

  • Premium
  • Padavan of Locus
  • ***
  • Posts: 252
  • Thanked: 1 times
    • View Profile
  • Device: Xcover 2
Re: sqlite to GEMF map conversion tool
« Reply #3 on: October 31, 2011, 18:52:09 »
I had some False Positives by Avira. I don't know if they have disappeared now, in V006?

Anyhow, the previous versions had some indexing problems, the maps were created, but the tiles were "a bit" out of order, sorry  :oops:
« Last Edit: January 01, 1970, 01:00:00 by Guest »
 

Online menion

  • Administrator
  • Professor of Locus
  • *****
  • Posts: 10117
  • Thanked: 74 times
    • View Profile
    • http://www.asamm.com
  • Device: SGS7
Re: sqlite to GEMF map conversion tool
« Reply #4 on: October 31, 2011, 20:58:37 »
stebu, I updated Avira to new version and still reporting error. But it's probably too early to receive their update. Anyway leave it be, I'm really not worried that there is some virus :). And thank you very much for your work. I'm sure that after some propagation, it will be more then useful tool. Also, if you want to propagate it more on new Locus homepage, I may create special page in support menu if you want ..
« Last Edit: January 01, 1970, 01:00:00 by Guest »
Ideas, wishes, problems
Advanced topics, public discussion, sharing of knowledges, testing beta versions: you're here!
 

Offline tomtomxp

  • More than Newbie
  • *
  • Posts: 11
    • View Profile
Re: sqlite to GEMF map conversion tool
« Reply #5 on: February 09, 2012, 00:50:39 »
is there a tool, to extract sqlite maps to bmp or ecw or any typical raster map to use/mod with global mapper or compegps??
« Last Edit: January 01, 1970, 01:00:00 by Guest »
 

Offline stebu

  • Premium
  • Padavan of Locus
  • ***
  • Posts: 252
  • Thanked: 1 times
    • View Profile
  • Device: Xcover 2
Re: sqlite to GEMF map conversion tool
« Reply #6 on: February 09, 2012, 13:19:04 »
Quote from: "tomtomxp"
is there a tool, to extract sqlite maps to bmp or ecw or any typical raster map to use/mod with global mapper or compegps??
Hi!
If you can use a Python script, I have one that reads Locus's sqlite maps and sends the tiles to disk files. There is no image conversion, so you'll get .png or .jpg files.
Let me know, if you are interested.
« Last Edit: January 01, 1970, 01:00:00 by Guest »
 

Offline tomtomxp

  • More than Newbie
  • *
  • Posts: 11
    • View Profile
Re: sqlite to GEMF map conversion tool
« Reply #7 on: March 03, 2012, 04:02:49 »
Hi, yes please ive heard of python script but might need a lil direction in executing it, cant be that hard if you know the command/script.
thanks
« Last Edit: January 01, 1970, 01:00:00 by Guest »
 

Offline stebu

  • Premium
  • Padavan of Locus
  • ***
  • Posts: 252
  • Thanked: 1 times
    • View Profile
  • Device: Xcover 2
Re: sqlite to GEMF map conversion tool
« Reply #8 on: March 03, 2012, 14:30:22 »
Quote from: "tomtomxp"
Hi, yes please ive heard of python script but might need a lil direction in executing it, cant be that hard if you know the command/script.
thanks
Yes, Python is amazingly powerful, yet easy to use (imho). The biggest problem I had was with changing my text editor to use spaces instead of tabs  8-)
Python site where you can download Python. My script uses the older version, so please use Python 2.7(.x).
You'll also be needing Sqlite, this you can get from http://http://code.google.com/p/pysqlite/downloads/list , the file pysqlite-2.6.3.win32-py2.7.exe  works for me. First install Python, then execute the downloded installer pysqlitexxxxx.exe.

the python scripts are zipped on my web-site. There's a bat file with the "keep" parameter, this leaves the sql-tiles in their folders as .png or .jpg files.

This should keep you busy for a while  ;)
« Last Edit: January 01, 1970, 01:00:00 by Guest »
 

Offline stebu

  • Premium
  • Padavan of Locus
  • ***
  • Posts: 252
  • Thanked: 1 times
    • View Profile
  • Device: Xcover 2
Re: sqlite to GEMF map conversion tool
« Reply #9 on: April 14, 2014, 15:42:26 »
My ISP shut down my home page.

Now you can (hopefully) get the conversion tools (V006) from Dropbox:

https://dl.dropboxusercontent.com/u/64901427/GemfToolV006.zip

The zip contains:
gemftool.html that has some description
readme.txt probably the same content as above
gemfread.exe
gemftool.exe

Some false positives have been reported by Avira. But I'm 99.9999% sure that they really are FALSE.
One ppm is reserved for NSA :)
 

Offline Bucky Kid

  • Padavan of Locus
  • ***
  • Posts: 255
  • Thanked: 3 times
    • View Profile
Re: sqlite to GEMF map conversion tool
« Reply #10 on: April 15, 2014, 07:10:50 »
Hi stebu, thanks for this tool. I wanted to ask what density and number of rounds should I use to get max compressed map. Have these values impact on performance?

And second question is why gemftool creates two sets of map:
For example I have 3GB big atlas named atlas.sqlite, After conversion I get atlas.gemf, atlas.gemf-1, atlas_.gemf and atlas_.gemf-1. What's the difference between maps suffixed by underscore and wihout it?
 

Offline stebu

  • Premium
  • Padavan of Locus
  • ***
  • Posts: 252
  • Thanked: 1 times
    • View Profile
  • Device: Xcover 2
Re: sqlite to GEMF map conversion tool
« Reply #11 on: April 15, 2014, 08:42:24 »
Hi stebu, thanks for this tool. I wanted to ask what density and number of rounds should I use to get max compressed map. Have these values impact on performance?

And second question is why gemftool creates two sets of map:
For example I have 3GB big atlas named atlas.sqlite, After conversion I get atlas.gemf, atlas.gemf-1, atlas_.gemf and atlas_.gemf-1. What's the difference between maps suffixed by underscore and wihout it?
It's over 2 years since I made or used Gemftool! So my answers might not be precise.

The more compressed (empty tiles squeezed out) the map is, the larger the header part is; more ranges of smaller map areas. I think the performance impact is on the app that uses the Gemf map, and I have little experience with this.

If you find that the Gemftool itself is running a long time, then reduce the number of rounds directly and/or reduce the goal density.

The files with underscores are temporary files and  may be deleted. In yor case atlas.gemf, atlas.gemf-1 are the result of the conversion. There's two files, because the size of the map exceeds 2GB, 3GB map is split into two files.