[MANUAL] - using SQL map files bigger then 2GB

[Menion]

I never worked on sqlite, and after I go though the usage of it, it seems that if you have a zoom level which across several db files, you need to have multiple connections to query for that zoom level and consolidate the result sets together. This is not very efficient is it? Is there any chance to split db files based on x instead of z? This of course depends on your querying needs on Locus, I dunno how you query the files of course  :lol:

Btw, I'm looking at the MOBAC source code and seems I can add a new atlascreator impl, although I guess MOBAC authors will not be too happy on that because I saw them rejecting your new format requests on their forum... :oops:

Hmm, i don't think is unefficient to hold more connections at once. As wrote mmcyclo, there is no speed issue with actual system and I don't see reason, why this should be wrong. But I'm not and database expert. Anyway, splitting by Z (zoom) looks for me as best idea, and also easier to implement.

About MOBAC, yes I looked at this also but they, as you read, do not agree with adding support for big maps, so it's up on us

Anyway, nice photos on homepage ykf

mmcyclo: I don't know exact value. It's question on any experienced Linux user who know limitations of this platform. But I expect limit at 2 147 483 648b

[skrz]

What program or procedure do you recommend to use for splitting a big SQL map?
I have 6GB .sqlitedb file, how should I split it?

[Menion]

there is actually no way how to do this. In this topic we discuss about developing such a tool ...

[ykf]

Menion, thanks for your appreciation on my art!  

There is a problem on splitting by z... when one level zoom is too big, e.g. for those google map on zoom level 19, just the zoom file alone can be bigger than 2GB. Anyway I'll do the MOBAC modification on each zoom level first, and then we'll think of how to solve this problem later.
The good news is MOBAC is pretty easy to understand and change, although the design seems outdated... why don't they use maven?!  :roll:

[ykf]

Menion, there's a question... will you still use the info table for the multi-db file implementation?

[Menion]

hi,
  sorry for late answer.No this table is mystery for me. Why should I want to know only first and last zoom when I need to know all zooms. Hmm, this brings me idea. What about creating this table as storage of zoom informations? Create there column named "zooms" which can contain just a string with zoom levels separated by ";", so for example "4;6;8;10". Hmm maybe I should use this also on created offline maps which will almost completely remove needed initialization because this is just one second task ...

[Mr Yod]

If a map file greater than 2GB doesn't work, why not reduce the dimension of the file itself?


Let me explain.
I made some tests using Mobile Atlas Creator making a map with the following features: google hybrid, 3x3 blocks with grid zoom 12 and zoom from 12 to 18 (not very big map, but big enough in storage terms).

I made that map using the "RMaps SQLite" option and the map file is 5.48GB (really HUGE for a map with an area of about 25km x 25km), then I made the same map using the "OruxMaps SQLite" option and the map file is 641MB (yes: 9 times smaller).

So: somehow the oruxmaps sqlite file is compressed and, in my opinion, some sort of compression could help both for the storage (less memory required) and for the use (the file can be opened).


EDIT: I looked at the MOBAC cache: google earth, google maps and google hybrid take 1.96GB of total space (1.78GB google earth, 1MB google maps and 179MB google hybrid); considering that I used the cache to make that map, how is possible that the rmaps sqlite file takes 5.48GB?

[Menion]

hey what is this? Really so big difference?? have to check

Anyway you can use bigger maps now, If you prepare them in small part, but I'm sure you know about it ...

[Mr Yod]

hey what is this? Really so big difference?? have to check

There's the "MOBAC's cache issue" too, like I explained in my edit at the end of my previous post
I made another test: I deleted all the MOBAC's cache and downloaded again the same area I used for the previus tests to see how much memory would take the cache.
The results are the following:
google earth: 450MB
google hybrid: 57MB
total memory used: 508MB

So: how's possible that rmaps sqlite file takes over 5GB?
The same space is taken if I use the "TrekBuddy tared atlas" option (I wanted to see if a tar archive would have taken less space)

Anyway you can use bigger maps now, If you prepare them in small part, but I'm sure you know about it ...

yes, I know: I already read it.

[MedoMys]

If I understand correctly first post of this thread, I have to create a directory in the directory Locus/Maps, copy some SQLite maps (of size less then 2 GB) into it and it should look like as a one compact map. This is what I tried (directory ZM10k, maps Brno_ZM10k.sqlitedb and CeskyRaj_ZM10k.sqlitedb), but such maps are not visible in offline maps list. Am I doing something wrong?

[Menion]

no, you understand it correctly. you should see map named as your directory, so ZM10k. So, is there?

[MedoMys]

No. Both maps disapeared when I've moved it into subdirectory ZM10k (they've worked correctly before it), and there isn't any new map named ZM10k in offline list. Some manual settings needed ?

[Menion]

nono, no settings needed. Weird. Try to clear cache directory if you're using market version. Hmm this is really weird

[MedoMys]

LocusCacheMap ? All subdirectories are empty now.
Tested function Empty cache in Locus also.
But the same behaviour.
Not Market version, I've version 0.9.28.9.

Edit: Testet both market versions, Free and Pro, unsuccessfull.

[MedoMys]

Any idea, how to solve it ?
I've tried test version 1.0.1 , reinitalized maps, but nothing has changed.
Last attempt - I've taken map created in Locus, made subdirectory and moved map into it. But this map has disappeared from the offline maps list after it.