GraphHopper - own offline routing file

Bucky Kid · July 30, 2015, 14:26:52

Hello. With help of VirtualBox I could create own offline routing files but compared to menion's files they're smaller (116MB vs. 152MB). I've tried to lookup a long route (400km) using both sets and with using own data set it took about 15minutes compared to menion's set which took about 5 minutes. How that's possible own data are smaller and take Locus so much longer to compute same route? I'm using PBF from geofabrik.

Anybody succeeded to run graphopper.sh on Windows?

john_percy · July 30, 2015, 17:22:48

I actually constructed mine using the function built into Emux's Atlas, but using a Java console window to get more memory when launching Atlas.
Command line from windows: java -Xmx1024M -jar Atlas.jar (make sure java is on the path, and launch from the Atlas directory)
Atlas is at: http://www.talent.gr/public/atlas/
osm.pbf files from: http://download.geofabrik.de/ (I used a Special Subregion from teh bottom of the page to get data for the entire British Isles)

karussell · July 31, 2015, 21:13:27

Hi,

thanks again for the feedback.

@menion: thanks for making this open source! I've added a link to the code in the Android section, maybe we get someone interested in this too:
https://github.com/graphhopper/graphhopper/blob/master/docs/android/index.md#locus-add-on

> my major problem is compiling of add-on

Sorry I'm not deep into Android or gradle

> inability to route over more files

That is already known but not on our soonish todo list:
https://github.com/graphhopper/graphhopper/issues/293

> 4. one last big issue for me as user ... I see here solution by creating own FlagEncoder with possibility to dynamically enable/disable certain osm tags.

That is our top prio issue after 0.5 release and we'll hopefully have it in ~October

> Anybody succeeded to run graphopper.sh on Windows?

This should be possible via cygwin. See e.g.
https://github.com/graphhopper/graphhopper/issues/449

And yes, the most things are known and we have to do a proper queuing

For example the turn instruction are currently too simplistic and only show up when some street properties change but this heuristic could create also too many instructions. But we have this on the high prio list!

> using own data set it took about 15minutes compared to menion's set which took about 5 minutes.

Maybe you disabled the speed mode? (prepare.chWeighting=no)

Without the speed mode the queries for routing can take longer BUT 5 minutes is definitely too long! E.g. through whole Germany it should be a few seconds even without the speed-mode, make sure you use the in-memory settings for higher speed (taking more RAM)

Regards,
Peter

Bucky Kid · August 05, 2015, 08:45:38

Quote from: karussell on July 31, 2015, 21:13:27> using own data set it took about 15minutes compared to menion's set which took about 5 minutes.

Maybe you disabled the speed mode? (prepare.chWeighting=no)

Without the speed mode the queries for routing can take longer BUT 5 minutes is definitely too long! E.g. through whole Germany it should be a few seconds even without the speed-mode, make sure you use the in-memory settings for higher speed (taking more RAM)

Regards,
Peter

Thanks Peter, I'm guessing I use untouched config.properties

Code Select

##### Vehicles #####
#
# Possible options: car,foot,bike,bike2,mtb,racingbike,motorcycle (comma separated)
# bike2 takes elevation data into account (like up-hill is slower than down-hill)
# and requires enabling graph.elevation.provider below
# graph.flagEncoders=car,foot,bike

# Enable turn restrictions for car or motorcycle. 
# Currently you need to additionally set prepare.chWeighting=no before using this (see below and #270)
graph.flagEncoders=car|turnCosts=true,foot,bike


##### Elevation #####
#
# To populate your graph with elevation data use SRTM, default is noop
# graph.elevation.provider=srtm
#
# default location for cache is /tmp/srtm
# graph.elevation.cachedir=./srtmprovider/
#
# If you have a slow disk or plenty of RAM change the default MMAP to:
# graph.elevation.dataaccess=RAM_STORE



##### Storage #####
#
# configure the memory access, use RAM_STORE for well equipped servers, 
# MMAP_STORE_SYNC could be used otherwise but will be a lot slower
graph.dataaccess=RAM_STORE

# if you want to reduce storage size and you don't need instructions for the resulting path use:
# osmreader.instructions=false

#### Speed-up Mode vs. Flexibility Mode ####
#
# By default the speed-up mode with the 'fastest' weighting is used. Internally a graph preparation via
# contraction hierarchies is done to speed routing up. This requires more RAM/disc space for holding the
# graph but less for every request. Also only the first vehicle of the flagEncoders list will be prepared.
# prepare.chWeighting=fastest
#
# Disable the speed-up mode (contraction hierarchies, CH) via enabling the flexibility mode:
prepare.chWeighting=no


##### Web #####
# if you want to support jsonp response type you need to add it explicitely here. By default it is disabled for 
# stronger security.
# web.jsonpAllowed=true

##### Extra Asamm #####
# http://stackoverflow.com/a/29976135/836138
osmreader.bytesForFlags=8

I would be curious what command line arguements and config.properties menion used to generate his data.
Since I tried to compute same route today using my data and it again took about 15 minutes.

karussell · August 05, 2015, 09:14:15

> prepare.chWeighting=no

Yes, you are using the flexibility mode, probably to support turn restrictions for car and multiple vehicles. Still strange that it takes 15 minutes - or is it on the mobile device?

The good news is that we've released the first release candidate yesterday for 0.5 which supports multiple vehicles in the speed mode (but still no turn costs).

Bucky Kid · August 05, 2015, 09:33:35

Computing 417km route optimized for bike-shortest took on my phone 10 minutes now (using my data).
Maybe as you said it may be problem of phone.

karussell · August 05, 2015, 09:37:43

On the phone this is not surprising. Again: either switch profiles to speed mode (using latest version) and disable turn costs for car. Then even longer roads should take less than 20 seconds. Or vote for our very first issue here:
https://github.com/graphhopper/graphhopper/issues/1

Bucky Kid · August 06, 2015, 12:48:38

Hello Peter, there's a bug in your config template
- osmreader.bytesForFlags is no more used

I have question yet about disk usage, it seems this version is very space hungry, my VM has 12 of 18GB free space available but crashes during compilation due to no space on disk. The question is if I can instruct your script to create temporary files somewhere outside internal filesystem.

karussell · August 06, 2015, 13:10:22

> osmreader.bytesForFlags is no more used

Where do you see this old option?

> but crashes during compilation due to no space on disk.

What do you mean with compilation? Compilation should be possible on <100MB easily

be it RAM or disc. GraphHopper is designed to be space efficient.

Or do you mean the import?

Bucky Kid · August 06, 2015, 14:02:03

Quote from: karussell on August 06, 2015, 13:10:22
> osmreader.bytesForFlags is no more used

Where do you see this old option?

Actually it seems to be not in config template but the scipt self hinted to change osmreader.bytesForFlags on exception. I had to lookup this change in change log.

Quote from: karussell on August 06, 2015, 13:10:22> but crashes during compilation due to no space on disk.

What do you mean with compilation? Compilation should be possible on <100MB easily
be it RAM or disc. GraphHopper is designed to be space efficient.

Or do you mean the import?

This is OK, my problem had resolved in the meantime (I had no more space on physical volume for virtual disc).

And I have yet one more question, if covering bike2 profile in graph.flagEncoders, I suppose the script requires SRTM data for the area processed at compile time. I got an error about invalid srtrm data. Where do I have to put corresponding SRTM files, and just to be sure, are they in .hgt or different format ?
Thanks, I hope I'm close to make successfull conversion.

karussell · August 06, 2015, 14:51:41

For elevation nothing is necessary, just include srtm enabling in the config.properties. See
https://github.com/graphhopper/graphhopper/blob/master/docs/core/elevation.md

Peter

Bucky Kid · August 06, 2015, 21:56:56

Hmm.strange. I'm using SRTM and CGIAR and both produce exception - unable to read PBF file.
Same PBF as from http://download.geofabrik.de/europe.html. With old version this problem did not occur.

Code Select

Exception in thread "PBF Reader" Exception in thread "main" java.lang.RuntimeException: Couldn't process file czech-republic-latest.osm.pbf, error: Couldn't map buffer 0 of 69 at position 100 for 72000000 bytes with offset 100, new fileLength:72351844
	at com.graphhopper.reader.OSMReader.writeOsm2Graph(OSMReader.java:323)
	at com.graphhopper.reader.OSMReader.readGraph(OSMReader.java:154)
	at com.graphhopper.GraphHopper.importData(GraphHopper.java:672)
	at com.graphhopper.GraphHopper.process(GraphHopper.java:642)
	at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:615)
	at com.graphhopper.tools.Import.main(Import.java:15)
Caused by: java.lang.RuntimeException: Couldn't map buffer 0 of 69 at position 100 for 72000000 bytes with offset 100, new fileLength:72351844
	at com.graphhopper.storage.MMapDataAccess.mapIt(MMapDataAccess.java:161)
	at com.graphhopper.storage.MMapDataAccess.ensureCapacity(MMapDataAccess.java:105)
	at com.graphhopper.storage.MMapDataAccess.create(MMapDataAccess.java:87)
	at com.graphhopper.storage.MMapDataAccess.create(MMapDataAccess.java:40)
	at com.graphhopper.reader.dem.CGIARProvider.getEle(CGIARProvider.java:204)
	at com.graphhopper.reader.OSMReader.getElevation(OSMReader.java:606)
	at com.graphhopper.reader.OSMReader.addNode(OSMReader.java:591)
	at com.graphhopper.reader.OSMReader.processNode(OSMReader.java:566)
	at com.graphhopper.reader.OSMReader.writeOsm2Graph(OSMReader.java:293)
	... 5 more
Caused by: java.io.IOException: Nepřípustný argument
	at sun.nio.ch.FileChannelImpl.map0(Native Method)
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:875)
	at com.graphhopper.storage.MMapDataAccess.newByteBuffer(MMapDataAccess.java:179)
	at com.graphhopper.storage.MMapDataAccess.mapIt(MMapDataAccess.java:153)
	... 13 more
java.lang.RuntimeException: Unable to read PBF file.
	at com.graphhopper.reader.pbf.PbfReader.run(PbfReader.java:53)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
	at com.graphhopper.reader.OSMInputFile.process(OSMInputFile.java:266)
	at com.graphhopper.reader.pbf.PbfDecoder.sendResultsToSink(PbfDecoder.java:107)
	at com.graphhopper.reader.pbf.PbfDecoder.processBlobs(PbfDecoder.java:172)
	at com.graphhopper.reader.pbf.PbfDecoder.run(PbfDecoder.java:185)
	at com.graphhopper.reader.pbf.PbfReader.run(PbfReader.java:49)
	... 1 more
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
	at com.graphhopper.reader.OSMInputFile.process(OSMInputFile.java:263)
	... 5 more

My prefs are:

graph.flagEncoders=car|turnCosts=true,foot,bike2,mtb
graph.elevation.provider=cgiar
graph.dataaccess=RAM_STORE
prepare.chWeighting=no
graph.bytesForFlags=8

Bucky Kid · August 07, 2015, 10:32:15

@menion: can you please update add-on GraphHopper to handle bike2, mtb
When I'm trying to navigate using these profiles, Locus crashes (I guess it's because of addon crasch)

Menion · August 07, 2015, 11:19:30

Nice discussion guys, will have to invest some time and read it precisely because I also suffer from some mentioned problems here.

And update, sorry not possible now ... http://forum.locusmap.eu/index.php?topic=4036.msg38588#msg38588 , my problem no. 1. After some update of Google/Java library (unfortunately do not know what exactly changed), I'm currently unable to generate new version of add-on.