Locus Map - forum

Some of the users and I have troubles viewing online maps (OSM Classic). "Access Blocked. We suspect this client is downloading maps in an unauthorised manner.."

This appears while trying to view the last two zoom levels (17 and 18). It appeared after the last or before the last update.

What could be the problem.

hmm nice ... from irc.openstreetmaps.org, I went to http://wiki.openstreetmap.org/wiki/Blocked_applications (http://wiki.openstreetmap.org/wiki/Blocked_applications) .. Locus in list, ach ... what the ..

That is kind-of bad, I believe. Is there anything You can do?

yes, I'll write to some OSM admin. Hope that someone cracked locus and download with this version more then is limit ...

I don't know how this can be possible ...

http://grant.dev.openstreetmap.org/tmp/ ... ek-20k.txt (http://grant.dev.openstreetmap.org/tmp/logs/03102011/yevaud-UA-hits-1week-20k.txt)

but locus is still in top ten downloaders from osm server. So since next version downloading from base osm server will be removed. this is not a problem. For downloading are much better servers. Mainly MapQuest

fine, donation to OSM sended http://donate.openstreetmap.org/comments/ (http://donate.openstreetmap.org/comments/), downloading OSM classic disabled, now contact someone from admins ...

That is discussed in the German OSM forum already since yesterday.
http://forum.openstreetmap.org/viewtopic.php?id=13959 (http://forum.openstreetmap.org/viewtopic.php?id=13959)

Most of them are against using Mobac, Locus, Oruxmap and all other "tools" to view and download tiles from the OSM tile servers, because OSM is not a map but a geo database.

thanks jusc. It's interesting that with mobac author, they discuss in just normal way. When someone from OSM discuss with me, it's usually with angry tone. Usually because Locus is also paid app and mobac no. Never mind, as I wrote before. I disabled direct downloading from OSM Classic. No other choice.

I was also (about a month ago) trying to set own tile server, but current server I use for vector map generating and rendering is not enough powerful for this task. I have to think about it a little and maybe I'll buy for locus users some more powerful tile server ...

Well, that sucks, but sounds like nothing you can do Menion.  I'd have to guess that it's an indication of how widely used Locus is, and even with the current restrictions (which IMO weren't very restrictive) the sheer number of users who might be grabbing tiles at any time are going to upset the OSM folks.  Only thing I could think of is further restricting Locus, perhaps X number of tiles per day and X number of tiles at once.  Setting up your own tile server would be a pretty big drain, it'd require a dedicated server, fast connection, and lots of bandwidth.  

In one way I see their point, they're probably always on the lookout for heavy access against their server and blocking anything which seems to be scraping it, but IMO they're missing part of the point in that they would want their maps to be widely used.  I've always had a bit of a sour taste from OSM and have always had more issues getting reliable or reasonably fast data back from it from various clients.  Even their website is slow, regardless of their restrictions.  I tried to draw a new bike path here once and i completely gave up on it, it took forever and the tools were horrible.

Sadly I have downloaded (over a few weeks of time) my town and a few I geocache in, and wouldn't want to re-download all of that from Mapquest or other sources.  I assume the database data already in place will continue to work?

As I thought about this more today, it occurred to me that maybe there is some compromise that can be made to get OSM to accept Locus.  Maybe OSM won't compromise, dunno..

First part is having two separate user agents in Locus, one for the online loading of maps the other for downloading of maps.  This is not to evade OSM filters; it would be communicated to them and agreement made.  The point being to ultimately have all "online" access unblocked, but give them the ability to block "downloads" in the future if they feel the need.

Second, greater restrictions on per-source tiles.  Maybe 2000 a day from a given source, such as any OSM, or any Mapquest.  I think this needs to be done regardless because other map providers may also feel Locus users are downloading too much.  Again, multiplied by the number of users, this number might not be sufficient, it might be 500, or it might be 5000, I don't know.  Perhaps with greater restrictions, OSM will open up "download" Tile access again.

Regardless, I hope they open up "online" access.  As per the original post it does look like anything lower than 17 works okay, and maybe that's sufficient for most things.  But we don't want them to limit things further, etc..

Quote from: "menion"I don't know how this can be possible ...

http://grant.dev.openstreetmap.org/tmp/ ... ek-20k.txt (http://grant.dev.openstreetmap.org/tmp/logs/03102011/yevaud-UA-hits-1week-20k.txt)

but locus is still in top ten downloaders from osm server. So since next version downloading from base osm server will be removed. this is not a problem. For downloading are much better servers. Mainly MapQuest

I wonder that I can find Locus only 22 times?


Sent from my GT-P1000 using Tapatalk

and what numbers are you expecting? :) older version are unfortunately faked as firefox ...

That has been the good old time. :-)

Sent from my GT-I9100 using Tapatalk

Yes, old good times with empty locus(@)asamm.cz inbox and no worries :D

from the German forum: http://forum.openstreetmap.org/viewtopi ... 98#p194498 (http://forum.openstreetmap.org/viewtopic.php?pid=194498#p194498)

translation from Google:

I allow myself a short time from the forum and TALK-DE to summarize:

Tiles servers are slow because the problem:
Number of connections is too high for universities (current issue)
Traffic and workload continues to increase (even more)

therefore delivery of the tiles had to be throttled (recognition of scrapers not reliably possible) so that left us just enough performance for mappers still remains. This is also partly affected services of other websites. Otherwise would, however, already hosting OSM risk.

Proposed solutions are:
Use of alternative services by Tiles Apps (Mapquest, Cloud Made, ...)
Build-own-Tiles App servers by operators to buy, or services
- It is suggested that documentation of Mapnik Wiki, PostGIS, ... to improve
- Kai has been building an out-of-the-box packages for Ubuntu Server
Apps donations by vendors
- Problem of overburdened hardware of the universities is not so well solved
- Some (such as Frederik, Firefishy ,...) OSM not see as a "service", but responsible only for the purpose of the project, namely to create a free world map. Draw maps and performant this can deploy to / should we not also accept donations
- And can even be a "premium" subscription would pay for the vendors so they can sustain the heavy load
Apps should switch to offline rendering of vector data
- There are of us still do not like simple technique, how to use finished tiles.

Has not anyone of you have already contacted the authors of the other tools included? Who would be so kind to address the times?

Hi, I'm from the (german) OSM community and wrote the quoted summary.

Yes it might be possible that the tone of the admins is sometimes rough (even to me as a member) but on the other hand I know that they are under high pressure. The problem is, that the universities, that currently host OSM for free will kick off the project, because their network hardware is to much stressed (connections and traffic). For that reasons we had to ban the most heavy used apps to make sure that the project (that creates a free database not deploys a webmap with similar performance than google maps) is still alive.

Of course this is annoying to endusers but on the other hand this tools doesn't fullfil the Tile usage policy and thats not fair for others and esp. to the volunteeres of the project who just want to do a good job
http://wiki.openstreetmap.org/wiki/Tile_usage_policy (http://wiki.openstreetmap.org/wiki/Tile_usage_policy)

So what are possible solutions (as pointed out above)?
-use another tiles hoster as Cloudmade or Mapquest http://wiki.openstreetmap.org/wiki/TMS (http://wiki.openstreetmap.org/wiki/TMS)
- buy a service account by a third party company http://wiki.openstreetmap.org/wiki/List ... Consulting (http://wiki.openstreetmap.org/wiki/List_of_Companies_offering_OSM_Consulting)
- setup your own rendering stack
- use only vector maps and render it on device

At this point donations won't fix the problem itself (max. connections in university hardware), but feel free to join the discussions here: http://www.openstreetmap.org/user/!i!/diary/15046 (http://www.openstreetmap.org/user/!i!/diary/15046)

Hello MandM,
  thanks for info. I'm not registered as part of german community and I don't personally speak german but please say them this (and this is also to others)

current (already done) steps:
--------------------
1. today I'll release new version 1.3.4
2. direct downloading from openstreemap.org server will be disable. I'll leave only viewing and caching data. This should reduce locus to only some mobile viewer
3. today I finished "new" vector maps, (update that finally cover whole planet) so I hope that many users will use them
4. I also moved MapQuest server on first place in list of maps, this should also help to reduce using of OSM classic server
5. And finally I donated amount that at least should cover some damage done due to enormous numbers of users

plans:
------------
1. I was yesterday playing for whole day with mapnik, openlayers, mod_tile and some other stuff and even when I never did something like this, seems it should work (I have already simple map of Czech republic in Locus from own server ... nice), so I decided that I'll create own tile server that will move completely all trafic from osm classic server to this (only for locus) server. One my friend is more experience with these server stuff so I'll meet him in monday and hope that we should start work on this soon ...

that's all

I added it to the forums (that are within the general OSM forums and so you can logon with your OSM account)

I've tried to figure out what an Amazon EC2 server would cost, but its calculator is using all terms I don't quite understand, even as a typical systems administrator..  I'd have to look more into what all of it means.

But I wonder if EC2 + http://tiledrawer.com/ (http://tiledrawer.com/) is a possible alternative?  Unless I have the calculator very wrong, I'm seeing something like $20-$40 a month, but again, I'm mostly guessing.

Anyone here use EC2 and can estimate better?

AFAIK there are some folks doing rendering on EC2 but currently don't know who that was...

you really mean 20-40$? Seems like nonsence ... by my simple compute it will be around 500$ per month. There is really a lot of users :). I'll try some small server to test how this all will works, but EC2 seems to be too expensive solution ...

Quote from: "menion"you really mean 20-40$? Seems like nonsence ... by my simple compute it will be around 500$ per month. There is really a lot of users :). I'll try some small server to test how this all will works, but EC2 seems to be too expensive solution ...

See, first off the calculator was a little confusing, and I have no idea how much actual data will be used ;)  I've looked around a little into how to set up my own tile server but I'd need to talk to the owner of the boxen I use..  Big machines on fat pipes, but not sure he'd want that sort of commitment.

That said, any idea how much data Locus users would download in an average month, order of magnitude...  10g.  50g.. 200g?

during september was downloaded around 1.2TB of vector maps
as I calculate, I expect around 2TB of online maps when there will be limit to 10k tiles per day. So quite a lot ... :)

Quote from: "menion"during september was downloaded around 1.2TB of vector maps
as I calculate, I expect around 2TB of online maps when there will be limit to 10k tiles per day. So quite a lot ... :)

Yikes, much higher than I expected.  Of course, 10k tiles per day is much higher than I expected too.

Yes, so you see whats the problem if we have 10 of this tools on our servers ;)

BTW Currently there is no feedback by the OSM Admins if Locus can be unblocked. I asked them now again...

You might be interested in the visualisation and compare
http://www.openstreetmap.org/user/!i!/diary/15190 (http://www.openstreetmap.org/user/!i!/diary/15190)

thanks, I already got echo from jusc here viewtopic.php?f=10&t=1229#p7296 (http://forum.asamm.cz/viewtopic.php?f=10&t=1229#p7296) . Pity that no one from German styled OSM maps or Classic openstreetmap.org not communicate. Have to write more emails at Monday. I sure that today next very unpopular update will make them happy :) (limited already limited downloading). At least someone ... because I'm going to take some pills on head from this all ... (don't worry, I hate pills) ...

Indeed I'm registret for the german mails, too. But we don't promote the tiles as we don't have that much resources.

I did some calculations on IRC based on munin stats (//http://munin.openstreetmap.org/) for konqui and albi, the two tile proxies. Based on 23 TB/month I estimated it at about 2500-3500/month to do with EC2 what the OSM tile servers do. About 1k/month for an instance, the remainder for bandwidth.

EC2 isn't a great option. It's good if you want to be able to rapidly scale up some types of services, but serving tiles isn't one of them. EC2 costs about $100/TB

Someone pointed to Hetzner (//http://www.hetzner.de/hosting/produkte_colocation/rack) as a cheaper option, being 6.9 euros/TB.