Locus Map - forum

Development => Other features => Topic started by: Menion on May 05, 2011, 20:54:47

Title: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 05, 2011, 20:54:47
Hmm, looks that this application bring me more troubles that I can handle ...

- read here (//http://gis.638310.n2.nabble.com/Paging-the-tile-server-admins-td6308590i20.html#a6333554). It's forum where is discuss about overloading of OSM servers during few last days (weeks). Looks that this is cause by Locus since all other map providers were removed. Some stupid users do not read and do not take some respect for OSM providers, and download more them 1 million tiles per day. So I have to stop this, and set in Locus some reduction on it.

So what you suggest?

I though about adding some daily limitation for downloading - around 10 000 tiles per day?

So what you think guys? I'm really tired thanks to all this troubles last weeks ...
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: berkley on May 05, 2011, 22:58:46
Hey,

I am really sorry for all this trouble. I've read the thread you've posted...

I think, OSM and Locus are in the same ship. You should contact them and find a solution that works for both sides.
First of all, you should find a way to tell the osm-server, that "locus" is contacting it... :P
A daily limitation for download would affect all users, even the ones which would "only" download 10.000 and one tile in a month. There are plenty of solutions. My first idea was, to handle this with user accounts. This means that, if you want to download something from OSM, you first have to login to OSM. With this solution OSM could handle theirself, how many tiles a user can download a day.

Anyway, hope this finds a good end.

Cheers, berkley
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 06, 2011, 06:09:13
this is not solution. people do not think much about server side, so this too huge downloading affect not only OSM servers, but the others also. I want to create some limitation on all servers ...
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: bobo007 on May 06, 2011, 09:07:28
Hi,
I guess when there is some (or total in extreme case) limitation that people find workarounds... One could be that they will start sharing downloaded map packages among them through download services as Rapidshare. One guy downloads (with older version of MOBAC e.g.) the files for some map area and others download this package. So this is good for the servers and not so good for users who cannot find files they want in the internet. I also think that most people need to download just smaller area for thei hiking and cycling trips so the idea of the day limit is OK for this case. Who wants the whole country then he/she downloads tha package as described above.
I agree that you could contact the OSM guys to help you find a solution for Locus. The download feature of Locus is no forbidden thing so you needn't be worried.
I also agree that Locus should identify itself with "Locus" ID to the servers, that's fair to the providers.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: jctmarques on May 06, 2011, 15:00:10
In my oppinion there should be no limitation if the user is connected with 3G. If it's on Wi-Fi you should decrease download speed to something acceptable to the servers. Or even start download at full speed and slow things every 1000 tiles.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: jusc on May 06, 2011, 16:59:56
Quote from: "menion"Hmm, looks that this application bring me more troubles that I can handle ...

- read here (//http://gis.638310.n2.nabble.com/Paging-the-tile-server-admins-td6308590i20.html#a6333554). It's forum where is discuss about overloading of OSM servers during few last days (weeks). Looks that this is cause by Locus since all other map providers were removed. Some stupid users do not read and do not take some respect for OSM providers, and download more them 1 million tiles per day. So I have to stop this, and set in Locus some reduction on it.

So what you suggest?

I though about adding some daily limitation for downloading - around 10 000 tiles per day?

So what you think guys? I'm really tired thanks to all this troubles last weeks ...

That is what I wrote before: viewtopic.php?f=10&t=496#p2441 (http://forum.asamm.cz/viewtopic.php?f=10&t=496#p2441)
The users who download 1 million tiles, don´t know anything about these limitations. So it  would be good to limt download or decelerate the download speed.
For me a 500MB flat per month is more then enough to use Locus in online mode if you stay in your own country. Offline mode is only necessary if you don´t want to pay for roaming.
AFAIk Mobac limited parallel download network connections from 4 to 2 for OSM maps.

by the way: NopMap is owner of the wanderreitkarte http://www.wanderreitkarte.de (http://www.wanderreitkarte.de)
J.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 06, 2011, 17:04:50
yes, this is exactly what you wrote. Unlimited downloading with limited speed is not good solution... so best seems to me download limit per day which cut of people who do not know limits ...
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: jusc on May 06, 2011, 17:17:50
hmm..
The question is: What does the server see?
If all users download as "Locus/version (Linux; U; Android; en-us)" as proposed, second user can´t download anything at the same day, because the admisn will block them?

J.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 06, 2011, 17:21:55
I don't how this will be solved. Hope that they do not cut off all locus users. Anyway current created limit is in application and is applied to all maps so every single user have own limit. Question is limit size. I don't know how big overload for server is generating 25k tiles ...

Anyway I'll wait for some response on OSM forum ...
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: jusc on May 06, 2011, 17:34:39
ok, I understand.

is there any possibilty to implement an "abo" for users the want to pay for map tiles? (It is used with apemap, which I don´t have  :D as I understood )
somthing like this: //abo.wanderreitkarte.de/10/537/340.png/ticket/47110815  If you understand German you can read it here http://www.wanderreitkarte.de/shop_abo_de.php (http://www.wanderreitkarte.de/shop_abo_de.php)

J.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 06, 2011, 17:39:42
yes, this is simply possible to do. 10€ is fair cost I think...
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: jusc on May 06, 2011, 17:47:22
I use this with mobac for maps about 20 x 20 km with mobac zoomlevel up to 16. The speed is not so fast as google, but for a private offer... it´s ok.
I fear if Locus comes always with the same "sender" he will block downloads, maybe online mode too...

J.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 06, 2011, 18:32:34
same sender? on wanderre.... If I understand this system correctly, you'll receive for 10€ key that allow you to download prerendered data. So It's up to you, to keep this key only for your private use. If so, you'll be simply indentified by this key and should be download data as you need (in some limits of course). Do I understand it correctly?
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: jusc on May 07, 2011, 11:14:38
sorry for this delay.
Yes with an abo it should work every time.
But I think if the sever identifies incoming Locus app, because Locus sends an identifier like: "Locus Version .....". Server admins can set up a firewall to block all requests from Locus even the online mode?

J.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: userloser on May 07, 2011, 21:08:31
Well, here's my two cents:

Of course, using a map means the end user of the map data has obligation to abide by the terms of use of the map provider. At the same time, your application enables me to use maps in very useful ways. However, if some of these interesting and useful ways violate the terms of use, that is my problem.

I don't subscribe to the point of view of some developers on the openstreetmaps dev list that an application should police its users. What I think is that the application should make the users aware of the terms of use, and let them choose their behavior.

Of course, these largely philosophical points can be debated to no end. This is how lawyers make a living. In practice, I try to abide by the terms of usage of most maps most of the time, but I admit that the occasional (rare) violation of the terms of use does have its uses if it doesn't impact negatively on the experience of other users.

So, if I had to define the responsibility of you as a developer, I'd say it is twofold: first, you should warn the users of potential terms of violation. Second, your application should allow a choice that violates the ToS, but should do its best to mitigate the adverse effects of such behavior.

In my opinion, most locus users who violate the terms of service do so without realizing it most of the time. Just providing a warning will probably mitigate most of the problems.

Maybe a good temporary option will be for Locus to require the map provider XML file to have some extra terms for each provider (e.g., allowed zoom levels, allowed maximum download rate, number of open connections, max tiles to download at once etc.) and give a warning to the user if a download that is initiated is actually violating those terms. Then, it should be up to the user to proceed or not. That will squarely put the responsibility where it belongs -- in the hands of the user -- as well as provide the user with enough information to make their choice. Alternatively, using something like robots.txt designed specifically for tile servers may also do the job well.

Mitigating the irresponsible user behavior is trickier. In my opinion, rather than limiting the number of tiles, a strategy like forced tarpitting would work best. You could implement a small delay between the download of different tiles, and increase that for egregious violations of terms of service, limiting the rate of download to what the user would do with a browser. This will allow the server load that a single client creates to decrease significantly, while letting those of us who want to keep an offline cache for emergencies to do so.

Also, I think that you should allow the user to customize whatever user agent string you're supplying to the server.

Of course, in the long run there should be a solution that is both useful and legal, but in the meantime a temporary compromise that can let end users use the maps while not putting excessive loads on the providers' servers is a good thing.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: jusc on May 08, 2011, 17:19:24
Online mode is already blocked for "Wanderreit"  :(
J.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: femto on May 08, 2011, 18:55:15
QuoteOnline mode is already blocked for "Wanderreit"  :(

This shows how dangerous the situation is currently for Locus and any other app which allows mass downloading of map tiles!

I agree with menion that adding a daily limit for downloading tiles is a good thing and I disagree strongly with everyone trying to fake the User Agent and any other tricks suggested. This won't make things better!

I also agree that the map providers should think about some sort of 'abo' which would allow the smartphone users to legally download tiles for offline use.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Hyperplane on May 08, 2011, 19:37:00
QuoteI also agree that the map providers should think about some sort of 'abo' which would allow the smartphone users to legally download tiles for offline use.

I concur. The best solution for all parties involved would be a subscription offered by the map providers allowing us to download a certain data/map volume within a  certain period of time. I would gladly pay a reasonable monthly or annual fee for this.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 08, 2011, 20:13:31
you can check already published Locus version. Limit is already there and also all stuff that are needed for valid use of all included online maps. Bad news for all download maniacs ... limit is applied also on maps added by provider.xml file. Locus send valid HTTP user-agent and I don't want to get emails that Locus blocks any other servers ...

you can read simple post about it here ... http://locus.asamm.cz/?p=400 (http://locus.asamm.cz/?p=400)
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: userloser on May 09, 2011, 07:28:13
Actually, that is bad news for everyone, and especially for the responsible users of locus.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 09, 2011, 07:46:32
Hmm I don't see it as a big problem. 25k tiles per day is very nice area don't you think? And also, you can download 25k today, 25k tomorrow, and so on and still on same map. So no problem at all. When you use some technique like download along path or around points, together with "zoom lock" button, I think that this is still very usable!
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: jusc on May 09, 2011, 08:28:27
I agree with http://locus.asamm.cz/?p=400 (http://locus.asamm.cz/?p=400) and I just installed 1.7.3 with limitation etc. but the "Wanderreit is still blocked in online mode. So only "good will" self limitaion doesn´t help.
I propose to change user agent to Mozilla p. e, for online mode.
J.
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 09, 2011, 09:43:34
Nono, no changing of user-agent. Sorry. Also, this is I think temporary solution from owner of Wanderreit. I need to wait for a while until I'll contact him with some offer. Currently, the soil is too hot on my side ...
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Minjin on May 09, 2011, 14:33:17
I had the same problem will using Nonimapview (desktop app like MOBAC).

A good idea is to use a delay in ms for downloading tiles (200ms, 500ms, etc) in that way downloading big maps is longer but works fine and then less tiles are downloaded in same times.

Maybe you can try, first to add a delay and see what append. Why not q bigger delay for downloading whole maps and smeller in normal (online) mode.

I guess it is not too bad if a map is charged in 30min in place of 10min. Maybe the delay can vary in function of size of maps to create. Big map = bigger delay...

With delay, online mode will not be affected as we do not load a lot of tiles in short times...  :D
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: spawny on May 31, 2011, 00:48:25
Hi Menion,
I just recognized the download limitation and understand it completely! However I have to complain about the zoom level, which is not suitable for cities in my opinion. Sometimes the names of the streets are not even visible or very hard to read on higher resolutions. I never downloaded more than 2000 tiles at once, which is good enough for the best parts of a town. I would appreciate it, if there only would be daily download limitation with full zoom. So far I had to install an old version of Locus to prepare a trip to some other place.

Anyway ... I really appreciate your hard work on that great program!
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on May 31, 2011, 05:49:39
hi spawny,
  I'm glad you understand. Limitation on zooms is from OSM Terms of usage, it's not my own decision. Anyway limit on last two zooms is applied only on classic OSM servers, but ArcGIS and Mapquest (and own maps) do not have limit on zooms! (if so, there is a mistake) :). So use some of alternative maps if you have problem with street reading ...
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: adrianTNT on July 18, 2011, 14:24:39
Hello Menion, are you familiar with Amazon Web Services? Amazon Cloud Front to be more specific, I think it might be used to help/solve the downloads problem.

http://aws.amazon.com/cloudfront/ (http://aws.amazon.com/cloudfront/)

I will try to explain what it does shortly.

You can have a "source domain" (locus domain or map provider's domain) and Cloud Front can read from that domain and cache content to deliver it to users from different servers across globe.

You get a domain like afdsfafew3r43ewwarefw.cloudfront.net and then you can set a cleaner cname like cdn.asamm.cz that points to that long domain.

Whenever someone access a file like http://cdn.asamm.cz/tile.jpg (http://cdn.asamm.cz/tile.jpg) they will get the file from your defined "source domain" like http://asamm.cz.tile.jpg (http://asamm.cz.tile.jpg) but loaded from a server from USA/Japan/Europe, depending on user's location, file will not be loaded from your server anymore.

Now I am not sure how map providers domain and data works, it might or might not be compatible with this, but since system is very simple it might work nicely.
Traffic is 0.12$/GB and lower with more traffic. Maybe Amazon can somehow become a sponsor OR use this service only for PRO users OR get donations from users and use them to pay for this traffic OR maybe users can BUY gb from you, (0.15-0.20$/gb is not bad).
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Chaos_99 on October 07, 2011, 12:17:46
Hi menion,

I just wanted to point out thet the mass downloading and server overloading problem just gone hot again in the OSM public blog.
Probably the same discussion is going on on the mailing list, but I don't follow that.

Seems that the server load is critical again and blocking external apps (stupid!) is a topic talked about. This will boil down to blocking mass download apps, but will still hit Locus.
Another view is that app developers should participate in hosting costs, at least if their apps are paid for. (Do you have donated to OSM recently? If so, you should make that public to strengthen your position.)

I'm trying to argue in favor of external apps and also mass downloading, which has its purpose and is a feature of OSM that I don't want to miss. But my voice doesn't count much.
Seems we have to find another solution in the long run. What about local offline rendering? Download vector data from OSM, render on your own PC and copy the tiles to Locus? (I haven't tried, but I think those Mapmaker tools just download tiles too, right?)

Well, just so you be informed,

Chaos
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on October 07, 2011, 12:34:51
Hi Chaos,
  I know I know ... this topic is discuss here viewtopic.php?f=10&t=1166 (http://forum.asamm.cz/viewtopic.php?f=10&t=1166)

  locus is currently blocked for some zoom layers without any notify to me and no warning even when all terms of use are valid. Yes I donated a few days ago not a small amount to cover some damage mainly few months ago. I'm trying as much to lower some downloading from servers (many maps to choose, now also direct vector map downloading from own server) but seems it's not enough ...
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: dreamteam on October 10, 2011, 17:11:36
Please find enclosed a information concerning osm map downloading I found today @ heise - sorry it is german.

Allthough they are using osm maps there is a charge for the map downloading:

>Der Download eines Landes kostet als In-App-Kauf 2,39 Euro, ein Kontinent 5,99 Euro und die weltweiten Karten 9,99 Euro. Einmal gekaufte >Karten kann man löschen und erst bei Bedarf wieder installieren, um Platz zu sparen.


I think we all need to make a difference that "Free" does not mean "no cost", because the infrastructure to make downloads available is creating costs.

So I think having at the end of the day we need to consider that there is a difference for those who are willing to pay and getting therefore a dedicated donwload volume or speed and those who don´t want or cannot pay and have to wait than longer.

I do not believe that the app developer at the end of the day is the responsible person for paying that "fee" because than the developer only can average the costs and increase the app price therefore and this would be alos not fair for the users who do not have the need for an ongoing high download demand.

If something does not costs anything it has also no value for most of the people - I am not saying that this is something good, but it is reality. Only if the downloads are costing money people will think more carefully if they need it or not

Best regards,

dreamteam


10.10.2011 16:10

Skobbler navigiert on- und offline

Die preiswerte Navigations-App Skobbler[1] ist unter dem Namen GPS Navigation 2[2] neu erschienen. Allerdings gibt es diese Version vorerst nur fürs iPhone; sie ist laut Hersteller kompatibel zu iOS 5 und kostet weiterhin 1,59 Euro. Sie unterstützt das Retina-Display des iPhone 4 und läuft auch auf dem iPad.

Unter der neuen Oberfläche von Skobbler verbirgt sich die Möglichkeit, Karten zur Offline-Navigation nachzukaufen

GPS Navigation 2 nutzt wie sein Vorgänger die regelmäßig kostenlos aktualisierten Karten von OpenStreetMap, wobei der aktuelle Abschnitt aus dem Internet geladen wird. Man kann aber auch Karten länder- oder kontinentweise herunterladen, um offline zu navigieren. Der Download eines Landes kostet als In-App-Kauf 2,39 Euro, ein Kontinent 5,99 Euro und die weltweiten Karten 9,99 Euro. Einmal gekaufte Karten kann man löschen und erst bei Bedarf wieder installieren, um Platz zu sparen.

Die Oberfläche der App wurde komplett überarbeitet. Nach dem Start erscheinen fünf große, kaum zu verfehlende Schaltflächen. Im Offline-Modus erleichtert eine Vorschlagsliste die Adresseingabe. Außerdem geht die Neuberechnung der Route zügiger, wenn keine Daten nachgeladen werden. Seine aktuelle Position kann man per E-Mail oder Facebook weitergeben.



(ad[3])







URL dieses Artikels:

http://www.heise.de/newsticker/meldung/ ... 58307.html (http://www.heise.de/newsticker/meldung/Skobbler-navigiert-on-und-offline-1358307.html)



Links in diesem Artikel:

  [1] http://www.skobbler.de/apps/navigation (http://www.skobbler.de/apps/navigation)
  [2] http://itunes.apple.com/de/app/skobbler ... 49116?mt=8 (http://itunes.apple.com/de/app/skobbler-navigation-und-umgebungssuche/id329349116?mt=8)
  [3] mailto:ad@ct.de

------------------------------------------------------------------------
Copyright 2011 Heise Zeitschriften Verlag
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Khaytsus on October 10, 2011, 17:58:12
Uh, what does that iphone app have to do with OSM or Locus?
Title: Re: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: dreamteam on October 10, 2011, 19:43:16
Quote from: "Khaytsus"Uh, what does that iphone app have to do with OSM or Locus?
The iphone app is downloading osm maps and locus is downloading osm maps

Menion is getting trouble because of this because too many locus users are  downloading too much

Have  you read the 1st post?

Best regards

Sent from my HTC Sensation Z710e using Tapatalk
Title: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Menion on October 11, 2011, 06:08:21
hi dreamteam, thanks for info. I have started with one friend work on own tile server that will switch osm classic to this, so I hope this will solve osm classic issue. Solution from iPhone app is also possible of course, but it's not exactly what I want to do. I'll rather allow downloading to Locus Pro users, and viewing for Locus Free users and we'll see how this will works
Title: Re: Re: MAP USAGE - SERVER OVERLOADING ...
Post by: Khaytsus on October 11, 2011, 14:00:15
Quote from: "dreamteam"
Quote from: "Khaytsus"Uh, what does that iphone app have to do with OSM or Locus?
The iphone app is downloading osm maps and locus is downloading osm maps

Menion is getting trouble because of this because too many locus users are  downloading too much

Have  you read the 1st post?
Of course I read the first post, but the link had nothing to do with OSM etc..  Perhaps it was translated badly..