MAP USAGE - SERVER OVERLOADING ...

Menion · May 05, 2011, 20:54:47

Hmm, looks that this application bring me more troubles that I can handle ...

- read here. It's forum where is discuss about overloading of OSM servers during few last days (weeks). Looks that this is cause by Locus since all other map providers were removed. Some stupid users do not read and do not take some respect for OSM providers, and download more them 1 million tiles per day. So I have to stop this, and set in Locus some reduction on it.

So what you suggest?

I though about adding some daily limitation for downloading - around 10 000 tiles per day?

So what you think guys? I'm really tired thanks to all this troubles last weeks ...

berkley · May 05, 2011, 22:58:46

Hey,

I am really sorry for all this trouble. I've read the thread you've posted...

I think, OSM and Locus are in the same ship. You should contact them and find a solution that works for both sides.
First of all, you should find a way to tell the osm-server, that "locus" is contacting it...

A daily limitation for download would affect all users, even the ones which would "only" download 10.000 and one tile in a month. There are plenty of solutions. My first idea was, to handle this with user accounts. This means that, if you want to download something from OSM, you first have to login to OSM. With this solution OSM could handle theirself, how many tiles a user can download a day.

Anyway, hope this finds a good end.

Cheers, berkley

Menion · May 06, 2011, 06:09:13

this is not solution. people do not think much about server side, so this too huge downloading affect not only OSM servers, but the others also. I want to create some limitation on all servers ...

bobo007 · May 06, 2011, 09:07:28

Hi,
I guess when there is some (or total in extreme case) limitation that people find workarounds... One could be that they will start sharing downloaded map packages among them through download services as Rapidshare. One guy downloads (with older version of MOBAC e.g.) the files for some map area and others download this package. So this is good for the servers and not so good for users who cannot find files they want in the internet. I also think that most people need to download just smaller area for thei hiking and cycling trips so the idea of the day limit is OK for this case. Who wants the whole country then he/she downloads tha package as described above.
I agree that you could contact the OSM guys to help you find a solution for Locus. The download feature of Locus is no forbidden thing so you needn't be worried.
I also agree that Locus should identify itself with "Locus" ID to the servers, that's fair to the providers.

jctmarques · May 06, 2011, 15:00:10

In my oppinion there should be no limitation if the user is connected with 3G. If it's on Wi-Fi you should decrease download speed to something acceptable to the servers. Or even start download at full speed and slow things every 1000 tiles.

jusc · May 06, 2011, 16:59:56

Quote from: "menion"Hmm, looks that this application bring me more troubles that I can handle ...

- read here. It's forum where is discuss about overloading of OSM servers during few last days (weeks). Looks that this is cause by Locus since all other map providers were removed. Some stupid users do not read and do not take some respect for OSM providers, and download more them 1 million tiles per day. So I have to stop this, and set in Locus some reduction on it.

So what you suggest?

I though about adding some daily limitation for downloading - around 10 000 tiles per day?

So what you think guys? I'm really tired thanks to all this troubles last weeks ...

That is what I wrote before: viewtopic.php?f=10&t=496#p2441
The users who download 1 million tiles, don´t know anything about these limitations. So it would be good to limt download or decelerate the download speed.
For me a 500MB flat per month is more then enough to use Locus in online mode if you stay in your own country. Offline mode is only necessary if you don´t want to pay for roaming.
AFAIk Mobac limited parallel download network connections from 4 to 2 for OSM maps.

by the way: NopMap is owner of the wanderreitkarte http://www.wanderreitkarte.de
J.

Menion · May 06, 2011, 17:04:50

yes, this is exactly what you wrote. Unlimited downloading with limited speed is not good solution... so best seems to me download limit per day which cut of people who do not know limits ...

jusc · May 06, 2011, 17:17:50

hmm..
The question is: What does the server see?
If all users download as "Locus/version (Linux; U; Android; en-us)" as proposed, second user can´t download anything at the same day, because the admisn will block them?

J.

Menion · May 06, 2011, 17:21:55

I don't how this will be solved. Hope that they do not cut off all locus users. Anyway current created limit is in application and is applied to all maps so every single user have own limit. Question is limit size. I don't know how big overload for server is generating 25k tiles ...

Anyway I'll wait for some response on OSM forum ...

jusc · May 06, 2011, 17:34:39

ok, I understand.

is there any possibilty to implement an "abo" for users the want to pay for map tiles? (It is used with apemap, which I don´t have

as I understood )
somthing like this: //abo.wanderreitkarte.de/10/537/340.png/ticket/47110815 If you understand German you can read it here http://www.wanderreitkarte.de/shop_abo_de.php

J.

Menion · May 06, 2011, 17:39:42

yes, this is simply possible to do. 10€ is fair cost I think...

jusc · May 06, 2011, 17:47:22

I use this with mobac for maps about 20 x 20 km with mobac zoomlevel up to 16. The speed is not so fast as google, but for a private offer... it´s ok.
I fear if Locus comes always with the same "sender" he will block downloads, maybe online mode too...

J.

Menion · May 06, 2011, 18:32:34

same sender? on wanderre.... If I understand this system correctly, you'll receive for 10€ key that allow you to download prerendered data. So It's up to you, to keep this key only for your private use. If so, you'll be simply indentified by this key and should be download data as you need (in some limits of course). Do I understand it correctly?

jusc · May 07, 2011, 11:14:38

sorry for this delay.
Yes with an abo it should work every time.
But I think if the sever identifies incoming Locus app, because Locus sends an identifier like: "Locus Version .....". Server admins can set up a firewall to block all requests from Locus even the online mode?

J.

userloser · May 07, 2011, 21:08:31

Well, here's my two cents:

Of course, using a map means the end user of the map data has obligation to abide by the terms of use of the map provider. At the same time, your application enables me to use maps in very useful ways. However, if some of these interesting and useful ways violate the terms of use, that is my problem.

I don't subscribe to the point of view of some developers on the openstreetmaps dev list that an application should police its users. What I think is that the application should make the users aware of the terms of use, and let them choose their behavior.

Of course, these largely philosophical points can be debated to no end. This is how lawyers make a living. In practice, I try to abide by the terms of usage of most maps most of the time, but I admit that the occasional (rare) violation of the terms of use does have its uses if it doesn't impact negatively on the experience of other users.

So, if I had to define the responsibility of you as a developer, I'd say it is twofold: first, you should warn the users of potential terms of violation. Second, your application should allow a choice that violates the ToS, but should do its best to mitigate the adverse effects of such behavior.

In my opinion, most locus users who violate the terms of service do so without realizing it most of the time. Just providing a warning will probably mitigate most of the problems.

Maybe a good temporary option will be for Locus to require the map provider XML file to have some extra terms for each provider (e.g., allowed zoom levels, allowed maximum download rate, number of open connections, max tiles to download at once etc.) and give a warning to the user if a download that is initiated is actually violating those terms. Then, it should be up to the user to proceed or not. That will squarely put the responsibility where it belongs -- in the hands of the user -- as well as provide the user with enough information to make their choice. Alternatively, using something like robots.txt designed specifically for tile servers may also do the job well.

Mitigating the irresponsible user behavior is trickier. In my opinion, rather than limiting the number of tiles, a strategy like forced tarpitting would work best. You could implement a small delay between the download of different tiles, and increase that for egregious violations of terms of service, limiting the rate of download to what the user would do with a browser. This will allow the server load that a single client creates to decrease significantly, while letting those of us who want to keep an offline cache for emergencies to do so.

Also, I think that you should allow the user to customize whatever user agent string you're supplying to the server.

Of course, in the long run there should be a solution that is both useful and legal, but in the meantime a temporary compromise that can let end users use the maps while not putting excessive loads on the providers' servers is a good thing.