Locus Map - forum

Development => Discussion/New features => Locus Map (4+) => Topic started by: Menion on September 08, 2020, 13:06:05

Title: Locus Map - cloud/sync server
Post by: Menion on September 08, 2020, 13:06:05
It's almost 9 months since we started the work on a cloud/sync solution (since Jan 2020).

And today, we are in a moment that all internal tests were successful and the first public test is possible. Locus Map team currently consists of seven people, but still, a lot of corner cases may not be covered. Because of this, any public tests are more than welcome.

Important: use on own risk!

Important: make a full backup of your database!! This version modifies the database upon start and also may cause a data loss in case of any serious problem in client-server communication.

Version: Google Drive (https://drive.google.com/drive/folders/1t1CYpIhHGZXjNBqOFUPQKTTvddj7VVCv?usp=sharing)

Privacy: we do not yet have complete "privacy policy" document. Anyway, be aware that based on GDPR regulations, we are trying to work with data as securely as possible. We never share, re-sell etc. any data, never. It's against our deep conviction (I personally guarantee). Once Locus Map 4 will be officially published, all uploaded data on the server (not on your device!) will be deleted.

Usage:
- by default, sync (synchronization between client-server) is disabled
- the best start-point is currently over Data Manager -> in the toolbar is the "Sync" button (will be improved later)
- after the first full sync is done, automatic-sync is enabled. After any change in the database, the sync will be done automatically (with 30 seconds delay). This may be disabled in sync-settings (to work fully manually if needed).

Reporting problems
As usually, best is reproduction of problem on my own device. This means:
- a sharing your original database backup + steps to reproduce problem
- or precise description that may lead to this problem
- or if the app crash, creating a bug report by well-known method here (https://docs.locusmap.eu/doku.php?id=manual:faq:how_to_create_debug_log)

Known missing functionality & bugs
- automatic notification on other users devices to download new changes

Web route planner & manager of own points/tracks is in preparation (the second part of sync-system). ETA for the Beta test of this part is currently 2-3 months. For now, this tool is mainly about "cloud data backup" and "syncing between own devices".

Looking forward to some feedback ;)
Title: Re: Locus Map - cloud/sync server
Post by: balloni55 on September 08, 2020, 13:48:02
Hello menion
thanks for neu sync version
My first experiences
- in portrait view not all text is visible
- the error text should show the reason for the error
- after enabeled wifi syc process finished but i get an error report
- if i now click the sync button again locus close
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 08, 2020, 13:57:06
You are fast, thanks for tests!

- cut texts ... hmm, even four lines are not enough. Oki, I'll improve it
- error when wi-fi is not enabled: ah, good point, thanks!
- the last one will be the worst. Error "-1" should never happen, it is something like "well, I really do not know what happens". If the app always crashes, should be possible to create a log for me? Thanks!
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 08, 2020, 14:18:57
OK I have my own backup strategy and will not upload my database - in my case my whole movements - unencrypted to a 3rd party. Would only do that if encryption/decryption was transparent and client side.
But I'd participate in testing if I could pick specific folders in my database for syncing - is that possible?
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 08, 2020, 14:36:18
This is not a problem. Whole "sync" system is 100% optional feature, not a requirement.

Selection of groups to sync is currently not possible. This feature is in my "ideas to consideration" list for version 2.
Title: Re: Locus Map - cloud/sync server
Post by: balloni55 on September 08, 2020, 14:42:04
Quote
should be possible to create a log for me? Thanks!
log send just now
Title: Re: Locus Map - cloud/sync server
Post by: janaton on September 08, 2020, 15:02:54
Hi there,
I am main developer of the backend part of synchronization solution. First of all, I would like to thank everyone for participating in the first public test and his/her feedback (positive makes us happy, negative will be pushing us to improve as much as we can ;) ).

@tapio I fully understand your concern. All data are encrypted when transferred between client and server (https, REST API). We use Google Cloud Platform for all our services that are powering synchronization solution (storage encrypted internally by Google). We believe that security provided by Google is good enough. Additionally we try to do our best to have our code as much secure as possible (code review process, up to date libraries, vulnerability scans, ..). Still, no one can guarantee 100% security. Once you upload something to the internet it can be possibly hacked and exposed to third parties ... But again, our biggest concern is keeping your data safe - we are not selling it or sharing with third parties!

Client side encryption/decryption is not possible because we need to have access to it (e.g. web planner - access to your library from web browser, ability to plan track and sync to device, ..).

As already mentioned by Menion, selecting specific folders for sync is not possible (sync all or nothing). If you would like to participate in testing you could
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 08, 2020, 15:44:32
Thx for explaining. Don't get me wrong, I trust you, my basic concern usually is, services get exploited and at some point we may find big data dumps available publicly - it always happens here and there... Well it's all about risk management,,, some stuff is OK for me to reside on the other end, other stuff only in encrypted form (with only me as the key owner).

But hey, since Alpha is having its own db here, I just remove "sensitive" folders and then sync.
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 08, 2020, 16:01:28
@balloni55
thanks for the log! Version 3.48.0.2/3 had a problem in methods that reported problem ::) so your log only contains info that the app crash when trying to report a problem  :).

New version MapGooglePlay_3.48.0.4_962_beta just uploaded.
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 08, 2020, 16:08:47
OK I learned client side changes, also deletions, are synced to the server.

A crash happened.

- Everything was in sync.
- I deleted all tracks in a folder
- Locus started auto sync as set
- Locus disappeared
- Started it again, top left notification visible again, crash
- Then it was possible to start

Initial ideas.
a) Track/point/folder overlays if in sync
b) Different modes. "Upload only, download only, mirror device to cloud, mirror cloud to device"

Haha, thx for your effort, I guess you opened a can of worms here. 😉
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 08, 2020, 16:15:55
New version, it happened after track deletions.

And let me add: congrats for going public after probably a lot of hard work.
Title: Re: Locus Map - cloud/sync server
Post by: Andrew Heard on September 09, 2020, 06:37:08
Important: make a full backup of your database!! This version modifies the database upon start and also may cause a data loss in case of any serious problem in client-server communication.
@menion - is the database shared with LM3Pro? I don't want to install this version of LM4 if it modifies the LM3Pro database (unless it stays compatible). It's probably OK because balloni55 & tapio haven't expressed concerns.

Congrats too on this important milestone.
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 09, 2020, 07:14:38
Not sure about the defaults. It's in Locus working directory (see misc settings).. /data/database/
I suggest you backup this folder. Or just create a backup.

My Locus instances use distinct paths.

@Menion, Locus UI does not display the  working dir. Just "SD card" is not enough.
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 09, 2020, 08:29:21
@tapio
may you please try you sync again? We identified and fixed issue on our server.

Ideas:
a) it needs a new "Data manager" screen (planned), but agree, will be useful
b) interesting idea, little dangerous. The question is if really needed. When the first sync after install happens, there is one alternative mode "clear device and download only", that should be enough I believe.

@Andrew Heard
as @tapio wrote, it depends on your settings (btw. full paths are under the "question mark" icon). In most cases, default settings for the Locus directory is currently the root of internal memory, so yes, databases are shared. It should not be a problem to use this shared database, but just to prevent any problems, suggest to keep backup in safe.
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 09, 2020, 08:53:45
I waited for an auto sync at first, but it did not do it. I think it should have tried it again.

Manual sync worked, thx.
Title: Re: Locus Map - cloud/sync server
Post by: janaton on September 09, 2020, 10:05:02
@tapio
Quote
Thx for explaining. Don't get me wrong, I trust you, my basic concern usually is, services get exploited and at some point we may find big data dumps available publicly - it always happens here and there... Well it's all about risk management
You are absolutely right. It is up to you if you decide to sync your data into cloud or not (do data backup on your own). Sync is and always will be opt-in service. When using synchronization you will benefit from integration with our web planner (work in progress).

Thanks for your time and checking/testing synchronization and your feedback :). We have already found some bugs that somehow passed during our internal testing round.

Quote
And let me add: congrats for going public after probably a lot of hard work.
Yes, we have spent a lot of time researching and designing this solution. Thanks a lot :). Hopefully no serious issue will be discovered  and we will go public soon and offer this great service to Locus community.
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 09, 2020, 12:30:37
I like your confirmation, that we worked hard and that it is a great service  ;D
Title: Re: Locus Map - cloud/sync server
Post by: balloni55 on September 09, 2020, 13:58:09
Quote
may you please try you sync again? We identified and fixed issue on our server.
sync stop after ~3 sec with message as schown in post#10
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 09, 2020, 15:30:15
Hello balloni,
hmm seems you have in database track with a single! trackpoint. Interesting :).

The new version 3.48.0.5 just uploaded, so please try it. Such track will be simply skipped.
Title: Re: Locus Map - cloud/sync server
Post by: balloni55 on September 09, 2020, 17:00:35
Now manually started sync work :) thanks

My settings Automatic sync >on   text: update the cloud content automatically after change.....

i´ve deleted some trash POIs, but no autosync start??
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 09, 2020, 17:49:15
The web planner could be very useful. I am really hoping for an easy to use/ nonspace wasting UI (having brouter-web in mind, it's unmatched IMO).
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 09, 2020, 18:17:01
I wanted to test syncing behaviour if I delete embedded WP from tracks.

a) Locus crashes/restarts when I delete an embedded WP. No sync.

b) Not related to syncing. Embedded Waypoints, unlike normal POI, cannot be deleted directly via the new POI screen. Makes their handling more clunky.
Title: Re: Locus Map - cloud/sync server
Post by: Andrew Heard on September 10, 2020, 01:40:02
Just my initial experience. With wifi/ mobile data disabled > Full sync > a reminder message to enable one or the other could be helpful.

1019 tracks & 272 points took ~20 minutes. All OK.

Any other testing possible? How to now share this data?
Title: Re: Locus Map - cloud/sync server
Post by: janaton on September 10, 2020, 10:34:03
Hi Andrew,
thanks for sharing your experience. Regarding the offline state, this is really good finding! I guess that it would make sense to disable "Sync button" + show some warning that you need connection for this operation. But let's wait for @menion, maybe this is already on the TODO list ;)

Another benefit of sync (other than backup) is actually possibility to keep multiple devices in sync. You just need to sign in with the same user on a different device (e.g. tablet or even emulator). All changes on both devices should be synced. If you modify same item in parallel then newest change wins. You can definitely test this scenario.

Be aware that right now, you have to pull new changes to your device manually ("Cloud sync" screen > "Quick sync"). We plan to add instant notifications about changes in cloud, soon. This notification should then automatically start sync which should keep your device fully synced with cloud without any extra effort.

Sharing data with other people using cloud is on our roadmap but not implemented right now. We need to finish web planner first which should be used to visualize your shared track (using dedicated url).
Title: Re: Locus Map - cloud/sync server
Post by: balloni55 on September 10, 2020, 18:52:25
First test with Locus MEmu emulator on PC and my device:
- sync data to "new" locus on emulator work well :)
- message about missing custom icons, only Locus and Garmin Icons are displayed
- due to the missing of custom icons it is not possible to do action with click on it> show/hide, some/all. It would be good to set up a missing custom icon and replace it with a standard icon.
- if i delete one or more POIs on EMU or device and run sync i get an error message with Code1404, although one POI is deleted at each sync
Title: Re: Locus Map - cloud/sync server
Post by: sigi-s on September 10, 2020, 20:21:57
Hi  menion, hi Locus developers!
Thanks for all your efforts, I can´t test now, but I am exited to see it running.
Although I trust you, Locus guys, I relay do not trust Google or the other big Tech giants.
 ?  Will there be a possibility in the future to sync to my OwnCloud / Nextcloud Webdav server?
 ?  When approx will the final version be released?
Thanks, Sigi
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 11, 2020, 09:22:58
@balloni55
ah amazing, the problem with deleting content after sync fixed, thanks!

The issue with icons is anyway weird. If the point does not find its own defined icon, it should automatically use the icon of the folder.

I've just uploaded new version "MapGooglePlay_3.48.0.6_964_beta", so give it a try. I believe it will works correctly now.

@sigi-s
Hi :). This is a good question we were thinking about ... for a quick moment. Sync as is created is not about simple copy of all your data to any place. It is more about deeper comparing of changes and solving possible conflict problems. Also about sharing to public (later) etc. So any other solution that our internal won't be possible, because whole system is tightly connected to our server logic.

For "backups" of your own data, you may use existing Google Drive/Dropbox/OneDrive (LM4) uploads over Backup manager. OwnCloud will be most probably added later as well.
Title: Re: Locus Map - cloud/sync server
Post by: lega4 on September 11, 2020, 13:12:13
First of all, thanks for starting work in this direction even with a delay of several years. Later is better than never :)

However, is my understanding correct that currently you're talking more about backup/sync between several smartphones? What I was expecting from a account/sync/cloud thingy is possibility to edit my points from the PC i.e. via Locus web interface (or Locus desktop app, no idea whether it exists, but you got the point) in order to get them on my smartphone. And get the new points I add on smartphone in this abstract "web interface" afterwards.

Is this completely out of scope for the implementation or do you consider having something like that eventually?

Just to be clear, my use case is typically "prepare travel plan with tracks and points" (on PC, just because it's so much superior over smartphone for such a complex task), then "do travel, during vacation add some newly discovered points to my map" (on the phone as I don't have PC during vacatioin) and then "analyze my track, organize all the points and publish it as i.e. blog post" (again on PC). And that's exactly what I was expecting from this cloud sync - to be able to edit my points from PC and seamlessly get the updates on the phone.
Title: Re: Locus Map - cloud/sync server
Post by: balloni55 on September 11, 2020, 13:30:26
any answer to my question #19

Quote
the problem with deleting content after sync fixed, thanks!
thanks, confirm fixed :)
but why did i see a red exclamation mark down left at hamburger button although the sync was successfull??? it change to a green dot if i open this note once.

Quote
The issue with icons is anyway weird. If the point does not find its own defined icon, it should automatically use the icon of the folder.
https://drive.google.com/file/d/1-FjkpXYi68ji7NIfnNxTZtGdJh_owTlt/view?usp=sharing

During synchronization no image attachments are transferred, understand
If I remove now the JPG icon with the exclamation mark and synchronize both devices the photo in my original POI will be deleted :o

Title: Re: Locus Map - cloud/sync server
Post by: janaton on September 11, 2020, 14:18:40
Hi lega4,
first of all, thanks. Implementation took some time but our resources are limited and we want to deliver quality over quantity :). But hopefully we are finally having right direction to do public release!

Cloud sync is exactly what you expect but we implement it in steps thus not everything is implemented - ready for public testing. I will try to briefly summarize our cloud roadmap:


When cloud sync and web planner is finished I believe that your trip planning workflow should be fully supported ;). I am already looking forward for it!
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 11, 2020, 19:53:40
@balloni55
ah so @janaton was correct when told me that there may be a problem with the icon of the group. Thanks for the video, I've added this to my tasks for the next days. Seems it works correctly for points, but not for groups with custom (but not available) icons.

Behavior with attachments is currently correct. They are not currently synchronized, anyway, it is a task with will work on, during the next months as well.
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on September 11, 2020, 22:31:08
@balloni55
...
I've just uploaded new version "MapGooglePlay_3.48.0.6_964_beta", so give it a try. I believe it will works correctly now.
...
Hi Menion,
looking at Google Drive I only see the 960 version from late August. How to get to the 964, pls ?
TXs and cheers
Michael
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 11, 2020, 23:11:58
Check link in the first post. I work on sync and routing separately, so every topic has its own versions. Once, sync will be considered as "final", I'll merge them.

Sent from my Pixel 2 using Tapatalk

Title: Re: Locus Map - cloud/sync server
Post by: Andrew Heard on September 12, 2020, 01:47:58
Just a very low priority thought, and I'm sure there is help topic elsewhere... With public sharing of tracks it becomes even more useful to implement a feature where the start or end of a recorded track can be optionally deleted so that the home address can be obfuscated/ hidden. Of course this is already possible via the track editor, but is too tedious for most users just to hide their home address.
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 12, 2020, 10:14:21
I wouldn't call it "backup" btw. - It syncs deletions to server. With the term backup, people would eg. expect to get accidentally deleted tracks back - which is not possible.
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on September 12, 2020, 19:20:46
Hi Menion and Janaton,
I am reading about conflict resolutions, even deletions and in particular the latter rings alarm bells.
To save users and Asamm from trouble (and believe me, when that goes out into the wild, life WILL become wild ...) Istrongly propose at least two things:
- any sync run shall create a separate, human readable log file which includes all points and track actions in a meaningful description
- any overwrite should not be a physical overwrite in the DB, but rather creating a new point/track and makr the earlier one "invisible"
- same for deletions
- then add a cleanup function that physically removes those flagged entries
- and if you want to be really nice, also offer to restore (with "restored-DATE") inserted into the object name
If you think this may puzzle users, you can offer such as "safe mode" :-)
Just my 2c
Cheers
Michael
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 12, 2020, 22:02:52
@Andrew Heard
good point, we probably forget on this important point. I've added this to list of things to discuss on our next meeting, thanks.

@tapio
correct, backup is little bit "missleading" :)

@michaelbechtold
hmm quite a complex change. I believe that such backups won't be necessary on the server-side. Ondřej (@janaton) already tried to do something similar, but in the end we declined this concept. I'll add this as a note to our discussion as well, thanks for the tip!
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on September 12, 2020, 22:37:20
Hi Menion,
I am not talking about server side. Today you notify about the last update. Locally on the device.
That could be a history of more than one.
When ypou tap the "?" you get some stats today. Very technical. That should be a drill down for each category: addition, deletion, change. All locally on the device, as you must have this information there anyway. Hence no complexity.
The flagging etc. I propose below is also LOCAL, not on the server side.
Resilience ...
Cheers
Michael
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on September 12, 2020, 23:08:02
Now some test results:
- tracks DB is nearly 1 GB in size
- waypoints has about 35,000 entries (6 MB)
- this is the sequence of events
- S10 has the latest DB, sync, runs through
- Note 3: started sync around 16:30 - still not finished, now at 8% of the points
- LG G5: it had an older version of the DBs, a bit smaller. Sync started - everything flattened
- Tab S6 lite: DB grew massively in size (now 1.4 GB), but seems to work at least. However, now with a new sync start, the points are flattened (groups are still there, but empty)
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on September 12, 2020, 23:29:46
New observation on the Tab: all groups are there twice, some appear even a third time. Many of them are not complete.
Quite a mess !
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 12, 2020, 23:52:20
Do I understand correctly, that you have more devices, everyone with your own version of the database and you started full sync on all of them? Then all your data are merged on the server and downloaded on all devices!

During the first sync, you have an options to perform "full-sync" so merge current device data with data already existing on the server or start sync as a new device, so all local data are deleted and data already existing on the server are downloaded -> and this option should be used on all secondary devices!
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on September 13, 2020, 08:36:46
Well, what you describe is a copy/paste, not a sync.
I had the suspicion that on the older devices there were some tracks that I had not manually transferred to the latest device (S10) in the past (waiting for your sync feature :-)).
So they would NOT be a fresh install, but rather a full-sync for a good reason.

Is there a way to access the server for my account and have a list of what is there ? You know, I do not need fancy features for this kind of fundamental testing.
Title: Re: Locus Map - cloud/sync server
Post by: janaton on September 14, 2020, 09:54:41
Hi michaelbechtold,
thanks for your time, testing sync and giving us feedback.I would like to reveal a little bit of technical side of sync algorithm and explain why sync is not working as you would expect.

Each object in device database (point, track) has to be uniquely identified. We use unique numeric ID for this purpose. If you have multiple devices (e.g. D1, D2) there is absolutely no guarantee that point[#1] (point with id=1) in D1 is the same as point[#1] in D2. This is true in the case when you transfer DB from D1 to D2 but in general we can not assume this.

Possibly, we could try to match objects by all properties but it is not correct, in my opinion, too. Let's imagine, that we have exactly the same point[#1] on D1 and D2. When you make minor modification to point[#1] on D1 then point[#1] on D2, which was previously the same, is now different one.

In order to match objects server side we added another ID to client DB -  UUID (example: 162d3708-b2d1-458d-8054-d8e866fb9b10) which is used as unique identification of object across all your devices (as explained, simple numeric ID can not be used for this).

After rolling out new Locus version including synchronization (UUIDs will be automatically generated) then you should be able copy and paste objects between devices and sync should have no issue with it (objects will be correctly paired using UUID) and synced correctly - no duplicates should occur.

Unfortunately, in your case, each device will generate different UUIDs on every device thus you will need to sync these changes manually, once. I will discuss this scenario with @Menion. Hopefully, we will by able to come up with some idea to make this merge easier ;).

I hope that my explanation is good enough for you and understandable. If not, do not hesitate to ask me :).
Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 14, 2020, 11:18:39
I can totally feel what MichaelBechtold is saying. Currently, the user cannot see what is about to happen and what is stored on server side. Backup/sync means a lot of complexity after all.
I am speaking very general. Experience is telling us, there will be a lot confusion, (assumed) data loss etc... A lot of users in need of support. Lots of pain for the dev side. I think it is crucial that a user can easily see/understand what exactly will happen to the data after he presses "sync".

Just one aspect of the pain: I am convinced users will at first understand this as a backup system, as it is happening with RAID systems. So when they accidentally delete a track on client side, they likely expect to get it back via a sync -which will not happen. Which will delete on server side as well.

You opened pandoras box :D  ;)
Title: Re: Locus Map - cloud/sync server
Post by: Frivold on September 14, 2020, 14:09:18
I have tested the new version now, and i like it very much! But one thing is missing, when I tap on a point in map, then I don't get the context-menu, that I am used to. (Fex guide too, detail etc).
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on September 14, 2020, 16:18:14
TXs Janaton, for your explanation. As I am in Comp.Science since 40 years, I exactly understand your message.
Yet, what I propose is a UUID based on a hash of the whole record (all point or track data, no device specifics inluded!). As we are not fighting off NSA here, it can be a "cheap" one.
Then all those special cases are gone, right ?
If the hash is identical across devices, your existing algorithm will automatically do the right thing - ignore. If not, sync.
And I still do not understand what is complex about careful handling of deletions (flagging, detailed sync history, user triggered cleanup).
TXs and cheers
Michael


Title: Re: Locus Map - cloud/sync server
Post by: janaton on September 14, 2020, 21:05:05
@michaelbechtold, thanks a lot for your idea regarding generating UUID based on the object hash. This won't work in all cases (e.g. object modified on one device) but should be good enough solution for most cases :). We will definitely discuss it with @Menion during this week.

I can not tell you details about complexity on the client because I am implementing only server side. Keeping older versions in the cloud was my initial idea. I wanted to have periodical job cleaning old object versions but I was convinced by team not to do it. Several reasons (I probably do not remember all of them):
.

If you have some thoughts why we should reconsider current logic (not keeping history), please, share it with us. I am sure we will have some time to open this discussion within the team once more :).
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on September 15, 2020, 07:19:07
TXs for your nightly response, Janaton.
Let me go through the points one by one:
1) Main concern -  we could accidentally mess up versions and expose something obsolete
Much more serious is a sloppy delete - from user side as well as on your side. A code that deletes ONLY if there is a specific flag set in a DB row if very straight forward and can be verified. A "free" delete as today is more risky in any regard. Risk x (# users ...)

2) Additional complexity - clean up made asynchronously (implement task manager, cleanup logic located away from sync code)
Yes, there is some more complexity. But one more daemon running on the server does not make a huge difference. And again, cleanup logic is very simple.

3) Increased storage requirements that you need to handle history
If you want to stay in control of this, then drop my idea to have users trigger the cleanup (and yes, users may forget forever ...), but rather establish a rule like "Restore up to one week from deletion". This limits the additional storage, and lets you stay in control. A restore (triggered by a user) is a simple removal of the deleted flag, followed by a sync.

 4) We agreed that we don't want to offer versioning functionality to user thus keeping history would be more or less for support and emergency situations (something would go terribly wrong).
I do NOT propose any versioning, but only a limited safeguard against accidental deletions. Big difference!

Regards to the whole team.
Michael

Title: Re: Locus Map - cloud/sync server
Post by: tapio on September 15, 2020, 15:46:36
Someone else mentioned it, on an otherwise working sync I see an intermediate exclamation mark on Locus main hamburger menu.

I have tested the new version now, and i like it very much! But one thing is missing, when I tap on a point in map, then I don't get the context-menu, that I am used to. (Fex guide too, detail etc).
Same what I saw? I missed the ... menu on embedded waypoints in tracks. No easy way to delete embedded WP.
Title: Re: Locus Map - cloud/sync server
Post by: Frivold on September 16, 2020, 10:08:09
I now understand that this thread is meant for cloud/sync, but i will tell that the "problem" suddenly works fine! Now i get a menu-line at bottom when I show points. I really like this version of Locus, and look forward to release. Do you have an idea when it is ready?

By the way, i see that many phrases in the app is in English, but the Norwegian translation is 100%. Is it something wrong, or haven't you put the strings into translation yet?
Title: Re: Locus Map - cloud/sync server
Post by: Frivold on September 16, 2020, 13:15:32
I now understand that this thread is meant for cloud/sync, but i will tell that the "problem" suddenly works fine! Now i get a menu-line at bottom when I show points. I really like this version of Locus, and look forward to release. Do you have an idea when it is ready?

By the way, i see that many phrases in the app is in English, but the Norwegian translation is 100%. Is it something wrong, or haven't you put the strings into translation yet?

Aaah! Now i know why it s suddenly working! When I have screen in full modus, the menu at bottom dont appear, when showing a point. But i turned full screen off at a moment yesterday, and then the menu work. So it is a little bug there...

Edit: The menu is there actually also in full screen, but just some pixels height, so we cant see the buttons.
Title: Re: Locus Map - cloud/sync server
Post by: slarti76 on September 30, 2020, 16:36:32
Hi there
Already a little late to the party here ;), was on holiday. Didn't get to trying out the Beta yet, but being a big supporter of the sync feature for years (and personally responsible for several naggings of Menion, I remember), I like what I read.
Some posts on the previous page reminded my of one of my proposals for sync, which I'm repeating here, in the hope you either have it already in your list, or hopefully add it: Selective sync
The basic idea is that, for every device, you can choose which folders to sync (points and tracks). This would provide an easy way to solve two of the main problems brought forward:
I would be OK with doing it based on groups instead of folders, too - just leaves the question how to handle root folders (without group): Always sync or always local?

What do you think?
Title: Re: Locus Map - cloud/sync server
Post by: freischneider on September 30, 2020, 17:43:34
I would be for it if you can only choose groups but also only a single folder. In general, I would treat folders and groups the same way.
For the future:
I would get rid of groups and replace them with a folder. In all systems I know there are folders and sub-folders. I understood why Locus introduced groups.
Title: Re: Locus Map - cloud/sync server
Post by: Menion on September 30, 2020, 20:02:16
@tapio
may you please explain in detail, which menu and where you miss? Thanks

@Frivold
Release date ... we all still believe till end of year. But web planner will be most probably feature, that took most of time and it's not clear if ...

Translations: it is possible. Not all texts are final so not yet uploaded for translation.

Full-screen mode ... I need some polishing, I'm aware of it  :-\

@slarti76
the idea to select the only a portion of the database for sync is already on my "think about it soon" list (last not yet solved main think about sync system). I understand this point, but I still do not see a simple solution, that will work correctly and won't be confusing. So I can't promise it will be in the first version, but I can promise, that I'll think about it soon!
Title: Re: Locus Map - cloud/sync server
Post by: jonny.blue on October 03, 2020, 16:24:51
Just started with sync on my two devices ... and that's really a great improvement. 😎

The only thing I detected with first tries ...:
when I create points (geocaches) on one device .... synchronization works  👍but the new points are unvisible on the second device.🙄
I manually have to find them and switch visibility state.
Title: Re: Locus Map - cloud/sync server
Post by: jonny.blue on October 04, 2020, 18:29:42
Today in field LM tried to sync despite my settings only allows that with active wifi connection ...
Title: Re: Locus Map - cloud/sync server
Post by: Menion on October 09, 2020, 14:03:02
Hi guys,
in new 3.48.2.3 Beta version, quite a lot of changes in UI and also in background was done. So who is interested, please give it a try.

Current known limitations
- missing sync for attachments
- missing notification on other devices that there is something new (needs to do it manually)
Title: Re: Locus Map - cloud/sync server
Post by: jonny.blue on October 09, 2020, 17:26:44
Can't see any download file (google drive link in first post) 🤔
Title: Re: Locus Map - cloud/sync server
Post by: jonny.blue on October 10, 2020, 11:35:01
Downloaded new version ... but the visibility problem is still there 🙄

Just to describe what I do and why it's important for me:

on tablet I prepare my geocaches in different folders. Some caches I hide (deactivate visibility) to see only those caches I want to solve when I'm outdoor with my mobile.
At the moment sync works good for folders and points ... but all caches on my later synced mobile are invisible.

Anyone any idea?
Is there anything I do wrong or is it just a bug (or maybe a feature 😁)
Title: Re: Locus Map - cloud/sync server
Post by: Menion on October 10, 2020, 11:55:23
Hi Jonny,
sorry, but this is intended behavior.
Sync system currently sync data, but not it's visibility state.
Because we will keep data up-to-date across all devices and also the web portal, we decided not to transfer visibility state to allow a unique set of visible items on every unique device.
Title: Re: Locus Map - cloud/sync server
Post by: freischneider on October 10, 2020, 12:21:46
Downloaded new version ... but the visibility problem is still there 🙄

Just to describe what I do and why it's important for me:

on tablet I prepare my geocaches in different folders. Some caches I hide (deactivate visibility) to see only those caches I want to solve when I'm outdoor with my mobile.
At the moment sync works good for folders and points ... but all caches on my later synced mobile are invisible.

Anyone any idea?
Is there anything I do wrong or is it just a bug (or maybe a feature 😁)

You could put the caches you want to be visible in a separate folder. And then only show this on the phone.
Title: Re: Locus Map - cloud/sync server
Post by: Erelen on October 11, 2020, 00:42:58
we decided not to transfer visibility state to allow a unique set of visible items on every unique device.
And what about order of folders and groups (for tracks and for points)? They doesn't sync by design or just not implemented yet?
Title: Re: Locus Map - cloud/sync server
Post by: Menion on October 12, 2020, 08:02:29
This is really good question Erelen. I personally suffer for many years because of inability to sort folders internally :). Unfortunately, there is still no internal parameter that keep folders in some "sort" order. The result is, that in certain situations, result on the second device may not match exactly. Quite unfortunate.

Probably best should be to sort folders always by it's name. For now. I'll think about it.
Title: Re: Locus Map - cloud/sync server
Post by: slarti76 on October 12, 2020, 08:39:34
This is really good question Erelen. I personally suffer for many years because of inability to sort folders internally :). Unfortunately, there is still no internal parameter that keep folders in some "sort" order. The result is, that in certain situations, result on the second device may not match exactly. Quite unfortunate.

Probably best should be to sort folders always by it's name. For now. I'll think about it.
They're not sorted? Seriously...? On my device, folders and groups always seem to be sorted by name, even when I create a new one. That can't really be "coincidence", can it?
Title: Re: Locus Map - cloud/sync server
Post by: Menion on October 12, 2020, 08:47:40
Of course, they are sorted ::) .. sorry, I was confused by Erelen's question because it looked like they are not. My mistake.

Oki, steps, back ... @Erelen, you see items sorted other than by name on the second device?
Title: Re: Locus Map - cloud/sync server
Post by: Erelen on October 19, 2020, 05:30:11
After some time, order on my tablet become the same as on the phone. But now I reproduce problem: reinstall Locus 4 on my tablet and delete Locus/data/Database. And after first sync ("Full download" mode) I see different order again.

And I see pattern: on my phone my "conventional" order - english named folders (in alphabetical order) at first and cyrillic folders (in alphabetical order too) after them.

But on the tablet cyrillic folders are first and then - english.
Title: Re: Locus Map - cloud/sync server
Post by: slarti76 on October 28, 2020, 07:24:54
Hi, so I finally tried out the sync on one of my phones. The thing I noticed is that when clicking on the sync icon, the next screen is always the same, no matter whether you already synced or not. It always says "Start" which is confusing - seeing this after you already synced for the first time makes you think it didn't work. Also, there's still the button "Later" which makes no sense after you already started syncing.
I think that, after you set up sync, a click on the icon should immediately bring up the settings, best with the Advanced Options integrated.

One more thing: On this phone I have two Google accounts. Now as I aready logged in with one of those accounts in the store, so sync used that one without asking. Is that a restriction by Google that you can only use one Google account in the whole app? I would understand, but if it's possible to choose separately, that would be more flexible.
Title: Re: Locus Map - cloud/sync server
Post by: Menion on October 29, 2020, 15:17:21
Hello @slarti76,
I've just published a new version of LM4 test version. I made quite a lot of changes in sync UI so give it a second try. Idea is that this screen should not be needed in the end, it should be completely automatic (not yet).

Google accounts: Locus Map don't care (do not know) which accounts are on your device. We use a single account for the whole Locus World (Locus Store, sync system, planner web planner, live tracking, etc). So it's just on your which account you choose (currently only possible to change in the side menu in Locus Store).Just keep in mind that synced data, purchased content in Store etc, are bound to your account!
Title: Re: Locus Map - cloud/sync server
Post by: slarti76 on October 29, 2020, 16:55:00
Hello @slarti76,
I've just published a new version of LM4 test version. I made quite a lot of changes in sync UI so give it a second try. Idea is that this screen should not be needed in the end, it should be completely automatic (not yet).
Yeah, ok, understand. Already looked at it, looks better.

Quote
Google accounts: Locus Map don't care (do not know) which accounts are on your device. We use a single account for the whole Locus World (Locus Store, sync system, planner web planner, live tracking, etc). So it's just on your which account you choose (currently only possible to change in the side menu in Locus Store).Just keep in mind that synced data, purchased content in Store etc, are bound to your account!
Ok, I kinda expected that. Not a problem, just curious. Probably would be better to move the selection of account to the settings (now that it's used in several places), but you seem to be aware of it.

Another question:
Is it planned to also sync icons and attached media at some point? At least icon ZIPs would be nice, and that shouldn't be too much data. With media, I can imagine that they could get quite big - and potentially difficult to sync if media from outside the Locus data directory is linked. How is this handled now?
Title: Re: Locus Map - cloud/sync server
Post by: Menion on October 29, 2020, 18:50:32
Accounts ... what is planned on near feature is something like "My account", probably in the main menu, instead of Locus Store, where you should be able to change nick, photo etc. So yes, we are aware of it.

Attachments and icons ... well, attachments are currently a weak spot. We have a system for them set up on paper, but because we want to do it properly, it will take some more time so it won't be finished for the first version. But it is with very high priority next in the row. Together with it is planner bigger change in Locus Map & work with attachments in UI (simplier, faster).

Icons ... well, I'm still not sure what to deal with them, sorry :). So for now, they won't be synced as well.
Title: Re: Locus Map - cloud/sync server
Post by: slarti76 on October 30, 2020, 09:01:04
Icons ... well, I'm still not sure what to deal with them, sorry :). So for now, they won't be synced as well.
Couldn't you just sync the ZIPs in data/icons? Would be useful in any case.
Title: Re: Locus Map - cloud/sync server
Post by: Andrew Heard on October 30, 2020, 22:10:05
Cloud Sync > tap More Info button > there is less info (no date/ time), but mainly same info on bigger screen.
Title: Re: Locus Map - cloud/sync server
Post by: Andrew Heard on October 30, 2020, 22:17:34
About 2 weeks ago I tried a manual cloud/sync for 1st time. Just 1 device. I assume it was OK - no errors.

So yesterday after a few more tracks & points I tried another manual sync. I was expecting the process to be fast because there were only 2 weeks of new tracks & points, but 40 minutes later I find that LM4 has created duplicates of every track, every point, every folder, every group.

Naturally this was quite annoying because A) when you delete a group you still have to delete each folder within the group, there is no option to delete all at once B) I then had to carefully pick which of each pair the latest changes, if any, were in, and delete the other. In one case I made a mistake & had to import.
Title: Re: Locus Map - cloud/sync server
Post by: jonny.blue on October 31, 2020, 19:42:06
Quote

sorry, but this is intended behavior.
Sync system currently sync data, but not it's visibility state.


I really want go back to this point ....

I thought sync is build for 'syncing' several devices to have the same data (points, tracks) on more than one device.

And in that way I would like to use it:
Building routes, loading and sorting geocaches on my tablet  ... and than go out with my synced mobile and see all those routes and caches ... and just walk and find them.😁
At the moment I have all my tracks and caches on my mobile but especially all those needed new points are invisible.
I have to enter every folder and set all to visible 😳

Do I use/understand 'sync' in a wrong manner?

What is the use case for a different visibility state?

Is there a workarround for this/maybe only my 🥴 problem?
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on October 31, 2020, 22:32:22
I did a quite systematic test with three devices  in summer and stumbled across the same issues.
Followed by a discussion wit Menion.
Bottom line: there needs to be ONE master for the initial SYNC, other need to be empty followers.
The reason why: the finger print is not by content, but by some meta data combination, incl. time stamps of the local device (or so).
I strongly proposed to go to a hash across contentand title (and ignore the usual suspects that may differ for thechnical reasons). Such would avoid any of those problems (after some deeper analysis and proper solution design).
But it is not the way they built it - until now ...
Cheers
Michael
Title: Re: Locus Map - cloud/sync server
Post by: slarti76 on November 02, 2020, 08:37:03
Quote

sorry, but this is intended behavior.
Sync system currently sync data, but not it's visibility state.


I really want go back to this point ....

I thought sync is build for 'syncing' several devices to have the same data (points, tracks) on more than one device.

And in that way I would like to use it:
Building routes, loading and sorting geocaches on my tablet  ... and than go out with my synced mobile and see all those routes and caches ... and just walk and find them.😁
At the moment I have all my tracks and caches on my mobile but especially all those needed new points are invisible.
I have to enter every folder and set all to visible 😳

Do I use/understand 'sync' in a wrong manner?

What is the use case for a different visibility state?

Is there a workarround for this/maybe only my 🥴 problem?
I personally support the "don't sync visibility" decision, because I use Locus in different ways on my phones (in the field) and tablet (planning, editing recorded tracks, making map images) and would be annoyed by synced visibility. But I also see your usecase. For me, this would really be something that needs to have a switch "Sync visibility of points/tracks".
From a development standpoint, I also understand if this is lower on the list for Menion for now (adds additional complexity, so best to first go without). But it should be on the list.
Title: Re: Locus Map - cloud/sync server
Post by: Menion on November 02, 2020, 08:56:45
@slarti76
sync of icons: it should be doable over basic sync of zip files and it probably ends like this, but I would rather discuss this with the rest of the team before I'll post some more info.

Anyway, custom icons & attachments are currently not synced, we know about it, we consider this as a limitation and will definitely work on it with higher priority as the next step.

@Andrew Heard
"More info" button > well, right. More info is not really "more info". Some stats may be useful here, to see exact progress...

Duplication: hmm, there had to be something that caused this problem! Your first and second sync were from the same device? There was no restore of the old manual backup between sync?

@jonny.blue
Sync of the visibility: sync system is not just between the device, but also for the web platform (web planner). As @slarti76 correctly wrote, all these "devices" are most probably used for the different use-cases. Automatic sync of item visibility for the web means always load all visible data = overkill for the web. It will also cause a lot more synchronization operations because not just editing of the point/track will trigger a sync, but also it's simple change in the visibility. I see a few more problems that may come with it. So sorry, no, it is not planned and it brings a lot more negative effects than benefits.

Anyway, as usually ... time and usage in the field may reveal unexpected results/usages that were not visible from the sofa :).

@michaelbechtold
I remember this proposal well. We discussed this for quite a long internally but were not satisfied with any possible solution based on generated hash from title/content etc. So yes, we are going the way, where the first used device should be a master device (it is automatic for new users, existing users have to be a little careful) and all other devices get a choice in updated UI, if they want to keep & merge current data or want's to remove local content completely and pull data from the server.

If we prepare UI for this "flow" correctly, it should be perfectly safe and 100% accurate. I'm anyway curios, what happen to Andrew because data duplication should never happen.
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on November 02, 2020, 18:55:25
I have data duplication, too - the DB is now nearly 2 GB in size ...
But I can remedy, as this device is not used for active tracking :-)
Title: Re: Locus Map - cloud/sync server
Post by: Andrew Heard on November 02, 2020, 23:01:41
@slarti76
Duplication: hmm, there had to be something that caused this problem! Your first and second sync were from the same device? There was no restore of the old manual backup between sync?
1st & 2nd sync were from same device - I only have LM4 installed on 1 device; "about" 2 weeks apart; no restore in between the 2 syncs; I haven't tried to sync again (risk) but next time I will do a full backup just before a test sync because this caused lots of work & nearly lost some data that was created in the 2 week period. Happy to continue testing but waiting for your advise.
Title: Re: Locus Map - cloud/sync server
Post by: Menion on November 03, 2020, 12:54:13
Both of you, hmm ... Michael, any idea how this may happen, as you already understand the system in the background. Two devices and merged data?

Andrew, this is a mystery to me. May you please share (privately or the public, no sensitive info inside) "Locus/data/database/sync.db" file? Inside are metadata for all sync tasks done on your device, thanks.
Title: Re: Locus Map - cloud/sync server
Post by: michaelbechtold on November 03, 2020, 23:59:29
Well, Menion, I know I had an older copy of the DB on that device which ended up with duplicates.
That did duplicate in my earlier testing phase and we discussed it. "Works as designed" in the end, right ?
Title: Re: Locus Map - cloud/sync server
Post by: Andrew Heard on November 04, 2020, 05:15:58
May you please share (privately or the public, no sensitive info inside) "Locus/data/database/sync.db" file?
I've emailed you the file. Thanks.
Title: Re: Locus Map - cloud/sync server
Post by: Menion on November 04, 2020, 09:43:01
@michalbechtold
well, yes, it's intended. Anyway, this should not happen with the latest version without user confirmation. When you init sync on your second device and this device detects any existing data on the server, it asks you if you want to duplicate data or replace local data with the server.

@Andrew Heard
thanks for the database. I've discussed this right now with @janaton (responsible for the server part) and we are unable to find any issue directly on our side. You are aware that Locus Map Free (LM4 alpha) and Locus Map Pro share the same data right? Isn't possible that you restored the backup/database in Locus Map Pro?
Title: Re: Locus Map - cloud/sync server
Post by: Andrew Heard on November 04, 2020, 23:19:28
You are aware that Locus Map Free (LM4 alpha) and Locus Map Pro share the same data right?
yes

Isn't possible that you restored the backup/database in Locus Map Pro?
do you mean sync? I haven't restored any backup. LM3Pro doesn't have any sync function right? I don't it's possible.

this should not happen with the latest version without user confirmation
is it possible to delete all cloud sync data before I retest? or what do you suggest I test next?