Messages

i seem to be getting the same problem of it failing after numerous HTTP 403 errors

ERROR dtm_england_294_548.tif (OSError, attempt 3): HTTP 403

version 1.48.2

Again great stuff. one more error i occasionally notice is:

Code Select Expand

✓ Step 3 finished in 44m25s  (cumulative 2h13m40s)
  (index sheet: reverse geocoding failed: URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>)
  lakes_opos_20m_g2p0_ombrage_z8-17_planche.png : index sheet (1000x968)
  lakes_vat_20m_g2p0_ombrage_z8-17_planche.png : index sheet (1000x968)
  Done! Folder: /Volumes/Reservoir-A/_Colonel/Maps/_Lidar2map/North Pennines
  Total time: 133m40s
  History saved: /Users/mus/Applications/historique.json  (20 entries)

✓ Terminé (code 0)
  History saved: /Users/mus/Applications/historique.json

thanks for your quick fixes. I did mean to ask what if anything you are using to merge .mbtiles files?

Hi Nico,
all has been working great apart from one thing a found.Im using bounding boxes and have an area of the Lake district selected for which the corner juts out into the sea near where there is a nuclear submarine manufacturing facility. There is an area of sea which im assuming belongs to that, but rather than a 'no coverage' being returned, i get lots of Http 500 errors then the process fails. if i reduce the box a little to exclude the area -3.421000, 54.138400, -3.376200, 54.210300 then all is good :

Code Select Expand

$ /Users/mus/Library/Application Support/lidar2map/lidar2map --provider gb-england --zone-bbox -3.512468,54.144741,-2.624122,54.959754 --zone-width 20 --zone-name Lakes --output-dir /Volumes/Reservoir-A/_Colonel/Maps/_Lidar2map/Lakes/ --cache-dir /Volumes/Reservoir-A/_Colonel/Maps/_Lidar2map/-cache/ --production-dir /Volumes/Reservoir-A/_Colonel/Maps/_Lidar2map/-production/ --lidar --download --workers 8 --shading vat:dist=20,gamma=2 --shading opos:dist=20,gamma=2,sweep=1 --svf-sweep --file-formats mbtiles --zoom-min 8 --zoom-max 17 --image-format jpeg --image-quality 85

  Log : /Users/mus/Applications/logs/lidar_20260821_101849_20630.log (publication atomique à la fin)
  History: entry '1787303928287-11242-gui' (in progress)
=======================================================
  LiDAR : Royaume-Uni (Angleterre) — LIDAR Composite DTM (EA)
  Pipeline rasterio + numpy (numba for SVF)
=======================================================
  Folder : /Volumes/Reservoir-A/_Colonel/Maps/_Lidar2map/Lakes/
  BBox WGS84 : -3.5125,54.1447 → -2.6241,54.9598
  BBox EPSG:27700 : 301301,472281 → 360133,563838
  Area: ~5386 km²
STEP:1/3 Downloading tiles
  -> Compression enabled
  Root    : /Volumes/Reservoir-A/_Colonel/Maps/_Lidar2map/Lakes
  Tiles  : /Volumes/Reservoir-A/_Colonel/Maps/_Lidar2map/-cache/lidar/gb
  Zone    : /Volumes/Reservoir-A/_Colonel/Maps/_Lidar2map/Lakes
  LiDAR tiles [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  0%  24/5391  19s
  ERROR dtm_england_301_489.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  2%  115/5391  1m37s
  ERROR dtm_england_302_489.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  3%  206/5391  2m37s
  ERROR dtm_england_303_489.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  283/5391  3m34s
  ERROR dtm_england_304_479.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  284/5391  3m39s
  ERROR dtm_england_304_481.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  285/5391  3m39s
  ERROR dtm_england_304_483.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  286/5391  3m40s
  ERROR dtm_england_304_482.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  287/5391  3m40s
  ERROR dtm_england_304_480.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  289/5391  3m41s
  ERROR dtm_england_304_484.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  290/5391  3m43s
  ERROR dtm_england_304_485.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  291/5391  3m44s
  ERROR dtm_england_304_486.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  295/5391  3m49s
  ERROR dtm_england_304_487.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  5%  296/5391  3m50s
  ERROR dtm_england_304_488.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [██░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  7%  379/5391  4m57s
  ERROR dtm_england_305_479.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [██░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  8%  480/5391  6m11s
  ERROR dtm_england_306_479.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [███░░░░░░░░░░░░░░░░░░░░░░░░░░░]  10%  569/5391  7m19s
  ERROR dtm_england_307_479.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [███░░░░░░░░░░░░░░░░░░░░░░░░░░░]  12%  656/5391  8m28s
  ERROR dtm_england_308_479.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [████░░░░░░░░░░░░░░░░░░░░░░░░░░]  13%  736/5391  9m23s
  ERROR dtm_england_309_472.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [████░░░░░░░░░░░░░░░░░░░░░░░░░░]  13%  737/5391  9m27s
  ERROR dtm_england_309_473.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [████░░░░░░░░░░░░░░░░░░░░░░░░░░]  13%  738/5391  9m27s
  ERROR dtm_england_309_474.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [████░░░░░░░░░░░░░░░░░░░░░░░░░░]  13%  740/5391  9m29s
  ERROR dtm_england_309_475.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [████░░░░░░░░░░░░░░░░░░░░░░░░░░]  13%  745/5391  9m32s
  ERROR dtm_england_309_476.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [████░░░░░░░░░░░░░░░░░░░░░░░░░░]  13%  748/5391  9m33s
  ERROR dtm_england_309_477.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [████░░░░░░░░░░░░░░░░░░░░░░░░░░]  13%  750/5391  9m34s
  ERROR dtm_england_309_478.tif (OSError, attempt 3): HTTP 500
  LiDAR tiles [███████░░░░░░░░░░░░░░░░░░░░░░░]  23%  1279/5391  17m24s
  ERROR dtm_england_314_511.tif (TimeoutError, attempt 3): The read operation timed out
  LiDAR tiles [███████░░░░░░░░░░░░░░░░░░░░░░░]  26%  1410/5391  21m39s
  ERROR dtm_england_316_483.tif (TimeoutError, attempt 3): The read operation timed out
  LiDAR tiles [██████████████████░░░░░░░░░░░░]  61%  3320/5391  46m45s
  ERROR dtm_england_337_482.tif (TimeoutError, attempt 3): The read operation timed out
  LiDAR tiles [██████████████████░░░░░░░░░░░░]  62%  3356/5391  48m56s
  ERROR dtm_england_337_524.tif (TimeoutError, attempt 3): The read operation timed out
  LiDAR tiles [██████████████████░░░░░░░░░░░░]  62%  3358/5391  49m31s
  ERROR dtm_england_337_529.tif (TimeoutError, attempt 3): The read operation timed out
  LiDAR tiles [██████████████████████████████] 100%  5391/5391  1h12m47s
  Downloaded: 5362  Cache: 129  Missing: 0  Errors: 29
  History saved: /Users/mus/Applications/historique.json  (9 entries)
[PYI-20630:ERROR] Failed to execute script '_loader' due to unhandled exception: 29 tile download error(s) - pipeline stopped before shading/tiling (rerun to retry the failed tiles; successful ones are cached)
[PYI-20630:ERROR] Traceback:
Traceback (most recent call last):
  File "_loader.py", line 36, in <module>
  File "<frozen runpy>", line 287, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/mus/Library/Application Support/lidar2map/_internal/lidar2map.py", line 11116, in <module>
    _dispatch()
  File "/Users/mus/Library/Application Support/lidar2map/_internal/lidar2map.py", line 11031, in _dispatch
    return main()    # --lidar / --osm / par défaut
          ^^^^^^
  File "/Users/mus/Library/Application Support/lidar2map/_internal/lidar2map.py", line 5564, in main
    _telecharger_dalles_zone(dalles_dict, bbox, dossier_dalles, dossier_ville, args)
  File "/Users/mus/Library/Application Support/lidar2map/_internal/lidar2map.py", line 6156, in _telecharger_dalles_zone
    return _telecharger_dalles_zone_impl(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "_terrain_download.py", line 255, in telecharger_dalles_zone
RuntimeError: 29 tile download error(s) - pipeline stopped before shading/tiling (rerun to retry the failed tiles; successful ones are cached)
UNHANDLED EXCEPTION:
Traceback (most recent call last):
  File "_loader.py", line 36, in <module>
  File "<frozen runpy>", line 287, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/mus/Library/Application Support/lidar2map/_internal/lidar2map.py", line 11116, in <module>
    _dispatch()
  File "/Users/mus/Library/Application Support/lidar2map/_internal/lidar2map.py", line 11031, in _dispatch
    return main()    # --lidar / --osm / par défaut
          ^^^^^^
  File "/Users/mus/Library/Application Support/lidar2map/_internal/lidar2map.py", line 5564, in main
    _telecharger_dalles_zone(dalles_dict, bbox, dossier_dalles, dossier_ville, args)
  File "/Users/mus/Library/Application Support/lidar2map/_internal/lidar2map.py", line 6156, in _telecharger_dalles_zone
    return _telecharger_dalles_zone_impl(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "_terrain_download.py", line 255, in telecharger_dalles_zone
RuntimeError: 29 tile download error(s) - pipeline stopped before shading/tiling (rerun to retry the failed tiles; successful ones are cached)

✗ Terminé (code 1)
  History saved: /Users/mus/Applications/historique.json


Thanks for your detailed explanation. Sorry about the photos, i accidently deleted them but the links should work again now. I'll test once the release is out.

Hi.
It is a non split mbtiles map, and the grey box is there no matter the zoom, but it gets proportionally bigger compared to the map tile coverage as you zoom out. It also disappears when you get to zoom level 8, and it disappears on 2 sides at zoom 18 (top and bottom) with one side grey and the other a black area.

https://photos.app.goo.gl/eUwJKJkAvjD6S2e66

https://photos.app.goo.gl/6bFmoyczov1nn7Dv9

Hi Col,

Thanks; all of that information is extremely useful, especially since you have access to a real Intel Mac.

I tried with release 1.31.5 and once

xattr-dr com.apple.quarantine /Applications/LIDAR2MAP.app

Has been applied the app runs and creates output absolutely fine, if a little slow, not sure if you have added numba etc to the build.

There is no tile border when the maps are used as an overlay, but when they are used as the main map and you zoom out beyond the tile border, there is now a large grey box, not sure if this is the map or a problem in locus. I tried this with VAT,LRM,SVF,O+ & O-.I dont have any other mbtiles maps to check this with, but it doesn't appear to happen with the other sqlitedb maps.

I just tried version 1.31.1 and it says this when I open it:

 "LIDAR2MAP.app" is damaged and can't be opened. You should move it to the Bin.

I can still view the contents of the app.

Thats great, Ill have a try tomorrow. Im on 15.7.7.

I did play around with your build scripts and got it to complile a working version, although i dont think everything is perfect, and it was a bit chaotic working with AI on it. To get numba working on an intel mac versions in the venv have to be python==3.12.13
numba==0.60.0 llvmlite==0.43.0 numpy 2.0.x as numba dropped intel support after this. Also I had to 'brew install libomp'

The original build i created was failing on the TIFF compression, _internal/pyproj/.dylibs/libtiff.6.dylib is called and doesnt work, _internal/rasterio/.dylibs/libtiff.6.dylib fixed the problem.

I did however notice that the maps created viewed fine unless zoomed out beyond the tile, in which case there would be a black border and flickering graphical artifacts. (maybe this is just my build)

I hope some of that info helps you, and if you need me to build/test let me know.

Amazing, this is exactly what I was looking for, I have an older mac though, any chance you could release an x64 version or an mac-x64 build script?

Since Android's changes to the file system, I'm looking for ways to backup and transfer photos and videos attached to Points. It would be really useful and a simple solution if the auto backup could also include this, I'm not sure why it doesn't, then it would be easy to transfer them to another device. Since this isn't possible how can I manually do this? as I can't access the contents of the folder. This makes me very nervous as it leaves me in the situation of potentially loosing many years of data.

I have used locus for many years and still have many photos in the data/photo_poi folder (older versions used this location) and wondering if there is any way to move them to the new location of data/media/photo without moving each photo manually ?

Also, the naming of the photos differs from whether they were taken while creating the point or adding to a point that has been named, eg.  xxxxxxx.jpg vs name_of_my_ point _xxxxxx.jpg it would be great to be able to be able to have them all renamed with the point name.

I realise that there are probably much more pressing features, but I often like to look through the folder to try and find a particular picture, and named by point name would make this easier.. and please my OCD.. 😂

any batch scripting ideas that won't break the database are welcomed😃

Yes, what you suggest would great.

Always a pleasure to use Locus..!

Hi,
Yes the in camera time zone does change the time the point displays. However I do find this behaviour odd, as the point relates to the time when the track was recorded..

Thanks for explaining.

Col

Hi,
here are the screenshots, you can see that the auto created track time shows the time on the phone I started the track record, and the info on point one is clearly one hour earlier..







The just over 1 minute discrepancy is the time it took to get a fix for first point..