Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

questions related to offline geotiff image and export option #200

Closed
jaigsingla opened this issue Jan 6, 2020 · 34 comments
Closed

questions related to offline geotiff image and export option #200

jaigsingla opened this issue Jan 6, 2020 · 34 comments

Comments

@jaigsingla
Copy link

jaigsingla commented Jan 6, 2020

Hi,
It looks very nice development. I have few queries:

1.Can we run this module on Geotiff image already available in my machine if yes, how? how to convert it to slippy map format ? do one requires full scene or multiple pieces of one scene?
2. Pls give guidelines for offline (without internet ) processing.

  1. If I have segmented version (classified building image in png format) of the images available and just want to use your post processing tool for saving in to geojson or shape file, how to do that ?

thanks

@daniel-j-h
Copy link
Collaborator

  1. Robosat only works with slippy map tiles; you can tile e.g. into 256x256 or 512x512 pixel tiles via

Here's a quick and unoptimized tiler script; if your use case gets more involved you should definitely look into better ways of doing this:

https://gist.github.com/daniel-j-h/69a967d63d833f74a123e2540bd146b9

  1. I don't understand what you mean by offline processing. All tools except for downloading satellite tiles from a remote server work "offline".

  2. Only use the post-processing tools then.

@ocourtin
Copy link
Contributor

ocourtin commented Jan 6, 2020

@jaigsingla, to complete @daniel-j-h answer, on imagery tiling:
You could also give a look on rsp tile, from https://github.com/datapink/robosat.pink
Usually faster than the others, able to deal with coverage imagery, and to skip nodata tiles...

HTH,

@jaigsingla
Copy link
Author

  1. Robosat only works with slippy map tiles; you can tile e.g. into 256x256 or 512x512 pixel tiles via

Here's a quick and unoptimized tiler script; if your use case gets more involved you should definitely look into better ways of doing this:

https://gist.github.com/daniel-j-h/69a967d63d833f74a123e2540bd146b9

  1. I don't understand what you mean by offline processing. All tools except for downloading satellite tiles from a remote server work "offline".
  2. Only use the post-processing tools then.

Hello daniel,
All tools are offline , I agree. But how to install and use this library on offline machine is a difficult task.

Request you provide all dependencies as a zip or virtual environment image >>

thanks

@daniel-j-h
Copy link
Collaborator

Look into

docker image save
docker image load

to create a self-contained docker image tarball, get it onto your machine, and load it again.

@jaigsingla
Copy link
Author

jaigsingla commented Jan 9, 2020

Hi daniel,
I could build everything offline without docker. Now i want to use it along with bavaria dataset :
Following is available in datasets:
Model as check point
geojson file of the heat map area
two comparison images

After going through your post, we need followings:
1.Any geo-referenced image
2. From building shape file, OSM extract in geoson format of that area.
3. these two files should be converted to slippy format using ./rs cover
4. rs rasterize on osm slippy format files
5. rs train like that

I want to simulate entire scenario using bavaria data-sets first but i am unable to fetch the same.
pls guide.

@daniel-j-h
Copy link
Collaborator

Please read the documentation for our tools and read through the diary posts here

https://github.com/mapbox/robosat#overview

they explain in detail how to run the pipeline, where to get the training data from, and so on.

@jaigsingla
Copy link
Author

ok thanks.
I am directly starting with my own data. i executed ./rs cover --zoom 20 .geojson building.tiles
but it gives error of xy() takes from 2 to 3 positional arguments but 5 were given.

@daniel-j-h
Copy link
Collaborator

Do you have a stack trace? Which version are you running, latest? How does your geojson file look like?

@jaigsingla
Copy link
Author

Stack trace. I used latest version only. geojson file is clipped over Ahmedabad, India .

0%| | 0/482 [00:00<?, ?feature/s]
Traceback (most recent call last):
File "/home/deep/anaconda3/envs/fastai/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/deep/anaconda3/envs/fastai/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/deep/building_extraction/robosat-master/robosat/tools/main.py", line 58, in
args.func(args)
File "/home/deep/building_extraction/robosat-master/robosat/tools/cover.py", line 30, in main
tiles.extend(map(tuple, burntiles.burn([feature], args.zoom).tolist()))
File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 62, in burn
all_touched=True)
File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/rasterio/env.py", line 397, in wrapper
return f(*args, **kwds)
File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/rasterio/features.py", line 280, in rasterize
for index, item in enumerate(shapes):
File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 59, in
((project_geom(geom['geometry']), 255) for geom in polys),
File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 12, in project_geom
for part in geom['coordinates']
File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 12, in
for part in geom['coordinates']
File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 11, in
[mercantile.xy(*coords) for coords in part]
TypeError: xy() takes from 2 to 3 positional arguments but 5 were given

@daniel-j-h
Copy link
Collaborator

How did you generate the GeoJSON? What type of features do you have in there? From what I can recall we were supporting "Polygon" features only due to the supermercado dependency limitations.

For our extract tool see

https://github.com/mapbox/robosat/blob/3ae9c7e1d58fc446b7c8df3835c628c40ca85ba2/robosat/osm/building.py

And supermercado upstream tickets

@jaigsingla
Copy link
Author

jaigsingla commented Jan 10, 2020

There is osm building shape file available over ahmedabad. Cropped subset over Ahmedabad was taken as an input and using QGIS I just saved this in geojson format and use for further processing.
recent_shapefile.zip

@daniel-j-h
Copy link
Collaborator

Yeap you have features in there which are not Polygon. The upstream supermercado library does not support that, see tickets linked above. I recommend you generate a GeoJSON file with Polygon features only.

@ocourtin
Copy link
Contributor

@jaigsingla You could also give a try to rsp rasterize resilient with all GeoJSON geometries types.

Cf:

@jaigsingla
Copy link
Author

@ocourtin can you share practical example of this ? and i guess we need to perform tilling on this also??

@jaigsingla
Copy link
Author

Yeap you have features in there which are not Polygon. The upstream supermercado library does not support that, see tickets linked above. I recommend you generate a GeoJSON file with Polygon features only.

@daniel-j-h but these are polygons only ??

@ocourtin
Copy link
Contributor

@ocourtin can you share practical example of this ?

https://github.com/datapink/robosat.pink/blob/v0.5.5/docs/101.md#retrieve-and-tile-labels

and i guess we need to perform tilling on this also??

If your labels are vector ones (GeoJSON or PostGIS), rsp rasterize performs the tiling for you.
If your labels are PNG ones (it could happens), rsp tile --label is the way to deal with...

HTH,

@jaigsingla
Copy link
Author

Hi daniel,
I could build everything offline without docker. Now i want to use it along with bavaria dataset :
Following is available in datasets:
Model as check point
geojson file of the heat map area
two comparison images

After going through your post, we need followings:
1.Any geo-referenced image
2. From building shape file, OSM extract in geoson format of that area.
3. these two files should be converted to slippy format using ./rs cover
4. rs rasterize on osm slippy format files
5. rs train like that

I want to simulate entire scenario using bavaria data-sets first but i am unable to fetch the same.
pls guide.

Hi daniel,
I spoiled earlier virtual env and now facing problem of could not find libspatialindex_c library file. I tried to install it offline using conda install --offline option . but , still error persists.

@daniel-j-h
Copy link
Collaborator

You'll need libspatialindex - but you really should just use the pre-built docker images so that we can guarantee you have all packages. If you want to install on your own, check out the Dockerfile.

@jaigsingla
Copy link
Author

jaigsingla commented Jan 23, 2020

solved again.

I could convert images and osm data into slippy format using ocourtin routeins. DIvided data into training, validation and images and labels also. Can i go ahead to run train routine of yours on that data now??

@daniel-j-h
Copy link
Collaborator

Sure why not.

@jaigsingla
Copy link
Author

ok.

what I did is as following:

  1. Created ds/training/images and ds/training/labels . similar to these ds/validation/images and ds/training/labels are created .
  2. calculated weights using ./rs weights dataset_building.toml
  3. run ./rs train using all parameters.
  4. after 50 epochs validation IOU shows 0.45 ??

@daniel-j-h
Copy link
Collaborator

Okay - so what? How big is your dataset? How did you create it? Did you manually check it if it looks good? What do you want to get out of it? You have to investigate a bit on your own here..

@jaigsingla
Copy link
Author

My dataset is not that big. it is small image file and osm data file as shared earlier too. I created data in slippy format using commands suggested by ocourtin. Manually , I check slippy files. it is properly placed in folders and looks ok. I want to generated vector shapes from image files. I need directions to investigate

@daniel-j-h
Copy link
Collaborator

My dataset is not that big.

Then what do you expect? We need a large dataset to learn from :)

There is only so much magic we can do here.

@jaigsingla
Copy link
Author

history-00050-of-00050
history-00020-of-00020

shall i proceed with predictions or what you say?

@daniel-j-h
Copy link
Collaborator

How big is your dataset? How many samples do you have in train and val?

@jaigsingla
Copy link
Author

Input image size is 777x674 with 4 bands.
i have converted to slippy format using zoom level 19.
i have used 90% for training and 10% validation.

@daniel-j-h
Copy link
Collaborator

If you have a single image with 777x674 pixel as your dataset, I don't think you will have much luck training machine learning models on it. It's way too small to learn anything interesting.

@jaigsingla
Copy link
Author

So should I try on a bigger data set. Can you share sample from your side so that I am sure to get validated output

@daniel-j-h
Copy link
Collaborator

Check out the diary posts in https://github.com/mapbox/robosat#overview

The Tanzania example is probably as low as you should go with a few thousand of 512x512 tile images. If you follow the Bavaria diary post you can get a much bigger dataset.

@hamzahkhan
Copy link

Hi,
It looks very nice development. I have few queries:

1.Can we run this module on Geotiff image already available in my machine if yes, how? how to convert it to slippy map format ? do one requires full scene or multiple pieces of one scene?
2. Pls give guidelines for offline (without internet ) processing.

  1. If I have segmented version (classified building image in png format) of the images available and just want to use your post processing tool for saving in to geojson or shape file, how to do that ?

thanks
@jaigsingla Any luck resolving your issue on converting segmented png to geojson? Stuck on same.

@jaigsingla
Copy link
Author

@hamzahkhan not tried yet.

@jaigsingla
Copy link
Author

@daniel-j-h pls make this Tanzania dataset available as a sample here. it is very difficult to fetch exact dataset without internet.

@daniel-j-h
Copy link
Collaborator

I no longer have access to the dataset and I'd have to fetch it from scratch myself, too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants