This repo contains remote sensing software developed in SMARTLAGOON project.
This repo contains scripts that allow to download and export to readable CSV format satellite data from NASA y ESA in batches by date and geopoints.
Due to the script is parametrise to request the APIs, its can be use to download netCDF4 files from EarthData (NASA) and Sentinel-3 CopernicusHub (ESA). However it has been just tested in follows enviroments:
MODIS Aqua, SNPP VIRRS, NOOA VIIRS for OC instrument and Level 2.
Sentinel-3 A y B for OLCI instrument and Level 2.
Scripts have been developed and test using Python3.6 and above on Ubuntu 16-18.
- Use requirements.txt to install library dependencies of scripts.
- You might to install some other external python dependencies to work with netCDF4 files.
- Script uses an elasticSearch instance to control downloaded files. It checks if requested point-date-satelliteProduct exists in a previous file downloaded.
- It is possible running the satellite_download.py with database field empty, but it is necessary to generate the csv and other processes.
Public APIs used by the script provide basic authentication process to request, so you must to provide a username and password by using config.ini.
For Sentinel product you should have an account in creodias.es:
https://portal.creodias.eu/register.php
The user and password will be add to config.ini of this project.
For NASA product MODIS, NOAA, SNPP: you should have an account in
https://cmr.earthdata.nasa.gov/
The user and password will be added to config.ini of the script. Also, you must authorize your user to access "OB.DAAC Data Access" in End-User License Agreement(EULA) in your user profile on earthdata. See:
https://urs.earthdata.nasa.gov , menu Applications->"authorized apps".
if you are able to work using elasticsearch, next describes the mapping to create the index:
{
"nc_files":{
"mappings":{
"properties":{
"agency":{
"type":"keyword"
},
"boundingBox":{
"type":"geo_shape"
},
"cloud":{
"type":"float"
},
"coords":{
"type":"float"
},
"date":{
"type":"date",
"format":"yyyy-MM-dd"
},
"file":{
"type":"keyword"
},
"instrument":{
"type":"keyword"
},
"lat":{
"type":"float"
},
"lon":{
"type":"float"
},
"path":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"polygonBox":{
"type":"geo_shape"
},
"satellite":{
"type":"keyword"
},
"time":{
"type":"date",
"format":"time"
},
"timeless":{
"type":"keyword"
},
"url":{
"type":"text"
},
"version":{
"type":"integer"
}
}
}
}
}
This file contains the basic parameters to request data to APIs. The structure is:
[ESA]
url_search:https://finder.creodias.eu/resto/api/collections/Sentinel3/search.json
url_auth:https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token
date_format: %%Y-%%m-%%dT%%H:%%M:%%S.%%fZ
user_name:
password:
[NASA]
url_search:https://cmr.earthdata.nasa.gov/search/granules.umm_json
date_format: %%Y-%%m-%%dT%%H:%%M:%%S.000Z
user_name:
password:
The user just need to fill user_name and passwords in the platforms to use.
Other platforms could be added here.
The scripts are based on json configuration called project. A project is a json file wherein is defined the parameters to requests the api and more such as this file contains the path wherein NC files will be saved and csv files will be generated. The name of this file is up to user, but the script call have to pass the project.json path as param.
This file contains a list of point to search. The CSV name is free since it have to be indicated in the project file to load. Required "name", "lat", "lon". Free to add more fields for personal proposes.
The scripts use loggin lib to write the results of the process. Also they print important messages (exceptions) in console. Log files are created in log/ path using the format: name of satellite (from project.json) + script_name.
import esa
file = 'file_nc/zip'
e = esa.ESA()
\#set the path of nc files
e.work_path = "path_of_file"
info = e.extract_data(file, lat, lon)