Skip to content
Bakinam Tarik Essawy edited this page Oct 15, 2015 · 35 revisions

Software Details for the VIC/iRODS Data Pre-Processing Pipeline

Bakinam T. Essawy*
Jonathon L. Goodall*
*Department of Civil and Environmental Engineering, University of Virginia, Charlottesville, Virginia

vicDataPrepro.r

processncdcPrcp.r

Description: Process precipitation data from study area.

Shell scripts used: run_convert_prcp.scr Argument file: None

Source code: convertPrcp.cpp Source code language: C++ Compiler used: g++ Operating system: Ubuntu

Input files: stationinfo.txt, one or more weather station files

Output files: prcp.daily, prcp.inf

readPreprocDly.r

Description: Formatting the combined data

Shell scripts used: inputPrcp.scr Argument files: prcp.daily, prcp.inf, and basin_prcp.fmt

Source code: read_prec_dly.f Source code language: FORTRAN Compiler used: f77 –o Additional information from the FORTRAN code: This program reads the output from the script previous script and formats the daily precipitation so the regrid program can read them. It uses the daily data and the station information files

Output file: basin_prcp.fmt

precTobAdj.r

Description: Time of adjustment of the combined data

Shell scripts used: prcp_tobAdj.scr Argument files: prec_tob_adj.input

Source code: prec_tob_adj.f Source code language: FORTRAN Compiler used: f77 –o Additional information from the FORTRAN code: This program reads the output from the read_prec_dly.f program and adjusts the daily precipitation to account for the time of observation for each gauge. This prepares a new file in the same format as the .fmt file for input to the regridding program.

Input files: basin_prcp.fmt, prcp.inf, and coop_tob.his

Output file: basin_prcp_adj.fmt

createMask.r

Description: Create mask (ascii file) for the study area

Shell scripts used: run_convert_tif_ascii.scr Argument file: None

Source code: convert_tif_ascii.py, tiff2ascii.py Source code language: Python Libraries used: sys, Image, logging, gdal

Input files: DEM.tif,

Output files: mask_125.asc

regridPrcp.r

Description: Regridding the precipitation data for the basin This script will store the content of the regridPrcp directory to the tmp directory under the name of prcp.log

Shell scripts used: regrd_prcp.scr Argument file: None

Input files: prcp.inf, basin_prcp_adj.fmt, mask_125.asc

Output files: adjprcp_grid.grd

mkMonthlyPrcp.r

Description: Estimate monthly precipitation from NCDC data

Shell scripts used: mk_monthly_ir.scr Argument files: None

Source code: mk_monthly.c Source code language: C Compiler used: gcc –o
Additional information from the C code: This program reads the output file from the regridding program (adjprcp_grid.grd). It then generates monthly values for jan,feb,mar,etc that will be used together with PRISM data to scale the timeseries in each gridcell

Input files: adjprcp_grid.grd

Output file: mk_monthly/*.monthly

Shell scripts used: run.mk_monthly.scr Argument files: None

Additional information from the additional script run.mk_monthly.scr: This script is used to run 'mk_monthly'. The output directory for 'jan.monthly, feb.monthly, etc, must be made before program execution. binflag=1 means data is in 2-byte signed binary formatmult is the data multiplier (saved as data*mult) -- only used for binary

Input files: 'mk_monthly/*

Output file: mk_mnth/monthly.*

mkPrism.r

Description: estimate monthly PRISM data

Shell scripts used: get_prism_ir.scr Argument files: None

Source code: get_prism.c Source code language: C Compiler used: gcc –o
Additional information from the script: This program reads the prism data (C.Daly et al 1993) and converts them into monthly gridded data at a chosen resolution. The program needs a maskfile as input together with the prism rawdata. The rawdata used in this program is located at /can-usa-prism/can_us_prism.jan . These files contains prism data for USA. If other prism rawdatafiles are used, then lines 28 to 38 should be changed in this program. If there are gridcells forexample in Canada that the prism data is not covering. The no_data value will be written in the outputfiles corresponding gridcell . The rescaling program will check for this void value at a later stage and the precip value will not be scaled for this gridcell.

Input files: mask_125.asc, prism-rawdata/can_us_prism.*

Output file: Prism/prisim.*

mkRescale.r

Description: Rescaling the precipitation data

Shell scripts used: rescale_ir.scr Argument file: None

Source code: convertPrcp.cpp Source code language: C Compiler used: gcc -o Operating system: Ubuntu

Additional information from the script: This program rescales the gridded data from the program regrid with the ratio of precipvalue in cell * (prism.jan/monthly.jan). This is done to achieve monthly precipitation in each grid cell equal to prism monthly meansInputfiles are the gridded data file, monthly values (mk_monthly) and prism monthly values (get_prism). Output is a new rescaled gridded file that can be turned into the final VIC input files.

Input files: mask_125.asc, mk_mnth/monthly., Prism/prism., adjprcp_grid.grd

Output files: adjprcp_rsm.rsc

processncdcTmax.r

Description: Process maximum temperature data from study area

Shell scripts used: run_convert_tmax.scr Argument file: None

Source code: convertTmax.cpp Source code language: C++ Compiler used: gcc -o Operating system: Ubuntu

Input files: stationinfo.txt, climate/*

Output files: Tmax/tmax.inf, Tmax/tmax.daily

readPreprocTmaxDly.r

Description: Formatting the combined data

Shell scripts used: inputTmax.scr Argument file: tmax.daily, tmax.inf and basin_tmax.fmt

Source code: read_temp_dly.f Source code language: FORTRAN Compiler used: f77 -o Operating system: Ubuntu

Additional information from the script: This program reads the output from the script inputPrcp.scr and formats the daily temperature so the regrid program can read them. Only the .fmt (cmb_prcp.fmt) file is output

Output files: basin_tmax.fmt

tmaxTobAdj.r

Description: Time of adjustment of the combined data

Shell scripts used: tmax_tobAdj.scr Argument file: tmax_tob_adj.input

Source code: tmax_tob_adj.f Source code language: FORTRAN Compiler used: f77 -o Operating system: Ubuntu

Additional information from the script: This program reads the output from the read_prec_dly.f program and adjusts the daily precipitation to account for the time of observation for each gauge. This prepares a new file in the same format as the .fmt file for input to the regridding program.

Input files: Tmax/basin_tmax.fmt, Tmax/tmax.inf and Tmax/tob/coop_tob.his Output files: basin_tmax_adj.fmt

regridTmax.r

Description: Regridding the maximum temperature data for the basin

Shell scripts used: regridTmax.scr Argument file: None

The shell script regrid/Tmax/*o , then the regrdTmax will be stored in the tmp directory under tmax.log. Input files: Tmax/tmax.inf, Basin/mask_125.asc and Tmax/basin_tmax_adj.fmt Output Files: Tmax/adjtmax_grid.grd

processncdcTmin.r

Description: Process minimum temperature data from study area

Shell scripts used: run_convert_tmin.scr Argument file:None

Source code: convertTmin.cpp Source code language: C++ Compiler used: g++ -o Operating system: Ubuntu

Input files: stationinfo.txt, tminncdc.inf and climate stations Output files: tmin.daily, tmin.inf

readPreprocTminDly.r

Description: Formatting the combined data

Shell scripts used: inputTmin.scr Argument file: Tmin/tmin.daily, Tmin/tmin.inf, Tmin/basin_tmin.fmt

Source code: read_temp_dly.f Source code language: FORTRAN Compiler used: f77 -o Operating system: Ubuntu

Additional information from the script: This program reads the output from the script inputPrcp_tmax/tmin.scr and formats the daily temperature so the regrid program can read them Only the *.fmt (cmb_prcp.fmt) file is output

Output files: basin_tmin.fmt

tminTobAdj.r

Description: Time of adjustment of the combined data

Shell scripts used: tmin_tobAdj.scr Argument file: tmin_tob_adj.input

Source code: tmin_tob_adj.f Source code language: FORTRAN Compiler used: f77 -o Operating system: Ubuntu

Additional information from the script: This program reads the output from the read_prec_dly.f program and adjusts the daily precipitation to account for the time of observation for each gauge. This prepares a new file in the same format as the .fmt file for input to the regridding program.

Output files: Tmin/basin_tmin.fmt

regridTmin.r

Description: Time of adjustment of the combined data

Shell scripts used: regrd_tmin.scr Argument file: None

The shell script regrid/Tmin/*o , then the regrdTmin will be stored in the tmp directory under tmin.log

Input files: Tmin/tmin.inf, Basin/mask_125.asc and Tmin/basin_tmin_adj.fmt Output Files: Tmin/adjtmin_grid.grd

vicInputMeteoData.r

Description: Run meteorological data (prcp_grid.rsc, tmax_grid.grd and tmin_grid.grd) to create gridded data for the study area.

Shell scripts used: run_vicinput_ir.scr Argument file: None

Source code: vicinput.c Source code language: C Compiler used: gcc -o Operating system: Ubuntu

Additional information from the script: This program reads the output from A.H.s regridding program and generates the vicinput files. The input needed is prcp, tmin, tmax,maskfile, outp_directory, binary_flag

Additional script is used to set the input files script: run_vicinput.scr

Input file: adjprcp_rsm.rsc, adjtmax_grid.grd, adjtmin_grid.grd, and mask_125.asc Output files: meteoCombined/data_*_

getNCARwind.r

Description: Collect wind speed data from NCEP/NCAR.

Shell scripts used: getwind.scr Argument file: None

This script sets the limits for the lat and long, output directory, and uses the ncks tool for conversion of the netCDF data to text

Additional script is used to set the input files script: run_vicinput.scr

Input file: gauss_t62_lat.list, gauss_t62_lon.list Output files: raw_wind/uwnd*.asc, raw_wind/vwnd*.asc

regridNCARwind.r

Description: Regridding the NCEP/NCAR wind speed data.

Shell scripts used: run_regrid_wind_ir.scr Argument file: None

Source code: regrid_wind.c Source code language: C Compiler used: gcc -o Operating system: Ubuntu

This program takes the yearly u-wind and v-wind NCEP/NCAR reanalysis data and regrids it from the t62-gaussian grid to a grid corresponding to the input mask file. The required inputs are those listed in the "run_regrid_wind.scr" file, and the files "gauss_t62_lat.list" and "gauss_t62_lon.list" must reside in the same directory as this file. These two files have the global lat/long coordinates of the reanalysis data. The output is one wind timeseries file for each grid cell active in the mask. This program also dumps two other files as output:

  1. wind_latlong.txt which is a list of file names for output
  2. cell_avg.out which is an average for each of 365 days for each cell These two files are used in the fill_data.c program, used to fill the wind data files with average daily wind speeds for each cell, for the years of VIC data preceding the reanalysis data.

Another script is used called run_regrid_wind.scr This script sets the limits example: start year, start month, end year, end month, min latitude, max latitude, min longitude Input file:raw_wind directory, gauss_t62_lat.list, gauss_t62_lon.list, Basin/mask_125.asc Output file: wind_latlong.txt, temp_Wind/

combineWind.r

Description: Combine the wind data with meteorological data

Shell scripts used: run_combine_wind_ir.scr Argument file: None

Source code: combine_wind.c Source code language: C Compiler used: gcc -o Operating system: Ubuntu Additional script: run_combine_wind.scr for the input data Input directory: meteoCombined, temp_wind, latlong99.txt Output Directory: forcingData

latlonSoil.r

Description: Adding preproc and append data into a combined file.

Shell scripts used: ldas_latlon.scr Argument file: mask_125.asc, latlon.txt

Source code: mask2latlon.c Source code language: C Compiler used: gcc -o Operating system: Ubuntu Output file: latlon.txt

ldasSoil.r

Description: Prepare soil data

Shell scripts used: ldas_soil.scr Argument file: soil_ldas.txt, latlon.txt, ldassoildata.txt

Source code: create_LDAS_soil_nearest.c Source code language: C Compiler used: gcc -o Operating system: Ubuntu Output file: latlon.txt

Additional Information from the script: Create a VIC soil file with the LDAS soil hydraulic parameters */ Takes two input files:

  1. LDAS domain soil file
  2. file containing latitude and longitude for each grid cell NOTE: soil files will be in the 53 column format of VIC 4.0 with 3 soil layers

Output files: ldassoildata.txt

ldasVeg.r

Description: Prepare vegetation data

Shell scripts used: ldas_veg.scr Argument file: ldas_lai.expanded.vegparams, ldassoildata.txt, ldasvegdata.txt

Source code: create_LDAS_veg_param.c Source code language: C Compiler used: gcc -o Operating system: Ubuntu Additional Information from the script: Extracts a vegetation parameter file to match the input soil parameter file. Set up for 1/8 degree LDAS VIC implementation set for 3 root zones and monthly LAI in vegetation parameter file.

Output files: ldasvegdata.txt