----------------------------------------------------------------------------- SYSTEM for INTEGRATING MULTI-PLATFORM data to BUILD the ATMPSPHERIC column ----------------------------------------------------------------------------- COLUMN BUILDING FRAMEWORK: WORKING LIST OF FEATURES INCLUDED/TO ADD: ----------------------------------------------------------------------------- ---------------------------------- Added in for v1.6.1 (23 April 2018): ---------------------------------- get_dpr_for_column.pro get_lev2bcmb_for_column.pro - These modules were updated to correct FOV assignments for GPM OPs when the Ku- or Ka-band swath edge is very near the column grid. Previously, when the edge of a swath was near (~10*column horizontal extent in km) the grid center, values in the nearest FOV were assigned to the column grid point. This has been updated so that for these types of cases, only values from the FOVs over the column grid are placed in the column. Specific new conditional tests for this relate to the variables MSxOK, MSyOK, HSxOK, HSyOK, col_in_MSswath, and col_in_HSswath within the modules. ----------------------------------------------------------------------------- --v1.6 ------ CHANGES MADE AFTER package_v1.5 : included in package_v1.6 ---- -- 19 MARCH 2018 ------------------------------------------------------------ ----------------------------------------------------------------------------- build_column.pro - added new option for setting time interval to include for sfc/ground-based platforms data: gauges, pluvios, 2DVDs, APUs/parsivels, MRRs "halftime_interval" variable: Set this as the number of minutes BOTH BEFORE AND AFTER the main platform's timestamp to be included in the column file for the gauges, disdrometers, and MRR data. get_lev2bcmb_for_column.pro - NEW MODULE adding support for GPM 2BCMB data products, available from PPS - these data are the output from the GPM combined (DPR + GMI) algorithm set_plat_values.pro - added support for GPM 2BCMB module - updated how timestamps & offsets are set for pluvios, 2DVD, APUs - these are now set in the respective modules to accomadate time interval update get_gauges_for_column.pro get_pluvio_for_column.pro get_apu_for_column.pro get_2dvd_for_column.pro get_mrr_for_column.pro - Each of these modules has been updated to include a time interval of data for the column file. The times included in the column file are now the minute corresponding to the main plat timestamp AND the halftime_interval number of minutes BOTH BEFORE & AFTER the main plat timestamp. - NOTE 1: The TIMESTAMP attribute for each of these platforms corresponds to the center of the included time interval. This timestamp will be nearly or exactly same as main_platform timestamp. - NOTE 2: The new attribute TIME_INTERVAL_WIDTH is the number of minutes before & after the timestamp - same as the halftime_interval variable set in build_column.pro - NOTE 2: There is now an additional (4th) dimension to the data fields from these platforms. These data in the SIMBA column files now have dimensions set up as [TIME x X_dir x Y_dir x Z-dir]. The new variable "t" has been added to describe the TIME dimension. Values of t are integers telling the number of minutes before (negative t) and after (positive t) the interval center time (interval center time = TIMESTAMP attribute for the platform). write_column_nc_file.pro - added support for the 2BCMB module - updated so version string is set in build_column.pro instead - updated so time intervals for gauge, pluvio, 2DVD, and MRR data included in output .nc file check_radar_range_to_column.pro - new function added to check range of column grid points from an input ground-based scanning radar - mainly added for D3R & DOW6 in OLYMPEX. ----------------------------------------------------------------------------- --v1.5 ------ CHANGES MADE AFTER package_v1.4 : included in package_v1.5 ---- -- 5 SEPTEMBER 2017 --------------------------------------------------------- ----------------------------------------------------------------------------- get_dpr_for_column.pro - fixed assignment for some multi-dim fields to ensure correct sub-field is used (eg: for piaNP from cloud water, ice, water vapor, and oxygen) - fixed handling of heights for 3D DPR fields: this was modified for v1.4 when the module was updated to run faster, but a bug when subesetting was introduced. This had been resolved. get_mrr_for_column.pro - updated for cases of MRR(s) avaialble but not in column box - updated attributes definitions for mrr_plat_info structure to use correct variables/values - updated module to not read in the .ave file if a unit is not located in the column box - this test was previously in the code after the file was read in (was not as efficient for time) - A MAJOR UPDATE: instead of individual bins and number concentrations, the mrr DSD profiles have been updated to profiles of Dm, log(Nw), and a computed reflectivity using functions written by Patrick Gatlin. Note that for v1.5, the /dependencies/ directory now includes: disdro_mrr_get_dsd_params.pro disdro_fit_dsd_norm_gamma.pro get_mrms_for_column.pro - updated way latitudes are assigned, per clarification from Pierre Kirstetter: first line in the lev2 files = MAX latitude write_column_nc_file.pro - added global attribute for the SIMBA version used to create column .nc file - updated APU sections for case of an APU is avaiable but has no data for the time of intereset (eg: APU 30 for OLYMPEX 3 Dec event) - updated how 2DVD var names are defined so case is consistently "twodvd" get_2dvd_for_column.pro - updated initialization of column data arrays to NaNs before populating get_olympex_sondes_for_column.pro - NEW MODULE for soundings data collected in OLYMPEX campaign. - Note: If non-OLYMPEX sounding data is used, the timestamps portion of this module may fail -- due to time convention used in OLYMPEX data files (seconds after 000000 UTC on 1 Jan 2015). For a more generic module, timestamp determiniations would need to be adjusted, but rest of format should be OK -- Assuming the data are kept in .nc files and follow same/similar convention as laid out by Ciesielski et al. 2012 (used for OLYMPEX sonde processing) ----------------------------------------------------------------------------- --v1.4 ------ CHANGES MADE AFTER package_v1.3 : included in package_v1.4 ---- -- 6 JUNE 2017 -------------------------------------------------------------- ----------------------------------------------------------------------------- get_apu_for_column.pro - fixed typo in conditional statement to set col_data_apu_dsd_ter_holder set_plat_values.pro & *_plat_info structures - updated to include more platform-specific attributes, now have: NPOL, 88D, D3R, DOW, MRR: lat, lon, elev, op mode, wavelenght, freq, beam width, gate size, timestamp, offset_vs_main 2DVD, APUs: lat, lon, op mode/disdo type, timestamp, offset_vs_main Gauges, Pluvios: lat, lon, op mode/gauge type, timestamp, offset_vs_main GMI, DPR: file type, algo name, obit #, data vers, operating frequencies, timestamp, offset_vs_main, timestamp_cntr get_lev1cgmi_for_column.pro - NEW MODULE to support L1C/L1C-R GMI PPS product - for subsetting the full orbit to area near column grid, the module MAKES THE ASSUMPTION THAT THE MAIN PLAT IS LOCATED WITHIN 10*(column_grid_horiz_limit) [km] OF THE COLUMN GRID CENTER LOCATION - eg: For a column grid w/ 5 km total extent, it is assumed the main_plat is within 50 km of the column grid center location. - NOTE ON LOCATIONS: The L1C GMI module locates the nearest pixel to each SIMBA column grid point to assign L1C values to the column data file. While this accounts from FOV size differences, an adjustment to pixel locations due to parallax effects has not yet been applied. GPROF GMI MODULE NAME CHANGE & IMPROVEMENTS: get_gmi_for_column.pro --> get_gprofgmi_for_column.pro - renamed to avoid confusion with new L1C gmi module - amended to include conditional for PPS data version as V04A & ITE114 do not contain the exact same fileds. Note that new Morris GPM readers are file for either version - but SIMBA had been looking for specific fields which may or may not be present, depending on the version. This update allows either V04A or ITE114/V05 input to be read correctly & the fields present in either version to be written to the column .nc file. - Note an update that speeds this module up, esp for larger column grids: As in the new L1C module, an ASSUMPTION is now made that the main platform is located within 10*(column_grid_horiz_limit) [km] of the column grid center location get_dpr_for_column.pro - Note an update that speeds this module up, esp for larger column grids: As in the new L1C module, an ASSUMPTION is now made that the main platform is located within 10*(column_grid_horiz_limit) [km] of the column grid center location get_mrms_for_column.pro - NEW MODULE for Level 2 MRMS preciprate/type product. These MRMS data are MRMS files for times when a GPM constellation satellite passes over the MRMS domain. Files are available from: hydro.ou.edu/projects/MRMS/level2/ - Details on the module and the new data structures are given in the _structs.txt & _pros.txt READMEs. get_gauges_for_column.pro - Removed a line (L. 247) that set a fake value to points w/ multiple gauges (eg: iowa-type/paired gauges) - this line was used for testing but wasn't removed when module was completed -- has been fixed. timestamp offsets: in some modules, ulong data types had been used instead of long data types, this has been corrected to 'long' in each case. Support added for all NEXRAD/88D radars in the GPM GV VN: - platform_location.pro - set_main_plat_values.pro - set_plat_values.pro - setup_main_plat.pro - new module that calls set_main_plat_values.pro This function performs "Step 2" in the main SIMBA procedure. - Because the grid_88d_for_column.pro module was set up for any 88d, it can handle any GPMGVDPQC'd 88D file. However, location and site ID information had to be ammened to .pros listed above. - Note that not all sites have been tested. Please send an email to stephanie.m.wingo@nasa.gov if problems occur when trying to use SIMBA with GPMGVDPQC'd data from any of the following radars: KABR KCAE KEAX KICT KLIX KMRX KTLH KAKQ KCCX KEVX KILN KLOT KMVX KTLX KAMX KCLX KFSD KILX KLSX KNQA KTWX KAPX KCRP KFTG KINX KLTX KOKX KTYX KARX KDDC KFWS KIWX KLZK KPAH PAEC KBMX KDGX KGRK KJAX KMHX KRAX PAIH KBOX KDLH KGRR KJGX KMKX KSGF PGUA KBRO KDMX KGSP KJKL KMLB KSHV PHKI KBUF KDOX KHGX KLCH KMOB KSRX PHMO KBYX KDVN KHTX KLGX KMQT KTBW TJUA NPOL, 88D, D3R modules & make_params_file.pro: - Field names & units have been hard set so 'unknown' should not appear in the column .nc attributes for ground-based scanning radar field names or units - Radx .params have been adjusted to properly handle discrete hydrometeor ID fields for radars that have this field available write_column_nc_file.pro - fixed typo in check_ID conditional test - updated to support new attributes for various platforms - updated for V04A or ITE114 data version for GMI GPROF - added sections for GMI L1C data & MRMS values list_column_inventory.pro - updated for compatibility with new/extended attributes & supported platforms get_2dvd_for_column.pro - updated so list of possible 2DVD units is derived from filenames in the two_dvd_dir, not a hard-coded list within the module - previously, user would need to update this list, but now that is not needed. Soundings module for OLYMPEX campaign -- this will be included in v1.5. ----------------------------------------------------------------------------- --v1.3 ------ CHANGES MADE AFTER package_v1.2 : included in package_v1.3 ---- --30 Mar 2017 --------------------------------------------------------------- ----------------------------------------------------------------------------- d3r_combine_gridded_rhis.pro - Added conditional tests through ka-band section for instances when all ka-band swp values are undefined. - Did not have this issue w/ ku-band swps, thus the ku-band section does NOT include these conditional tests. d3r_check_scantype.pro New function added to perform a test verifying that input D3R scans in files_D3R_Ku & files_D3R_Ka are all either RHI or PPI sweeps get_dpr_for_column.pro Updated how Nw fileds (HS & NS swaths) are handled to remove 10*log(Nw) multiplier factor - values written to column file for log(Nw) fields now are in conventional range (values ~1-5) get_gmi_for_column.pro Now uses more generic reader (for any GPROF PPS file), and reader version in /dependencies/ should support soon-to-be-released new format, see DMarks emails 21 Mar 2017: read_2agrpof_hdf5.pro get_2agprofgmi_datasets.pro get_mrr_for_column.pro Now allows multiple MRR units within the column box to be used - handles similar to muleiple 88Ds or gauges, etc. Updated filtering/interpolation process - mainly to ensure any negative Nxx (drop densities) are removed since these values have no physical meaning list_column_inventory.pro Now also outputs a simplified map .png file, but note: ground platforms located beyond the range of the map windong on the plot are not shown, however these platforms may still be available in column .nc file - check inventory .txt file and/or the .nc header Several updates for applying SIMBA to OLYMPEX data: - platform_location.pro: added several locations: 'NPOL-OLYMPEX', 'KLGX', 'AmandaPark', OLYMPEX APUs, OLYMPEX Pluvios, DOW6 in OLYMPEX - set_plat_values.pro: various updates for OLYMPEX, additional attributes added for satellite sensors (GMI, DPR), other minor improvements - set_main_plat_values.pro: now also allows DOW6 or KLGX to as main_plat options - get_apu_for_column.pro: updated for compatibility w/ OLYMPEX data - supports if one or more available APU units does not have the main_plat timestamp avail in the processed data files. - added support for *raindsd* and rainparameter_min. files - get_2dvd_for_column.pro: updated for compatibility w/ OLYMPEX data - supports both 50/100% files, and meas/terminal fall speed files. Also updated so that if a file is/isn't available, that's what determines if the data structure for fields in that file type get sent to column .nc, instead of just keeping NaNs for series(s) of fields that DNE for a particular 2DVD unit. - fixed bug where dsd fields from terminal fall speeds --two_dvd_info_and_data.col_data_2dvd_raindsd_ter fields were getting written same as from files of dsd from measured fall speeds (difference between Ali's _raindsd. & _raindsd_ter. files was not getting written to column .nc properly -- this is resolved. - get_gauges_for_column.pro: minor updates for OLYMPEX - returns -1 back if all gauges outside of column box or if no .gmin files contain the time of interest - updated ASCII read in template to read in negative longitudes for OLYMPEX (was starting 1 char too far) - get_pluvio_for_column.pro: NEW MODULE for OLYMPEX data - works similar to gagues & APU modules - grid_dow6_for_column.pro: NEW MODULE for OLYMPEX data - NOTE: need to use updated RSL routines: see emails w/ Bart from 3 & 7 March rsl_cfradial_to_radar.pro rsl_new_ray.pro rsl_radar_to_uf.pro - make_params_file.pro: updated for DOW6 - write_column_nc_file.pro: updated for compatibility w/ updated/new modules -for updated apus structures from get_apu_for_column.pro -for updated 2dvd structures from get_2dvd_for_column.pro -for updated gagues_info_and_data possibilitiy of -1 -for new pluvio_info_and_data from get_pluvio_for_column.pro -for new dow6_plat_info & col_data_dow6 from grid_do6_for_column.pro ----------------------------------------------------------------------------- --v1.2 ------ CHANGES MADE AFTER package_v1.1 : included in package_v1.2 ---- --30 Jan 2017 --------------------------------------------------------------- ----------------------------------------------------------------------------- - write_column_nc_file.pro : - corrected typo "NCDF_VARPUR" to "NCDF_VARPUT" in section for writing gauges data to the column .nc file - ammended to also write a header.txt file w/ the ncdump -h result for the column .nc file - updated for if two_dvd_info_and_data structure is not defined or gets returned as -1 - set_plat_values.pro: added case statement for plat_name eq 'MRR-02' - get_2dvd_for_column.pro: ammended this module for instances with *raindsd_ter* 2DVD file isn't available...will check also for *raindsd.* or the *raindsd_50* file to send to set_plat_values.pro - build_column.pro, grid_d3r_for_column.pro, make_params_file.pro Updated/created code to automatically generate .params files to use for running Radx. Setting hard coded paths to manually generated .params files is no longer needed. - build_column.pro, grid_*_for_column.pro Updated to set full_grid_dir & full_vol_dir at top of build_column.pro and pass into the gridding modules, rather than having these hard coded within each of the modules. NOTE: full_vol_dir only applies for D3R. - get_dpr_for_column.pro, write_column_nc_file.pro: Updated to support extra dimensional 2ADPR fields, including Dm, Nw, TPW consituents - dependencies dir: Now includes a few items previously overlooked but are called from the example plotting routine (col_box_test_plots.pro). "package_v1.2" Current working SIMBA framework: AS OF 30 JAN 2017 ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- --v1.1 ------ CHANGES MADE AFTER package_v1.0 : included in package_v1.1 ---- --30 Nov 2016 --------------------------------------------------------------- ----------------------------------------------------------------------------- - Updated grid_d3r_for_column.pro to delete temp/hold dirs while completeing "Step 1" to combine RHI swps to single volume - Added DPR (2ADPR) as get_dpr_for_column.pro NOTE: Need to verify way of getting hts...agrees w/ FrezLvl recorded in file...good sign... Am using: [binNovalues-1] for vert subs since the binNos are 1-based and IDl subs are 0-based... (see working notes 25-27 Oct 2016) - Added 2DVD support: get_2dvd_for_column.pro Assumes using files of processed 2DVD generated by Ali Tokay, similar to the APUs. NOTES: - 2DVD unit locations are assumed to be the default WFF PRF sites, will have to change if they are moved/for field campaign data sets. Like most of the platforms, the location info (lat/lon) is set in platform_location.pro - As for APUs: currently, only main_plat_timestamp is used, but next version would like to add a user-set time interval to be included as well. - Added GAUGES support: get_gauges_for_column.pro Assumes using .GMIN processed gauge data (1-min, spline technique) generated by Jerry Wang (see Wang et al. 2008 JTECH). NOTES: - Gauge unit locations are all in the .gmin files, so these are NOT INCLUDED in platform_location.pro - Since have to test .gmin files for both time & location AFTER reading in, this module DOES NOT call set_plat_values.pro, but instead sets up the _plat_info values directly in the module - As for APUs & 2DVDs: currently, only main_plat_timestamp is used, but in next version would like to a user-set interval for +/- main_plat_timestamp to include, too. - Updated list_column_inventory.pro to only work as having the column .nc file passed in, then the inventory is written to a .txt file instead of to terminal (as it had been). This way, users won't have to open the .pro at all in order to get the inventory, and it is saved alreeady to a file. - Framework Name: "SIMBA" System for Integrating Multi-platform data to Build the Atmospheric column "package_v1.1" Current working SIMBA framework: AS OF 30 NOV 2016 ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- -- v1.0 -- 12 Sep 2016 -- column building framework ------------------------- ----------------------------------------------------------------------------- v1.0: goal is to get a working framework that can ingest multiple platforms' data and write a new .nc file with only obs in the use-defined column. Keep simple so can get to a v1 data product quicker. Add in more attributes/platform/orig files information: --> will prolly need/want to change how plat_info structure is set: currently, it is set at bottom, outside of the plat_type sections, but i think will be better in long run to set it up WITHIN ea plat_type section so ea platform can have sep set of params/specs Scanning radars: scaning sequence, beamwidth, gate spacing, dist to first gate, freq/wavelength, MRR: orig gate spacing, avging interval, sampling rate, radar's ht ASL, max(gate_hts) Need to include a data inventory utility that can read in the column .nc files and print out a list of what plats are/are not avail in that file --> this is the list_column_inventory.pro Would also be good to have a simple plotting tool that could provide... -kinda did this in my sanity checking...can provide my col_box_test_plots.pro as a simple/starter example. "package_v1.0" Current working framework: AS OF 12 SEPT 2016 Supported platforms: NPOL, NEXRAD (KDOX, KAKQ), D3R, APUs, MRR, GMI/GPROF ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- ::STILL TO DO:: - **STILL FOR OLYMPEX UPDATES**: verify D3R module ok...-- done. verify MRR hts handled ok-- done. add soundings module ADD MODULES FOR: (running list) - SOUNDINGS! (OLYMPEX) - L1C GMI Tbs -- done. - update DPR - add 2AKA, 2AKU, CMB... - MRMS - done. - sfc data/wind obs for v2.0: Next big version of the framework, Things to Include: (1) create the Radx .params files on the fly for each radar based on the scan type (PPI or RHI) and the column grid set up parameters (---this was included in v1.2---) (2) For APUs: in addition to only having the APU values at the single minutes corresponding to the main_plat time, add in fieds/structures so can include main_plat time +/- 10 min of time for the APU values -- call these something like APU interval data vs. APU main_time data -->> same thing for 2DVD data: add an interval option (3) For MRR: as with APUs, allow user-set +/- # mins interval for extracting data. Maybe also add a threshold for data_quality? (4) For GMI: Walt suggests also including L1C product(s) for Tb info among each of the GMI channels -- to do this, may end up having to rename get_gmi_for_column.pro to get_gmigprof, and then also have a get_gmi_lev1c .pro... -- done. (5) May wana revisit how RHIs are handled?? D3R raised a few questions: if RHI azimuth(s) don't go directly thru/very near column grid points, does data get put at any points? For v1, I am gridding the RHIs and just using the resulting .nc files to then locate points in the column grid/interpolate to column grid...but does this actually "miss" some data that could still be represented? or is the way of defining the grid a limitation? ie: RHI azimuths are not all at multiples of 45 deg, so can have instances where first point [xsub1, ysub1] away from the grid center [or radar, really] won't have data, but pts farther away will... (6) For DPR, but really all: should consider/check/update how any discrete fields are being handled...neighbors? (7) Idea: include separate fields for original data? maybe even as seperate groups so could have original dims/dcoords?? (8) MAKE BIG LOOPS IN GMI & DPR MODULES MORE EFFICIENT IF AT ALL POSSIBLE! -- v1.4: update to use search area to limit where looking -- done. (9) Revamp/Re-order way column .nc file is written: to save memory: instead of one big write_column_nc_file.pro to be called only once after structures from every platform are created, could try to write (or append) to the file after have the structure for another platform, then delete the structure from memory..? ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- -----------------------------------------------------------------------------