Update of ESMF version causes mizuRoute to abort

[ekluzek]

And update of the ESMF version on both Cheyenne and izumi cause mizuRoute to abort in the initialization phase inside of mizuRoute.

The PET file shows the following..

.onnected using mesh
20240110 181907.429 INFO             PET0 (lnd_import_export:fldlist_realize)(lnd_import_export:realize_fields):clmImport Field = Flrr_volrmch is connected using mesh
20240110 181915.508 INFO             PET0 (lnd_comp_nuopc):(InitializeRealize)  done
20240110 181915.508 INFO             PET0 (rof_comp_nuopc):(InitializeRealize)  called
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_Glue.C:189 ESMCI_meshaddnodes() Value unrecognized or out of range  -  node ids must be >= 1
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_Glue.C:391 ESMCI_meshaddnodes() Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:574 ESMCI_mesh_create_from_ESMFMesh_ Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:670 ESMCI_mesh_create_from_ESMFMesh_ Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:284 ESMCI_mesh_create_from_file() Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 ESMCI_MeshCap.C:2600 MeshCap::meshcreatefromfilenew() Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 ESMF_Mesh.F90:1969 ESMF_MeshCreateFromFile() Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 rof_comp_nuopc.F90:591 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 ERROR            PET0 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:2898 Value unrecognized or out of range  - Phase 'IPDv01p3' Initialize for modelComp 4: ROF did not return ESMF_SUCCESS
20240110 181915.769 ERROR            PET0 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:1982 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 ERROR            PET0 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:2898 Value unrecognized or out of range  - Phase 'IPDv02p3' Initialize for modelComp 1: ESM0001 did not return ESMF_SUCCESS
20240110 181915.769 ERROR            PET0 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:1987 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 ERROR            PET0 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:486 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 ERROR            PET0 esmApp.F90:134 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 INFO             PET0 Finalizing ESMF

[nmizukami]

Hi Erik, yes, it is clearly from the esmf function, but do you know what exactly is the problem? is it node id problem (usually starting 0, but does not like 0 any more?)? Let me know if there is anything I should look into!

[ekluzek]

The ccs_config issue for this is that documents the ESMF updates that result in the problem is here:

ESMCI/ccs_config_cesm#146

[ekluzek]

Here's some more information on one of the failing cases and what the mesh file looks like:

SMS_Ld1_D_P2x1.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default.20240110_230530_klwpfm]$ ncdump -v elementConn  /fs/cgd/csm/inputdata/share/meshes/5x5pt-amazon_navy_ESMFmesh_cd5_c20210107.nc
netcdf \5x5pt-amazon_navy_ESMFmesh_cd5_c20210107 {
dimensions:
	nodeCount = 36 ;
	elementCount = 25 ;
	maxNodePElement = 4 ;
	coordDim = 2 ;
	origGridRank = 2 ;
variables:
	int origGridDims(origGridRank) ;
	double nodeCoords(nodeCount, coordDim) ;
		nodeCoords:units = "degrees" ;
	int elementConn(elementCount, maxNodePElement) ;
		elementConn:long_name = "Node indices that define the element connectivity" ;
		elementConn:_FillValue = -1 ;
	int numElementConn(elementCount) ;
		numElementConn:long_name = "Number of nodes per element" ;
	double centerCoords(elementCount, coordDim) ;
		centerCoords:units = "degrees" ;
	double elementArea(elementCount) ;
	int elementMask(elementCount) ;
		elementMask:units = "unitless" ;

// global attributes:
		:gridType = "unstructured mesh" ;
		:version = "0.9" ;
		:inputFile = "domain.lnd.5x5pt-amazon_navy.SCRIP.070121.nc" ;
		:timeGenerated = "Thu Jan  7 13:58:21 2021" ;
data:

 elementConn =
  1, 2, 7, 8,
  2, 3, 9, 7,
  3, 4, 10, 9,
  4, 5, 11, 10,
  5, 6, 12, 11,
  8, 7, 13, 14,
  7, 9, 15, 13,
  9, 10, 16, 15,
  10, 11, 17, 16,
  11, 12, 18, 17,
  14, 13, 19, 20,
  13, 15, 21, 19,
  15, 16, 22, 21,
  16, 17, 23, 22,
  17, 18, 24, 23,
  20, 19, 25, 26,
  19, 21, 27, 25,
  21, 22, 28, 27,
  22, 23, 29, 28,
  23, 24, 30, 29,
  26, 25, 31, 32,
  25, 27, 33, 31,
  27, 28, 34, 33,
  28, 29, 35, 34,
  29, 30, 36, 35 ;
}

[ekluzek]

Here are the other mesh files...

./xmlquery -p MESH                      
Results in group run_domain
	ATM_DOMAIN_MESH: /fs/cgd/csm/inputdata/share/meshes/5x5pt-amazon_navy_ESMFmesh_cd5_c20210107.nc
	GLC_DOMAIN_MESH: UNSET
	ICE_DOMAIN_MESH: UNSET
	LND_DOMAIN_MESH: /fs/cgd/csm/inputdata/share/meshes/5x5pt-amazon_navy_ESMFmesh_cd5_c20210107.nc
	MASK_MESH: /fs/cgd/csm/inputdata/share/meshes/5x5pt-amazon_navy_ESMFmesh_cd5_c20210107.nc
	OCN_DOMAIN_MESH: UNSET
	ROF_DOMAIN_MESH: /fs/cgd/csm/inputdata/rof/mizuRoute/meshes/r05_amazon_c110308_ctrcrd_cdf5_ESMFmesh_c20200624.nc
	WAV_DOMAIN_MESH: UNSET

[ekluzek]

And here's the mesh that has just the small squares around the center point of the HRU's

ncdump -v hru_id,elementConn /fs/cgd/csm/inputdata/rof/mizuRoute/meshes/r05_amazon_c110308_ctrcrd_cdf5_ESMFmesh_c20200624.nc
netcdf r05_amazon_c110308_ctrcrd_cdf5_ESMFmesh_c20200624 {
dimensions:
	elementCount = 748 ;
	connectionCount = 2992 ;
	nodeCount = 2992 ;
	coordDim = 2 ;
variables:
	char ocgis_coordinate_system ;
		ocgis_coordinate_system:proj4 = "+proj=longlat +datum=WGS84 +no_defs " ;
		ocgis_coordinate_system:ocgis_role = "coordinate_system" ;
	double hru_id(elementCount) ;
	int elementConn(connectionCount) ;
		elementConn:long_name = "Node indices that define the element connectivity." ;
		elementConn:start_index = 0LL ;
	int numElementConn(elementCount) ;
		numElementConn:long_name = "Number of nodes per element." ;
	double nodeCoords(nodeCount, coordDim) ;
		nodeCoords:units = "degrees" ;
		nodeCoords:long_name = "Node coordinate values indexed by element connectivity." ;
		nodeCoords:coordinates = "x y" ;
	double centerCoords(elementCount, coordDim) ;
		centerCoords:_FillValue = NaN ;
		centerCoords:units = "degrees" ;

// global attributes:
		:gridType = "unstructured" ;
		:version = "0.9" ;
		:history = "Mon Jun 22 16:06:21 2020: ncks -A -v centerCoords out.reorder.nc output/ESMF_unstruct_mosart0.5_amazon_land_360.nc\n",
			"Fri May  8 16:35:35 2020: ncrename -v CELLVALUE,hru_id ESMF_unstruct_mosart0.5_amazon_land_360.nc" ;
		:NCO = "netCDF Operators version 4.7.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
		:history_of_appended_files = "Mon Jun 22 16:06:21 2020: Appended file out.reorder.nc had no \"history\" attribute\n",
			"" ;
data:

 hru_id = 123694, 122974, 122254, 121534, 120814, 120094, 119374, 118654, 
    117934, 117214, 116494, 115774, 115054, 114334, 113614, 112894, 112174, 
    111454, 110734, 110014, 109294, 108574, 123695, 122975, 122255, 121535, 
    120815, 120095, 119375, 118655, 117935, 117215, 116495, 115775, 115055, 
    114335, 113615, 112895, 112175, 111455, 110735, 110015, 109295, 108575, 
    123696, 122976, 122256, 121536, 120816, 120096, 119376, 118656, 117936, 
    117216, 116496, 115776, 115056, 114336, 113616, 112896, 112176, 111456, 
    110736, 110016, 109296, 108576, 123697, 122977, 122257, 121537, 120817, 
    120097, 119377, 118657, 117937, 117217, 116497, 115777, 115057, 114337, 
    113617, 112897, 112177, 111457, 110737, 110017, 109297, 108577, 123698, 
    122978, 122258, 121538, 120818, 120098, 119378, 118658, 117938, 117218, 
    116498, 115778, 115058, 114338, 113618, 112898, 112178, 111458, 110738, 
    110018, 109298, 108578, 123699, 122979, 122259, 121539, 120819, 120099, 
    119379, 118659, 117939, 117219, 116499, 115779, 115059, 114339, 113619, 
    112899, 112179, 111459, 110739, 110019, 109299, 108579, 123700, 122980, 
    122260, 121540, 120820, 120100, 119380, 118660, 117940, 117220, 116500, 
    115780, 115060, 114340, 113620, 112900, 112180, 111460, 110740, 110020, 
    109300, 108580, 123701, 122981, 122261, 121541, 120821, 120101, 119381, 
    118661, 117941, 117221, 116501, 115781, 115061, 114341, 113621, 112901, 
    112181, 111461, 110741, 110021, 109301, 108581, 123702, 122982, 122262, 
    121542, 120822, 120102, 119382, 118662, 117942, 117222, 116502, 115782, 
    115062, 114342, 113622, 112902, 112182, 111462, 110742, 110022, 109302, 
    108582, 123703, 122983, 122263, 121543, 120823, 120103, 119383, 118663, 
    117943, 117223, 116503, 115783, 115063, 114343, 113623, 112903, 112183, 
    111463, 110743, 110023, 109303, 108583, 123704, 122984, 122264, 121544, 
    120824, 120104, 119384, 118664, 117944, 117224, 116504, 115784, 115064, 
    114344, 113624, 112904, 112184, 111464, 110744, 110024, 109304, 108584, 
    123705, 122985, 122265, 121545, 120825, 120105, 119385, 118665, 117945, 
    117225, 116505, 115785, 115065, 114345, 113625, 112905, 112185, 111465, 
    110745, 110025, 109305, 108585, 123706, 122986, 122266, 121546, 120826, 
    120106, 119386, 118666, 117946, 117226, 116506, 115786, 115066, 114346, 
    113626, 112906, 112186, 111466, 110746, 110026, 109306, 108586, 123707, 
    122987, 122267, 121547, 120827, 120107, 119387, 118667, 117947, 117227, 
    116507, 115787, 115067, 114347, 113627, 112907, 112187, 111467, 110747, 
    110027, 109307, 108587, 123708, 122988, 122268, 121548, 120828, 120108, 
    119388, 118668, 117948, 117228, 116508, 115788, 115068, 114348, 113628, 
    112908, 112188, 111468, 110748, 110028, 109308, 108588, 123709, 122989, 
    122269, 121549, 120829, 120109, 119389, 118669, 117949, 117229, 116509, 
    115789, 115069, 114349, 113629, 112909, 112189, 111469, 110749, 110029, 
    109309, 108589, 123710, 122990, 122270, 121550, 120830, 120110, 119390, 
    118670, 117950, 117230, 116510, 115790, 115070, 114350, 113630, 112910, 
    112190, 111470, 110750, 110030, 109310, 108590, 123711, 122991, 122271, 
    121551, 120831, 120111, 119391, 118671, 117951, 117231, 116511, 115791, 
    115071, 114351, 113631, 112911, 112191, 111471, 110751, 110031, 109311, 
    108591, 123712, 122992, 122272, 121552, 120832, 120112, 119392, 118672, 
    117952, 117232, 116512, 115792, 115072, 114352, 113632, 112912, 112192, 
    111472, 110752, 110032, 109312, 108592, 123713, 122993, 122273, 121553, 
    120833, 120113, 119393, 118673, 117953, 117233, 116513, 115793, 115073, 
    114353, 113633, 112913, 112193, 111473, 110753, 110033, 109313, 108593, 
    123714, 122994, 122274, 121554, 120834, 120114, 119394, 118674, 117954, 
    117234, 116514, 115794, 115074, 114354, 113634, 112914, 112194, 111474, 
    110754, 110034, 109314, 108594, 123715, 122995, 122275, 121555, 120835, 
    120115, 119395, 118675, 117955, 117235, 116515, 115795, 115075, 114355, 
    113635, 112915, 112195, 111475, 110755, 110035, 109315, 108595, 123716, 
    122996, 122276, 121556, 120836, 120116, 119396, 118676, 117956, 117236, 
    116516, 115796, 115076, 114356, 113636, 112916, 112196, 111476, 110756, 
    110036, 109316, 108596, 123717, 122997, 122277, 121557, 120837, 120117, 
    119397, 118677, 117957, 117237, 116517, 115797, 115077, 114357, 113637, 
    112917, 112197, 111477, 110757, 110037, 109317, 108597, 123718, 122998, 
    122278, 121558, 120838, 120118, 119398, 118678, 117958, 117238, 116518, 
    115798, 115078, 114358, 113638, 112918, 112198, 111478, 110758, 110038, 
    109318, 108598, 123719, 122999, 122279, 121559, 120839, 120119, 119399, 
    118679, 117959, 117239, 116519, 115799, 115079, 114359, 113639, 112919, 
    112199, 111479, 110759, 110039, 109319, 108599, 123720, 123000, 122280, 
    121560, 120840, 120120, 119400, 118680, 117960, 117240, 116520, 115800, 
    115080, 114360, 113640, 112920, 112200, 111480, 110760, 110040, 109320, 
    108600, 123721, 123001, 122281, 121561, 120841, 120121, 119401, 118681, 
    117961, 117241, 116521, 115801, 115081, 114361, 113641, 112921, 112201, 
    111481, 110761, 110041, 109321, 108601, 123722, 123002, 122282, 121562, 
    120842, 120122, 119402, 118682, 117962, 117242, 116522, 115802, 115082, 
    114362, 113642, 112922, 112202, 111482, 110762, 110042, 109322, 108602, 
    123723, 123003, 122283, 121563, 120843, 120123, 119403, 118683, 117963, 
    117243, 116523, 115803, 115083, 114363, 113643, 112923, 112203, 111483, 
    110763, 110043, 109323, 108603, 123724, 123004, 122284, 121564, 120844, 
    120124, 119404, 118684, 117964, 117244, 116524, 115804, 115084, 114364, 
    113644, 112924, 112204, 111484, 110764, 110044, 109324, 108604, 123725, 
    123005, 122285, 121565, 120845, 120125, 119405, 118685, 117965, 117245, 
    116525, 115805, 115085, 114365, 113645, 112925, 112205, 111485, 110765, 
    110045, 109325, 108605, 123726, 123006, 122286, 121566, 120846, 120126, 
    119406, 118686, 117966, 117246, 116526, 115806, 115086, 114366, 113646, 
    112926, 112206, 111486, 110766, 110046, 109326, 108606, 123727, 123007, 
    122287, 121567, 120847, 120127, 119407, 118687, 117967, 117247, 116527, 
    115807, 115087, 114367, 113647, 112927, 112207, 111487, 110767, 110047, 
    109327, 108607 ;

 elementConn = 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 
    18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 
    36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 
    54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 
    72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 
    90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 
    106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 
    120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 
    134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 
    148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 
    162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 
    176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 
    190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 
    204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 
    218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 
    232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 
    246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 
    260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 
    274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 
    288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 
    302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 
    316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 
    330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 
    344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 
    358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 
    372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 
    386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 
    400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 
    414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 
    428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 
    442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 
    456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 
    470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 
    484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 
    498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 
    512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 
    526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 
    540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 
    554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 
    568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 
    582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 
    596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 
    610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 
    624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 
    638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 
    652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 
    666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 
    680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 
    694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 
    708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 
    722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 
    736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 
    750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 
    764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 
    778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 
    792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 
    806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 
    820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 
    834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 
    848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 
    862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 
    876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 
    890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 
    904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 
    918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 
    932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 
    946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 
    960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 
    974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 
    988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 
    1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 
    1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 
    1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 
    1038, 1039, 1040, 1041, 1042, 1043, 1044, 1045, 1046, 1047, 1048, 1049, 
    1050, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 
    1062, 1063, 1064, 1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 
    1074, 1075, 1076, 1077, 1078, 1079, 1080, 1081, 1082, 1083, 1084, 1085, 
    1086, 1087, 1088, 1089, 1090, 1091, 1092, 1093, 1094, 1095, 1096, 1097, 
    1098, 1099, 1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 
    1110, 1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118, 1119, 1120, 1121, 
    1122, 1123, 1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 
    1134, 1135, 1136, 1137, 1138, 1139, 1140, 1141, 1142, 1143, 1144, 1145, 
    1146, 1147, 1148, 1149, 1150, 1151, 1152, 1153, 1154, 1155, 1156, 1157, 
    1158, 1159, 1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1169, 
    1170, 1171, 1172, 1173, 1174, 1175, 1176, 1177, 1178, 1179, 1180, 1181, 
    1182, 1183, 1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 
    1194, 1195, 1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1205, 
    1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 
    1218, 1219, 1220, 1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 
    1230, 1231, 1232, 1233, 1234, 1235, 1236, 1237, 1238, 1239, 1240, 1241, 
    1242, 1243, 1244, 1245, 1246, 1247, 1248, 1249, 1250, 1251, 1252, 1253, 
    1254, 1255, 1256, 1257, 1258, 1259, 1260, 1261, 1262, 1263, 1264, 1265, 
    1266, 1267, 1268, 1269, 1270, 1271, 1272, 1273, 1274, 1275, 1276, 1277, 
    1278, 1279, 1280, 1281, 1282, 1283, 1284, 1285, 1286, 1287, 1288, 1289, 
    1290, 1291, 1292, 1293, 1294, 1295, 1296, 1297, 1298, 1299, 1300, 1301, 
    1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 
    1314, 1315, 1316, 1317, 1318, 1319, 1320, 1321, 1322, 1323, 1324, 1325, 
    1326, 1327, 1328, 1329, 1330, 1331, 1332, 1333, 1334, 1335, 1336, 1337, 
    1338, 1339, 1340, 1341, 1342, 1343, 1344, 1345, 1346, 1347, 1348, 1349, 
    1350, 1351, 1352, 1353, 1354, 1355, 1356, 1357, 1358, 1359, 1360, 1361, 
    1362, 1363, 1364, 1365, 1366, 1367, 1368, 1369, 1370, 1371, 1372, 1373, 
    1374, 1375, 1376, 1377, 1378, 1379, 1380, 1381, 1382, 1383, 1384, 1385, 
    1386, 1387, 1388, 1389, 1390, 1391, 1392, 1393, 1394, 1395, 1396, 1397, 
    1398, 1399, 1400, 1401, 1402, 1403, 1404, 1405, 1406, 1407, 1408, 1409, 
    1410, 1411, 1412, 1413, 1414, 1415, 1416, 1417, 1418, 1419, 1420, 1421, 
    1422, 1423, 1424, 1425, 1426, 1427, 1428, 1429, 1430, 1431, 1432, 1433, 
    1434, 1435, 1436, 1437, 1438, 1439, 1440, 1441, 1442, 1443, 1444, 1445, 
    1446, 1447, 1448, 1449, 1450, 1451, 1452, 1453, 1454, 1455, 1456, 1457, 
    1458, 1459, 1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 
    1470, 1471, 1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1481, 
    1482, 1483, 1484, 1485, 1486, 1487, 1488, 1489, 1490, 1491, 1492, 1493, 
    1494, 1495, 1496, 1497, 1498, 1499, 1500, 1501, 1502, 1503, 1504, 1505, 
    1506, 1507, 1508, 1509, 1510, 1511, 1512, 1513, 1514, 1515, 1516, 1517, 
    1518, 1519, 1520, 1521, 1522, 1523, 1524, 1525, 1526, 1527, 1528, 1529, 
    1530, 1531, 1532, 1533, 1534, 1535, 1536, 1537, 1538, 1539, 1540, 1541, 
    1542, 1543, 1544, 1545, 1546, 1547, 1548, 1549, 1550, 1551, 1552, 1553, 
    1554, 1555, 1556, 1557, 1558, 1559, 1560, 1561, 1562, 1563, 1564, 1565, 
    1566, 1567, 1568, 1569, 1570, 1571, 1572, 1573, 1574, 1575, 1576, 1577, 
    1578, 1579, 1580, 1581, 1582, 1583, 1584, 1585, 1586, 1587, 1588, 1589, 
    1590, 1591, 1592, 1593, 1594, 1595, 1596, 1597, 1598, 1599, 1600, 1601, 
    1602, 1603, 1604, 1605, 1606, 1607, 1608, 1609, 1610, 1611, 1612, 1613, 
    1614, 1615, 1616, 1617, 1618, 1619, 1620, 1621, 1622, 1623, 1624, 1625, 
    1626, 1627, 1628, 1629, 1630, 1631, 1632, 1633, 1634, 1635, 1636, 1637, 
    1638, 1639, 1640, 1641, 1642, 1643, 1644, 1645, 1646, 1647, 1648, 1649, 
    1650, 1651, 1652, 1653, 1654, 1655, 1656, 1657, 1658, 1659, 1660, 1661, 
    1662, 1663, 1664, 1665, 1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 
    1674, 1675, 1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685, 
    1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695, 1696, 1697, 
    1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 
    1710, 1711, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 
    1722, 1723, 1724, 1725, 1726, 1727, 1728, 1729, 1730, 1731, 1732, 1733, 
    1734, 1735, 1736, 1737, 1738, 1739, 1740, 1741, 1742, 1743, 1744, 1745, 
    1746, 1747, 1748, 1749, 1750, 1751, 1752, 1753, 1754, 1755, 1756, 1757, 
    1758, 1759, 1760, 1761, 1762, 1763, 1764, 1765, 1766, 1767, 1768, 1769, 
    1770, 1771, 1772, 1773, 1774, 1775, 1776, 1777, 1778, 1779, 1780, 1781, 
    1782, 1783, 1784, 1785, 1786, 1787, 1788, 1789, 1790, 1791, 1792, 1793, 
    1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 
    1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 
    1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 
    1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 
    1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 
    1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 
    1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 
    1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 
    1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 
    1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 
    1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 
    1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 
    1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 
    1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 
    1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 
    1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 
    1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 
    1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 
    2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 
    2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 
    2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 
    2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 
    2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 
    2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 
    2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 
    2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 
    2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 
    2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 
    2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 
    2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 
    2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 
    2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 
    2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 
    2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 
    2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 
    2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 
    2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 
    2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 
    2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 
    2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 
    2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 
    2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 
    2298, 2299, 2300, 2301, 2302, 2303, 2304, 2305, 2306, 2307, 2308, 2309, 
    2310, 2311, 2312, 2313, 2314, 2315, 2316, 2317, 2318, 2319, 2320, 2321, 
    2322, 2323, 2324, 2325, 2326, 2327, 2328, 2329, 2330, 2331, 2332, 2333, 
    2334, 2335, 2336, 2337, 2338, 2339, 2340, 2341, 2342, 2343, 2344, 2345, 
    2346, 2347, 2348, 2349, 2350, 2351, 2352, 2353, 2354, 2355, 2356, 2357, 
    2358, 2359, 2360, 2361, 2362, 2363, 2364, 2365, 2366, 2367, 2368, 2369, 
    2370, 2371, 2372, 2373, 2374, 2375, 2376, 2377, 2378, 2379, 2380, 2381, 
    2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 
    2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 
    2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 
    2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 
    2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2439, 2440, 2441, 
    2442, 2443, 2444, 2445, 2446, 2447, 2448, 2449, 2450, 2451, 2452, 2453, 
    2454, 2455, 2456, 2457, 2458, 2459, 2460, 2461, 2462, 2463, 2464, 2465, 
    2466, 2467, 2468, 2469, 2470, 2471, 2472, 2473, 2474, 2475, 2476, 2477, 
    2478, 2479, 2480, 2481, 2482, 2483, 2484, 2485, 2486, 2487, 2488, 2489, 
    2490, 2491, 2492, 2493, 2494, 2495, 2496, 2497, 2498, 2499, 2500, 2501, 
    2502, 2503, 2504, 2505, 2506, 2507, 2508, 2509, 2510, 2511, 2512, 2513, 
    2514, 2515, 2516, 2517, 2518, 2519, 2520, 2521, 2522, 2523, 2524, 2525, 
    2526, 2527, 2528, 2529, 2530, 2531, 2532, 2533, 2534, 2535, 2536, 2537, 
    2538, 2539, 2540, 2541, 2542, 2543, 2544, 2545, 2546, 2547, 2548, 2549, 
    2550, 2551, 2552, 2553, 2554, 2555, 2556, 2557, 2558, 2559, 2560, 2561, 
    2562, 2563, 2564, 2565, 2566, 2567, 2568, 2569, 2570, 2571, 2572, 2573, 
    2574, 2575, 2576, 2577, 2578, 2579, 2580, 2581, 2582, 2583, 2584, 2585, 
    2586, 2587, 2588, 2589, 2590, 2591, 2592, 2593, 2594, 2595, 2596, 2597, 
    2598, 2599, 2600, 2601, 2602, 2603, 2604, 2605, 2606, 2607, 2608, 2609, 
    2610, 2611, 2612, 2613, 2614, 2615, 2616, 2617, 2618, 2619, 2620, 2621, 
    2622, 2623, 2624, 2625, 2626, 2627, 2628, 2629, 2630, 2631, 2632, 2633, 
    2634, 2635, 2636, 2637, 2638, 2639, 2640, 2641, 2642, 2643, 2644, 2645, 
    2646, 2647, 2648, 2649, 2650, 2651, 2652, 2653, 2654, 2655, 2656, 2657, 
    2658, 2659, 2660, 2661, 2662, 2663, 2664, 2665, 2666, 2667, 2668, 2669, 
    2670, 2671, 2672, 2673, 2674, 2675, 2676, 2677, 2678, 2679, 2680, 2681, 
    2682, 2683, 2684, 2685, 2686, 2687, 2688, 2689, 2690, 2691, 2692, 2693, 
    2694, 2695, 2696, 2697, 2698, 2699, 2700, 2701, 2702, 2703, 2704, 2705, 
    2706, 2707, 2708, 2709, 2710, 2711, 2712, 2713, 2714, 2715, 2716, 2717, 
    2718, 2719, 2720, 2721, 2722, 2723, 2724, 2725, 2726, 2727, 2728, 2729, 
    2730, 2731, 2732, 2733, 2734, 2735, 2736, 2737, 2738, 2739, 2740, 2741, 
    2742, 2743, 2744, 2745, 2746, 2747, 2748, 2749, 2750, 2751, 2752, 2753, 
    2754, 2755, 2756, 2757, 2758, 2759, 2760, 2761, 2762, 2763, 2764, 2765, 
    2766, 2767, 2768, 2769, 2770, 2771, 2772, 2773, 2774, 2775, 2776, 2777, 
    2778, 2779, 2780, 2781, 2782, 2783, 2784, 2785, 2786, 2787, 2788, 2789, 
    2790, 2791, 2792, 2793, 2794, 2795, 2796, 2797, 2798, 2799, 2800, 2801, 
    2802, 2803, 2804, 2805, 2806, 2807, 2808, 2809, 2810, 2811, 2812, 2813, 
    2814, 2815, 2816, 2817, 2818, 2819, 2820, 2821, 2822, 2823, 2824, 2825, 
    2826, 2827, 2828, 2829, 2830, 2831, 2832, 2833, 2834, 2835, 2836, 2837, 
    2838, 2839, 2840, 2841, 2842, 2843, 2844, 2845, 2846, 2847, 2848, 2849, 
    2850, 2851, 2852, 2853, 2854, 2855, 2856, 2857, 2858, 2859, 2860, 2861, 
    2862, 2863, 2864, 2865, 2866, 2867, 2868, 2869, 2870, 2871, 2872, 2873, 
    2874, 2875, 2876, 2877, 2878, 2879, 2880, 2881, 2882, 2883, 2884, 2885, 
    2886, 2887, 2888, 2889, 2890, 2891, 2892, 2893, 2894, 2895, 2896, 2897, 
    2898, 2899, 2900, 2901, 2902, 2903, 2904, 2905, 2906, 2907, 2908, 2909, 
    2910, 2911, 2912, 2913, 2914, 2915, 2916, 2917, 2918, 2919, 2920, 2921, 
    2922, 2923, 2924, 2925, 2926, 2927, 2928, 2929, 2930, 2931, 2932, 2933, 
    2934, 2935, 2936, 2937, 2938, 2939, 2940, 2941, 2942, 2943, 2944, 2945, 
    2946, 2947, 2948, 2949, 2950, 2951, 2952, 2953, 2954, 2955, 2956, 2957, 
    2958, 2959, 2960, 2961, 2962, 2963, 2964, 2965, 2966, 2967, 2968, 2969, 
    2970, 2971, 2972, 2973, 2974, 2975, 2976, 2977, 2978, 2979, 2980, 2981, 
    2982, 2983, 2984, 2985, 2986, 2987, 2988, 2989, 2990, 2991 ;
}

[nmizukami]

Hi Erik (@ekluzek), hru_id is in double (I expected id to be int32). This might be problem? Not sure why it is in double (must be from python)..... but it is easy to change a type from double to int32? I wonder the other meshes also have ID in double precision.

UPDATE: I went through all the other mesh ID to check the type. all but HDMA amazon meth are in int64.

[nmizukami]

ah, looking at PET log, ESMF is complaining mesh node id (index?) not computing node.... that is strange (I never expect node index become negative)

[ekluzek]

Bob Oehmke from ESMF figured out the problem is that elementConn starts at 0 rather than the expected 1. This used to work, but because of potential issues more checking was added, and it now fails. ESMF is going to add an update where starting at 0 will also work. And also add clearer error checking so that it would easier to catch the problem.

@nmizukami has updates to the mesh files so they start at 1, so we'll try that out and verify this fixes the problem.

[ekluzek]

Trying out the 5x5_amazon case on Izumi starting elementConn at 1 works!

PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default CREATE_NEWCASE
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default XML
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default SETUP
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default SHAREDLIB_BUILD time=22
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default MODEL_BUILD time=133
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default SUBMIT
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default RUN time=148
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default MEMLEAK insuffiencient data for memleak test
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default SHORT_TERM_ARCHIVER

[nmizukami]

We need to move ESMF mesh with 1-index from naoki's directory to cesm data directory.

But also ESFM may allow us to use 0-index base MESH in next update??

[ekluzek]

ESMF told us that the fix for this went into ESMF 8.6.1. Which is great.

However, in terms of CESM this hasn't been brought in as of now. ccs_config has NOT updated to using that version on either Derecho nor Izumi. Derecho uses 8.6.0 and Izumi 8.5.0. There are a couple beta tags of 8.6.1 on Derecho, and a beta tag of 8.7.0 also so those are the only possibilities. On Izumi 8.5.0 is the latest version available, so trying it out would require installing a newer version of ESMF on at least Izumi, as well as using a beta version on Derecho.

[ekluzek]

FYI: Here's the PR that fixed it in ESMF esmf-org/esmf#147 and the first beta tag with it was 8.6.1.b02 and that is available on Derecho. It does look like there is a 8.6.1 library on Derecho, but I'm not sure what it's compatible with.