Adding Tagging Criteria

[SantyE]

Hello all, I wanted to implement some specific tagging criteria for refinement. I was able to do this by modifying Tagging.H, Tagging.cpp, and PeleC.cpp. Initially I would get the "refine_eb_type can only be 'static' or 'adaptive'" error message when trying to run a case with my additional criteria in these files (not actually implemented in the input file). This seemed weird as I never specified the eb type and even when I specifically set it to "static" the error continued. I removed some pre-existing criteria so that the total number of tagging parameters were the same before and after my changes and it worked as expected then. Is this sort of behavior expected when editing the Tagging files? I'm not sure if somewhere in the code it expects a specific number of criteria or if that was just coincidental and the problem is somewhere I've missed. I looked at the merge that added tagging on low temperature thresholds and my approach looked pretty much the same.

The change I implemented was refining on cells that met two criteria at once - a density gradient and a temperature value. I tied them both to a single maximum refinement level, maybe that is relevant.

Thanks in advance for any input!

[marchdf]

Thank you for submitting this question! I would have to see the source changes to understand why you saw the error that you did. However, for user defined tagging, we actually recommend that you define a tagging routine in your prob.H file. You can see a dummy example here. You have access to the full state there and can implement any tagging you would like. I hope this helps!

[SantyE]

Thanks for the response, I'll try to add tagging on the prob.H file then! Just for the sake of curiosity, here are the changes I made:

Tagging.H -

struct TaggingParm
{
  amrex::Real dualdengrad = 1.0e10; // added tag
  amrex::Real dualvorterr = 1.0e10; // added tag
  amrex::Real denerr = 1.0e10;
  amrex::Real dengrad = 1.0e10;
  amrex::Real denratio = 1.0e10;
  amrex::Real presserr = 1.0e10;
  amrex::Real pressgrad = 1.0e10;
  amrex::Real velerr = 1.0e10;
  amrex::Real velgrad = 1.0e10;
  amrex::Real vorterr = 1.0e10;
  amrex::Real temperr = 1.0e10;
  amrex::Real lotemperr = -1.0e10;
  amrex::Real tempgrad = 1.0e10;
  amrex::Real ftracerr = 1.0e10;
  amrex::Real ftracgrad = 1.0e10;
  amrex::Real vfracerr = 1.0e10;
  amrex::Real detag_eb_factor = 3.0;

  std::string eb_refine_type = "static";

  int max_denvorterr_lev = 10; // added tag
  int max_denerr_lev = 10;
  int max_dengrad_lev = 10;
  int max_denratio_lev = 10;
  int max_presserr_lev = 10;
  int max_pressgrad_lev = 10;
  int max_velerr_lev = 10;
  int max_velgrad_lev = 10;
  int max_vorterr_lev = 10;
  int max_temperr_lev = 10;
  int max_lotemperr_lev = 10;
  int max_tempgrad_lev = 10;
  int max_ftracerr_lev = 10;
  int max_ftracgrad_lev = 10;
  int max_vfracerr_lev = 10;
  int max_eb_refine_lev = 10;
  int min_eb_refine_lev = -1;
  int adapt_eb_refined_lev = -1;
};

AMREX_GPU_DEVICE
AMREX_FORCE_INLINE
void
tag_dualerror(
  const int i,
  const int j,
  const int k,
  amrex::Array4<char> const& tag,
  amrex::Array4<amrex::Real const> const& field1,
  amrex::Array4<amrex::Real const> const& field2,
  const amrex::Real field1grad,
  const amrex::Real field2err,
  char tagval) noexcept
{
  // Tag on regions of high field1 gradient
  AMREX_D_TERM(
    amrex::Real ax = amrex::Math::abs(field1(i + 1, j, k) - field1(i, j, k));
    ax = amrex::max<amrex::Real>(
      ax, amrex::Math::abs(field1(i, j, k) - field1(i - 1, j, k)));
    , amrex::Real ay = amrex::Math::abs(field1(i, j + 1, k) - field1(i, j, k));
    ay = amrex::max<amrex::Real>(
      ay, amrex::Math::abs(field1(i, j, k) - field1(i, j - 1, k)));
    , amrex::Real az = amrex::Math::abs(field1(i, j, k + 1) - field1(i, j, k));
    az = amrex::max<amrex::Real>(
      az, amrex::Math::abs(field1(i, j, k) - field1(i, j, k - 1)));)
#if AMREX_SPACEDIM > 1
  if (amrex::max<amrex::Real>(AMREX_D_DECL(ax, ay, az)) >= field1grad &&
    field2(i, j, k) >= field2err)
#elif AMREX_SPACEDIM == 1
  if (ax >= field1grad && field2(i,j,k) >= field2err)
#endif
  {
    tag(i, j, k) = tagval;
  }
}

Tagging.cpp -

void
PeleC::read_tagging_params()
{
  amrex::ParmParse pp("tagging");

  pp.query("denerr", tagging_parm->denerr);
  pp.query("max_denerr_lev", tagging_parm->max_denerr_lev);
  pp.query("dengrad", tagging_parm->dengrad);
  pp.query("max_dengrad_lev", tagging_parm->max_dengrad_lev);
  pp.query("denratio", tagging_parm->denratio);
  pp.query("max_denratio_lev", tagging_parm->max_denratio_lev);

  pp.query("presserr", tagging_parm->presserr);
  pp.query("max_presserr_lev", tagging_parm->max_presserr_lev);
  pp.query("pressgrad", tagging_parm->pressgrad);
  pp.query("max_pressgrad_lev", tagging_parm->max_pressgrad_lev);

  pp.query("velerr", tagging_parm->velerr);
  pp.query("max_velerr_lev", tagging_parm->max_velerr_lev);
  pp.query("velgrad", tagging_parm->velgrad);
  pp.query("max_velgrad_lev", tagging_parm->max_velgrad_lev);

  pp.query("vorterr", tagging_parm->vorterr);
  pp.query("max_vorterr_lev", tagging_parm->max_vorterr_lev);

  pp.query("temperr", tagging_parm->temperr);
  pp.query("max_temperr_lev", tagging_parm->max_temperr_lev);
  pp.query("lotemperr", tagging_parm->lotemperr);
  pp.query("max_lotemperr_lev", tagging_parm->max_lotemperr_lev);
  pp.query("tempgrad", tagging_parm->tempgrad);
  pp.query("max_tempgrad_lev", tagging_parm->max_tempgrad_lev);

  pp.query("dualdengrad", tagging_parm->dualdengrad);
  pp.query("dualvorterr", tagging_parm->dualvorterr);
  pp.query("max_denvorterr_lev", tagging_parm->max_denvorterr_lev);

  pp.query("ftracerr", tagging_parm->ftracerr);
  pp.query("max_ftracerr_lev", tagging_parm->max_ftracerr_lev);
  pp.query("ftracgrad", tagging_parm->ftracgrad);
  pp.query("max_ftracgrad_lev", tagging_parm->max_ftracgrad_lev);

  pp.query("vfracerr", tagging_parm->vfracerr);
  pp.query("max_vfracerr_lev", tagging_parm->max_vfracerr_lev);

  pp.query("eb_refine_type", tagging_parm->eb_refine_type);
  pp.query("max_eb_refine_lev", tagging_parm->max_eb_refine_lev);
  pp.query("eb_detag_factor", tagging_parm->detag_eb_factor);
  tagging_parm->detag_eb_factor *= std::sqrt(2.0);
  if (
    (tagging_parm->eb_refine_type != "static") &&
    (tagging_parm->eb_refine_type != "adaptive")) {
    amrex::Abort("refine_eb_type can only be 'static' or 'adaptive'");
  }
  if (tagging_parm->eb_refine_type == "adaptive") {
    tagging_parm->min_eb_refine_lev = 0;
    pp.query("min_eb_refine_lev", tagging_parm->min_eb_refine_lev);
    tagging_parm->adapt_eb_refined_lev = tagging_parm->min_eb_refine_lev;
  }
}

PeleC.cpp -

 // Tagging density gradient and vorticity error
      S_derData.setVal<amrex::RunOn::Device>(0.0, datbox);
      pc_dermagvort(
        tilebox, S_derData, ncp, Sfab.nComp(), S_data[mfi], geom, time, bc,
        level);
      if (level < tagging_parm->max_denvorterr_lev) {
        const amrex::Real captured_dualdengrad = tagging_parm->dualdengrad;
        const amrex::Real captured_dualvorterr = 
          tagging_parm->dualvorterr * std::pow(2.0, level);
        amrex::ParallelFor(
          tilebox, [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept {
            tag_dualerror(i, j, k, tag_arr, Sfab, S_derarr, captured_dualdengrad,
              captured_dualvorterr, tagval);
          });
      }

This would cause the "refine_eb_type can only be 'static' or 'adaptive'" message even when I didn't flag to refine on the new parameters. Since I added two tagging values and one tagging level parameter, I commented out velerr, velgrad, and max_velerr_lev (I kept max_velgrad_lev) and it worked as expected.

[marchdf]

It's not obvious to me why the code you posted would not work... But let me know how things in prob.H work out for you.

[SantyE]

Thanks for taking a look, yeah not sure... I'll let you know if I can get prob.H to work, thanks again!