How do you specify coordinates for MueLu (Zoltan)?

[abodado]

@jhux2

I am trying to use Zoltan.

Question 1:
I see there might be a way to specify the coordinates to MueLu once a Hierarchy has been created and then specify:
H->GetLevel(0)->Set("Coordinates", coordinates);
Do the coordinates have to have any specific structure? I was going to use a 3 column Tpetra Multivector. Does that suffice?

Question 2:
What if I use a different method to build the preconditioner, like "MueLu::CreateTpetraPreconditioner".
Specifically:
RCP<op_type> M = MueLu::CreateTpetraPreconditioner<SC, LO, GO, NO>(mA, *paramList);
where op_type is Tpetra::Operator<SC, LO, GO, NO>;
What is the approach to specify the coordinates for the fine level?

[jhux2]

@abodado Specify coordinates as a multivector. 1st vector is X, 2nd is Y, 3rd is Z. There should be one coordinate per mesh node. (So if you are working with a system of PDEs, you should not duplicate coordinates.)

You set the coordinates in a parameter sublist:

Teuchos::ParameterList& userData = paramList.sublist("user data");
userData.set("Coordinates",coordMV);
RCP<op_type> M = MueLu::CreateTpetraPreconditioner<SC, LO, GO, NO>(mA, *paramList);

[abodado]

@jhux2 , thanks. I tried that just now, but I got an error that Coordinates couldn't be found.

terminate called after throwing an instance of 'MueLu::Exceptions::RuntimeError'
what(): CoordinatesTransferFactory::DeclareInput: (LevelID = 0) unable to find or generate requested data "Coordinates" with generating factory "null" [0]
during request for data " Coordinates" on level 0 by factory CoordinatesTransferFactory [0x537e7a0]
during request for data " Coordinates" on level 1 by factory RebalanceTransferFactory [0x5384ad0]
during request for data " P" on level 1 by factory NoFactory [0x53674a0]

Here is an excerpt from my code in case any obvious error jumps out. The coordinates I create are not practical, it's just a test case.

    // Create a Tpetra::MultiVector for coordinates
    RCP<multiVect_type> zoltanCoords = rcp(new multiVect_type(rowMap,3));

    int lowGlobIndx = rowMap->getMinGlobalIndex();
    int highGlobIndx = rowMap->getMaxGlobalIndex();

    for (int mtxRowIndex=lowGlobIndx; mtxRowIndex<=highGlobIndx; mtxRowIndex++) {
      zoltanCoords->replaceGlobalValue(mtxRowIndex,0,1.0f);
      zoltanCoords->replaceGlobalValue(mtxRowIndex,1,1.0f);
      zoltanCoords->replaceGlobalValue(mtxRowIndex,2,1.0f);
    }

    Teuchos::ParameterList paramList;
  
     // set some params (not all shown)
     paramList.set("max levels", 4);
     paramList.set("smoother: type", "RELAXATION");
     paramList.sublist("smoother: params").set("relaxation: type", "Gauss-Seidel");
     paramList.set("aggregation: type", "uncoupled");
     paramList.set("repartition: enable", true);

     Teuchos::ParameterList userData = paramList.sublist("user data");
     userData.set("Coordinates",zoltanCoords);

    RCP<op_type> M = MueLu::CreateTpetraPreconditioner<SC, LO, GO, NO>(mA, paramList);

[cgcgcg]

Just a guess, but could you try this?

Teuchos::ParameterList& userData = paramList.sublist("user data");
userData.set("Coordinates",zoltanCoords);

[abodado]

@cgcgcg , thanks! Getting somewhere now.

terminate called after throwing an instance of 'Teuchos::Exceptions::InvalidParameterType'
what(): Error! An attempt was made to access parameter "Coordinates" of type "Tpetra::MultiVector<double,int,long long,Tpetra::KokkosCompat::KokkosDeviceWr apperNode<Kokkos::Serial, Kokkos::HostSpace>>*"
in the parameter (sub)list "ANONYMOUS->user data"
using the incorrect type "Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSp ace> > >"!

Throw number = 2

I thought this meant that I should wrap the Tpetra mv into an xpetra mv. I did so, but got this:

terminate called after throwing an instance of 'Teuchos::Exceptions::InvalidParameterType'
what(): Error! An attempt was made to access parameter "Coordinates" of type "Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >*"
in the parameter (sub)list "ANONYMOUS->user data"
using the incorrect type "Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >"!

Throw number = 2

I would appreciate any additional thoughts on this.

[cgcgcg]

Hm. Could you run this through a debugger? We should have code that wraps the Tpetra MV to Xpetra for you automatically:

Trilinos/packages/muelu/adapters/tpetra/MueLu_CreateTpetraPreconditioner.hpp

Lines 92 to 100 in 8bded3a

	if (userList.isParameter("Coordinates")) {
	RCP<Xpetra::MultiVector<typename Teuchos::ScalarTraits<Scalar>::coordinateType, LO, GO, NO> > coordinates = Teuchos::null;
	try {
	coordinates = TpetraMultiVector_To_XpetraMultiVector<typename Teuchos::ScalarTraits<Scalar>::coordinateType, LO, GO, NO>(userList.get<RCP<Tpetra::MultiVector<typename Teuchos::ScalarTraits<Scalar>::coordinateType, LocalOrdinal, GlobalOrdinal, Node> > >("Coordinates"));
	} catch (Teuchos::Exceptions::InvalidParameterType&) {
	coordinates = userList.get<RCP<Xpetra::MultiVector<typename Teuchos::ScalarTraits<Scalar>::coordinateType, LocalOrdinal, GlobalOrdinal, Node> > >("Coordinates");
	}
	userList.set<RCP<Xpetra::MultiVector<typename Teuchos::ScalarTraits<Scalar>::coordinateType, LO, GO, NO> > >("Coordinates", coordinates);
	}

[abodado]

@cgcgcg , Thanks! I see. So what I understand is that it is acceptable to pass in the Tpetra Multivector without wrapping beforehand. Somehow I just got everything to work. I did end up wrapping it to an Xpetra vector and made some other minor adjustments with RCP definitions. I'll play around with this a bit more and try to diagnose what went wrong initially. Thanks to you and @jhux2 for your help!
FYI, I must also say that adding the ampersand in this line was critical. I has missed that!
Teuchos::ParameterList& userData = paramList.sublist("user data");

[cgcgcg]

It looks like the coordinates were passed as a pointer to MV, not as an RCP<MV>?

[abodado]

Yes, I think that's correct.