3rd API proposal

[mvachhar]

API proposal combining ideas from proposal 1 and 2 and simplifying everything to a single Peering object between VPCs.

[Fredi-raspall]

I don't understand this statement. In the general case where there's address
overlapping (general is common here), you need to guarantee uniqueness. This is
not just about routing it's about identity. Assume 3 vpcs, vpc-1, vpc-2, vpc-3,
where vpc-2 and vpc-3 have overlapping spaces and vpc-1 peers with both. Each of
them exposes a DISTINCT service that vpc-1 needs to consume. The subnets of vpc-2
and vpc-3 have to be disambiguated by exposing a distinct "external" identities.
Otherwise the gateway will not know where to send a packet received from vpc-1.
Ofc if vpc-2 and vpc-3 expose the same service, and vpc-1 does not care which is picked
then fine. However, then we'll have to address the issue of selecting to which of them
the traffic is sent, which, without state would need to resort to some form of hash
over the packet. So:
- the uniqueness restriction is there (even if I believe it could be relaxed in some cases)
- I would consider this ECMP kind of case a more advanced one for later.

[mvachhar]

Your analysis is exactly correct, if two VPCs expose a different service using the same address, they'll get ECMP routing between the two and very confusing behavior. But, in order to support multiple externals exposing the same routes, and to treat externals just like VPCs, this should be the default behavior, IMHO.

We can later warn the user about the issue, but how do we really know if it is intentional or not? I'm open to suggestions to allow the multi-home multi-external case and to address your concern, but I thought we could deal with the restriction later since we have meaningful and useful behavior in the overlap case.

[Fredi-raspall]

If you have externals, the same applies. If the externals use private addressing,
you need to know the identity of them to distinguish them. If they are public,
agreed, they could have overlap. Here, we have a new problem, that stems from trying to model externals as
VPCs. In the fabric case, a VPC represents a set of unique, distinguishable destinations.
By modelling multiple externals as VPCs, you're trying to model something differnt with
the same abstraction. You're trying to model multiple "means to get to"
unique, distinguishable destinations, not the destinations themselves. Unsure if an
alternative should be modelling this as a distinct entity to which you can peer with / have a virtual link to.
In that particular case, agreed, ECMP is an option and you could tweak routing to
prefer one over the other.

[mvachhar]

So is your suggestion to treat overlap from externals differently than normal VPCs? We can do that and may be a good middle ground between the two solutions, though like you mentioned it is sometimes useful to even allow overlap between private VPCs.

[Fredi-raspall]

I don't understand the next example. Are we trying to provide transit
between the externals? ... or to the externals?
In the first case, this is just a plain IP router.

[mvachhar]

Yes, this is to configure a plain IP router in the case that gateway is used without fabric.

[Fredi-raspall]

The major issue I see with the above is that it mixes all together and breaks the multi-tenancy.

• you're specifying what ips can use the peering and how on the peering itself.
• if you have to peer with many other vpcs, you have to repeat that all around, for each peering you create.
• A model with a pif (don't call it like that, see it just as: 1) an indirection 2) anchor) allows VPCs independently choose what they expose to the outside, how and what
  internal entities can reach the outside and how, independently of what other vpcs choose to do, and be able to change that at will regardless of the rest.

To me:

• each VPC should be able to specify:
  - what sets of internal addresses can go outside
  - which of them can be reachable from the outside, in general
• If a vpc needs to "except the above", it can just create more "pifs" (reachable entities).
• Suppose a VPC hosted MS365. MS folks would just specify a single pif or the like (and not 10K) and then just specify which remote parties would be able to access it,
  regardless of anything else.
• Each side owns its thing.
• Then, if you want to allow somebody to reach that thing you create a peering / virtual-link / peering-policy. That just tells the gateway upon reception of a packet
  if the packet is allowed to be sent to the destination (MS365), how to get there, and enables the return path.
• The same applies for a "consuming-services" vpc. It may consume services from 10 distinct other vpcs. You'd
  • specify who in the vpc can consume the service, once.
  • then add links to the 10 services. Each link would allow each of the 10 service vpcs to reach you back.

[mvachhar]

The plan here was to allow reference to external objects to specify the various parts that get repeated to address this concern. In the simple no-reuse case I think it is not friendly to make people configure lots of objects just to link two VPCs which is why we simplified to this.

[Fredi-raspall]

I think we're running in circles. The more proposals I see, the more I like the original one. True, it had rough edges and stuff that was not defined. But it had good properties, and I don't think the rough edges justifies a complete amendment. Unsure if we should just stick to it in the parts that were clear. I think we should focus on that and move on. It' s not just about the time. A complete rewrite of the model is going to confuse people a lot. I'd propose enumerating the defects you believe it had and discuss them.

[mvachhar]

I think we're running in circles. The more proposals I see, the more I like the original one. True, it had rough edges and stuff that was not defined. But it had good properties, and I don't think the rough edges justifies a complete amendment. Unsure if we should just stick to it in the parts that were clear. I think we should focus on that and move on. It' s not just about the time. A complete rewrite of the model is going to confuse people a lot. I'd propose enumerating the defects you believe it had and discuss them.

The two main defects the original PIF proposal had are that apart from static NAT, it is not clear how to add firewall rules and such so that everything semantically makes sense. Plus, the original proposal's IP address restrictions meant that you could not use the same IPs to talk to an external and expose services using those IPs. Without those restrictions, you could no longer clearly define which PIF or peering policy would be in effect for a given packet.

I also found the whole producer/consumer thing very confusing, and I like that in your proposal 2 that you got rid of that stuff.

Once I realized that my proposal (proposal-1) had the same implied/explicit route overlap issue that the original proposal had, and that I objected to in your proposal, it made sense to move to something more like what you proposed, but we wanted to centralize the configuration so there is no confusion about what policy is applied when (apart from the ECMP routing, which we can discuss). So we dropped the PIFs.

I don't see the value in PIFs apart from config reuse, and we can achieve that config reuse in other ways without creating the impression that there is something more going on where we are "selecting" interfaces, etc. Moreover, we force users to configure 3 objects instead of 1, which is less convenient when there is no policy reuse. Let's discuss more today.

[mvachhar]

How much cleanup do we want before merge? Or do we merge and then have another PR with more concrete examples and work on clarifying exact syntax, etc.?

[Frostman]

How much cleanup do we want before merge? Or do we merge and then have another PR with more concrete examples and work on clarifying exact syntax, etc.?

@mvachhar I think we can probably mainly clean up right here, but it doesn't matter to me