Accept CEP 002

[kosack]

Please use this PR to comment on a proposed ctapipe enhancement proposal (CEP): CEP-002: change the internal event structure.

Recall that ctapipe releases before 1.0.0 are subject to small and large changes in the API, and this is one such large change that has been discussed for several years now. Note that this affects any code that uses ctapipe, and therefore adopting it will require some changes to local code and scripts. This CEP has already been internally discussed in #2304, and is now open for public comment.

Please read the proposal for CEP-002 in the ctapipe docs and give constructive comments here if anything is ambiguous or missing.

Details about what is a CEP and how we incorporate them can be found in the documentation as CEP-001

[maxnoe]

Docs failure is fixed in main, please rebase

[mexanick]

Can you please clarify how the events will be indexed and connected?
In the structure below, are the SubarrayEventIndex and TelescopeEventIndex independent? If yes, how the telescope-specific data shall be gathered to form a Subarray event, through tel:Map[...]?

SubarrayEvent
- index: SubarrayEventIndex
- simulation: SimulatedShower
- dl0: DL0Subarray
- dl1: DL1Subarray
- dl2: DL2Subarray
- dl3: DL3Subarray
- tel: Map[tel_id -> TelescopeEvent]

TelescopeEvent
- index: TelescopeEventIndex
- simulation: TelescopeSimulation
- r0: R0Telescope
- r1: R1Telescope
- dl0: DL0Telescope
- dl1: DL1Telescope
- dl2: DL2Telescope
- dl3: DL3Telescope

Also, with the new structure, will ctapipe-merge tool support merging of DL1 events (especially DL1a, required for Deep Learning models)?

[maxnoe]

In the structure below, are the SubarrayEventIndex and TelescopeEventIndex independent?

I don't understand whay you mean with "independent" here.

These two containers contain the "primary key"
identifying the subarray or telescope event respectively, i.e. obs_id / event_id for SubarrayIndexContainer and obs_id, event_id, tel_id for TelescopeIndexContainer.

If yes, how the telescope-specific data shall be gathered to form a Subarray event, through tel:Map[...]?

The key of .tel is the telescope id, as before and written in the snippet you posted.

Also, with the new structure, will ctapipe-merge tool support merging of DL1 events (especially DL1a, required for Deep Learning models)?

This CEP is concerned with the in-memory structure of the data (ctapipe containers), not the file structure.
So your question is independent of this CEP, but yes, we do plan to implement merging files per multiple telescopes into one file for the whole subarray, as this will also be how ACADA will take actual CTA data.

[maxnoe]

If we want to merge this now, @kosack needs to remove the "draft" version of it we only kept so that the diff here renders nicely and not just "file renamed without changes".

[maxnoe]

As discussed in today's meeting, I removed the CEP 3 from this PR and will open an individual PR for this.

So I think this is ready to be merged to formalize the acceptance.