Skip to content

2019PostMortem

Jump to bottom
Gerry Demaret edited this page Apr 7, 2019 · 4 revisions

FOSDEM 2019 video post mortem

Buildup

  • buildup was messy:
    • general understaffing on friday (vasil)
    • people not assigned beforehand (vasil)
    • people doing buildup not the same as the people responsible for the building or teardown (vasil)
    • had to do at least 2 passes (for example change of H's audio setup) (vasil)
    • should have better and communicated procedure for buildup (like labeling where cameras go, etc.) (vasil)
  • moving the flightcases
    • suggestion: attach some straps to pull the cases instead of bending over and pushing the cases (your back will thank you) (basti)
    • suggestion2: unload the flightcases from the van near the destination buildings instead of k (basti)
  • The time for the video setup was not enough for good check of everything. We've lost too much time just waiting our order to come. This can be fixed either with possibility for earlier start of the video setup, or more experienced people (kindly promised by Marian for the next year). (neter)
  • The sound check by 1-2 people in NOC during the setup is a bottleneck. I had (maybe) 2 rooms in H that was not sound checked during the setup because of that. This can be fixed either with more people in NOC and communication in IRC for the sound check (there was many moments during the setup when the radio was full with other talks), or some method the per-building team to make the sound check in place (isn't the check with headphones on the camera good enough?). (neter)
    • I actually did the sound check on camera in 2 rooms, but Delyan was not ok with the levels, even thou the sound on the headphones was fine. So it would be better to check the sound on the testing laptop from the stream, but we need to have a simple way to see the same thing, Delyan was watching. (marian)
      • And the fun part is, we have two types of outputs - camera and room, so both have to be ok and should be independent. With the remote controller mixers I plan to have two really independent outputs, just because of that, and also see to having one set of measurements/levels that we can try to attain. (vasil)
        • If you have the levels... I can make a few Raspberries with a small display that, when connected will show you the levels per channel. (marian)
          • I forgot to mention it with the headphones on the camera check (which is about the quality of the sound), but if the Grafana graphs are enough and we have them during the setup, they can be used in addition for levels check and it has good view on mobile screen, so with these two checks we will have full stack quality+levels checks and we need only headphones and a phone, without additional hardware. (neter)
  • Finally, I didn't know that some of the BG guys, that were helping last year, were not helping this year. If I knew, I would have trained and invited more video volunteers. We were lacking at least 5-6 experienced video guys. I'll make sure, this is not a problem next year! (marian)
  • The redistributed boxes were great. (luc)
  • Way too relaxed, buildup lasted until 21:00 which was late (luc)
  • Network in U was a disaster, not laid down to the proper positions, We also had no idea what cables we required to go with the videoboxes (xlr, extension cords). We ended up being way too rushed to record this information. (luc)
  • Perhaps next year, after the equipment is labelled, people are assigned to whole buildings (2 teams for U), so buildings can be done in parallel and not in succession. This also makes it easier to streamline the individual tasks of setting up a devroom. I assume that this was not done this year due to the fact that not as many people turned up as expected. (luc)
  • Network was delayed quite a lot in H building (vladimir)
    • Cables were pulled and crimped (vladimir)
    • Switch was there and powered up (vladimir)
    • No cables were connected (had to ask about 7 times untill somebody connected them) (vladimir)
      • Maybe a network connectivity dashboard so we can see such issues (with some alerting) (vladimir)
      • A standardized plan which port is for what (e.g. por 1-2 UP, port 3-16 WIFI, port 17-32 VIDEO) (vladimir)
  • If possible start dropping video equipment early (will also help network team to know where to pull cables) (vladimir)
  • Voctops with dhcp for in room checking before network is really up (vladimir)
    • e.g. drop boxes cam etc, connect voctop and test if stuff is fine in place (vladimir)
    • Be damn sure that those voctop's are reconfigured the moment they are not needed or network will kill us (vladimir)
  • Was nice to have a list per box what is where, but maybe some more logic in it (or maybe i missed it). (vladimir)
    • Related boxes should be sequentially numbered (vladimir)
    • Boxes per building should also be sequentially numbered (vladimir)

Teardown

  • teardown was messy:
    • maybe we forgot the lamps in H (vasil)
    • van with video equipment delayed us (vasil)
  • we need better organization of the budgetcam check (vasil)
  • Teardown of the K building was great, handled by graphics devroom people (luc)

Audio

  • this "exporting the internal mic to end users" was a horrible thing (vasil)
  • two mics == in-room mixer, that wasn't communicated properly beforehand (vasil)
  • ULB's audio still needs improvements, although only the big rooms were left (vasil)
  • we need to plan audio to have a single auto-gain stage, hopefilly the sennheisers (vasil)
  • tie pin mics & PA system don't play well, we should have either headsets or feedback destroyer (vasil)
  • some devroom managers played a lot with the audio mixers (vasil)
  • multiple rooms (mostly in U) had a noisy PA (vasil)
  • some day we need to look into noise detection (vasil)
  • the handheld mics didn't have MUTE, which caused a lot of loop problems in U building. (marian)
  • speaker voice not caught due to muting / not powering up the devices (basti)
    • suggestion: tape the mute button so it can't be used (basti)
    • suggestion2: do not use mics that have a mute button (like the handhelds) (basti)
    • suggestion3: instructions to devroom managers for how to power up everything (including instructions to speakers to actually wear the microphones as some didn't ...) (basti)
    • suggestion4: show/flash icon on the cam/slide box that shows bad condition (like no sound from mic...) (basti)
  • the tie mics pic too little if the gain is not high, and at that point they hear the speakers from the PA systems. It was a lot of playing around to get the correct combination. (marian)
  • ambient sounds way to high in some rooms (basti)
    • suggestion: turn off auto setting for internal camera and set a manual level that does not catch every door open/close, cough and so on (basti)
    • suggestion2: turn off the internal mic and use only the handheld mics (dont' know how much this will cost) (basti)
    • suggestion3: split audio metrics per channel (L/R) so we can alert, graph and see too much deviation between channels (basti)
    • suggestion4: add room/building label to the audio metrics (for building dashboards per building ;) ) (basti)
    • suggestion5: listen to the audio then just relying on metrics on screen (basti)

People

  • coordination/monitoring
    • not enough people, no very clear tasks (vasil)
    • irc isn't the best option, should look into matrix/rocketchat (vasil)
    • no single dashboard with all status (vasil)
    • A lot of new tools were available (dashboards etc) but not communicated (vladimir)
    • No central place where such info can be found (vladimir)
    • Access to some restricted resources - HOW? (vladimir)
  • volunteers in general
    • more (vasil)
    • more trained ones (vasil)
    • maybe more events through the year, in BG and BE (vasil)
    • usage of volunteers.fosdem.org (vasil)
  • paulk may be able to help with setting up on Friday (luc)
  • Volunteers came and left too quickly; once we had figured out how to set mixers up quickly, after the second devroom, the volunteer person said he was out of time and left. This put us back on square one. (luc)

Rooms

  • H.1301 leaked (vasil)
  • UB5.132 was too far, if we'll be using rooms there, maybe we need a third team in U (vasil)
  • under the stairs of AW "rained" (so a tarp was suggested for next year) (vasil)
  • some rooms had issues with the projector turning off/switching sources by itself (vasil)
  • no proper procedure for moving rooms (vasil)
  • an extra building is expected, so floorplans will be essential (vasil)
    • if that leads to dropping one of the current buildings, it'd be great, but probably too much to hope for (vasil)
  • low light conditions on cam videos (basti)
    • suggestion: maybe some dedicated lighting from above (on an autopole for example) to light up the speaker(s) - could use powersocket with the slides box
  • UD2118a had its projector constantly switching the input source. (marian)
  • UD2120 had so much echo, that the tie mic was unusable :( (marian)
  • UB2152(Lamere) had a lot of noise on the speakers (marian)
  • UA2220 had buzz in the speakers, when the lights were on. (marian)
  • UA2114 I don't know if it was the mixer or the receivers of the mics there, but just a gentle touch on the receiver either fixed or created a lot of buzz in the audio. I had to tape them a little bit in order to remove the buzz for one talk, but on the next, someone had already touched them and the buzz was back :( (marian)
  • UB5142 was way past reachable. I mean, that the time from either UA or UD video checkpoint is around 10-12min. We were happy that there were no issues with that room, but if there were any issues, we would have lost the whole talk that is affected. (marian)
  • From a devroom manager pov, all was amazing. Apart from one issue with the scaler/splitter playing up during one talk, we had no hiccups. (luc)
  • It is becomming a recurring topic that devroom managers are generally undereducated and tend to play with stuff they must not (vladimir)
    • What about some prerecorded general training and instrunctions what to do and more importantly NOT to do (vladimir)
  • In H and U we generally want some way to box out the equipment (e.g. do not cross line) (vladimir)

Monitoring

  • monitoring's updates were very helpful (vasil)
  • TCP works (vasil)
    • except when there was no timeout and weird circumstances broke it (vasil)
      • led to loss of streamdumps (vasil)
      • and some local recordings (vasil)
  • no proper way of checking the streams from the FOSDEM wifi suggestion: run some cable from the video vlan to the spots where the building teams are placed and plug laptops to that lan (basti)

Review

  • Review went smooth as well for us. (luc)
  • find people to help Wouter, as it seems insane for him to handle all review fallout (vasil)
  • audio desync issues on reviewed videos: audio is in sync in the source files, but not in the reviewed videos (gerry)

Ideas for next year

  • put laptops in NOC, use their displays for monitoring (vasil)
  • remote-controlled audio mixers (the experiment was successful) (vasil)
    • (bonus) Yeah, remotely controlled audio mixer will be so good. Almost every lecturer has his own best sound level and some rooms are constantly overcrowded and very often you need a complex gymnastics to fix this. (neter)
  • amps & speakers at least for some rooms (vasil)
    • I know that Vasil has told you about my setup with the JBL EON208P, but keep in mind that the amplifier is not very powerful and the whole set is not cheap at all :( (marian)
  • The DevRoom managers played too much with the mixers and in a few rooms with the video cables. Maybe, we should consider locking at least the video cables to the boxes :( (marian)

Extra hardware

Clone this wiki locally