Skip to content

03.08.2021 TT Agenda

Jump to bottom
Nic edited this page Mar 8, 2021 · 3 revisions

People

@nniicc @adam3smith @stahs @qqmyers

Discussion Topics

Admin

  • TAB meeting was good - longer discussion of DR recovery and backup
  • Its Reserved Instance time of the year - SK check with SO on current needs

Seba & Jim Coordination

  • Plan for automated cleaning of older files in /tmp (which tool, who does the work)
  • Should stage be updated to match dev (with ~5.4 changes) or just style fixes that could go to prod ~now?
  • discuss DataCite 4.x support (question from Sebastian)

Seba

  • Item

Jim

  • Report '
  • AnnoRep Update from SK (keeping Jim in loop)

Assigned Tasks / Decisions

  • SK - Reserved instances renewal - Same as 2020
  • SO - Look into version control on S3 buckets
  • SO - Look into /tmp tools for purging so we don't run out of disc
  • SK + NW - find out date for downtime so Seba can prep EBS
  • JM - Put ~5.4 on stage along with style change issues

Notes

Seba work accomplished:

  • Seba adjusted scripts to send backups to ICT + Verified (this is including DV databases from S3). Uses similar logic of duply we had originally, just updated.

  • ICT backups are verified and working as of Sunday 03.07

  • This will be documented in backup page on Infrastructre docs in Github...

  • The backup ICT vpn set up is non-trivial for Seba locally

  • And the backups are not as accessible for entire team - but we CAN get them ... Will work going forward on making these more accessible and documenting this ...

Renew reserved instances ...

  • Should we just renew blankly or is there anything to adjust
  • If you buy RI of an instance class - the size doesn't matter that much (C5, XLarge or 2Xlarge...)
  • Current -T1 and T3 are burstable and cheap ...
    • Metrics we are nowhere near performance bottlenecks ... There is nothing cheaper either so we're good.

Temp repo cleaning

  • What should we use in order to do these purges
  • And who should hold the bag...
    • Production usage went up to 74% ... So not immediate because we can manually delete...
    • When does storage alert kick in? 80%

EBS situation...

  • Combining EBS volumes are the next step
  • Let's discuss when can we take downtime - Probably around 30 min ...
  • Cost of backups ... Writing out full s3 copies or are we using duply? There is no change to Duply config for ICT backups...
    • For S3 Dataverse backup - there is no increase or movement in cost (that we observed)
    • DR backups ... It's a pool from S3 that we are going to use...

Is there any S3 built in backup that we can use to do backup in addition to ICT ...

  • Having a warmish back up on ICT would add layer of security
  • Rsync style backup would be helpful there...

Jim - seven 9's worth of backup - and there is versioning that you can put that into place - all of our files are write once

SK - is there an ability to perma-delete (for use case when we download, delete, and reupload)

Are we paying for additional S3 storage ... ?

  • We don't think so - but we should check on the version control for the bucket
Clone this wiki locally