backlog/git-lite

[bwalsh]

Epic

As a release manager, I need a feature tracking system to identify and prioritize missing features with real value for the users.

Missing / incomplete features

LOE: level of effort

easy:

• rm remove a file - we can do it via utilities file rm However, this is not really "git like" LOE: easy
• some commands depend on server [init, utilities access sign status] we should immediately log "working" to console (in case server is slow) LOE: easy
• log show commit log - we can do it via utilities file ls --is_metadata However, this is not really "git like" LOE:easy
• diff - no equivalent LOE:moderate ✅

moderate:

• pull "some" - currently pull retrieves all the files, need to add some ability to pull only some files based on some parameters ✅ TBD: e.g.

  • pull
  • pull -object_ids
  • pull -patient_ids
    • -specimen_ids

• reset commit - no equivalent LOE:moderate

  • remove 1 or more commit records from completed, save in pending
  • re-run push's publish step

hard:

• orphaned metadata analyse server database(s), flag orphaned records, allow user to remove them, or automatically purge them. LOE: hard

unknown:

• g3t status - when performed after push, each time user executes g3t status the status command will check k8s job command if job status in [Unknown, Running]. Once the job status changes, the response is cached and not checked again. However, if the user never executes g3t status for more than N minutes and then executes it. The job logs expires, we have no way of knowing status plus the gen3 client library logs a bunch of errata to the screen.
  @lbeckman314 : is there a workaround for this at the k8s level. i.e. long lived logs see here and k8s doc

[lbeckman314]

[ ] g3t status - when performed after push, each time user executes g3t status the status command will check k8s job command if job status in [Unknown, Running]. Once the job status changes, the response is cached and not checked again. However, if the user never executes g3t statu for more than N minutes and then executes it. The job logs expires, we have no way of knowing status plus the gen3 client library logs a bunch of errata to the screen.

@lbeckman314 : is there a workaround for this at the k8s level. i.e. long lived logs see here and k8s doc

Just updated Sower so that jobs are kept for 24 hours after completing. This can be changed by updating sower.sowerConfig.TTLSecondsAfterFinished:

sower:
  sowerConfig:
    - name: fhir-import-export
      action: fhir_import_export
      TTLSecondsAfterFinished: 86400   <--- Job TTL (24 hours here, 1 hour if not specified)

Open Sower PR:

uc-cdis/sower#42

[bwalsh]

@lbeckman314 As I recall the TTLSecondsAfterFinished didn't work?

[bwalsh]

orphaned metadata:

• Identify any files in indexd that do not have DocumentReference node
• Identify DocumentReference that do not have indexd node
• Identify bucket entries that do not have indexd node