*: split dumnati to commons, graph-builder, and policy-engine #19

zonggen · 2020-07-08T21:04:55Z

This changes splits dumnati to two binaries and a library crate. No content changes are added and only directory structure changes.

Related: #3
Signed-off-by: Allen Bai [email protected]

zonggen · 2020-07-08T21:10:23Z

The updated directory structure is:

.
├── commons
│   ├── Cargo.toml
│   └── src
│       ├── graph.rs
│       ├── lib.rs
│       ├── metadata.rs
│       └── metrics.rs
├── dumnati
│   ├── Cargo.toml
│   └── src
│       ├── graph.rs
│       ├── main.rs
│       ├── metadata.rs
│       ├── metrics.rs
│       ├── policy.rs
│       └── scraper.rs
├── graph-builder
│   ├── Cargo.toml
│   └── src
│       ├── main.rs
│       ├── policy.rs
│       └── scraper.rs
└──policy-engine
    ├── Cargo.toml
    └── src
        ├── main.rs
        ├── policy.rs
        └── scraper.rs

The dumnati directory remains untouched and its content has been split into commons, graph-builder and policy-engine. @lucab is this overall structure somehow reasonable?

lucab · 2020-07-09T10:03:56Z

@zonggen yes the hierarchy looks ok. I'd just call the two binaries fcos-graph-builder and fcos-policy-engine so that it makes it easier to slowly move in the other repository without impacting existing code upfront.

commons/Cargo.toml

graph-builder/src/main.rs

policy-engine/src/main.rs

lucab · 2020-07-09T11:43:49Z

By the way you'll need to bring #18 into this.

zonggen · 2020-07-09T20:43:53Z

rebased on top of master
renamed to fcos-graph-builder and fcos-policy-engine, also for the variables mentioned in previous reviews
moved scraper logic out of fcos-policy-engine so that it turns an empty graph now
re-adjusted structure a little bit by moving policy under ./commons. So now the structure looks like:

├── commons
│   ├── Cargo.toml
│   └── src
│       ├── graph.rs
│       ├── lib.rs
│       ├── metadata.rs
│       ├── policy.rs
│       └── metrics.rs
├── dumnati
│   ├── Cargo.toml
│   └── src
│       ├── graph.rs
│       ├── main.rs
│       ├── metadata.rs
│       ├── metrics.rs
│       ├── policy.rs
│       └── scraper.rs
├── fcos-graph-builder
│   ├── Cargo.toml
│   └── src
│       ├── main.rs
│       └── scraper.rs
└──fcos-policy-engine
    ├── Cargo.toml
    └── src
        └── main.rs

zonggen · 2020-07-09T21:01:22Z

Verify the behaviors:

fcos-graph-builder:

GET http://127.0.0.1:8080/v1/graph?basearch=x86_64&stream=stable

{
  "nodes": [
    {
      "version": "31.20200108.3.0",
      "metadata": {
        "org.fedoraproject.coreos.releases.age_index": "0",
        "org.fedoraproject.coreos.scheme": "checksum"
      },
      "payload": "113aa27efe1bbcf6324af7423f64ef7deb0acbf21b928faec84bf66a60a5c933"
    },
    {
      "version": "31.20200113.3.1",
      "metadata": {
        "org.fedoraproject.coreos.releases.age_index": "1",
        "org.fedoraproject.coreos.scheme": "checksum"
      },
      "payload": "f480038412cba26ab010d2cd5a09ddec736204a6e9faa8370edaa943cf33c932"
    },
    {
      "version": "31.20200118.3.0",
      "metadata": {
        "org.fedoraproject.coreos.releases.age_index": "2",
        "org.fedoraproject.coreos.scheme": "checksum"
      },
      "payload": "093f7da6ffa161ae1648a05be9c55f758258ab97b55c628bea5259f6ac6e370e"
    },
    {
      "version": "31.20200127.3.0",
      "metadata": {
        "org.fedoraproject.coreos.releases.age_index": "3",
        "org.fedoraproject.coreos.scheme": "checksum"
      },
      "payload": "2ba7bb9678d86fc469f1920a03c270d7057e0b93d06fcd1a437f1f79bdc26d83"
    },
    {
      "version": "31.20200210.3.0",
      "metadata": {
        "org.fedoraproject.coreos.releases.age_index": "4",
        "org.fedoraproject.coreos.scheme": "checksum"
      },
      "payload": "4ea6beed22d0adc4599452de85820f6e157ac1750e688d062bfedc765b193505"
    },
    {
      "version": "31.20200223.3.0",
      "metadata": {
        "org.fedoraproject.coreos.scheme": "checksum",
        "org.fedoraproject.coreos.releases.age_index": "5"
      },
      "payload": "3477082298eb5e2213433415c78f0919a991b40e62726b8fde434d244c1ec1b6"
    },
    {
      "version": "31.20200310.3.0",
      "metadata": {
        "org.fedoraproject.coreos.releases.age_index": "6",
        "org.fedoraproject.coreos.scheme": "checksum"
      },
      "payload": "436592e6eb93e899bebab8dbd17514c85be683390ef8bbce8c6d96069ce4c543"
    },
    {
      "version": "31.20200323.3.2",
      "metadata": {
        "org.fedoraproject.coreos.scheme": "checksum",
        "org.fedoraproject.coreos.releases.age_index": "7"
      },
      "payload": "9b322b9cead87ebf80881b75560e481b6e5a26966814b01bf0c12b61fa363708"
    },
    {
      "version": "31.20200407.3.0",
      "metadata": {
        "org.fedoraproject.coreos.scheme": "checksum",
        "org.fedoraproject.coreos.releases.age_index": "8"
      },
      "payload": "89e17cc21b6aa3bea8959d1e6957fda157168d57ba6805d8a36142184edc2901"
    },
    {
      "version": "31.20200420.3.0",
      "metadata": {
        "org.fedoraproject.coreos.scheme": "checksum",
        "org.fedoraproject.coreos.releases.age_index": "9"
      },
      "payload": "b3fc3a3e8513d7e424d0ced1e2517484cb766d238951f2fdec3da2fed3522efb"
    },
    {
      "version": "31.20200505.3.0",
      "metadata": {
        "org.fedoraproject.coreos.releases.age_index": "10",
        "org.fedoraproject.coreos.scheme": "checksum"
      },
      "payload": "01f074cc6cd88d8d2b43f821da692f2367c101eb4377802cb35092bde0ef02f7"
    },
    {
      "version": "31.20200517.3.0",
      "metadata": {
        "org.fedoraproject.coreos.releases.age_index": "11",
        "org.fedoraproject.coreos.scheme": "checksum"
      },
      "payload": "967b7b8d624e6d10ff51c2e81ef198fae966c567ac2e9b479771c693d0987949"
    },
    {
      "version": "32.20200601.3.0",
      "metadata": {
        "org.fedoraproject.coreos.scheme": "checksum",
        "org.fedoraproject.coreos.updates.rollout": "true",
        "org.fedoraproject.coreos.updates.start_value": "1",
        "org.fedoraproject.coreos.releases.age_index": "12"
      },
      "payload": "b51037798e93e5aae5123633fb596c80ddf30302b5110b0581900dbc5b2f0d24"
    },
    {
      "version": "32.20200615.3.0",
      "metadata": {
        "org.fedoraproject.coreos.updates.barrier_reason": "https://github.com/coreos/fedora-coreos-tracker/issues/484",
        "org.fedoraproject.coreos.updates.duration_minutes": "4320",
        "org.fedoraproject.coreos.updates.rollout": "true",
        "org.fedoraproject.coreos.updates.start_value": "0",
        "org.fedoraproject.coreos.updates.barrier": "true",
        "org.fedoraproject.coreos.releases.age_index": "13",
        "org.fedoraproject.coreos.updates.start_epoch": "1593540000",
        "org.fedoraproject.coreos.scheme": "checksum"
      },
      "payload": "abb4b978999bd689ff36114e99c737be16026bc154a4428d4c84a340758c5b01"
    }
  ],
  "edges": [
    [
      0,
      13
    ],
    [
      1,
      13
    ],
    [
      2,
      13
    ],
    [
      3,
      13
    ],
    [
      4,
      13
    ],
    [
      5,
      13
    ],
    [
      6,
      13
    ],
    [
      7,
      13
    ],
    [
      8,
      13
    ],
    [
      9,
      13
    ],
    [
      10,
      13
    ],
    [
      11,
      13
    ],
    [
      12,
      13
    ],
    [
      0,
      12
    ],
    [
      1,
      12
    ],
    [
      2,
      12
    ],
    [
      3,
      12
    ],
    [
      4,
      12
    ],
    [
      5,
      12
    ],
    [
      6,
      12
    ],
    [
      7,
      12
    ],
    [
      8,
      12
    ],
    [
      9,
      12
    ],
    [
      10,
      12
    ],
    [
      11,
      12
    ]
  ]
}

GET http://127.0.0.1:9080/metrics

# HELP fcos_cincinnati_gb_scraper_graph_final_edges Number of edges in the cached graph, after processing
# TYPE fcos_cincinnati_gb_scraper_graph_final_edges gauge
fcos_cincinnati_gb_scraper_graph_final_edges{stream="next"} 9
fcos_cincinnati_gb_scraper_graph_final_edges{stream="stable"} 25
fcos_cincinnati_gb_scraper_graph_final_edges{stream="testing"} 29
# HELP fcos_cincinnati_gb_scraper_graph_final_releases Number of releases in the cached graph, after processing
# TYPE fcos_cincinnati_gb_scraper_graph_final_releases gauge
fcos_cincinnati_gb_scraper_graph_final_releases{stream="next"} 10
fcos_cincinnati_gb_scraper_graph_final_releases{stream="stable"} 14
fcos_cincinnati_gb_scraper_graph_final_releases{stream="testing"} 30
# HELP fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp UTC timestamp of last graph refresh
# TYPE fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp gauge
fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp{stream="next"} 1594327749
fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp{stream="stable"} 1594327749
fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp{stream="testing"} 1594327749
# HELP fcos_cincinnati_gb_scraper_upstream_scrapes_total Total number of upstream scrapes
# TYPE fcos_cincinnati_gb_scraper_upstream_scrapes_total counter
fcos_cincinnati_gb_scraper_upstream_scrapes_total{stream="next"} 8
fcos_cincinnati_gb_scraper_upstream_scrapes_total{stream="stable"} 8
fcos_cincinnati_gb_scraper_upstream_scrapes_total{stream="testing"} 8
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1594327539

fcos-policy-engine:

GET http://127.0.0.1:8081/v1/graph?basearch=x86_64&stream=stable

{
  "nodes": [],
  "edges": []
}

GET http://127.0.0.1:9081/metrics

# HELP fcos_cincinnati_pe_v1_graph_incoming_requests_total Total number of incoming HTTP client request to /v1/graph
# TYPE fcos_cincinnati_pe_v1_graph_incoming_requests_total counter
fcos_cincinnati_pe_v1_graph_incoming_requests_total 1
# HELP fcos_cincinnati_pe_v1_graph_rollout_wariness Per-request rollout wariness.
# TYPE fcos_cincinnati_pe_v1_graph_rollout_wariness histogram
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0"} 0
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.1"} 0
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.2"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.30000000000000004"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.4"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.5"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.6000000000000001"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.7000000000000001"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.8"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.9"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="1"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="+Inf"} 1
fcos_cincinnati_pe_v1_graph_rollout_wariness_sum 0.18848315746051456
fcos_cincinnati_pe_v1_graph_rollout_wariness_count 1
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1594328133

, which look sane to me.

lucab · 2020-07-10T07:47:29Z

Thanks! Both output and patch looks fine. Travis is red due to a few minor linting errors. Other than that, this should be ok to land.

lucab · 2020-07-14T08:36:08Z

@zonggen while rebasing, you may also want to include #20 here.

zonggen · 2020-07-14T13:33:57Z

Re-based on top of master.
Also adjust according to Travis CI https://travis-ci.com/github/zonggen/fedora-coreos-cincinnati/jobs/360895796 where failing at cargo clippy -- -D warnings

This changes splits dumnati to two binaries and a library crate. No content changes are added and only directory structure changes. Makes fcos-policy-engine a dummy crate for now and adds additional logic with follow-up PRs. Related: coreos#3 Signed-off-by: Allen Bai <[email protected]>

lucab

LGTM

lucab reviewed Jul 9, 2020

View reviewed changes

zonggen force-pushed the split-dumnati-binary branch from d3e77b6 to 2e64c82 Compare July 9, 2020 20:33

zonggen force-pushed the split-dumnati-binary branch from 2e64c82 to 1548e5b Compare July 9, 2020 21:04

zonggen changed the title ~~[WIP] *: split dumnati to commons, graph-builder, and policy-engine~~ *: split dumnati to commons, graph-builder, and policy-engine Jul 9, 2020

zonggen force-pushed the split-dumnati-binary branch from 1548e5b to 9e0ddd9 Compare July 14, 2020 13:33

zonggen force-pushed the split-dumnati-binary branch from 9e0ddd9 to 2438a73 Compare July 14, 2020 14:25

lucab approved these changes Jul 14, 2020

View reviewed changes

zonggen merged commit 3bdc9db into coreos:master Jul 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: split dumnati to commons, graph-builder, and policy-engine #19

*: split dumnati to commons, graph-builder, and policy-engine #19

zonggen commented Jul 8, 2020

zonggen commented Jul 8, 2020

lucab commented Jul 9, 2020

lucab commented Jul 9, 2020

zonggen commented Jul 9, 2020

zonggen commented Jul 9, 2020 •

edited

Loading

lucab commented Jul 10, 2020 •

edited

Loading

lucab commented Jul 14, 2020

zonggen commented Jul 14, 2020 •

edited

Loading

lucab left a comment

*: split dumnati to commons, graph-builder, and policy-engine #19

*: split dumnati to commons, graph-builder, and policy-engine #19

Conversation

zonggen commented Jul 8, 2020

zonggen commented Jul 8, 2020

lucab commented Jul 9, 2020

lucab commented Jul 9, 2020

zonggen commented Jul 9, 2020

zonggen commented Jul 9, 2020 • edited Loading

fcos-graph-builder:

fcos-policy-engine:

lucab commented Jul 10, 2020 • edited Loading

lucab commented Jul 14, 2020

zonggen commented Jul 14, 2020 • edited Loading

lucab left a comment

Choose a reason for hiding this comment

zonggen commented Jul 9, 2020 •

edited

Loading

lucab commented Jul 10, 2020 •

edited

Loading

zonggen commented Jul 14, 2020 •

edited

Loading