Question about graphql::Outlier defined in src/graphql/outlier.rs

[minshao]

The following code is from src/graphql/outlier.rs

#[derive(Debug, Deserialize, SimpleObject)]
#[graphql(complex)]
pub(super) struct Outlier {
    #[graphql(skip)]
    pub(super) id: i64, //timestamp
    #[graphql(skip)]
    pub(super) events: Vec<i64>,
    pub(super) size: i64,
    pub(super) model_id: i32,
}

pub trait FromKeys: Sized {
    /// Creates a new instance from the given keys.
    ///
    /// # Errors
    ///
    /// Returns an error if the key or value cannot be deserialized.
    fn from_keys(keys: &[Box<[u8]>]) -> Result<Self>;
}

impl FromKeys for Outlier {
    fn from_keys(keys: &[Box<[u8]>]) -> Result<Self> {
        let size = i64::try_from(keys.len())?;
        let mut events: Vec<i64> = Vec::new();
        let keys: Vec<_> = keys.iter().take(DEFAULT_OUTLIER_SIZE).collect();

        let Some(first_key) = keys.first() else {
            return Err(anyhow!("Failed to read outlier's first key").into());
        };
        let (model_id, timestamp, _, event, _) =
            bincode::DefaultOptions::new().deserialize::<RankedOutlierKey>(first_key)?;
        events.push(event);

        for key in keys {
            let (_, _, _, event, _) =
                bincode::DefaultOptions::new().deserialize::<RankedOutlierKey>(key)?;
            events.push(event);
        }
        events.sort_unstable();
        Ok(Self {
            id: timestamp,
            events,
            size,
            model_id,
        })
    }
}

It seems that the purpose of the Outlier definition is to showcase the id along with the associated events it represents. However, the from_keys function appears to be attempting something different:

1. It assigns the timestamp as the id: In the RankedOutlier key, the timestamp denotes the time when the outlier is generated, not the specific ID of the outlier. Since all outliers affected within the same Batch share the same timestamp, it essentially becomes a part of the batch information. The actual ID of the outlier is the third value in the tuple extracted from the key, referred to as event in from_keys.
2. It uses the outlier id as the event represented by the outlier: As clarified earlier, the third value in the tuple signifies the id of the outlier. Combining model_id with id uniquely identifies an outlier. The events associated with that outlier are currently stored in the PostgresQL database.

I find this a bit perplexing and am wondering whether I've overlooked something.

[syncpark]

@kimhanbeom Do you remember the id related things of outlier that @minshao said above?
@minshao I've talked about this outlier structure with @kimhanbeom and @MW-Kim several months ago. After some more talks about your question, we will add comment here.

[minshao]

Thanks for the help!

[syncpark]

In review: src/agent/request.rs: Outlier events are stored into outlier map like following.

fn update_outliers(
    store: &Store,
    model_id: i32,
    timestamp: i64,
    outliers: &[OutlierInfo],
) -> Result<()> {
    let map = store.outlier_map();

    for outlier in outliers {
        let key = bincode::DefaultOptions::new().serialize(&(
            model_id,
            timestamp,
            outlier.rank,
            outlier.id,
            &outlier.source,
        ))?;
        let value = bincode::DefaultOptions::new().serialize(&(outlier.distance, DEFAULT_SAVED))?;
        map.insert(&key, &value)?;
    }
    Ok(())
}

in review-web/src/graphql/outlier.rs: RankedOutlierKey is produced from Outlier event like following.

type RankedOutlierKey = (i32, i64, i64, i64, String);  // model_id, timestamp, rank, id, source
impl From<PreserveOutliersInput> for RankedOutlierKey {
    fn from(input: PreserveOutliersInput) -> Self {
        (
            input.model_id,
            input.timestamp,
            input.rank,
            input.id,
            input.source,
        )
    }
}

type RankedOutlierKey = (i32, i64, i64, i64, String); => The order is model_id, timestamp, rank, id, source.

The outlier event retrieved from the outlier map have the key of tuple RankedOutlierKey.
The first model_id and the second timestamp is the unique key for the outliers generated in a round.

The code impl FromKeys for Outlier { ... have no problem. But It confuses us little, I'd like to change the type name RankedOutlierKey to OutlierKey.

[minshao]

Can you help me understand the meaning of Outlier and the meaning of its field: id, events, size? By "meaning", I'm referring to how we would explain the data to the user, i.e. what they represent.

[syncpark]

Can you help me understand the meaning of Outlier and the meaning of its field: id, events, size? By "meaning", I'm referring to how we would explain the data to the user, i.e. what they represent.

The outlier events are used in our Web UI and also 3-rd party UIs too. In our Web UI, outlier events are used in two ways.

1. Firstly, outlier events are displayed to describe a round of a model beside of model and clusters. In this UI, the outlier events are displayed to show it's content to the user. Users will gain knowledge about what type of outlier events was generated in the latest round.
   

2. Secondly, there is RankedOutlier UI. In the UI, the outlier events can be filtered or searched in order by distance values, and users can select some of the outlier events and add descriptions for the other use. We wanted to give users an idea of ​​which outlier events to see first.

The Outlier structure is used in above 1., and RankedOutlier structure is used in above 2.

---

The fields of Outlier structure:

• id field of Outlier structure is used to identify outlier events generated in same round.
• events field contains the timestamp of raw events and it's used to retrieve the raw events from the Giganto.
• size is for total count of outlier events of the round. This value is displayed in the Web UI too.

[minshao]

Thanks for the explanation, it helps a lot!

p.s. I'd like to mention that RankedOutlierKey::source is also needed to fetch raw event from Giganto. (If settings in Giganto hasn't been changed, that is.)

[syncpark]

Yes you're right.

Although outlier events are currently handled together with the model, a separate source is not specified, but since there are cases where they are linked with 3-rd party software, the source field is likely to be necessary.