Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new-log-viewer: Add support for exporting decoded logs as a file. #72

Merged
merged 26 commits into from
Sep 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
b127bbf
Export Logs draft
Henry8192 Sep 15, 2024
b9ab9dd
Apply suggestions from code review
Henry8192 Sep 16, 2024
da3f0be
Move blob & numChunks to LogExportManager; Move downloadDecompressedL…
Henry8192 Sep 16, 2024
5abb640
Download logs can only be downloaded once; Need to check corner case …
Henry8192 Sep 17, 2024
4b5c276
logs can be downloaded multiple times
Henry8192 Sep 17, 2024
20d98a6
plausible fix for empty logs
Henry8192 Sep 17, 2024
f6023cf
address comments in code review
Henry8192 Sep 17, 2024
7bf9432
Apply suggestions from code review
Henry8192 Sep 17, 2024
73bc71f
address changes from code review
Henry8192 Sep 17, 2024
406b76c
Introduce 8px padding to .menu-bar for better spacing. Add .menu-bar-…
Henry8192 Sep 18, 2024
27659ff
change exportedFileName format; address two exports in ExportLogsButt…
Henry8192 Sep 18, 2024
a69e3d0
Apply suggestions from code review
Henry8192 Sep 18, 2024
8bcf115
fix issues in review
Henry8192 Sep 18, 2024
e2cc493
fix lint warnings; address the remaining issues in code review
Henry8192 Sep 18, 2024
c105eaf
Merge branch 'main' into export-logs
junhaoliao Sep 18, 2024
69ca0b7
Minor comment fixes.
kirkrodrigues Sep 19, 2024
0744f99
loadChunk: Refactor docstring and error.
kirkrodrigues Sep 19, 2024
bc4d463
StateContextProvider: Group states and refs; Reorder exportProgress s…
kirkrodrigues Sep 19, 2024
6b51ef9
Clear export progress when a new file is loaded.
kirkrodrigues Sep 19, 2024
5e5d24d
loadChunk: Apply renaming suggestions from review.
kirkrodrigues Sep 19, 2024
73bd8fa
Address suggestions from code review
Henry8192 Sep 19, 2024
c34ed9b
suppress todo warnings
Henry8192 Sep 19, 2024
1e99e07
Use EXPORT_LOGS_PROGRESS_VALUE_MIN instead of STATE_DEFAULT.exportPro…
kirkrodrigues Sep 19, 2024
623b9ed
Edit TODO about empty fileName check.
kirkrodrigues Sep 19, 2024
7d2051d
Revert "Use EXPORT_LOGS_PROGRESS_VALUE_MIN instead of STATE_DEFAULT.e…
kirkrodrigues Sep 20, 2024
f1699d6
Reset export progress before loading file.
kirkrodrigues Sep 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions new-log-viewer/src/components/MenuBar/ExportLogsButton.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
import {useContext} from "react";

import {
CircularProgress,
Typography,
} from "@mui/joy";

import DownloadIcon from "@mui/icons-material/Download";

import {StateContext} from "../../contexts/StateContextProvider";
import {
EXPORT_LOG_PROGRESS_VALUE_MAX,
EXPORT_LOG_PROGRESS_VALUE_MIN,
} from "../../services/LogExportManager";
import SmallIconButton from "./SmallIconButton";


/**
* Represents a button for triggering log exports and displays the progress.
*
* @return
*/
const ExportLogsButton = () => {
const {exportLogs, exportProgress, fileName} = useContext(StateContext);

return (
<SmallIconButton
disabled={
// eslint-disable-next-line no-warning-comments
// TODO: Replace `"" === fileName` with a more specific context variable that
// indicates whether the file has been loaded.
(null !== exportProgress && EXPORT_LOG_PROGRESS_VALUE_MAX !== exportProgress) ||
"" === fileName
Henry8192 marked this conversation as resolved.
Show resolved Hide resolved
}
onClick={exportLogs}
>
{null === exportProgress || EXPORT_LOG_PROGRESS_VALUE_MIN === exportProgress ?
<DownloadIcon/> :
<CircularProgress
determinate={true}
thickness={3}
value={exportProgress * 100}
variant={"solid"}
color={EXPORT_LOG_PROGRESS_VALUE_MAX === exportProgress ?
"success" :
"primary"}
>
{EXPORT_LOG_PROGRESS_VALUE_MAX === exportProgress ?
<DownloadIcon
color={"success"}
sx={{fontSize: "14px"}}/> :
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
<Typography level={"body-xs"}>
{Math.ceil(exportProgress * 100)}
</Typography>}
</CircularProgress>}
</SmallIconButton>
);
};

export default ExportLogsButton;
10 changes: 6 additions & 4 deletions new-log-viewer/src/components/MenuBar/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,15 @@ import {
Typography,
} from "@mui/joy";

import Description from "@mui/icons-material/Description";
import DescriptionIcon from "@mui/icons-material/Description";
import FileOpenIcon from "@mui/icons-material/FileOpen";
import Settings from "@mui/icons-material/Settings";
import SettingsIcon from "@mui/icons-material/Settings";

import {StateContext} from "../../contexts/StateContextProvider";
import {CURSOR_CODE} from "../../typings/worker";
import {openFile} from "../../utils/file";
import SettingsModal from "../modals/SettingsModal";
import ExportLogsButton from "./ExportLogsButton";
import NavigationBar from "./NavigationBar";
import SmallIconButton from "./SmallIconButton";

Expand Down Expand Up @@ -58,7 +59,7 @@ const MenuBar = () => {
flexGrow={1}
gap={0.5}
>
<Description/>
<DescriptionIcon/>
<Typography level={"body-md"}>
{fileName}
</Typography>
Expand All @@ -74,8 +75,9 @@ const MenuBar = () => {
<SmallIconButton
onClick={handleSettingsModalOpen}
>
<Settings/>
<SettingsIcon/>
</SmallIconButton>
<ExportLogsButton/>
</Sheet>
<SettingsModal
isOpen={isSettingsModalOpen}
Expand Down
84 changes: 69 additions & 15 deletions new-log-viewer/src/contexts/StateContextProvider.tsx
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
/* eslint max-lines: ["error", 400] */
import React, {
createContext,
useCallback,
Expand All @@ -7,6 +8,7 @@ import React, {
useState,
} from "react";

import LogExportManager, {EXPORT_LOG_PROGRESS_VALUE_MIN} from "../services/LogExportManager";
import {Nullable} from "../typings/common";
import {CONFIG_KEY} from "../typings/config";
import {SEARCH_PARAM_NAMES} from "../typings/url";
Expand All @@ -20,7 +22,10 @@ import {
WORKER_RESP_CODE,
WorkerReq,
} from "../typings/worker";
import {getConfig} from "../utils/config";
import {
EXPORT_LOGS_CHUNK_SIZE,
getConfig,
} from "../utils/config";
import {
clamp,
getChunkNum,
Expand All @@ -37,12 +42,15 @@ import {
interface StateContextType {
beginLineNumToLogEventNum: BeginLineNumToLogEventNumMap,
fileName: string,
loadFile: (fileSrc: FileSrcType, cursor: CursorType) => void,
loadPage: (newPageNum: number) => void,
exportProgress: Nullable<number>,
logData: string,
numEvents: number,
numPages: number,
pageNum: Nullable<number>
pageNum: Nullable<number>,

exportLogs: () => void,
loadFile: (fileSrc: FileSrcType, cursor: CursorType) => void,
loadPage: (newPageNum: number) => void,
}
const StateContext = createContext<StateContextType>({} as StateContextType);

Expand All @@ -51,13 +59,16 @@ const StateContext = createContext<StateContextType>({} as StateContextType);
*/
const STATE_DEFAULT: Readonly<StateContextType> = Object.freeze({
beginLineNumToLogEventNum: new Map<number, number>(),
exportProgress: null,
fileName: "",
loadFile: () => null,
loadPage: () => null,
logData: "Loading...",
numEvents: 0,
numPages: 0,
pageNum: 0,

exportLogs: () => null,
loadFile: () => null,
loadPage: () => null,
});

interface StateContextProviderProps {
Expand Down Expand Up @@ -130,44 +141,82 @@ const workerPostReq = <T extends WORKER_REQ_CODE>(
const StateContextProvider = ({children}: StateContextProviderProps) => {
const {filePath, logEventNum} = useContext(UrlContext);

// States
const [fileName, setFileName] = useState<string>(STATE_DEFAULT.fileName);
const [logData, setLogData] = useState<string>(STATE_DEFAULT.logData);
const [numEvents, setNumEvents] = useState<number>(STATE_DEFAULT.numEvents);
const beginLineNumToLogEventNumRef =
useRef<BeginLineNumToLogEventNumMap>(STATE_DEFAULT.beginLineNumToLogEventNum);
const [exportProgress, setExportProgress] =
useState<Nullable<number>>(STATE_DEFAULT.exportProgress);

// Refs
const logEventNumRef = useRef(logEventNum);
const numPagesRef = useRef<number>(STATE_DEFAULT.numPages);
const pageNumRef = useRef<Nullable<number>>(STATE_DEFAULT.pageNum);

const logExportManagerRef = useRef<null|LogExportManager>(null);
const mainWorkerRef = useRef<null|Worker>(null);

const handleMainWorkerResp = useCallback((ev: MessageEvent<MainWorkerRespMessage>) => {
const {code, args} = ev.data;
console.log(`[MainWorker -> Renderer] code=${code}`);
switch (code) {
case WORKER_RESP_CODE.CHUNK_DATA:
if (null !== logExportManagerRef.current) {
const progress = logExportManagerRef.current.appendChunk(args.logs);
setExportProgress(progress);
}
break;
case WORKER_RESP_CODE.LOG_FILE_INFO:
setFileName(args.fileName);
setNumEvents(args.numEvents);
break;
case WORKER_RESP_CODE.NOTIFICATION:
// eslint-disable-next-line no-warning-comments
// TODO: notifications should be shown in the UI when the NotificationProvider
// is added
console.error(args.logLevel, args.message);
break;
case WORKER_RESP_CODE.PAGE_DATA: {
setLogData(args.logs);
beginLineNumToLogEventNumRef.current = args.beginLineNumToLogEventNum;
const lastLogEventNum = getLastLogEventNum(args.beginLineNumToLogEventNum);
updateLogEventNumInUrl(lastLogEventNum, logEventNumRef.current);
break;
}
case WORKER_RESP_CODE.NOTIFICATION:
// eslint-disable-next-line no-warning-comments
// TODO: notifications should be shown in the UI when the NotificationProvider
// is added
console.error(args.logLevel, args.message);
break;
default:
console.error(`Unexpected ev.data: ${JSON.stringify(ev.data)}`);
break;
}
}, []);

const exportLogs = useCallback(() => {
if (null === mainWorkerRef.current) {
console.error("Unexpected null mainWorkerRef.current");

return;
}
if (STATE_DEFAULT.numEvents === numEvents && STATE_DEFAULT.fileName === fileName) {
console.error("numEvents and fileName not initialized yet");

return;
}

setExportProgress(EXPORT_LOG_PROGRESS_VALUE_MIN);
logExportManagerRef.current = new LogExportManager(
Henry8192 marked this conversation as resolved.
Show resolved Hide resolved
Henry8192 marked this conversation as resolved.
Show resolved Hide resolved
Math.ceil(numEvents / EXPORT_LOGS_CHUNK_SIZE),
fileName
);
workerPostReq(
mainWorkerRef.current,
WORKER_REQ_CODE.EXPORT_LOG,
{decoderOptions: getConfig(CONFIG_KEY.DECODER_OPTIONS)}
);
}, [
numEvents,
fileName,
]);

const loadFile = useCallback((fileSrc: FileSrcType, cursor: CursorType) => {
if ("string" !== typeof fileSrc) {
updateWindowUrlSearchParams({[SEARCH_PARAM_NAMES.FILE_PATH]: null});
Expand All @@ -185,6 +234,8 @@ const StateContextProvider = ({children}: StateContextProviderProps) => {
cursor: cursor,
decoderOptions: getConfig(CONFIG_KEY.DECODER_OPTIONS),
});

setExportProgress(STATE_DEFAULT.exportProgress);
}, [
handleMainWorkerResp,
]);
Expand Down Expand Up @@ -274,13 +325,16 @@ const StateContextProvider = ({children}: StateContextProviderProps) => {
<StateContext.Provider
value={{
beginLineNumToLogEventNum: beginLineNumToLogEventNumRef.current,
exportProgress: exportProgress,
fileName: fileName,
loadFile: loadFile,
loadPage: loadPage,
logData: logData,
numEvents: numEvents,
numPages: numPagesRef.current,
pageNum: pageNumRef.current,

exportLogs: exportLogs,
loadFile: loadFile,
loadPage: loadPage,
}}
>
{children}
Expand Down
69 changes: 69 additions & 0 deletions new-log-viewer/src/services/LogExportManager.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
import {downloadBlob} from "../utils/file";


const EXPORT_LOG_PROGRESS_VALUE_MIN = 0;
const EXPORT_LOG_PROGRESS_VALUE_MAX = 1;

/**
* Manager for exporting logs as a file.
*/
class LogExportManager {
Henry8192 marked this conversation as resolved.
Show resolved Hide resolved
/**
* Internal buffer which stores decoded chunks of log data.
*/
readonly #chunks: string[] = [];

/**
* Total number of chunks to export.
*/
readonly #numChunks: number;

/**
* Name of the file to export to.
Henry8192 marked this conversation as resolved.
Show resolved Hide resolved
*/
readonly #exportedFileName: string;

constructor (numChunks: number, fileName: string) {
this.#numChunks = numChunks;
this.#exportedFileName = `exported-${new Date().toISOString()
.replace(/[:.]/g, "-")}-${fileName}.log`;
}

/**
* Appends the provided chunk of logs into an internal buffer. If the number of chunks reaches
* the specified limit, triggers a download.
*
* @param chunk
* @return The current download progress as a float between 0 and 1.
*/
appendChunk (chunk: string): number {
if (0 === this.#numChunks) {
this.#download();

return EXPORT_LOG_PROGRESS_VALUE_MAX;
}
this.#chunks.push(chunk);
if (this.#chunks.length === this.#numChunks) {
this.#download();
this.#chunks.length = 0;
Henry8192 marked this conversation as resolved.
Show resolved Hide resolved

return EXPORT_LOG_PROGRESS_VALUE_MAX;
}

return this.#chunks.length / this.#numChunks;
}

/**
* Triggers a download of the accumulated chunks.
*/
#download () {
const blob = new Blob(this.#chunks, {type: "text/plain"});
downloadBlob(blob, this.#exportedFileName);
}
}

export default LogExportManager;
export {
EXPORT_LOG_PROGRESS_VALUE_MAX,
EXPORT_LOG_PROGRESS_VALUE_MIN,
};
32 changes: 32 additions & 0 deletions new-log-viewer/src/services/LogFileManager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
CursorType,
FileSrcType,
} from "../typings/worker";
import {EXPORT_LOGS_CHUNK_SIZE} from "../utils/config";
import {getUint8ArrayFrom} from "../utils/http";
import {getChunkNum} from "../utils/math";
import {formatSizeInBytes} from "../utils/units";
Expand Down Expand Up @@ -152,6 +153,37 @@ class LogFileManager {
this.#decoder.setDecoderOptions(options);
}

/**
* Loads log events in the range
* [`beginLogEventIdx`, `beginLogEventIdx + EXPORT_LOGS_CHUNK_SIZE`), or all remaining log
* events if `EXPORT_LOGS_CHUNK_SIZE` log events aren't available.
*
* @param beginLogEventIdx
* @return An object containing the log events as a string.
* @throws {Error} if any error occurs when decoding the log events.
*/
loadChunk (beginLogEventIdx: number): {
logs: string,
} {
const endLogEventIdx = Math.min(beginLogEventIdx + EXPORT_LOGS_CHUNK_SIZE, this.#numEvents);
const results = this.#decoder.decode(
beginLogEventIdx,
endLogEventIdx
);

if (null === results) {
throw new Error(
`Failed to decode log events in range [${beginLogEventIdx}, ${endLogEventIdx})`
);
}

const messages = results.map(([msg]) => msg);

return {
logs: messages.join(""),
};
}

/**
* Loads a page of log events based on the provided cursor.
*
Expand Down
Loading
Loading