Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge with fix-2.1.0 #518

Open
wants to merge 25 commits into
base: feat/backup
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
cd045a8
Refactor: Removed unused function
ArceDanielShok Aug 1, 2024
2bed40b
Merge pull request #505 from internxt/fix/upload
ArceDanielShok Aug 1, 2024
6d921fa
Fixed issue with updating folders as described in ticket PB-2665
ArceDanielShok Aug 21, 2024
97d7886
fixed some bugs
ArceDanielShok Aug 27, 2024
9ab2305
add persist queueManager
ArceDanielShok Aug 28, 2024
eb5c46e
delete queue manager persist after logout
ArceDanielShok Aug 29, 2024
67cb45e
fix some smell issues
ArceDanielShok Sep 3, 2024
3dfd35f
Implemented the changes for ticket PB-2692. Integrated the new endpoi…
ArceDanielShok Sep 4, 2024
7609c84
update package json
ArceDanielShok Sep 4, 2024
50aa255
Worked on a fix for the download process, resolved the refresh token …
ArceDanielShok Sep 6, 2024
3a47024
fix logout error and update package version
ArceDanielShok Sep 9, 2024
e62d161
Added validations to prevent temporary files in the app
ArceDanielShok Sep 11, 2024
52a26b3
update start sync remote start
ArceDanielShok Sep 18, 2024
ffac4e1
update package json
ArceDanielShok Sep 18, 2024
7c57024
delete log
ArceDanielShok Sep 23, 2024
9bbd0d7
Implemented the removal of items from the database once the server re…
ArceDanielShok Sep 25, 2024
310b92b
delete comment and update rewind time
ArceDanielShok Sep 30, 2024
5f7cdf8
merge with fix-2.1.0
ArceDanielShok Sep 30, 2024
423b334
merge with fix on release 2.1.3
ArceDanielShok Oct 1, 2024
c61760e
update issue page
ArceDanielShok Oct 1, 2024
a18c669
delete get usage photo
ArceDanielShok Oct 2, 2024
c762018
delete get usage photo
ArceDanielShok Oct 2, 2024
f578d2a
update version
ArceDanielShok Oct 2, 2024
cc0ae1c
Completed the task of implementing manual backup for one or more folders
ArceDanielShok Oct 3, 2024
510eb5a
resolve smell issue
ArceDanielShok Oct 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .erb/configs/webpack.config.backups.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ const configuration: webpack.Configuration = {

target: 'electron-renderer',

module: {
rules: [{ test: /\.node$/, loader: 'node-loader' }],
},

entry: [
'core-js',
'regenerator-runtime/runtime',
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "internxt-drive",
"version": "2.1.0",
"version": "2.2.0",
"author": "Internxt <[email protected]>",
"description": "Internxt Drive client UI",
"license": "AGPL-3.0",
Expand Down
2 changes: 1 addition & 1 deletion release/app/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "internxt-drive",
"version": "2.1.0",
"version": "2.2.0",
"description": "Internxt Drive client UI",
"main": "./dist/main/main.js",
"author": "Internxt <[email protected]>",
Expand Down
142 changes: 90 additions & 52 deletions src/apps/backups/Backups.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import LocalTreeBuilder from '../../context/local/localTree/application/LocalTreeBuilder';
import { LocalTree } from '../../context/local/localTree/domain/LocalTree';
import { File } from '../../context/virtual-drive/files/domain/File';
import { Folder } from '../../context/virtual-drive/folders/domain/Folder';
import { SimpleFolderCreator } from '../../context/virtual-drive/folders/application/create/SimpleFolderCreator';
import { BackupInfo } from './BackupInfo';
import { BackupsIPCRenderer } from './BackupsIPCRenderer';
Expand All @@ -17,12 +18,14 @@
FoldersDiff,
FoldersDiffCalculator,
} from './diff/FoldersDiffCalculator';
import { getParentDirectory, relative, relativeV2 } from './utils/relative';
import { getParentDirectory, relativeV2 } from './utils/relative';
import { DriveDesktopError } from '../../context/shared/domain/errors/DriveDesktopError';
import { UserAvaliableSpaceValidator } from '../../context/user/usage/application/UserAvaliableSpaceValidator';
import { FileDeleter } from '../../context/virtual-drive/files/application/delete/FileDeleter';
import { RemoteTreeBuilder } from '../../context/virtual-drive/remoteTree/application/RemoteTreeBuilder';
import { RemoteTree } from '../../context/virtual-drive/remoteTree/domain/RemoteTree';
import { FolderDeleter } from '../../context/virtual-drive/folders/application/delete/FolderDeleter';
import { LocalFolder } from '../../context/local/localFolder/domain/LocalFolder';

@Service()
export class Backup {
Expand All @@ -32,6 +35,7 @@
private readonly fileBatchUploader: FileBatchUploader,
private readonly fileBatchUpdater: FileBatchUpdater,
private readonly remoteFileDeleter: FileDeleter,
private readonly remoteFolderDeleter: FolderDeleter,
private readonly simpleFolderCreator: SimpleFolderCreator,
private readonly userAvaliableSpaceValidator: UserAvaliableSpaceValidator
) {}
Expand All @@ -44,37 +48,23 @@
): Promise<DriveDesktopError | undefined> {
Logger.info('[BACKUPS] Backing:', info);

Logger.info('[BACKUPS] Generating local tree');
const localTreeEither = await this.localTreeBuilder.run(
info.pathname as AbsolutePath
);

Logger.debug('[BACKUPS] Local tree either', localTreeEither);

if (localTreeEither.isLeft()) {
Logger.error('[BACKUPS] local tree is left', localTreeEither);
return localTreeEither.getLeft();
}

const local = localTreeEither.getRight();

Logger.info('[BACKUPS] Generating remote tree', info.folderId);
const remote = await this.remoteTreeBuilder.run(info.folderId, true);

Logger.debug('[BACKUPS] Remote tree', JSON.stringify(remote));

Logger.debug('[BACKUPS] Remote tree file', remote.files);

Logger.debug('[BACKUPS] Remote tree folder', remote.folders);

const foldersDiff = FoldersDiffCalculator.calculate(local, remote);

Logger.debug('[BACKUPS] Folders diff', foldersDiff);

const filesDiff = DiffFilesCalculator.calculate(local, remote);

Logger.debug('[BACKUPS] Files diff', filesDiff);

await this.isThereEnoughSpace(filesDiff);

const alreadyBacked =
Expand All @@ -88,7 +78,7 @@
alreadyBacked
);

await this.backupFolders(foldersDiff, local, remote);
await this.backupFolders(foldersDiff, local, remote, abortController);

await this.backupFiles(filesDiff, local, remote, abortController);

Expand Down Expand Up @@ -128,47 +118,27 @@
private async backupFolders(
diff: FoldersDiff,
local: LocalTree,
remote: RemoteTree
remote: RemoteTree,
abortController: AbortController
) {
Logger.info('[BACKUPS] Backing folders');
Logger.info('[BACKUPS] Folders added', diff.added.length);

await Promise.all(
diff.added.map(async (localFolder) => {
const relativePath = relativeV2(local.root.path, localFolder.path);

if (relativePath === '/') {
return; // Ignorar la carpeta raíz
}

const remoteParentPath = getParentDirectory(
local.root.path,
localFolder.path
);
const parentExists = remote.has(remoteParentPath);

if (!parentExists) {
return;
}

const parent = remote.getParent(remoteParentPath);
const existingItems = remote.has(relativePath);

if (existingItems) {
return;
}
const { added, deleted } = diff;

const folder = await this.simpleFolderCreator.run(
relativePath,
parent.id
);

remote.addFolder(parent, folder);
const deleteFolder = await this.deleteRemoteFolders(
deleted,
abortController
);

this.backed++;
BackupsIPCRenderer.send('backups.progress-update', this.backed);
})
Logger.debug('[BACKUPS] start upload', deleted.length);
const uploadFolder = await this.uploadAndCreateFolder(
local.root.path,
added,
remote
);

return await Promise.all([deleteFolder, uploadFolder]);
}

private async backupFiles(
Expand All @@ -182,7 +152,12 @@
const { added, modified, deleted } = filesDiff;

Logger.info('[BACKUPS] Files added', added.length);
await this.uploadAndCreate(local.root.path, added, remote, abortController);
await this.uploadAndCreateFile(
local.root.path,
added,
remote,
abortController
);

Logger.info('[BACKUPS] Files modified', modified.size);
await this.uploadAndUpdate(modified, local, remote, abortController);
Expand All @@ -191,7 +166,7 @@
await this.deleteRemoteFiles(deleted, abortController);
}

private async uploadAndCreate(
private async uploadAndCreateFile(
localRootPath: string,
added: Array<LocalFile>,
tree: RemoteTree,
Expand Down Expand Up @@ -260,4 +235,67 @@
this.backed += deleted.length;
BackupsIPCRenderer.send('backups.progress-update', this.backed);
}

private async deleteRemoteFolders(
deleted: Array<Folder>,
abortController: AbortController
) {
for (const folder of deleted) {
if (abortController.signal.aborted) {
return;
}

// eslint-disable-next-line no-await-in-loop
await this.remoteFolderDeleter.run(folder);
}

this.backed += deleted.length;
BackupsIPCRenderer.send('backups.progress-update', this.backed);
}

private async uploadAndCreateFolder(
localRootPath: string,
added: Array<LocalFolder>,
tree: RemoteTree
): Promise<void> {
for (const localFolder of added) {
const relativePath = relativeV2(localRootPath, localFolder.path);

Logger.debug('[BACKUPS] Relative path of folder', relativePath);

if (relativePath === '/') {
Logger.debug('[BACKUPS] Ignoring root folder');
continue; // Ignorar la carpeta raíz
}

const remoteParentPath = getParentDirectory(
localRootPath,
localFolder.path
);
const parentExists = tree.has(remoteParentPath);

if (!parentExists) {
Logger.debug('[BACKUPS] Parent folder does not exist');
continue;
}

const parent = tree.getParent(remoteParentPath);
const existingItems = tree.has(relativePath);

if (existingItems) {
Logger.debug('[BACKUPS] Folder already exists');
continue;
}

const folder = await this.simpleFolderCreator.run(

Check warning on line 290 in src/apps/backups/Backups.ts

View workflow job for this annotation

GitHub Actions / 🧪 Lint and test

Unexpected `await` inside a loop
relativePath,
parent.id
);

tree.addFolder(parent, folder);

this.backed++;
BackupsIPCRenderer.send('backups.progress-update', this.backed);
}
}
}
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
import { ContainerBuilder } from 'diod';
import { SimpleFolderCreator } from '../../../../context/virtual-drive/folders/application/create/SimpleFolderCreator';
import { RemoteFileSystem } from '../../../../context/virtual-drive/folders/domain/file-systems/RemoteFileSystem';
import { HttpRemoteFileSystem } from '../../../../context/virtual-drive/folders/infrastructure/HttpRemoteFileSystem';
import { AuthorizedClients } from '../../../shared/HttpClient/Clients';
import { HttpRemoteFolderSystem } from '../../../../context/virtual-drive/folders/infrastructure/HttpRemoteFolderSystem';
import { RemoteFolderSystem } from '../../../../context/virtual-drive/folders/domain/file-systems/RemoteFolderSystem';
import { FolderDeleter } from '../../../../context/virtual-drive/folders/application/delete/FolderDeleter';

export async function registerFolderServices(builder: ContainerBuilder) {
builder
.register(RemoteFileSystem)
.register(RemoteFolderSystem)
.useFactory((c) => {
const clients = c.get(AuthorizedClients);
return new HttpRemoteFileSystem(
return new HttpRemoteFolderSystem(
// @ts-ignore

Check warning on line 14 in src/apps/backups/dependency-injection/virtual-drive/registerFolderServices.ts

View workflow job for this annotation

GitHub Actions / 🧪 Lint and test

Do not use "@ts-ignore" because it alters compilation errors
clients.drive,
clients.newDrive
);
Expand All @@ -18,4 +19,5 @@
.private();

builder.registerAndUse(SimpleFolderCreator);
builder.registerAndUse(FolderDeleter);
}
11 changes: 8 additions & 3 deletions src/apps/main/auth/handlers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,12 @@ import Logger from 'electron-log';
import { AccessResponse } from '../../renderer/pages/Login/service';
import { applicationOpened } from '../analytics/service';
import eventBus from '../event-bus';
import { setupRootFolder } from '../virtual-root-folder/service';
import {
clearRootVirtualDrive,
setupRootFolder,
} from '../virtual-root-folder/service';
import { getWidget } from '../windows/widget';
import { createTokenSchedule } from './refresh-token';
import { checkUserData, createTokenSchedule } from './refresh-token';
import {
canHisConfigBeRestored,
encryptToken,
Expand Down Expand Up @@ -49,6 +52,7 @@ ipcMain.handle('get-token', () => {

export function onUserUnauthorized() {
eventBus.emit('USER_WAS_UNAUTHORIZED');
eventBus.emit('USER_LOGGED_OUT');

logout();
Logger.info('[AUTH] User has been logged out because it was unauthorized');
Expand All @@ -62,6 +66,7 @@ ipcMain.on('user-logged-in', async (_, data: AccessResponse) => {
if (!canHisConfigBeRestored(data.user.uuid)) {
await setupRootFolder();
}
await clearRootVirtualDrive();

setIsLoggedIn(true);
eventBus.emit('USER_LOGGED_IN');
Expand All @@ -79,7 +84,7 @@ eventBus.on('APP_IS_READY', async () => {
if (!isLoggedIn) {
return;
}

await checkUserData();
encryptToken();
applicationOpened();
await createTokenSchedule();
Expand Down
43 changes: 39 additions & 4 deletions src/apps/main/auth/refresh-token.ts
Original file line number Diff line number Diff line change
@@ -1,20 +1,23 @@
import Logger from 'electron-log';

import { getClient } from '../../shared/HttpClient/main-process-client';
import { getNewTokenClient } from '../../shared/HttpClient/main-process-client';
import { TokenScheduler } from '../token-scheduler/TokenScheduler';
import { onUserUnauthorized } from './handlers';
import {
getUser,
obtainTokens as obtainStoredTokens,
setUser,
updateCredentials,
} from './service';
import axios from 'axios';

const authorizedClient = getClient();
const newAuthorizedClient = getNewTokenClient();

async function obtainTokens() {
try {
Logger.debug('[TOKEN] Obtaining new tokens');
const res = await authorizedClient.get(
`${process.env.API_URL}/user/refresh`
const res = await newAuthorizedClient.get(
`${process.env.NEW_DRIVE_URL}/drive/users/refresh`
);

return res.data;
Expand All @@ -34,6 +37,7 @@ async function refreshToken() {

updateCredentials(token, newToken);

Logger.debug('[TOKEN] Refreshed tokens', token, newToken);
return [token, newToken];
}

Expand All @@ -48,3 +52,34 @@ export async function createTokenSchedule(refreshedTokens?: Array<string>) {
createTokenSchedule(await refreshToken());
}
}

async function getRootFolderMetadata(rootFolderid: number) {
try {
const res = await newAuthorizedClient.get(
`${process.env.NEW_DRIVE_URL}/drive/folders/${rootFolderid}/metadata`
);

Logger.info('[AUTH] Got root folder metadata', res.data);
return res.data;
} catch (err) {
Logger.error('[AUTH] Could not get root folder metadata', err);
if (axios.isAxiosError(err)) {
Logger.error('[Is Axios Error]', err.response?.data);
}
return null;
}
}

export async function checkUserData(): Promise<void> {
const user = getUser();
if (user?.root_folder_id && !user?.rootFolderId) {
const rootFolderMetadata = await getRootFolderMetadata(user.root_folder_id);
if (rootFolderMetadata) {
setUser({
...user,
rootFolderId: rootFolderMetadata.uuid,
});
}
refreshToken();
}
}
Loading
Loading