Skip to content

Commit

Permalink
Honeypots E2E tests (cvat-ai#8638)
Browse files Browse the repository at this point in the history
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->

<!-- Provide a general summary of your changes in the Title above -->

### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->

### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->

### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](

https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary

([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),

[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),

[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and

[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))

### License

- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
  Feel free to contact the maintainers if that's a concern.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Introduced new commands for creating dummy tasks and configuring
quality settings in headless testing.
- Added functions for generating task and ground truth job
specifications, enhancing task management capabilities.

- **Bug Fixes**
- Updated frame selection logic for ground truth jobs to improve
accuracy in tests.

- **Documentation**
- Updated copyright notice to reflect the current year and organization.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
  • Loading branch information
klakhov authored Nov 22, 2024
1 parent 3d7e2c7 commit 7070875
Show file tree
Hide file tree
Showing 3 changed files with 180 additions and 82 deletions.
182 changes: 102 additions & 80 deletions tests/cypress/e2e/features/ground_truth_jobs.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,24 @@

/// <reference types="cypress" />

import { defaultTaskSpec } from '../../support/default-specs';

context('Ground truth jobs', () => {
const caseId = 'Ground truth jobs';
const labelName = 'car';
const taskName = `Annotation task for Case ${caseId}`;
const attrName = `Attr for Case ${caseId}`;
const textDefaultValue = 'Some default value for type Text';
const taskName = 'Annotation task for Ground truth jobs';

const jobOptions = {
jobType: 'Ground truth',
frameSelectionMethod: 'Random',
fromTaskPage: true,
};

const defaultValidationParams = {
frameCount: 3,
mode: 'gt',
frameSelectionMethod: 'random_uniform',
};

const groundTruthRectangles = [
{
id: 1,
Expand Down Expand Up @@ -64,8 +69,8 @@ context('Ground truth jobs', () => {
let jobID = null;
let taskID = null;

// With seed = 1, frameCount = 4, totalFrames = 10 - predifined ground truth frames are:
const groundTruthFrames = [0, 1, 5, 6];
// With seed = 1, frameCount = 4, totalFrames = 100 - predifined ground truth frames are:
const groundTruthFrames = [10, 23, 71, 87];

function checkRectangleAndObjectMenu(rectangle, isGroundTruthJob = false) {
if (isGroundTruthJob) {
Expand Down Expand Up @@ -97,36 +102,33 @@ context('Ground truth jobs', () => {
cy.get('.cvat-quality-control-management-tab').should('exist').and('be.visible');
}

function createAndOpenTask(serverFiles, validationParams = null) {
const { taskSpec, dataSpec, extras } = defaultTaskSpec({
taskName, serverFiles, labelName, validationParams,
});
return cy.headlessCreateTask(taskSpec, dataSpec, extras).then((taskResponse) => {
taskID = taskResponse.taskID;
if (validationParams) {
[groundTruthJobID, jobID] = taskResponse.jobIDs;
} else {
[jobID] = taskResponse.jobIDs;
}
}).then(() => {
cy.visit(`/tasks/${taskID}`);
cy.get('.cvat-task-details').should('exist').and('be.visible');
});
}

before(() => {
cy.visit('auth/login');
cy.login();
});

describe('Testing ground truth basics', () => {
const imagesCount = 10;
const imageFileName = 'ground_truth_1';
const width = 800;
const height = 800;
const posX = 10;
const posY = 10;
const color = 'gray';
const archiveName = `${imageFileName}.zip`;
const archivePath = `cypress/fixtures/${archiveName}`;
const imagesFolder = `cypress/fixtures/${imageFileName}`;
const directoryToArchive = imagesFolder;
const serverFiles = ['bigArchive.zip'];

before(() => {
cy.visit('/tasks');
cy.imageGenerator(imagesFolder, imageFileName, width, height, color, posX, posY, labelName, imagesCount);
cy.createZipArchive(directoryToArchive, archivePath);
cy.createAnnotationTask(taskName, labelName, attrName, textDefaultValue, archiveName);
cy.openTask(taskName);
cy.url().then((url) => {
taskID = Number(url.split('/').slice(-1)[0].split('?')[0]);
});
cy.get('.cvat-job-item').first().invoke('attr', 'data-row-id').then((val) => {
jobID = val;
});
createAndOpenTask(serverFiles);
});

after(() => {
Expand Down Expand Up @@ -196,35 +198,80 @@ context('Ground truth jobs', () => {
});
});

describe('Testing creating task with quality params', () => {
const imagesCount = 3;
const imageFileName = `image_${taskName.replace(' ', '_').toLowerCase()}`;
const width = 800;
const height = 800;
const posX = 10;
const posY = 10;
const color = 'gray';
const archiveName = `${imageFileName}.zip`;
const archivePath = `cypress/fixtures/${archiveName}`;
const imagesFolder = `cypress/fixtures/${imageFileName}`;
const directoryToArchive = imagesFolder;
const attrName = 'gt_attr';
const defaultAttrValue = 'GT attr';
const multiAttrParams = false;
const forProject = false;
const attachToProject = false;
const projectName = null;
const expectedResult = 'success';
const projectSubsetFieldValue = null;
const advancedConfigurationParams = false;

before(() => {
cy.contains('.cvat-header-button', 'Tasks').should('be.visible').click();
cy.url().should('include', '/tasks');
cy.imageGenerator(imagesFolder, imageFileName, width, height, color, posX, posY, labelName, imagesCount);
cy.createZipArchive(directoryToArchive, archivePath);
});

afterEach(() => {
cy.goToTaskList();
cy.deleteTask(taskName);
});

function createTaskWithQualityParams(qualityParams) {
cy.createAnnotationTask(
taskName,
labelName,
attrName,
defaultAttrValue,
archiveName,
multiAttrParams,
advancedConfigurationParams,
forProject,
attachToProject,
projectName,
expectedResult,
projectSubsetFieldValue,
qualityParams,
);
cy.openTask(taskName);
cy.get('.cvat-job-item').first()
.find('.ant-tag')
.should('have.text', 'Ground truth');
}

it('Create task with ground truth job', () => {
createTaskWithQualityParams({
validationMode: 'Ground Truth',
});
});

it('Create task with honeypots', () => {
createTaskWithQualityParams({
validationMode: 'Honeypots',
});
});
});

describe('Testing ground truth management basics', () => {
const serverFiles = ['images/image_1.jpg', 'images/image_2.jpg', 'images/image_3.jpg'];

before(() => {
cy.headlessCreateTask({
labels: [{ name: labelName, attributes: [], type: 'any' }],
name: taskName,
project_id: null,
source_storage: { location: 'local' },
target_storage: { location: 'local' },
}, {
server_files: serverFiles,
image_quality: 70,
use_zip_chunks: true,
use_cache: true,
sorting_method: 'lexicographical',
}).then((taskResponse) => {
taskID = taskResponse.taskID;
[jobID] = taskResponse.jobIDs;
}).then(() => (
cy.headlessCreateJob({
task_id: taskID,
frame_count: 3,
type: 'ground_truth',
frame_selection_method: 'random_uniform',
})
)).then((jobResponse) => {
groundTruthJobID = jobResponse.jobID;
}).then(() => {
createAndOpenTask(serverFiles, defaultValidationParams).then(() => {
cy.visit(`/tasks/${taskID}/quality-control#management`);
cy.get('.cvat-quality-control-management-tab').should('exist').and('be.visible');
cy.get('.cvat-annotations-quality-allocation-table-summary').should('exist').and('be.visible');
Expand Down Expand Up @@ -312,35 +359,10 @@ context('Ground truth jobs', () => {
});

describe('Regression tests', () => {
const imagesCount = 20;
const imageFileName = 'ground_truth_2';
const width = 100;
const height = 100;
const posX = 10;
const posY = 10;
const color = 'gray';
const archiveName = `${imageFileName}.zip`;
const archivePath = `cypress/fixtures/${archiveName}`;
const imagesFolder = `cypress/fixtures/${imageFileName}`;
const directoryToArchive = imagesFolder;
const serverFiles = ['bigArchive.zip'];

before(() => {
cy.visit('/tasks');
cy.imageGenerator(imagesFolder, imageFileName, width, height, color, posX, posY, labelName, imagesCount);
cy.createZipArchive(directoryToArchive, archivePath);
cy.createAnnotationTask(
taskName,
labelName,
attrName,
textDefaultValue,
archiveName,
false,
{ multiJobs: true, segmentSize: 1 },
);
cy.openTask(taskName);
cy.url().then((url) => {
taskID = Number(url.split('/').slice(-1)[0].split('?')[0]);
});
createAndOpenTask(serverFiles);
});

afterEach(() => {
Expand Down
17 changes: 15 additions & 2 deletions tests/cypress/support/commands.js
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ Cypress.Commands.add(
projectName = '',
expectedResult = 'success',
projectSubsetFieldValue = 'Test',
qualityConfigurationParams = null,
) => {
cy.url().then(() => {
cy.get('.cvat-create-task-dropdown').click();
Expand Down Expand Up @@ -215,6 +216,9 @@ Cypress.Commands.add(
if (advancedConfigurationParams) {
cy.advancedConfiguration(advancedConfigurationParams);
}
if (qualityConfigurationParams) {
cy.configureTaskQualityMode(qualityConfigurationParams);
}
cy.get('.cvat-submit-continue-task-button').scrollIntoView();
cy.get('.cvat-submit-continue-task-button').click();
if (expectedResult === 'success') {
Expand Down Expand Up @@ -291,7 +295,7 @@ Cypress.Commands.add('headlessCreateObjects', (objects, jobID) => {
});
});

Cypress.Commands.add('headlessCreateTask', (taskSpec, dataSpec) => {
Cypress.Commands.add('headlessCreateTask', (taskSpec, dataSpec, extras) => {
cy.window().then(async ($win) => {
const task = new $win.cvat.classes.Task({
...taskSpec,
Expand All @@ -310,7 +314,7 @@ Cypress.Commands.add('headlessCreateTask', (taskSpec, dataSpec) => {
task.remoteFiles = dataSpec.remote_files;
}

const result = await task.save();
const result = await task.save(extras || {});
return cy.wrap({ taskID: result.id, jobIDs: result.jobs.map((job) => job.id) });
});
});
Expand Down Expand Up @@ -897,6 +901,15 @@ Cypress.Commands.add('advancedConfiguration', (advancedConfigurationParams) => {
}
});

Cypress.Commands.add('configureTaskQualityMode', (qualityConfigurationParams) => {
cy.contains('Quality').click();
if (qualityConfigurationParams.validationMode) {
cy.get('#validationMode').within(() => {
cy.contains(qualityConfigurationParams.validationMode).click();
});
}
});

Cypress.Commands.add('removeAnnotations', () => {
cy.contains('.cvat-annotation-header-button', 'Menu').click();
cy.get('.cvat-annotation-menu').within(() => {
Expand Down
63 changes: 63 additions & 0 deletions tests/cypress/support/default-specs.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
// Copyright (C) 2024 CVAT.ai Corporation
//
// SPDX-License-Identifier: MIT

function defaultTaskSpec({
labelName,
taskName,
serverFiles,
validationParams,
}) {
const taskSpec = {
labels: [
{ name: labelName, attributes: [], type: 'any' },
],
name: taskName,
project_id: null,
source_storage: { location: 'local' },
target_storage: { location: 'local' },
};

const dataSpec = {
server_files: serverFiles,
image_quality: 70,
use_zip_chunks: true,
use_cache: true,
sorting_method: (validationParams && validationParams.mode === 'gt_pool') ? 'random' : 'lexicographical',
};

const extras = {};
if (validationParams) {
const convertedParams = {};
if (validationParams.frames) {
convertedParams.frames = validationParams.frames;
}
if (validationParams.frameSelectionMethod) {
convertedParams.frame_selection_method = validationParams.frameSelectionMethod;
}
if (validationParams.frameCount) {
convertedParams.frame_count = validationParams.frameCount;
}
if (validationParams.framesPerJobCount) {
convertedParams.frames_per_job_count = validationParams.framesPerJobCount;
}
if (validationParams.mode) {
convertedParams.mode = validationParams.mode;
}
if (validationParams.randomSeed) {
convertedParams.random_seed = validationParams.randomSeed;
}

extras.validation_params = convertedParams;
}

return {
taskSpec,
dataSpec,
extras,
};
}

module.exports = {
defaultTaskSpec,
};

0 comments on commit 7070875

Please sign in to comment.