add readme

Signed-off-by: Ivy Zhang <[email protected]>

Author: crazydemo

tests/integration/test_lists/qa/README.md

@@ -0,0 +1,109 @@
+# Description
+
+This folder contains QA test definitions for TensorRT-LLM, which are executed on a weekly/release schedule. These tests focus on end-to-end validation, accuracy verification, disaggregated testing, and performance benchmarking.
+
+## Test Categories
+
+QA tests are organized into three main categories:
+
+### 1. Functional Tests
+Functional tests include E2E (end-to-end), accuracy, and disaggregated test cases:
+
+- **E2E Tests**: Complete workflow validation from model loading to inference output
+- **Accuracy Tests**: Model accuracy verification against reference implementations
+- **Disaggregated Tests**: Distributed deployment and multi-node scenario validation
+
+### 2. Performance Tests
+Performance tests focus on benchmarking and performance validation:
+- Baseline performance measurements
+- Performance regression detection
+- Throughput and latency benchmarking
+- Resource utilization analysis
+
+### 3. Triton Backend Tests
+Triton backend tests validate the integration with NVIDIA Triton Inference Server:
+- Backend functionality validation
+- Model serving capabilities
+- API compatibility testing
+- Integration performance testing
+
+## Dependencies
+
+The following Python packages are required for running QA tests:
+
+```bash
+pip install mako oyaml rouge_score lm_eval
+```
+
+### Dependency Details
+
+- **mako**: Template engine for test generation and configuration
+- **oyaml**: YAML parser with ordered dictionary support
+- **rouge_score**: ROUGE evaluation metrics for text generation quality assessment
+- **lm_eval**: Language model evaluation framework
+
+## Test Files
+
+This directory contains various test configuration files:
+
+### Functional Test Lists
+- `examples_test_list.txt` - Primary test list for single node multi-GPU scenarios (all new test cases should be added here)
+- `llm_sanity_test.txt` - Subset of examples for quick torch flow validation
+- `benchmark_test_list.txt` - Long-running single node test cases
+- `llm_multinodes_function_test.txt` - Multi-node functional test cases
+- `llm_release_gb20x.txt` - GB20X release test cases
+- `llm_release_rtx_pro_6000.txt` - RTX Pro 6000 specific tests
+
+### Performance Test Files
+- `trt_llm_release_perf_test.yml` - Main performance test configuration
+- `trt_llm_release_perf_cluster_test.yml` - Cluster-based performance tests
+- `trt_llm_release_perf_sanity_test.yml` - Performance sanity checks
+- `trt_llm_release_perf_l2_test.yml` - L2 level performance tests
+- `trt_llm_integration_perf_test.yml` - Integration performance tests
+- `trt_llm_integration_perf_sanity_test.yml` - Integration performance sanity checks
+- `llm_release_perf_multinode_test.txt` - Multi-node performance tests
+
+### Triton Backend Tests
+- `llm_triton_integration_test.txt` - Triton backend integration tests
+
+### Release-Specific Tests
+- `llm_release_digits_func.txt` - Functional tests for DIGITS release
+- `llm_release_digits_perf.txt` - Performance tests for DIGITS release
+
+## Test Execution Schedule
+
+QA tests are executed on a regular schedule:
+
+- **Weekly**: Automated regression testing
+- **Release**: Comprehensive validation before each release
+- **On-demand**: Manual execution for specific validation needs
+
+## Running Tests
+
+### Manual Execution
+
+To run specific test categories:
+
+```bash
+# direct to defs folder
+cd tests/integration/defs
+# Run all fp8 functional test
+pytest --no-header -vs --test-list=../test_lists/qa/examples_test_list.txt -k fp8
+# Run a single test case
+pytest -vs accuracy/test_cli_flow.py::TestLlama3_1_8B::test_auto_dtype
+```
+
+### Automated Execution
+
+QA tests are typically executed through CI/CD pipelines with appropriate test selection based on:
+
+- Release requirements
+- Hardware availability
+- Test priority and scope
+
+## Test Guidelines
+
+### Adding New Test Cases
+- **Primary Location**: For functional testing, new test cases should be added to `examples_test_list.txt` first
+- **Categorization**: Test cases should be categorized based on their scope and execution time
+- **Validation**: Ensure test cases are properly validated before adding to any test list