Skip to content
/ QASMBench Public

A low-level OpenQASM benchmark suite for NISQ evaluation and simulation. Please see our paper for details.

License

Notifications You must be signed in to change notification settings

pnnl/QASMBench

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

57 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

QASMBench Benchmark Suite

QASMBench is an OpenQASM benchmark suite for NISQ evaluation. The .qasm code can be directly loaded in IBM Quantum Experience for execution. Please see our TQC paper (DOI:10.1145/3550488 or on arXiv) for details.

If you need specialized circuits with the number of qubits being configurable, or circuits in alternative representations such as Cirq, QSharp, Qiskit, and PyQuil, please see our NWQBench.

alt text

Current version

Latest version: 1.4

About QASMBench

The rapid development of quantum computing (QC) in the NISQ era urgently demands a light-weighted, low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers, and assemblers, and the capability of quantum simulators in a classical computer. QASMBench is a low-level, easy-to-use benchmark suite based on the OpenQASM-2 assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, etc., trading-off between generality and usability. Most of the QASMBench application code can be launched and verified in IBM-Q directly. For simulation purposes, you may also want to use our State-Vector simulator (SV-Sim) and Density-Matrix simulator (DM-Sim) to run on GPU/CPU HPC clusters.

To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. We provide a script under the metric folder to analyze the OpenQASM circuit and report the metrics we defined. For each benchmark, the README.md lists the metrics which include what we defined in the paper and those defined in SupermarQ.

QASMBench Benchmarks

Depending on the number of qubits used, QASMBench includes three categories. For the introduction of the benchmarking routines under each category, please see our paper for detail. For each benchmark in the following tables, we list its name, brief description, and the algorithm category it belongs to, which is based on this Nature paper by adding the categories of quantum arithmetic, quantum machine learning, and quantum communication. We try to update QASMBench with respect to IBMQ roadmap.

alt text

The 'Gates' here refers to the number of Standard OpenQASM-2 gates (see our paper) but excluding those gates in a branching if statement. It is known that physical qubits in a NISQ device follow a certain topology. Since the 2-qubit gates such as CNOT (i.e., CX) can only be performed between two adjacent physical qubits, a series of SWAP operations can be required to move the relevant qubits until they become directly-connected. Therefore, we list the number of CNOT gates in the tables.

Small-scale

Quantum circuits using 2 to 10 qubits.

Benchmark Description Algorithm Qubits Gates CNOT Reference
deutsch Deutsch algorithm with 2 qubits for f(x) = x Hidden Subgroup 2 5 1 OpenQASM
iswap An entangling swapping gate Logical Operation 2 9 2 OpenQASM
quantumwalks Quantum walks on graphs with up to 4 nodes Quantum Walk 2 11 3 Repo
grover Grover’s algorithm Search and Optimization 2 16 2 AgentANAKIN
ipea Iterative phase estimation algorithm Hidden Subgroup 2 68 30 OpenQASM
dnn 3 layer quantum neural network sample Machine Learning 2 226 42 Ref
teleportation Quantum teleportation Quantum Communication 3 8 2 Ref
qaoa Quantum approximate optimization algorithm Search and Optimization 3 15 6 Repo
toffoli Toffoli gate Logical Operation 3 18 6 Scaffold
linearsolver Solver for a linear equation of one qubit Linear Equation 3 19 4 Ref
fredkin Controlled-swap gate Logical Operation 3 19 8 Scaffold
wstate W-state preparation and assessment Logical Operation 3 30 9 OpenQASM
basis_change Transform the single-particle basis of an linearly connected electronic structure Quantum Simulation 3 53 10 OpenFermion
qrng Quantum random number generator Quantum Arithmetic 4 4 0 Paper, Repo
cat_state Coherent superposition of two coherent states with opposite phase Logical Operation 4 4 3 Scaffold
inverseqft Performs an exact inversion of quantum Fourier transform Hidden Subgroup 4 8 0 OpenQASM
adder Quantum ripple-carry adder Quantum Arithmetic 4 23 10 Scaffold
hs4 Hidden subgroup problem Hidden Subgroup 4 28 4 Scaffold
bell Circuit equivalent to Bell inequality test Logic Operation 4 33 7 Cirq
qft Quantum Fourier transform Hidden Subgroup 4 36 12 OpenQASM
variational Variational ansatz for a Jellium Hamiltonian with a linear-swap network Quantum Simulation 4 54 16 OpenFermion
vqe Variational quantum eigensolver Linear Equation 4 89 9 Scaffold
vqe_uccsd Variational quantum eigensolver with UCCSD Linear Equation 4 220 88 Scaffold
basis_trotter Implement Trotter steps for molecule LiH at equilibrium geometry Quantum Simulation 4 1626 582 OpenFermion
qec_sm Repetition code syndrome measurement Error Correction 5 5 4 OpenQASM
lpn Learning parity with noise Machine Learning 5 11 2 sampaio96
qec_en Quantum repetition code encoder Error Correction 5 25 10 sampaio96
shor Shor’s algorithm Hidden Subgroup 5 64 30 Qiskit
pea Phase estimation algorithm Hidden Subgroup 5 98 42 OpenQASM
error_correctiond3 Error correction with distance 3 and 5 qubits Error Correction 5 114 49 Ref
simons Simon’s algorithm Hidden Subgroup 6 44 14 AgentANAKIN
qaoa Quantum approximate optimization algorithm Search and Optimization 6 270 54 Cirq
vqe_uccsd Variational quantum eigensolver with UCCSD Linear Equation 6 2282 1052 Scaffold
hhl Using HHL algorithm to solve linear system of equations Linear Equation 7 689 196 Qiskit HHL
bb84 A quantum key distribution circuit Quantum Communication 8 27 0 Cirq
dnn 16-dimension quantum neural network sample Machine Learning 8 1008 192 Ref
vqe_uccsd Variational quantum eigensolver with UCCSD Linear Equation 8 10808 5488 Scaffold
qpe Quantum phase estimation algorithm Hidden Subgroup 9 123 43 AgentANAKIN
adder Quantum ripple-carry adder Quantum Arithmetic 10 142 65 OpenQASM
ising Ising model simulation via QC Quantum Simulation 10 480 90 Scaffold
hhl Using HHL algorithm to solve linear system of equations Linear Equation 10 186795 72449 Qiskit HHL

Medium-scale

Quantum circuits using 11 to 27 qubits.

Benchmark Description Algorithm Qubits Gates CNOT Reference
seca Shor's error correction algorithm for teleportation Error Correction 11 216 84 AgentANAKIN
sat Boolean satisfiability problem Search and Optimization 11 679 252 OpenQASM
cc Counterfeit coin finding problem Search and Optimization 12 22 11 OpenQASM
multiply Performing 3×5 in a quantum circuit Quantum Arithmetic 13 98 40 AgentANAKIN
gcm Generator coordinate method Quantum Chemistry 13 3148 762 GCM
bv Bernstein-Vazirani algorithm Hidden Subgroup 14 41 13 OpenQASM
hhl Using HHL algorithm to solve linear system of equations Linear Equation 14 3726506 1042859 HHL
qf21 Using quantum phase estimation to factor the number 21 Hidden Subgroup 15 311 115 AgentANAKIN
multiplier Quantum multiplier Quantum Arithmetic 15 574 246 Cirq
factor247 Factorizing 247 to 13x19 with preiod=12 Hidden Subgroup 15 610573 273071 Reproduced from Ref with modification
dnn quantum neural network sample Machine Learning 16 2016 384 Ref
qec9xz Quantum error correction 9-qubit code Error Correction 17 53 32 Ref
qft Quantum Fourier transform Hidden Subgroup 18 783 306 OpenQASM
bigadder Quantum ripple-carry adder Quantum Arithmetic 18 284 130 OpenQASM
square_root Computing the square root of an number via amplitude amplification Quantum Arithmetic 18 2300 898 Scaffold
bv Bernstein-Vazirani algorithm Hidden Subgroup 19 56 18 OpenQASM
qram Bucket brigade qRAM prototype circuit Quantum Architecture 20 223 92 Ref
bwt Binary Welded Tree: a quantum walk algorithm in continuous time domain Quantum Walk 21 462001 174800 QASMBench
cat_state Coherent superposition of two coherent states with opposite phase Logical Operation 22 22 21 QASMBench
ghz_state Greenberger-Horne-Zeilinger (GHZ) state for max entanglement Logical Operation 23 23 22 QASMBench
vqe Variational quantum eigensolver Quantum Simulation 24 2306072 1538240 QASMBench
swap_test Swap test to measure quantum state distance Machine Learning 25 230 96 QASMBench
knn Quantum K-nearest neighbor Search and Optimization 25 230 96 Ref
ising Ising model simulation via QC Quantum Simulation 26 280 50 QASMBench
wstate W-state preparation and assessment Logical Operation 27 157 52 QASMBench

Large-scale

Quantum circuits using 28 to 433 qubits or more.

Benchmark Description Algorithm Qubits Gates CNOT Reference
vqe_uccsd Variational quantum eigensolver with UCCSD Linear Equation 28 399482 296648 QASMBench
adder Quantum adder Quantum Arithmetic 28, 64, 118, 433 424, 988, 1834, 6769 195, 455, 845, 3120 QASMBench
bv Bernstein-Vazirani algorithm Hidden Subgroup 30, 70, 140, 280 78, 176, 352, 712 18, 36, 72, 152 QASMBench
bwt Binary Welded Tree: a quantum walk algorithm in continuous time domain Quantum Walk 37, 57, 97, 177 1649201, 3145201, 6113201, 12049201 632400, 1209200, 2353200, 4641200 QASMBench
cat Coherent superposition of two coherent states with opposite phase Logical Operation 35, 65, 130, 260 35, 65, 130, 260 34, 64, 129, 259 QASMBench
cc Counterfeit coin finding problem Search and Optimization 32, 64, 151, 301 62, 126, 300, 600 31, 63, 150, 300 QASMBench
dnn quantum neural network sample Machine Learning 33, 51 608, 959 248, 392 Ref
ghz Greenberger-Horne-Zeilinger (GHZ) state for max entanglement Logical Operation 40, 78, 127, 255 40, 78, 127, 255 39, 77, 126, 254 QASMBench
ising Ising model simulation via QC Quantum Simulation 34, 66, 98, 420 368, 720, 1072, 4614 66, 130, 194, 838 QASMBench
knn Quantum K-nearest neighbor Search and Optimization 31, 67, 129, 341 287, 629, 1218, 3232 120, 264, 512, 1260 Ref
multiplier Quantum multiplier Quantum Arithmetic 45, 75, 350, 400 5981, 17077, 383844, 501877 2574, 7350, 165200, 216000 QASMBench
qft Quantum Fourier transform Hidden Subgroupe 29, 63, 160, 320 2059, 9828, 63760, 255520 812, 3906, 25440, 102080 QASMBench
qugan Quantum generative adversarial network Machine Learning 39, 71, 111, 395 759, 1415, 2235, 8057 296, 552, 872, 3144 Ref
square_root Computing the square root of an number via amplitude amplification Quantum Arithmetic 45, 60 138794, 1061939 54151, 415123 QASMBench
swap_test Swap test to measure quantum state distance Machine Learning 41, 83, 115, 361 382, 781, 1085, 3422 160, 328, 456, 1440 QASMBench
wstate W-state preparation and assessment Logical Operation 36, 76, 118, 380 211, 451, 703, 2275 70, 150, 234, 758 QASMBench
quantum telecloning Prepare quantum telecloning state Quantum Cloning 201, 2001 Ref1, Ref2, Ref3
QAOA on MAX-3-SAT QAOA with random angles on random 3-SAT problems QAOA 100, 1000, 10000 Ref
quantum volume Random Quantum Volume Benchmark Circuits Quantum Volume 32, 100, 1000 Ref1, Ref2, Ref3

qelib1.inc

OpenQASM header file that defines all the gates. Please see OpenQASM and our paper for details.

QASMBenchmark Suite Structure

Each benchmark folder includes the following file:

  • bench.qasm: OpenQASM source file.
  • bench.png: Visualization of the circuit from IBM QE.
  • res_bench.png: Running results from IBM QE quantum backends (mainly 5-qubit Burlington, 15-qubit Melbourne, and 27-qubit Paris).

Tests

The QASMBench circuits can be directly uploaded and verified on IBM-Q NISQ quantum device.

Classic HPC simulation

You may also want to use our state-vector and density-matrix quantum circuit simulator (SV-Sim and DM-Sim) for simulating the QASMBench benchmark circuits efficiently on modern CPU (Intel X86, AMD X86, IBM Power), GPU (NVIDIA GPU and AMD GPU) and Xeon-Phi workstations or HPC clusters (e.g., ORNL Summit/Frontier, ANL Theta, and NERSC Cori/Perlmutter Supercomputers).

Metrics

We propose a set of circuit-based evaluation metrics representing various features of a quantum application. These metrics are designed such that through them certain estimations can be performed on executing a particular circuit over a particular NISQ device. The metrics serve as useful indicators of how a quantum circuit can stress a NISQ hardware device. Please see our paper for the math formula and analysis.

Circuit Width

Circuit width is defined as the number of qubits that enter the superposition state at least once within an application’s lifespan. Qubits that are measured in the interim of a circuit and re-enter superposition are only counted as one qubit towards the circuit width. Circuit width dictates the spatial capacity required for a quantum device in order to run the quantum circuit. The following figure shows the circuit width of QASMBench: alt text

Circuit Depth

Circuit depth is defined as the minimum time-evolution steps required to complete a quantum application. Time evolution is the process of completing all gates defined at time T=T(j), and once these are completed, the circuit moves onto time T = T(j+1), where the following gates are to be processed. Circuit depth can be computed by decomposing OpenQASM code into a n(q) x T matrix Q, where Q(q(i),t(j)) is the time-evolution steps to complete the gate on qubit i at time j. The sum of the maximum time in each column is then equal to the minimum time required for a quantum application. The following figure shows the circuit depth of QASMBench: alt text

Gate Density

Gate density, or operation density, describes the occupancy of gate slots along the time-evolution steps of a quantum circuit. As certain qubits might need to wait for other qubits in the time evolution (i.e, gate dependency), they remain idle by executing the identity gate (ID gate). Consequently, if a gate slot is empty due to dependency, it implies a lower occupancy for the quantum hardware. This is similar to a classical processor, where data dependency introduces pipeline bubbles and reduced occupancy. We propose Gate Density to measure the likely occupancy of a circuit when mapping to quantum hardware. The following figure shows the gate density of QASMBench: alt text

Retention Lifespan

Retention Lifespan describes the maximum lifespan of a qubit within a system and is motivated by the T1 and T2 coherence time of a quantum device. A longer lifespan of a quantum system implies more decay to the ground state (T1) and state-transition due to environment noise (T2), thus is more susceptible to information loss. Therefore, we propose taking the qubit with the longest lifespan to determine the system’s retention lifespan. Using this metric, one can estimate if a particular circuit can be executed in a NISQ device with high fidelity, given its T1/T2 coherence time. Note, all IBM-Q machines offer T1/T2 coherence time as status indicators for the hardware. As circuit depth can grow substantially, we introduce the log operator to shrink the scale. The following figure shows the retention lifespan of QASMBench: alt text

Measurement Density

Measurement density assesses the importance of measurements in a circuit. A higher measurement count implies the fact that each measurement might be of relatively less importance (e.g., periodic measurement in QEC, or measurement over ancilla qubits), whereas for applications with fewer measurements, the measurement may be of utmost importance. The importance also increases when a measurement accounts for a wider and/or deeper circuit. A good example is the SWAP test, where the circuit can be very large but only one measurement is taken to report the similarity. Consequently, this measurement is extremely important to the application. Since the circuit depth/width can be large and the importance of measurement decays when the circuit depth/width keeps on increasing, we add a log to shrink the scale. The following figure shows the retention lifespan of QASMBench: alt text

Entanglement Variance

Entanglement Variance measures the balance of entanglement across the qubits of a circuit. Circuits with a higher Entanglement Variance indicate that certain qubits are more connected than other qubits (i.e., using more 2-qubit gates such as CX than others). This metric implies that when the circuit is mapped to a NISQ device: (i) fewer SWAP gates are needed if those hotspot qubits are mapped to the central vertices in the NISQ device topology, such as Qubit-1 in ibmq-belem and Qubit-2 in ibmq-yorkton). A higher entanglement variance implies a higher potential benefit from a good logic-physical qubit-mapping through quantum transpilation. If the entanglement variance is zero, the little benefit should be expected from a better transpilation strategy; (ii) Given 2-qubit gate is one of the major sources introducing error, a higher Entanglement Variance implies uneven error introduction among qubits. The following figure shows the entanglement variance of QASMBench: alt text

OpenQASM

OpenQASM (Open Quantum Assembly Language) is a low-level quantum intermediate representation (IR) for quantum instructions, similar to the traditional Hardware-Description-Language (HDL) like Verilog and VHDL. OpenQASM is the open-source unified low-level assembly language for IBM quantum machines publically available on the cloud that have been investigated and verified by many existing research works. Several popular quantum software frameworks use OpenQASM as one of their output-formats, including Qiskit, Cirq, Scaffold, ProjectQ, etc.

Qiskit

The Quantum Information Software Kit (Qiskit) is a quantum software developed by IBM. It is based on Python. OpenQASM can be generated from Qiskit via:

QuantumCircuit.qasm()

Cirq

Cirq is a quantum software framework from Google. OpenQASM can be generated from Cirq (not fully compatible) via:

cirq.Circuit.to_qasm()

Scaffold

Scaffold is a quantum programming language embedded in the C/C++ programming language based on the LLVM compiler toolchain. A Scaffold program can be compiled by Scaffcc to OpenQASM via "-b" compiler option.

ProjectQ

ProjectQ is a quantum software platform developed by Steiger et al. from ETH Zurich. The official website is here. ProjectQ can generate OpenQASM when using IBM quantum machines as the backends:

IBMBackend.get_qasm()

Authors

Ang Li, Pacific Northwest National Laboratory (PNNL)

Samuel Stein, Pacific Northwest National Laboratory (PNNL)

Sriram Krishnamoorthy, Pacific Northwest National Laboratory (PNNL)

James Ang, Pacific Northwest National Laboratory (PNNL)

And also the original authors that developed these quantum routines.

Citation format

For research articles, please cite our paper:

  • Ang Li, Samuel Stein, Sriram Krishnamoorthy, and James Ang. "QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation." ACM Transactions on Quantum Computing (2022). DOI:10.1145/3550488, [arXiv:2005.13018].

Bibtex:

@article{li2022qasmbench,
  title={QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation},
  author={Li, Ang and Stein, Samuel and Krishnamoorthy, Sriram, and Ang, James},
  journal={ACM Transactions on Quantum Computing},
  year={2022},
  publisher={ACM New York, NY}
}

License

This project is licensed under the BSD License, see LICENSE file for details.

Acknowledgments

We acknowledge the support from the many developers and the open-source community.

PNNL-IPID: 31924-E, IR: PNNL-SA-153380, PNNL-SA-162867, ECCN:EAR99

This work was supported by the DOE Office of Science National Quantum Information Science Research Centers, Co-design Center for Quantum Advantage (C2QA) under contract number DE-SC0012704. This research used resources from the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. This research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory, operated under Contract No. DE-AC02-05CH11231 using NERSC award ERCAP0022228.

We also acknowledge support from Microsoft’s Azure Quantum for providing credits and access to the ion-trap quantum hardware used in our evaluation. The Pacific Northwest National Laboratory is operated by Battelle for the U.S. Department of Energy under contract DE-AC05-76RL01830.

Contributing

Please contact us If you'd like to add your circuits to the benchmark suite or if you'd like to remove your circuits from the suite.

About

A low-level OpenQASM benchmark suite for NISQ evaluation and simulation. Please see our paper for details.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published