Skip to content

Commit

Permalink
Merge branch 'master' into parallel-muhash
Browse files Browse the repository at this point in the history
  • Loading branch information
@coderofstuff
coderofstuff authored Oct 9, 2024
2 parents 7cfb861 + 1274e9c commit 0556835
Show file tree
Hide file tree
Showing 26 changed files with 1,083 additions and 934 deletions.
1,182 changes: 485 additions & 697 deletions Cargo.lock
View file

Large diffs are not rendered by default.

7 changes: 3 additions & 4 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,7 @@ regex = "1.10.2"
ripemd = { version = "0.1.3", default-features = false }
rlimit = "0.10.1"
rocksdb = "0.22.0"
rv = "0.16.4"
secp256k1 = { version = "0.29.0", features = [
"global-context",
"rand-std",
Expand All @@ -243,8 +244,6 @@ sha3 = "0.10.8"
slugify-rs = "0.0.3"
smallvec = { version = "1.11.1", features = ["serde"] }
sorted-insert = "0.2.3"
statest = "0.2.2"
statrs = "0.13.0" # TODO "0.16.0"
subtle = { version = "2.5.0", default-features = false }
sysinfo = "0.31.2"
tempfile = "3.8.1"
Expand All @@ -253,8 +252,8 @@ thiserror = "1.0.50"
tokio = { version = "1.33.0", features = ["sync", "rt-multi-thread"] }
tokio-stream = "0.1.14"
toml = "0.8.8"
tonic = { version = "0.12.2", features = ["tls-webpki-roots", "gzip", "transport"] }
tonic-build = { version = "0.12.2", features = ["prost"] }
tonic = { version = "0.12.3", features = ["tls-webpki-roots", "gzip", "transport"] }
tonic-build = { version = "0.12.3", features = ["prost"] }
triggered = "0.1.2"
uuid = { version = "1.5.0", features = ["v4", "fast-rng", "serde"] }
wasm-bindgen = { version = "0.2.93", features = ["serde-serialize"] }
Expand Down
93 changes: 48 additions & 45 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,15 @@ The default branch of this repository is `master` and new contributions are cons
## Installation
<details>
<summary>Building on Linux</summary>

1. Install general prerequisites

```bash
sudo apt install curl git build-essential libssl-dev pkg-config
sudo apt install curl git build-essential libssl-dev pkg-config
```

2. Install Protobuf (required for gRPC)

```bash
sudo apt install protobuf-compiler libprotobuf-dev #Required for gRPC
```
Expand All @@ -36,16 +36,16 @@ The default branch of this repository is `master` and new contributions are cons
llvm python3-clang
```
3. Install the [rust toolchain](https://rustup.rs/)
If you already have rust installed, update it by running: `rustup update`

If you already have rust installed, update it by running: `rustup update`
4. Install wasm-pack
```bash
cargo install wasm-pack
```
4. Install wasm32 target
```bash
rustup target add wasm32-unknown-unknown
```
```
5. Clone the repo
```bash
git clone https://github.com/kaspanet/rusty-kaspa
Expand All @@ -55,52 +55,52 @@ The default branch of this repository is `master` and new contributions are cons



<details>
<details>
<summary>Building on Windows</summary>


1. [Install Git for Windows](https://gitforwindows.org/) or an alternative Git distribution.

2. Install [Protocol Buffers](https://github.com/protocolbuffers/protobuf/releases/download/v21.10/protoc-21.10-win64.zip) and add the `bin` directory to your `Path`


3. Install [LLVM-15.0.6-win64.exe](https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.6/LLVM-15.0.6-win64.exe)

Add the `bin` directory of the LLVM installation (`C:\Program Files\LLVM\bin`) to PATH

set `LIBCLANG_PATH` environment variable to point to the `bin` directory as well

**IMPORTANT:** Due to C++ dependency configuration issues, LLVM `AR` installation on Windows may not function correctly when switching between WASM and native C++ code compilation (native `RocksDB+secp256k1` vs WASM32 builds of `secp256k1`). Unfortunately, manually setting `AR` environment variable also confuses C++ build toolchain (it should not be set for native but should be set for WASM32 targets). Currently, the best way to address this, is as follows: after installing LLVM on Windows, go to the target `bin` installation directory and copy or rename `LLVM_AR.exe` to `AR.exe`.

4. Install the [rust toolchain](https://rustup.rs/)
If you already have rust installed, update it by running: `rustup update`

If you already have rust installed, update it by running: `rustup update`
5. Install wasm-pack
```bash
cargo install wasm-pack
```
6. Install wasm32 target
```bash
rustup target add wasm32-unknown-unknown
```
```
7. Clone the repo
```bash
git clone https://github.com/kaspanet/rusty-kaspa
cd rusty-kaspa
```
</details>
</details>


<details>
<details>
<summary>Building on Mac OS</summary>


1. Install Protobuf (required for gRPC)
```bash
brew install protobuf
```
2. Install llvm.
2. Install llvm.

The default XCode installation of `llvm` does not support WASM build targets.
To build WASM on MacOS you need to install `llvm` from homebrew (at the time of writing, the llvm version for MacOS is 16.0.1).
```bash
Expand Down Expand Up @@ -133,23 +133,23 @@ To build WASM on MacOS you need to install `llvm` from homebrew (at the time of
source ~/.zshrc
```
3. Install the [rust toolchain](https://rustup.rs/)
If you already have rust installed, update it by running: `rustup update`

If you already have rust installed, update it by running: `rustup update`
4. Install wasm-pack
```bash
cargo install wasm-pack
```
4. Install wasm32 target
```bash
rustup target add wasm32-unknown-unknown
```
```
5. Clone the repo
```bash
git clone https://github.com/kaspanet/rusty-kaspa
cd rusty-kaspa
```

</details>
</details>

<details>

Expand Down Expand Up @@ -184,7 +184,8 @@ To build WASM on MacOS you need to install `llvm` from homebrew (at the time of
<summary>
Kaspa CLI + Wallet
</summary>
`kaspa-cli` crate provides cli-driven RPC interface to the node and a

`kaspa-cli` crate provides a cli-driven RPC interface to the node and a
terminal interface to the Rusty Kaspa Wallet runtime. These wallets are
compatible with WASM SDK Wallet API and Kaspa NG projects.

Expand Down Expand Up @@ -235,7 +236,7 @@ cargo run --release --bin kaspad -- --testnet
```
**Testnet 11**
For participation in the 10BPS test network (TN11), see the following detailed [guide](docs/testnet11.md).
<details>
Expand All @@ -249,7 +250,7 @@ cargo run --release --bin kaspad -- --configfile /path/to/configfile.toml
# or
cargo run --release --bin kaspad -- -C /path/to/configfile.toml
```
- The config file should be a list of \<CLI argument\> = \<value\> separated by newlines.
- The config file should be a list of \<CLI argument\> = \<value\> separated by newlines.
- Whitespace around the `=` is fine, `arg=value` and `arg = value` are both parsed correctly.
- Values with special characters like `.` or `=` will require quoting the value i.e \<CLI argument\> = "\<value\>".
- Arguments with multiple values should be surrounded with brackets like `addpeer = ["10.0.0.1", "1.2.3.4"]`.
Expand Down Expand Up @@ -297,17 +298,17 @@ wRPC
**Sidenote:**
Rusty Kaspa integrates an optional wRPC
subsystem. wRPC is a high-performance, platform-neutral, Rust-centric, WebSocket-framed RPC
subsystem. wRPC is a high-performance, platform-neutral, Rust-centric, WebSocket-framed RPC
implementation that can use [Borsh](https://borsh.io/) and JSON protocol encoding.
JSON protocol messaging
is similar to JSON-RPC 1.0, but differs from the specification due to server-side
JSON protocol messaging
is similar to JSON-RPC 1.0, but differs from the specification due to server-side
notifications.
[Borsh](https://borsh.io/) encoding is meant for inter-process communication. When using [Borsh](https://borsh.io/)
both client and server should be built from the same codebase.
both client and server should be built from the same codebase.
JSON protocol is based on
JSON protocol is based on
Kaspa data structures and is data-structure-version agnostic. You can connect to the
JSON endpoint using any WebSocket library. Built-in RPC clients for JavaScript and
TypeScript capable of running in web browsers and Node.js are available as a part of
Expand All @@ -316,27 +317,21 @@ wRPC
</details>
<details>
## Benchmarking & Testing
<details>
<details>
<summary>Simulation framework (Simpa)</summary>
Logging in `kaspad` and `simpa` can be [filtered](https://docs.rs/env_logger/0.10.0/env_logger/#filtering-results) by either:
The current codebase supports a full in-process network simulation, building an actual DAG over virtual time with virtual delay and benchmarking validation time (following the simulation generation).
The current codebase supports a full in-process network simulation, building an actual DAG over virtual time with virtual delay and benchmarking validation time (following the simulation generation).
To see the available commands
```bash
```bash
cargo run --release --bin simpa -- --help
```
```
The following command will run a simulation to produce 1000 blocks with communication delay of 2 seconds and 8 BPS (blocks per second) while attempting to fill each block with up to 200 transactions.
The following command will run a simulation to produce 1000 blocks with communication delay of 2 seconds and 8 BPS (blocks per second) while attempting to fill each block with up to 200 transactions.
```bash
cargo run --release --bin simpa -- -t=200 -d=2 -b=8 -n=1000
Expand All @@ -347,7 +342,7 @@ cargo run --release --bin simpa -- -t=200 -d=2 -b=8 -n=1000
<details>
<details>
<summary>Heap Profiling</summary>
Expand All @@ -362,7 +357,7 @@ It will produce `{bin-name}-heap.json` file in the root of the workdir, that can
</details>
<details>
<details>
<summary>Tests</summary>
Expand All @@ -384,12 +379,21 @@ cd rusty-kaspa
cargo nextest run --release
```
</details>
<details>
<summary>Lints</summary>
```bash
cd rusty-kaspa
./check
```
</details>
<details>
<details>
<summary>Benchmarks</summary>
Expand All @@ -400,7 +404,7 @@ cargo bench
</details>
<details>
<details>
<summary>Logging</summary>
Expand All @@ -415,4 +419,3 @@ Logging in `kaspad` and `simpa` can be [filtered](https://docs.rs/env_logger/0.1
In this command we set the `loglevel` to `INFO`.
</details>
3 changes: 1 addition & 2 deletions components/addressmanager/Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,4 @@ thiserror.workspace = true
tokio.workspace = true

[dev-dependencies]
statrs.workspace = true
statest.workspace = true
rv.workspace = true
17 changes: 8 additions & 9 deletions components/addressmanager/src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -520,8 +520,7 @@ mod address_store_with_cache {
use kaspa_database::create_temp_db;
use kaspa_database::prelude::ConnBuilder;
use kaspa_utils::networking::IpAddress;
use statest::ks::KSTest;
use statrs::distribution::Uniform;
use rv::{dist::Uniform, misc::ks_test as one_way_ks_test, traits::Cdf};
use std::net::{IpAddr, Ipv6Addr};

#[test]
Expand Down Expand Up @@ -591,10 +590,11 @@ mod address_store_with_cache {
assert!(num_of_buckets >= 12);

// Run multiple Kolmogorov–Smirnov tests to offset random noise of the random weighted iterator
let num_of_trials = 512;
let num_of_trials = 2048; // Number of trials to run the test, chosen to reduce random noise.
let mut cul_p = 0.;
// The target uniform distribution
let target_u_dist = Uniform::new(0.0, (num_of_buckets) as f64).unwrap();
let target_uniform_dist = Uniform::new(1.0, num_of_buckets as f64).unwrap();
let uniform_cdf = |x: f64| target_uniform_dist.cdf(&x);
for _ in 0..num_of_trials {
// The weight sampled expected uniform distibution
let prioritized_address_distribution = am
Expand All @@ -603,13 +603,12 @@ mod address_store_with_cache {
.take(num_of_buckets)
.map(|addr| addr.prefix_bucket().as_u64() as f64)
.collect_vec();

let ks_test = KSTest::new(prioritized_address_distribution.as_slice());
cul_p += ks_test.ks1(&target_u_dist).0;
cul_p += one_way_ks_test(prioritized_address_distribution.as_slice(), uniform_cdf).1;
}

// Normalize and adjust p to test for uniformity, over average of all trials.
let adjusted_p = (0.5 - cul_p / num_of_trials as f64).abs();
// we do this to reduce the effect of random noise failing this test.
let adjusted_p = ((cul_p / num_of_trials as f64) - 0.5).abs();
// Define the significance threshold.
let significance = 0.10;

Expand All @@ -619,7 +618,7 @@ mod address_store_with_cache {
adjusted_p,
significance
);
assert!(adjusted_p <= significance)
assert!(adjusted_p <= significance);
}
}
}
5 changes: 5 additions & 0 deletions consensus/Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,16 @@ serde_json.workspace = true
flate2.workspace = true
rand_distr.workspace = true
kaspa-txscript-errors.workspace = true
kaspa-addresses.workspace = true

[[bench]]
name = "hash_benchmarks"
harness = false

[[bench]]
name = "check_scripts"
harness = false

[features]
html_reports = []
devnet-prealloc = ["kaspa-consensus-core/devnet-prealloc"]
Loading

0 comments on commit 0556835

Please sign in to comment.