Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds lazy reader support for blobs #629

Merged
merged 36 commits into from
Sep 1, 2023
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
e0a83d8
Top-level nulls, bools, ints
zslayton Jul 16, 2023
89f79aa
Consolidate impls of AsUtf8 w/helper fn
zslayton Jul 25, 2023
840be4d
Improved TextBufferView docs, removed DataSource
zslayton Jul 25, 2023
5db1ff0
Adds lazy text floats
zslayton Jul 27, 2023
07d4a70
Adds LazyRawTextReader support for comments
zslayton Jul 27, 2023
181e0a5
Adds LazyRawTextReader support for reading strings
zslayton Jul 28, 2023
357ca8f
clippy fixes
zslayton Jul 28, 2023
716ff34
Fix a couple of unit tests
zslayton Jul 29, 2023
e29fec5
Less ambitious float eq comparison
zslayton Jul 29, 2023
8f79a36
Adds LazyRawTextReader support for reading symbols
zslayton Aug 1, 2023
4cb9b2b
Adds more doc comments
zslayton Aug 1, 2023
54470d2
More doc comments
zslayton Aug 1, 2023
78014e7
Adds `LazyRawTextReader` support for reading lists
zslayton Aug 3, 2023
a6a3aa8
Adds `LazyRawTextReader` support for structs
zslayton Aug 10, 2023
4fc9078
More doc comments
zslayton Aug 10, 2023
11174ac
Adds `LazyRawTextReader` support for reading IVMs
zslayton Aug 10, 2023
719dbaa
Initial impl of a LazyRawAnyReader
zslayton Aug 11, 2023
f603872
Improved comments.
zslayton Aug 11, 2023
4696ca5
Adds LazyRawTextReader support for annotations
zslayton Aug 11, 2023
c7129ac
Adds lazy reader support for timestamps
zslayton Aug 14, 2023
44435ea
Lazy reader support for s-expressions
zslayton Aug 18, 2023
d50e05b
Fixed doc comments
zslayton Aug 18, 2023
8283422
Fix internal doc link
zslayton Aug 18, 2023
0f01099
Adds lazy reader support for decimals
zslayton Aug 19, 2023
b60f1fe
Fixed bad unit test example case
zslayton Aug 20, 2023
915c83a
clippy fixes
zslayton Aug 20, 2023
fe922ff
Adds lazy reader support for blobs
zslayton Aug 20, 2023
4b53bb3
Merge remote-tracking branch 'origin/main' into lazy-timestamps
zslayton Aug 23, 2023
60d5a17
Incorporates review feedback
zslayton Aug 23, 2023
db9718d
Matcher recognizes +00:00 as Zulu
zslayton Aug 23, 2023
37264a3
Merge remote-tracking branch 'origin/lazy-timestamps' into lazy-sexps
zslayton Aug 23, 2023
74b8baf
Merge remote-tracking branch 'origin/lazy-sexps' into lazy-decimals
zslayton Aug 23, 2023
d716c9f
Merge remote-tracking branch 'origin/lazy-decimals' into lazy-blobs
zslayton Aug 23, 2023
8cecdda
Allow blobs with interleaved whitespace
zslayton Aug 28, 2023
b28df56
clippy suggestions
zslayton Aug 28, 2023
8ea5c9d
Merge remote-tracking branch 'origin/main' into lazy-blobs
zslayton Sep 1, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ num-bigint = "0.4.3"
num-integer = "0.1.44"
num-traits = "0.2"
arrayvec = "0.7"
smallvec = "1.9.0"
smallvec = {version ="1.9.0", features = ["const_generics"]}
digest = { version = "0.9", optional = true }
sha2 = { version = "0.9", optional = true }

Expand Down
21 changes: 12 additions & 9 deletions examples/lazy_read_all_values.rs
Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
#[cfg(feature = "experimental-lazy-reader")]
use ion_rs::IonResult;

#[cfg(not(feature = "experimental-lazy-reader"))]
fn main() {
println!("This example requires the 'experimental-lazy-reader' feature to work.");
}

#[cfg(feature = "experimental-lazy-reader")]
use ion_rs::IonResult;

#[cfg(feature = "experimental-lazy-reader")]
fn main() -> IonResult<()> {
lazy_reader_example::read_all_values()
}

#[cfg(feature = "experimental-lazy-reader")]
mod lazy_reader_example {
use std::fs::File;
use std::process::exit;

use memmap::MmapOptions;

use ion_rs::lazy::r#struct::LazyBinaryStruct;
use ion_rs::lazy::reader::LazyBinaryReader;
use ion_rs::lazy::sequence::LazyBinarySequence;
use ion_rs::lazy::value::LazyBinaryValue;
use ion_rs::lazy::value_ref::ValueRef;
use ion_rs::IonResult;
use memmap::MmapOptions;
use std::fs::File;
use std::process::exit;

pub fn read_all_values() -> IonResult<()> {
let args: Vec<String> = std::env::args().collect();
Expand Down Expand Up @@ -53,14 +53,17 @@ mod lazy_reader_example {
fn count_value_and_children(lazy_value: &LazyBinaryValue) -> IonResult<usize> {
use ValueRef::*;
let child_count = match lazy_value.read()? {
List(s) | SExp(s) => count_sequence_children(&s)?,
List(s) => count_sequence_children(s.iter())?,
SExp(s) => count_sequence_children(s.iter())?,
Struct(s) => count_struct_children(&s)?,
_ => 0,
};
Ok(1 + child_count)
}

fn count_sequence_children(lazy_sequence: &LazyBinarySequence) -> IonResult<usize> {
fn count_sequence_children<'a, 'b>(
lazy_sequence: impl Iterator<Item = IonResult<LazyBinaryValue<'a, 'b>>>,
) -> IonResult<usize> {
let mut count = 0;
for value in lazy_sequence {
count += count_value_and_children(&value?)?;
Expand Down
6 changes: 3 additions & 3 deletions src/binary/binary_writer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ impl<W: Write> IonWriter for BinaryWriter<W> {
panic!("Cannot set symbol ID ${symbol_id} as annotation. It is undefined.");
}
}
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text),
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text.as_ref()),
};
self.raw_writer.add_annotation(symbol_id);
}
Expand All @@ -145,7 +145,7 @@ impl<W: Write> IonWriter for BinaryWriter<W> {
));
}
}
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text),
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text.as_ref()),
};
self.raw_writer.write_symbol(symbol_id)
}
Expand All @@ -159,7 +159,7 @@ impl<W: Write> IonWriter for BinaryWriter<W> {
panic!("Cannot set symbol ID ${symbol_id} as field name. It is undefined.");
}
}
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text),
RawSymbolTokenRef::Text(text) => self.get_or_create_symbol_id(text.as_ref()),
};
self.raw_writer.set_field_name(text);
}
Expand Down
Loading
Loading