Skip to content

Commit

Permalink
Make clear that correct_cursor not part of API yet
Browse files Browse the repository at this point in the history
  • Loading branch information
smheidrich committed Jul 5, 2023
1 parent e4feeae commit 252d5a7
Showing 1 changed file with 7 additions and 6 deletions.
13 changes: 7 additions & 6 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,13 @@ enum State {
/// stream: Python file-like object / stream to read JSON from. Can be
/// either in text mode or in binary mode (so long as the bytes are valid
/// UTF-8).
/// correct_cursor: Whether it is required that the cursor is left in the
/// correct position (behind the last processed character) after
/// park_cursor() has been called. If set to False, performance for
/// unseekable streams is drastically improved at the cost of the cursor
/// ending up in places unrelated to the actual tokenization progress. For
/// seekable streams, the improvement shouldn't be noticable.
/// correct_cursor: *(not part of API yet, may be removed at any point)*
/// Whether it is required that the cursor is left in the correct position
/// (behind the last processed character) after park_cursor() has been
/// called. If set to False, performance for unseekable streams is
/// drastically improved at the cost of the cursor ending up in places
/// unrelated to the actual tokenization progress. For seekable streams, the
/// improvement shouldn't be noticable.
#[pyclass]
#[pyo3(text_signature = "(stream, *, correct_cursor=True)")]
struct RustTokenizer {
Expand Down

0 comments on commit 252d5a7

Please sign in to comment.