bufio

Lookahead_Reader provides io lookahead. This is useful for tokenizers/parsers. Lookahead_Reader is similar to bufio.Reader, but unlike bufio.Reader, Lookahead_Reader's buffer size will EXACTLY match the specified size, whereas bufio.Reader's buffer size may differ from the specified size. This makes sure that the buffer will not be accidentally read beyond the expected size.

Read_Writer stores pointers to a Reader and a Writer

Reader is a buffered wrapper for an io.Reader

Extra errors returns by scanning procedures

Split_Proc is the signature of the split procedure used to tokenize the input.

import "core:bytes" Writer is a buffered wrapper for an io.Writer

lookahead_reader_consume drops the first n populated bytes from the Lookahead_Reader.

lookahead_reader_peek returns a slice of the Lookahead_Reader which holds n bytes If the Lookahead_Reader cannot hold enough bytes, it will read from the underlying reader to populate the rest. NOTE: The returned buffer is not a copy of the underlying buffer

lookahead_reader_peek_all returns a slice of the Lookahead_Reader populating the full buffer If the Lookahead_Reader cannot hold enough bytes, it will read from the underlying reader to populate the rest. NOTE: The returned buffer is not a copy of the underlying buffer

reader_buffered returns the number of bytes that can be read from the current buffer

reader_destroy destroys the underlying buffer with its associated allocator if and only if (⟺) that allocator has been set

reader_discard skips the next n bytes, and returns the number of bytes that were discarded

reader_init initializes using an allocator

reader_init initializes using a user provided bytes buffer buf

reader_peek returns the next n bytes without advancing the reader The bytes stop being valid on the next read call If reader_peek returns fewer than n bytes, it also return an error explaining why the read is short The error will be .Buffer_Full if n is larger than the internal buffer size

reader_read reads data into p The bytes are taken from at most one read on the underlying Reader, which means n may be less than len(p)

reader_read_byte reads and returns a single byte If no byte is available, it return an error

reader_read_bytes reads until the first occurrence of delim from the Reader It returns an allocated slice containing the data up to and including the delimiter

reader_read_rune reads a single UTF-8 encoded unicode character and returns the rune and its size in bytes If the encoded rune is invalid, it consumes one byte and returns utf8.RUNE_ERROR (U+FFFD) with a size of 1

reader_read_slice reads until the first occurrence of delim in the input, returning a slice pointing at the bytes in the internal buffer. The returned slice is only valid until the next read call. If the buffer fills without finding delim, it returns .Buffer_Full. If the underlying reader returns an error before finding delim, that error is returned. Because the returned data will be overwritten by the next I/O operation, reader_read_bytes or reader_read_string is usually preferred.

reader_read_slice returns err != nil if and only if line does not end in delim.

reader_read_string reads until the first occurrence of delim from the Reader It returns an allocated string containing the data up to and including the delimiter

reader_reset resets the read and write positions, and the error values

reader_size returns the number of bytes in the backing buffer

reader_to_stream converts a Reader into an io.Stream

reader_unread_byte unreads the last byte. Only the most recently read byte can be unread

reader_unread_rune unreads the last rune. Only the most recently read rune can be unread

scan advances the Scanner

scan_bytes is a splitting procedure that returns each byte as a token

scan_lines is a splitting procedure that returns each line of text stripping of any trailing newline and an optional preceding carriage return (\r?\n). A new line is allowed to be empty.

scan_runes is a splitting procedure that returns each UTF-8 encoded rune as a token. The lsit of runes return is equivalent to that of iterating over a string in a 'for in' loop, meaning any erroneous UTF-8 encodings will be returned as U+FFFD. Unfortunately this means it is impossible for the "client" to know whether a U+FFFD is an expected replacement rune or an encoding of an error.

scan_words is a splitting procedure that returns each Unicode-space-separated word of text, excluding the surrounded spaces. It will never return return an empty string.

Returns the most recent token created by 'scan'. The underlying array may point to data that may be overwritten by another call to 'scan'. Treat the returned value as if it is immutable.

Returns the first non-EOF error that was encountered by the scanner

Initializes a Scanner buffer an allocator buf_allocator

Initializes a Scanner buffer a user provided bytes buffer buf

scanner_scan is an alias of scan

Returns the most recent token created by 'scan'. The underlying array may point to data that may be overwritten by another call to 'scan'. Treat the returned value as if it is immutable.

writer_available returns how many bytes are unused in the buffer

writer_buffered returns the number of bytes that have been writted into the current buffer

writer_destroy destroys the underlying buffer with its associated allocator if and only if (⟺) that allocator has been set

writer_flush writes any buffered data into the underlying io.Writer

Initialized a Writer with an allocator

Initialized a Writer with a user provided buffer buf

writer_read_from is to support io.Reader_From types If the underlying writer supports the io,read_from, and b has no buffered data yet, this procedure calls the underlying read_from implementation without buffering

writer_size returns the size of underlying buffer in bytes

writer_to_stream converts a Writer into an io.Stream

writer_to_stream converts a Writer into an io.Stream

writer_write writes the contents of p into the buffer It returns the number of bytes written If n < len(p), it will return an error explaining why the write is short

writer_write_byte writes a single byte

writer_write_rune writes a single unicode code point, and returns the number of bytes written with any error

writer_write_string writes a string into the buffer It returns the number of bytes written If n < len(p), it will return an error explaining why the write is short