A fastbuf is a stream (or file) abstraction optimized for both speed and flexibility.

Fastbufs can represent many different kinds of objects: regular files, network sockets, file descriptors in general, or various memory buffers. These objects are handled by different fastbuf back-ends.

Once you have a fastbuf, you can access it by functions similar to those of stdio.h, or you can use a variety of fastbuf front-ends providing various formatted operations.

There are also fastbuf wrappers, which serve as both back-ends and front-ends, taking one stream and converting it to another on the fly.

Please keep in mind that fastbufs do not allow arbitrary mixing of reads and writes on the same stream. If you need to mix them, you have to call bflush() inbetween and remember that the file position reported by btell() points after the flushed buffer, which is not necessarily the same as after the data you’ve really read.

Fastbufs can also participate in the libucw resource management system. You can tie a fastbuf to a resource in the current resource pool by fb_tie(). When the pool gets cleaned up, the fastbuf is automatically closed. If you call bclose() explicitly, the resource is removed, too.



Internal structure

Generally speaking, a fastbuf consists of a buffer and a set of callbacks. All front-end functions operate on the buffer and if the buffer becomes empty or fills up, they ask the corresponding callback to handle the situation. Back-ends then differ just in the definition of the callbacks.

The state of the fastbuf is represented by a struct fastbuf, which is a simple structure describing the state of the buffer (the pointers buffer, bufend), the front-end cursor (bptr), the back-end cursor (bstop), position of the back-end cursor in the file (pos), some flags (flags) and pointers to the callback functions.

The buffer can be in one of the following states:

  1. Flushed:

    | unused                             | free space                |
    ^              ^                     ^                           ^
    buffer      <= bstop (BE pos)     <= bptr (FE pos)            <= bufend
    • This schema describes a fastbuf after its initialization or bflush().

    • There is no cached data and we are ready for any read or write operation (well, only if the back-end supports it).

    • The interval [bptr, bufend] can be used by front-ends for writing. If it is empty, the spout callback gets called upon the first write attempt to allocate a new buffer. Otherwise the fastbuf silently comes to the writing mode.

    • When a front-end needs to read something, it calls the refill callback.

    • The pointers can be either all non-NULL or all NULL.

    • bstop == bptr in most back-ends, but it is not necessary. Some in-memory streams take advantage of this.

  2. Reading:

    | read data                          | unused                    |
    ^               ^                    ^                           ^
    buffer       <= bptr (FE pos)     <= bstop (BE pos)           <= bufend
    • If we try to read something, we get to the reading mode.

    • No writing is allowed until a flush operation. But note that bflush() will simply set bptr to bstop before spout and it breaks the position of the front-end’s cursor, so the user should seek afwards.

    • The interval [buffer, bstop] contains a block of data read by the back-end. bptr is the front-end’s cursor which points to the next character to be read. After the last character is read, bptr == bstop and the refill callback gets called upon the next read attempt to bring further data. This gives us an easy way how to implement bungetc().

  3. Writing:

    | unused                | written data   | free space            |
    ^            ^                           ^                       ^
    buffer    <= bstop (BE pos)            < bptr (FE pos)        <= bufend
    • This schema corresponds to the situation after a write attempt.

    • No reading is allowed until a flush operation.

    • The bptr points at the position where the next character will be written to. When we want to write, but bptr == bufend, we call the spout hook to flush the witten data and get an empty buffer.

    • bstop usually points at the beginning of the written data, but it is not necessary.

Rules for back-ends:

  • Front-ends are only allowed to change the value of bptr, some flags and if a fatal error occurs, then also bstop. Back-ends can rely on it.

  • buffer <= bstop <= bufend and buffer <= bptr <= bufend.

  • pos should be the real position in the file corresponding to the location of bstop in the buffer. It can be modified by any back-end’s callback, but the position of bptr (pos + (bptr - bstop)) must stay unchanged after refill or spout.

  • Failed callbacks (except close) should use bthrow().

  • Any callback pointer may be NULL in case the callback is not implemented.

  • Callbacks can change not only bptr and bstop, but also the location and size of the buffer; the fb-mem back-end takes advantage of it.

  • Initialization:

    • out: buffer <= bstop <= bptr <= bufend (flushed).

    • fb_tie() should be called on the newly created fastbuf.

  • refill:

    • in: buffer <= bstop <= bptr <= bufend (reading or flushed).

    • out: buffer <= bptr <= bstop <= bufend (reading).

    • Resulting bptr == bstop signals the end of file. The next reading attempt will again call refill which can succeed this time.

    • The callback must also return zero on EOF (iff bptr == bstop).

  • spout:

    • in: buffer <= bstop <= bptr <= bufend (writing or flushed).

    • out: buffer <= bstop <= bptr < bufend (flushed).

  • seek:

    • in: buffer <= bstop <= bptr <= bufend (flushed).

    • in: (ofs >= 0 && whence == SEEK_SET) || (ofs <= 0 && whence == SEEK_END).

    • out: buffer <= bstop <= bptr <= bufend (flushed).

  • close:

    • in: buffer <= bstop <= bptr <= bufend (flushed or after bthrow()).

    • close must always free all internal structures, even when it throws an exception.

struct fastbuf {
  byte *bptr, *bstop;                           /* State of the buffer */
  byte *buffer, *bufend;                        /* Start and end of the buffer */
  char *name;                                   /* File name (used for error messages) */
  ucw_off_t pos;                                /* Position of bstop in the file */
  uint flags;                                   /* See enum fb_flags */
  int (*refill)(struct fastbuf *);              /* Get a buffer with new data, returns 0 on EOF */
  void (*spout)(struct fastbuf *);              /* Write buffer data to the file */
  int (*seek)(struct fastbuf *, ucw_off_t, int);/* Slow path for @bseek(), buffer already flushed; returns success */
  void (*close)(struct fastbuf *);              /* Close the stream */
  int (*config)(struct fastbuf *, uint, int);   /* Configure the stream */
  int can_overwrite_buffer;                     /* Can the buffer be altered? 0=never, 1=temporarily, 2=permanently */
  struct resource *res;                         /* The fastbuf can be tied to a resource pool */

This structure contains the state of the fastbuf. See the discussion above for how it works.

enum fb_flags {
  FB_DEAD = 0x1,                                /* Some fastbuf's method has thrown an exception */
  FB_DIE_ON_EOF = 0x2,                          /* Most of read operations throw "fb.eof" on EOF */

Fastbuf flags

struct fastbuf *fb_tie(struct fastbuf *b);

Tie a fastbuf to a resource in the current resource pool. Returns the pointer to the same fastbuf.

Fastbuf on files

If you want to use fastbufs to access files, you can choose one of several back-ends and set their parameters.

enum fb_type {
  FB_STD,                               /* Standard buffered I/O */
  FB_DIRECT,                            /* Direct I/O bypassing system caches (see fb-direct.c for a description) */
  FB_MMAP                               /* Memory mapped files */

Back-end types

struct fb_params {
  enum fb_type type;                    /* The chosen back-end */
  uint buffer_size;                     /* 0 for default size */
  uint keep_back_buf;                   /* FB_STD: optimize for bi-directional access */
  uint read_ahead;                      /* FB_DIRECT options */
  uint write_back;
  struct asio_queue *asio;

When you open a file fastbuf, you can use this structure to select a back-end and set its parameters. If you want just an "ordinary" file stream, you can happily pass NULL instead and the defaults from the configuration file (or hard-wired defaults if no config file has been read) will be used.

extern struct cf_section fbpar_cf;

Configuration section with which you can fill the fb_params

extern struct fb_params fbpar_def;

The default fb_params

struct fastbuf *bopen_file(const char *name, int mode, struct fb_params *params);

Opens a file with file mode mode (see the man page of open()). Use params to select the fastbuf back-end and its parameters or pass NULL if you are fine with defaults.

Raises ucw.fb.open if the file does not exist.

struct fastbuf *bopen_file_try(const char *name, int mode, struct fb_params *params);

Like bopen_file(), but returns NULL on failure.

struct fastbuf *bopen_tmp_file(struct fb_params *params);

Opens a temporary file. It is placed with other temp files and it is deleted when closed. Again, use NULL for params if you want the defaults.

struct fastbuf *bopen_fd_name(int fd, struct fb_params *params, const char *name);

Creates a fastbuf from a file descriptor fd and sets its filename to name (the name is used only in error messages). When the fastbuf is closed, the fd is closed as well. You can override this behavior by calling bconfig().

static inline struct fastbuf *bopen_fd(int fd, struct fb_params *params);

Same as above, but with an auto-generated filename.

void bfilesync(struct fastbuf *b);

Flushes all buffers and makes sure that they are written to the disk.

Fastbufs on regular files

If you want to use the FB_STD back-end and not worry about setting up any parameters, there is a couple of shortcuts.

struct fastbuf *bopen(const char *name, uint mode, uint buflen);

Equivalent to bopen_file() with FB_STD back-end.

struct fastbuf *bopen_try(const char *name, uint mode, uint buflen);

Equivalent to bopen_file_try() with FB_STD back-end.

struct fastbuf *bopen_tmp(uint buflen);

Equivalent to bopen_tmp_file() with FB_STD back-end.

struct fastbuf *bfdopen(int fd, uint buflen);

Equivalent to bopen_fd() with FB_STD back-end.

struct fastbuf *bfdopen_shared(int fd, uint buflen);

Like bfdopen(), but it does not close the fd on bclose().

Temporary files

Usually, bopen_tmp_file() is the best way how to come to a temporary file. However, in some specific cases you can need more, so there is also a set of more general functions.

void temp_file_name(char *name_buf, int *open_flags);

Generates a temporary filename and stores it to the name_buf (of size at least * TEMP_FILE_NAME_LEN). If open_flags are not NULL, flags that should be OR-ed with other flags to open() will be stored there.

The location and style of temporary files is controlled by the configuration. By default, the system temp directory ($TMPDIR or /tmp) is used.

If the location is a publicly writeable directory (like /tmp), the generated filename cannot be guaranteed to be unique, so open_flags will include O_EXCL and you have to check the result of open() and iterate if needed.

This function is not specific to fastbufs, it can be used separately.

int open_tmp(char *name_buf, int open_flags, int mode);

Opens a temporary file and returns its file descriptor. You specify the file mode and open_flags passed to open().

If the name_buf (of at last TEMP_FILE_NAME_LEN chars) is not NULL, the filename is also stored in it.

This function is not specific to fastbufs, it can be used separately.

void bfix_tmp_file(struct fastbuf *fb, const char *name);

Sometimes, a file is created as temporary and then moved to a stable location. This function takes a fastbuf created by bopen_tmp_file() or bopen_tmp(), marks it as permanent, closes it and renames it to name.

Please note that it assumes that the temporary file and the name are on the same volume (otherwise, rename() fails), so you might want to configure a special location for the temporary files beforehand.

Fastbufs on file fragments

The fblim back-end reads from a file handle, but at most a given number of bytes. This is frequently used for reading from sockets.

struct fastbuf *bopen_limited_fd(int fd, uint bufsize, uint limit);

Create a fastbuf which reads at most limit bytes from fd.

Fastbufs on in-memory streams

The fbmem back-end keeps the whole contents of the stream in memory (as a linked list of memory blocks, so address space fragmentation is avoided).

First, you use fbmem_create() to create the stream and the fastbuf used for writing to it. Then you can call fbmem_clone_read() to get an arbitrary number of fastbuf for reading from the stream.

struct fastbuf *fbmem_create(uint blocksize);

Create stream and return its writing fastbuf.

struct fastbuf *fbmem_clone_read(struct fastbuf *f);

Given a writing fastbuf, create a new reading fastbuf.

Fastbufs on static buffers

The fbbuf back-end stores the stream in a given block of memory. This is useful for parsing and generating of complex data structures.

void fbbuf_init_read(struct fastbuf *f, byte *buffer, uint size, uint can_overwrite);

Creates a read-only fastbuf that takes its data from a given buffer. The fastbuf structure is allocated by the caller and pointed to by f. The buffer and size specify the location and size of the buffer.

In some cases, the front-ends can take advantage of rewriting the contents of the buffer temporarily. In this case, set can_overwrite as described in Internals. If you do not care, keep can_overwrite zero.

It is not possible to close this fastbuf. This implies that no tying to resources takes place.

void fbbuf_init_write(struct fastbuf *f, byte *buffer, uint size);

Creates a write-only fastbuf which writes into a provided memory buffer. The fastbuf structure is allocated by the caller and pointed to by f. An attempt to write behind the end of the buffer causes the ucw.fb.write exception.

Data are written directly into the buffer, so it is not necessary to call bflush() at any moment.

It is not possible to close this fastbuf. This implies that no tying to resources takes place.

static inline uint fbbuf_count_written(struct fastbuf *f);

Calculates, how many bytes were already written into the buffer.

Fastbuf on recyclable growing buffers

The fbgrow back-end keeps the stream in a contiguous buffer stored in the main memory, but unlike fbmem, the buffer does not have a fixed size and it is expanded to accomodate all data.

At every moment, you can use fastbuf->buffer to gain access to the stream.

struct fastbuf *fbgrow_create(uint basic_size);

Create the growing buffer pre-allocated to basic_size bytes.

struct fastbuf *fbgrow_create_mp(struct mempool *mp, uint basic_size);

Create the growing buffer pre-allocated to basic_size bytes.

void fbgrow_reset(struct fastbuf *b);

Reset stream and prepare for writing.

void fbgrow_rewind(struct fastbuf *b);

Prepare for reading (of already written data).

uint fbgrow_get_buf(struct fastbuf *b, byte **buf);

Can be used in any state of b (for example when writing or after fbgrow_rewind()) to return the pointer to internal buffer and its length in bytes. The returned buffer can be invalidated by further requests.

Fastbuf on memory pools

The write-only fbpool back-end also keeps the stream in a contiguous buffer, but this time the buffer is allocated from within a memory pool.

struct fbpool {
  struct fastbuf fb;
  struct mempool *mp;

Structure for fastbufs & mempools.

void fbpool_init(struct fbpool *fb);

Initialize a new fbpool. The structure is allocated by the caller, so bclose() should not be called and no resource tying takes place.

void fbpool_start(struct fbpool *fb, struct mempool *mp, size_t init_size);

Start a new continuous block and prepare for writing (see mp_start()). Provide the memory pool you want to use for this block as mp.

void *fbpool_end(struct fbpool *fb);

Close the block and return the address of its start (see mp_end()). The length can be determined by calling mp_size(mp, ptr).

Atomic files for multi-threaded programs

This fastbuf backend is designed for cases when several threads of a single program append records to a common file and while the record can mix in an arbitrary way, the bytes inside a single record must remain uninterrupted.

In case of files with fixed record size, we just allocate the buffer to hold a whole number of records and take advantage of the atomicity of the write() system call.

With variable-sized records, we need another solution: when writing a record, we keep the fastbuf in a locked state, which prevents buffer flushing (and if the buffer becomes full, we extend it), and we wait for an explicit commit operation which write()s the buffer if the free space in the buffer falls below the expected maximum record length.

Please note that initialization of the clones is not thread-safe, so you have to serialize it yourself.

struct fastbuf *fbatomic_open(const char *name, struct fastbuf *master, uint bufsize, int record_len);

Open an atomic fastbuf. If master is NULL, the file name is opened. If it is non-null, a new clone of an existing atomic fastbuf is created.

If the file has fixed record length, just set record_len to it. Otherwise set record_len to the expected maximum record length with a negative sign (you need not fit in this length, but as long as you do, the fastbuf is more efficient) and call fbatomic_commit() after each record.

You can specify record_len, if it is known (for optimisations).

The file is closed when all fastbufs using it are closed.

static inline void fbatomic_commit(struct fastbuf *b);

Declare that you have finished writing a record. This is required only if a fixed record size was not specified.

Null fastbufs

struct fastbuf *fbnull_open(uint bufsize);

Creates a new "/dev/null"-like fastbuf. Any read attempt returns an EOF, any write attempt is silently ignored.

void fbnull_start(struct fastbuf *b, byte *buf, uint bufsize);

Can be used by any back-end to switch it to the null mode. You need to provide at least one byte long buffer for writing.

bool fbnull_test(struct fastbuf *b);

Checks whether a fastbuf has been switched to the null mode.

Fastbufs atop other fastbufs

Imagine some code which does massive string processing. It takes an input buffer, writes a part of it into an output buffer, then some other string and then the remaining part of the input buffer. Or anything else where you copy all the data at each stage of the complicated process.

This backend takes multiple fastbufs and concatenates them formally into one. You may then read them consecutively as they were one fastbuf at all.

This backend is read-only.

This backend is seekable iff all of the supplied fastbufs are seekable.

You aren’t allowed to do anything with the underlying buffers while these are connected into fbmulti.

The fbmulti is inited by fbmulti_create(). It returns an empty fbmulti. Then you call fbmulti_append() for each fbmulti.

If bclose() is called on fbmulti, all the underlying buffers get closed recursively.

If you want to keep an underlying fastbuf open after bclose, just remove it by fbmulti_remove where the second parameter is a pointer to the removed fastbuf. If you pass NULL, all the underlying fastbufs are removed.

After fbmulti_remove, the state of the fbmulti is undefined. The only allowed operation is either another fbmulti_remove or bclose on the fbmulti.

struct fastbuf *fbmulti_create(void);

Create an empty fbmulti

void fbmulti_append(struct fastbuf *f, struct fastbuf *fb);

Append a fb to fbmulti

void fbmulti_remove(struct fastbuf *f, struct fastbuf *fb);

Remove a fb from fbmulti

Configuring stream parameters

enum bconfig_type {
  BCONFIG_IS_TEMP_FILE,                 /* 0=normal file, 1=temporary file, 2=shared fd */
  BCONFIG_KEEP_BACK_BUF,                /* Optimize for bi-directional access */

Parameters that could be configured.

int bconfig(struct fastbuf *f, uint type, int data);

Configure a fastbuf. Returns previous value.

Universal functions working on all fastbuf’s

void bclose(struct fastbuf *f);

Close and free fastbuf. Can not be used for fastbufs not returned from function (initialized in a parameter, for example the one from fbbuf_init_read).

void bthrow(struct fastbuf *f, const char *id, const char *fmt, ...) FORMAT_CHECK(printf,3,4) NONRET;

Throw exception on a given fastbuf

void bflush(struct fastbuf *f);

Write data (if it makes any sense, do not use for in-memory buffers).

void bseek(struct fastbuf *f, ucw_off_t pos, int whence);

Seek in the buffer. See man fseek for description of whence. Only for seekable fastbufs.

void bsetpos(struct fastbuf *f, ucw_off_t pos);

Set position to pos bytes from beginning. Only for seekable fastbufs.

void brewind(struct fastbuf *f);

Go to the beginning of the fastbuf. Only for seekable ones.

ucw_off_t bfilesize(struct fastbuf *f);

How large is the file? -1 if not seekable.

static inline ucw_off_t btell(struct fastbuf *f);

Where am I (from the beginning)?

static inline int bgetc(struct fastbuf *f);

Return next character from the buffer.

static inline int bpeekc(struct fastbuf *f);

Return next character from the buffer, but keep the current position.

static inline int beof(struct fastbuf *f);

Have I reached EOF?

static inline void bungetc(struct fastbuf *f);

Return last read character back. Only one back is guaranteed to work.

static inline void bputc(struct fastbuf *f, uint c);

Write a single character.

static inline uint bavailr(struct fastbuf *f);

Return the length of the cached data to be read. Do not use directly.

static inline uint bavailw(struct fastbuf *f);

Return the length of the buffer available for writing. Do not use directly.

static inline uint bread(struct fastbuf *f, void *b, uint l);

Read at most l bytes of data into b. Returns number of bytes read. 0 means end of file.

static inline uint breadb(struct fastbuf *f, void *b, uint l);

Reads exactly l bytes of data into b. If at the end of file, it returns 0. If there are data, but less than l, it raises ucw.fb.eof.

static inline void bwrite(struct fastbuf *f, const void *b, uint l);

Writes buffer b of length l into fastbuf.

char *bgets(struct fastbuf *f, char *b, uint l);

Reads a line into b and strips trailing \n. Returns pointer to the terminating 0 or NULL on EOF. Raises ucw.fb.toolong if the line is longer than l.

char *bgets0(struct fastbuf *f, char *b, uint l);

The same as bgets(), but for 0-terminated strings.

int bgets_nodie(struct fastbuf *f, char *b, uint l);

Returns either length of read string (excluding the terminator) or -1 if it is too long. In such cases exactly l bytes are read.

uint bgets_bb(struct fastbuf *f, struct bb_t *b, uint limit);

Read a string, strip the trailing \n and store it into growing buffer b. Raises ucw.fb.toolong if the line is longer than limit.

char *bgets_mp(struct fastbuf *f, struct mempool *mp);

Read a string, strip the trailing \n and store it into buffer allocated from a memory pool.

#define bgets_stk(fb) \
  ({ struct bgets_stk_struct _s; _s.f = (fb); for (bgets_stk_init(&_s); _s.cur_len; _s.cur_buf = alloca(_s.cur_len), bgets_stk_step(&_s)); _s.cur_buf; })

Read a string, strip the trailing \n and store it on the stack (allocated using alloca()).

static inline void bputs(struct fastbuf *f, const char *b);

Write a string, without 0 or \n at the end.

static inline void bputs0(struct fastbuf *f, const char *b);

Write string, including terminating 0.

static inline void bputsn(struct fastbuf *f, const char *b);

Write string and append a newline to the end.

static inline void bbcopy(struct fastbuf *f, struct fastbuf *t, uint l);

Copy l bytes of data from fastbuf f to fastbuf t. UINT_MAX (~0U) means all data, even if more than UINT_MAX bytes remain.

static inline int bskip(struct fastbuf *f, uint len);

Skip len bytes without reading them.

Direct I/O on buffers

static inline uint bdirect_read_prepare(struct fastbuf *f, byte **buf);

Begin direct reading from fastbuf’s internal buffer to avoid unnecessary copying. The function returns a buffer buf together with its length in bytes (zero means EOF) with cached data to be read.

Some back-ends allow the user to modify the data in the returned buffer to avoid unnecessary. If the back-end allows such modifications, it can set f->can_overwrite_buffer accordingly:

  • 0 if no modification is allowed,

  • 1 if the user can modify the buffer on the condition that the modifications will be undone before calling the next fastbuf operation

  • 2 if the user is allowed to overwrite the data in the buffer if bdirect_read_commit_modified() is called afterwards. In this case, the back-end must be prepared for trimming of the buffer which is done by the commit function.

The reading must be ended by bdirect_read_commit() or bdirect_read_commit_modified(), unless the user did not read or modify anything.

static inline void bdirect_read_commit(struct fastbuf *f, byte *pos);

End direct reading started by bdirect_read_prepare() and move the cursor at pos. Data in the returned buffer must be same as after bdirect_read_prepare() and pos must point somewhere inside the buffer.

static inline void bdirect_read_commit_modified(struct fastbuf *f, byte *pos);

Similar to bdirect_read_commit(), but accepts also modified data before pos. Note that such modifications are supported only if f->can_overwrite_buffer == 2.

static inline uint bdirect_write_prepare(struct fastbuf *f, byte **buf);

Start direct writing to fastbuf’s internal buffer to avoid copy overhead. The function returns the length of the buffer in buf (at least one byte) where we can write to. The operation must be ended by bdirect_write_commit(), unless nothing is written.

static inline void bdirect_write_commit(struct fastbuf *f, byte *pos);

Commit the data written to the buffer returned by bdirect_write_prepare(). The length is specified by pos which must point just after the written data. Also moves the cursor to pos.

Formatted output

int bprintf(struct fastbuf *b, const char *msg, ...);

printf into a fastbuf.

int vbprintf(struct fastbuf *b, const char *msg, va_list args);

vprintf into a fastbuf.


Fastbufs on network sockets with timeouts.

struct fbsock_params {
  int fd;
  int fd_is_shared;
  uint bufsize;
  uint timeout_ms;
  void (*err)(void *data, uint flags, char *msg);
  void *data;                   // Passed to the err callback

Configuration of socket fastbuf.

enum fbsock_err_flags {
  FBSOCK_READ = 1,              // Happened during read
  FBSOCK_WRITE = 2,             // Happened during write
  FBSOCK_TIMEOUT = 4,           // The error is a timeout

Description of a socket error

struct fastbuf *fbsock_create(struct fbsock_params *par);

Create a new socket fastbuf. All information is passed by par.


Reading and writing of unicode characters.

Invalid codes are replaced by UNI_REPLACEMENT when reading.

static inline int bget_utf8(struct fastbuf *b);

Read a single utf8 character from range [0, 0xffff].

static inline int bget_utf8_32(struct fastbuf *b);

Read a single utf8 character (from the whole unicode range).

static inline void bput_utf8(struct fastbuf *b, uint u);

Write a single utf8 character from range [0, 0xffff].

static inline void bput_utf8_32(struct fastbuf *b, uint u);

Write a single utf8 character (from the whole unicode range).

static inline int bget_utf16_be(struct fastbuf *b);

Read an utf16 character from fastbuf. Big endian version.

static inline int bget_utf16_le(struct fastbuf *b);

Read an utf16 character from fastbuf. Little endian version.

static inline void bput_utf16_be(struct fastbuf *b, uint u);

Write an utf16 character to fastbuf. Big endian version.

static inline void bput_utf16_le(struct fastbuf *b, uint u);

Write an utf16 character to fastbuf. Little endian version.


We define several functions to read or write binary integer values.

The name patterns for such routines are:

  • TYPE bget \#\# NAME \#\# ENDIAN(struct fastbuf *f);

  • void bput \#\# NAME \#\# ENDIAN(struct fastbuf *f, TYPE value);

where NAME together with TYPE can be:

  • w for 16-bit unsigned integers stored in sequences of 2 bytes, the TYPE is int

  • l for 32-bit unsigned integers stored in sequences of 4 bytes, the TYPE is uint

  • 5 for 40-bit unsigned integers stored in sequences of 5 bytes, the TYPE is u64

  • q for 64-bit unsigned integers stored in sequences of 8 bytes, the TYPE is u64

and supported ENDIAN suffixes are:

  • empty for the default order of bytes (defined by CPU)

  • _le for little-endian

  • _be for big-endian

If we fail to read enough bytes because of EOF, the reading function returns (TYPE)-1.


When debugging a program, you might wonder what strange characters are there in the output, or you might want to spice up the input with Unicode snowmen to make the program freeze.

In such situations, you can wrap your input or output stream in the hex wrapper, which converts between strange characters and their hexadecimal representation.

struct fastbuf *fb_wrap_hex_out(struct fastbuf *f);

Creates an output hex wrapper for the given fastbuf. Printable ASCII characters written to the wrapper are copied verbatim to f. Control characters, whitespace and everything outside ASCII are transcribed hexadecimally as <XY>. A newline is appended at the end of the output.

struct fastbuf *fb_wrap_hex_in(struct fastbuf *f);

Creates an input hex wrapper for the given fastbuf. It reads characters from f and translates hexadecimal sequences <XY>. All other characters are copied verbatim.


All standard back-ends and front-ends raise exceptions on errors. All such exceptions live in the ucw.fb subtree. The following exceptions are defined:


Unexpected end of file (e.g., when the FB_DIE_ON_EOF flag is set)


Memory mapping failed (e.g., the mmap syscall has failed)


Opening failed (file does not exist and similar problems)


Read error (e.g., the read syscall has failed or the stream is write-only)


Seek error (e.g., file not seekable, or a seek behind EOF)


Creation of temporary file failed


Object (typically a line) is too long


Write error (e.g., the write syscall has failed or the stream is read-only)