Platymuus

Runtime-fmt, a runtime formatting crate

Wed, 05 Apr 2017 15:42:00 -0700

runtime-fmt is a small, nightly-only, safe Rust crate which provides variants of the six standard formatting macros which validate their format strings at runtime. It has feature parity with std::fmt and accepts exactly the same syntax for format strings and arguments, with the additional feature of supporting format strings which are not known at compile time.

#[macro_use] extern crate runtime_fmt;

fn main() {
	let format_string = "Hello, {}!";
	rt_println!(format_string, "world").unwrap();
	rt_println!("bogus {} reference").unwrap_err();
	rt_println!("bogus }{ format string").unwrap_err();
}

Applications include internationalization, configuration, and modding. For example, designers might be allowed to adjust text with interspersed values in a video game, or command-line users could specify what shape the output should be.

The core implementation is under 300 SLOC, plus 550 or so to embed libfmt_macros.

Implementation

runtime-fmt is safe. It achieves direct feature-parity with the standard formatting macros by using the same code that rustc uses to parse format strings. Somewhat awkwardly, this code is included inline in the crate rather than linked in, because it is distributed only dynamically, and most Rust users don’t want to link libstd dynamically. On top of format string parsing, rustc includes elaborate machinery for converting the parsed format string into an AST for macro expansion. Because runtime-fmt does not need to work on the AST level, and can make assumptions about the values returned by libfmt_macros, this conversion is vastly simplified.

All format specifiers are supported. Recognizing at runtime whether a given argument supports a given format string is achieved through specialization, implementing internal shadow traits of the standard formatting traits which return None when unavailable, and surfacing this as an error to users.

All errors are encapsulated in an enum Error, returned within a Result from all the macros. Syntax errors are returned as-is, including descriptions similar to those generated by rustc, and other types of error variants are available, including std::io::Error and std::fmt::Error variants which may be returned when using rt_write! or rt_writeln!.

The crate requires nightly for obvious reasons; access to the formatting machinery isn’t stable, and might never be. That said, it changes rarely, and the crate is unlikely to break often. The use of specialization described earlier also requires nightly.

Test Suite

This crate passes the main body of tests from rustc’s test suite, as well as a small smattering of tests written specifically for runtime-fmt. I intend to continue to expand the test suite, both to include more tests from rustc and to more rigorously ensure that format string and argument combinations which ought to error do.

Future Expansion

Some performance is left on the table in the initial implementation. Particularly, format strings are re-parsed each time the string is to be formatted. I have an idea on how to fix this, revolving around specifying a struct or tuple type in rt_format_args! once during parsing, where the format string is parsed and types are resolved and checked, and then supplying values of that struct type which can be formatted quickly by the prebuilt formatter.

Most of std::fmt is also available in core. The library currently uses heap allocation in many instances, but it may be possible to adapt it to use static buffers or other trickery in those situations and thereby allow the library to be used under #[no_std], just like the builtin formatting macros.

Addendum: thanks illegalprime on IRC for pointing out that the snippet at the top of the blog post didn’t demonstrate error handling.

Sketch of overloaded short-circuiting operators

Sun, 25 Sep 2016 11:30:00 -0700

Rust’s std::ops module provides a variety of traits for overloading Rust’s operators, but conspicuously skips the && and || operators. While I don’t believe these operators are that important to overload, I wanted to attempt to address one of the barriers to that functionality, namely short-circuiting, and provide a starting point for someone who actually cares to write an RFC.

As a refresher, short-circuiting is the behavior that these operators sometimes don’t evaluate their right-hand sides at all, depending on the value of their left-hand sides (true || x is always true, false && x is always false). This prevents representing these operators with function calls, because a functions’s arguments must be evaluated before it can be called. Even a closure isn’t enough - because an expression like lhs && return is valid, but cannot be transformed to and(lhs, || return), which has different semantics.

A more involved approach is needed. Fundamentally, these short-circuiting operators evaluate the left-hand side and depending on that value either short-circuit to a result or evaluate the right-hand side and combine it with the left-hand side in an operation. To represent these two steps, our trait will need two methods, one to represent the short-circuit check and one for the actual operation. For the rest of this post, I will be speaking about the && operator; the || operator works pretty much the same.

trait And<Rhs=Self> {
	type Output;

	fn and_short(&self) -> Option<Self::Output>;
	// alternate:
	fn and_short(self) -> Result<Self::Output, Self>;

	fn and(self, rhs: Rhs) -> Self::Output;
}

In this sketch, and_short takes &self because if it returns None, we need to pass that same self to and. If we wanted to allow moving in and_short, its signature could instead be changed to the listed alternate, where Ok(v) is a short-circuited result and Err(s) returns the value to be used as the self value of and.

The desugaring of lhs && rhs then becomes fairly straightforward. To eliminate ambiguity, I’ve represented it as a macro:

macro_rules! and {
	($lhs:expr, $rhs:expr) => {{
		let lhs = $lhs;
		match And::and_short(&lhs) {
			Some(value) => value,
			None => And::and(lhs, $rhs),
		}
	}}
}

The left-hand operand is always evaluated immediately, then and_short is called. If it returns Some, the right-hand operand is not evaluted; if it returns None, the right-hand side is evaluated and and is called. Here’s how an implementation for bool might look, if we needed to implement it in Rust:

impl And for bool {
	type Output = bool;
	fn and_short(&self) -> Option<bool> {
		match *self {
			false => Some(false),
			true => None,
		}
	}
	fn and(self, rhs: bool) -> bool {
		match (self, rhs) {
			(true, true) => true,
			_ => false,
		}
	}
}

And here’s what an implementation for a hypothetical ternary logic value might look like:

enum Tri {
	False = -1,
	None = 0,
	True = 1,
}

impl And for Tri {
	type Output = Tri;
	fn and_short(&self) -> Option<Tri> {
		match *self {
			Tri::False => Some(Tri::False),
			_ => None,
		}
	}
	fn and(self, rhs: Tri) -> Tri {
		match (self, rhs) {
			(Tri::False, _) |
			(_, Tri::False) => Tri::False,
			(Tri::None, _) |
			(_, Tri::None) => Tri::None,
			(Tri::True, Tri::True) => Tri::True,
		}
	}
}

Note that it’s important for consistency of behavior that for values of self where and_short returns Some(v), and returns v for any value of rhs.

A full example, including an Or trait is available as a gist and can be run on the playground. Thanks to those on the #rust IRC who inspired me to want to write this post and poked holes in my earlier ideas.

Wrapping callbacks without userdata

Fri, 29 Jul 2016 08:08:00 -0700

It’s easy to find articles describing how to wrap Rust closures to pass to C libraries that will carry a void* to your callback, but what can you do when no userdata value is provided? This post is based on work I’ve done for the lua crate.

The Situation

Let’s imagine that the C library provides an API roughly analogous to the following:

type ffi_Object = ...; // opaque type representing foreign Object
type ffi_Callback = unsafe extern fn(*mut ffi_Object) -> u32;
extern {
	fn ffi_push_callback(_: *mut ffi_Object, _: ffi_Callback);
}

We currently have a safe wrapper type named Object and we want to enable consumers of our library to push callbacks onto it without writing any unsafe code, but how can we do that without a userdata pointer? While the lack of a userdata pointer prevents us from carrying any state with the callback (ruling out closures), it’s still possible to wrap ordinary functions.

The Solution

First, let’s think about the signature we want the users of our wrapper to conform to. The FFI demands unsafe extern fn(*mut ffi_Object) -> u32, but we want to allow such callbacks to be totally safe. This means no unsafe or extern, and we should replace *mut ffi_Object with a safe analogue. Since the callback does not take ownership, this is &mut Object.

Now we’re going to need to define a function that converts from our safe signature to the unsafe signature. At first, you might try to write:

fn wrap_callback(callback: fn(&mut Object) -> u32) -> ffi_Callback { ... }

but this leads to a dead end. A fn(&mut Object) -> u32 is a function pointer, a runtime value, state! Exactly what we can’t pack into an ffi_Callback. We’re going to need to build, at compile time, a different ffi_Callback for each function we’re ever passed. Luckily, Rust has the useful property that each function gets its own unique type, so we can use a type parameter - our wrap_callback function is going to be generic. These unique types can’t be written directly, so let’s add a dummy parameter _: F to allow inference to fill them in for us.

fn wrap_callback<F: Fn(&mut Object) -> u32>(_: F) -> ffi_Callback {

Now that we know the function’s signature, what about its contents? First, let’s not forget to reintroduce a restriction we just accidentally loosened: this signature accepts closures with state, and it’s not possible for us to support that. Let’s be sure we’re being passed ordinary functions (or equally-valid stateless closures):

	assert!(mem::size_of::<F>() == 0);

It’s a little unfortunate that we have to runtime assert this and can’t check it at compile time, but it’s the best we can do in current Rust. Make sure this requirement is displayed in your library’s documentation.

Alright, now what? Well, we need to define a function with the signature demanded by ffi_Callback. Unlike wrap_callback, which accepts a function as a type parameter and returns an ffi_Callback, this function will accept the same and be an ffi_Callback.

	unsafe extern fn wrapped<F: Fn(&mut Object) -> u32>(ptr: *mut ffi_Object) -> u32 {

First, let’s convert our *mut ffi_Object into a safe Object we can use:

		let mut object = Object::from_ptr(ptr);

Now we need to obtain a value of type F in order to call it. We just asserted that F is zero-sized, so there’s exactly one possible value, but the compiler doesn’t know that. We’ll have to unsafely construct that value somehow - probably the simplest way is mem::zeroed. Then, we’re going to call that function with a reference to the Object, and save the result:

		//           | generate value ||  call it  |
		let result = mem::zeroed::<F>()(&mut object);

Lastly, since our Object is an owned type, we need to avoid dropping it, then return the result we just generated. A future post will cover how to handle this in a less risky way.

		mem::forget(object);
		result
	}

Now we’re back in wrap_callback, ready to return the function satisfying ffi_Closure we just built:

	wrapped::<F>
}

And that’s it! Using it is simple, and as we hoped, contains no use of unsafe:

fn our_callback(object: &mut Object) -> u32 { ... }
fn push_our_callback(object: &mut Object) {
	let wrapped: ffi_Callback = wrap_callback(our_callback);
	object.push_callback(wrapped);
}

Depending on your needs, you can make wrap_callback public or use it internally.

Full Example

Here’s the code in this article all put together, including a dummy version of Object which I omitted for brevity. If you want to experiment, you can use this code on the Rust Playground. I’m interested in feedback and can accept it by email or the comments section of wherever you found this post.

#![allow(non_camel_case_types)]
use std::mem;

// The Situation
struct ffi_Object;
type ffi_Callback = unsafe extern fn(*mut ffi_Object) -> u32;
extern fn ffi_push_callback(_: *mut ffi_Object, _: ffi_Callback) {}

// Dummy wrapper
struct Object {
	ptr: *mut ffi_Object
}

impl Object {
	fn new() -> Object {
		// initialize ffi_Object here
		Object { ptr: std::ptr::null_mut() }
	}
	fn from_ptr(ptr: *mut ffi_Object) -> Object {
		Object { ptr: ptr }
	}
	fn push_callback(&mut self, cb: ffi_Callback) {
		ffi_push_callback(self.ptr, cb);
	}
}

impl Drop for Object {
	fn drop(&mut self) {
		// destroy ffi_Object here
	}
}

// The Solution
fn wrap_callback<F: Fn(&mut Object) -> u32>(_: F) -> ffi_Callback {
	assert!(mem::size_of::<F>() == 0);

	unsafe extern fn wrapped<F: Fn(&mut Object) -> u32>(ptr: *mut ffi_Object) -> u32 {
		let mut object = Object::from_ptr(ptr);
		let result = mem::transmute::<_, &F>(&())(&mut object);
		mem::forget(object);
		result
	}

	wrapped::<F>
}

// Usage
fn our_callback(_: &mut Object) -> u32 {
	0
}
fn main() {
	let mut object = Object::new();
	let wrapped: ffi_Callback = wrap_callback(our_callback);
	object.push_callback(wrapped);
}

Addendum: Safer, Less Usable

If you’re not satisfied with the runtime assert, it can be eliminated if you’re willing to sacrifice usability:

// Definition
trait Callback {
	fn call(object: &mut Object) -> i32;
}

fn wrap_callback<F: Callback>() -> ffi_Callback {
	unsafe extern fn wrapped<F: Callback>(object: *mut ffi_Object) -> i32 {
		let mut object = Object::from_ptr(ptr);
		let result = F::call(&mut object);
		mem::forget(object);
		result
	}
	wrapped::<F>
}

// Usage
struct OurCallback;
impl Callback for OurCallback {
	fn call(object: &mut Object) -> u32 {
		...
	}
}

fn push_our_callback(object: &mut Object) {
	object.push_callback(wrap_callback::<OurCallback>());
}

Thanks to Lalaland on #rust for this suggestion.

Addendum: The Past

There was once a time when those unique per-function types we made use of were not zero-sized! This was just an implementation flaw and has since been resolved, but it was possible to work around it using closures that captured no state, and were zero-sized:

fn push_our_callback(object: &mut Object) {
	object.push_callback(wrap_callback(|s| our_callback(s)));
}