This guide assumes that you are reasonably familiar with C and Lily. This guide explains why this tool is necessary, how manifest files work, and provides some examples of extending Lily in C.
Bindgen is a tool for writing Lily extensions in C. The Lily interpreter does not provide a mechanism to directly use foreign libraries. Instead, a foreign library (dll, dylib, so) must export information tables for the interpreter.
Bindgen takes care of generating those tables. To use bindgen, pass it the path
to a manifest file. For a given manifest file (ex: manifest/tutorial.lily),
bindgen will generate a bindings file (ex: src/lily_tutorial_bindings.h).
You will need to have Lily installed, because coretools (bindgen and docgen) use a C library (spawni) that links to Lily's library. The easiest way to get them is to download garden (the package manager) and have it install coretools. Garden is available here:
https://gitlab.com/FascinatedBox/lily-garden
Garden does not have any dependencies. Execute run_garden.lily with
install FascinatedBox/lily-coretools to get and prepare coretools.
Bindgen and docgen are located in the src directory of coretools.
Lily uses a loading mechanism called dynaload. Simply put, it only loads the symbols of a foreign module once they are referenced. This is done to save both time and memory. Dynaload is used with all foreign modules, including predefined modules where it makes a substantial difference.
The dynaload mechanism requires two tables to be present within a module that is loaded. One table holds stringified definitions of symbols, and the other holds a load function for any given symbol. Bindgen generates these tables so you don't have to think about them.
Since bindgen generates the tables for the interpreter's prelude, it needs to support the full range of interpreter symbol types. To make that work, the interpreter has a hidden manifest mode. Manifest mode allows defining symbols, but not initializing them.
Bindgen then uses introspection to determine what the subinterpreter picked up. From that information, it generates relevant bindings. This same strategy is used by docgen, and is what allows docgen to work on both native and foreign code.
The examples in this guide assume the following structure.
tutorial/
CMakeLists.txt
test.lily
cmake/
Findlily.cmake
manifest/
tutorial.lily
src/
lily_tutorial.c
--- CMakeLists.txt ---
cmake_minimum_required(VERSION 3.0.0)
project("lily-tutorial")
set(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake)
find_package(lily)
lily_add_library(tutorial src/lily_tutorial.c)
--- Findlily.cmake ---
(This file is in the interpreter's repository.)
--- tutorial.lily ---
(This line must be at the top of this file.)
import manifest
--- lily_tutorial.c ---
#include "lily.h"
#include "lily_tutorial_bindings.h"
(This line must be below all declarations.)
LILY_DECLARE_TUTORIAL_CALL_TABLE
--- test.lily ---
import "src/tutorial"
Run bindgen.lily against the tutorial.lily file to generate the missing
lily_tutorial_bindings.h file. Afterward, build the project.
The project is working if lily test.lily does not produce any output.
The language is designed to provide several assurances to native code. During native code execution, the interpreter is able to enforce those assurances. Extensions, on the other hand, are expected to operate within those assurances. The interpreter's extension api is designed to help mitigate issues, but it cannot get rid of them entirely. This section details how the api is designed, and common expectations.
Unless explicitly stated otherwise, functions in the interpreter's C api
must not be sent NULL.
If a function takes an index, it starts at 0 and must not be negative.
New values are created by pushing a value onto the stack. Reference counting is handled automatically. The interpreter will not start a gc sweep because of a value being pushed onto the stack.
Variable initialization functions are executed at parse time, and must finish by
having one extra value on the stack. They can use the interpreter's config and
push values, but must not otherwise modify the interpreter. All other
functions must call lily_return_* at the end. The most common ones are
lily_return_top and lily_return_unit.
String values must always be valid utf-8 and zero terminated. Extensions
can use lily_is_valid_utf8 to verify before creating a String. On the other
hand, ByteString allows invalid utf-8 and may not be zero terminated.
List, Tuple, classes, and non-empty variants are all treated as containers
and can use lily_con_* functions. All containers, aside from List, cannot be
resized once they have been created.
When creating a container, an element count is required. All elements specified
must be filled. An easy way to verify is to check if print crashes, since
print is not designed to handle uninitialized values. Since print does not
include the properties of classes, make sure to print the properties too.
Empty variants such as None are not containers. Use lily_arg_isa with
an autogenerated ID_* macro to check an argument before using
lily_arg_container.
The api is not all-inclusive. If you need functionality that is not present, feel free to open an issue or ask.
The simplest definition is one that does not take arguments and returns Unit.
----- manifest
define no_op
----- source
void lily_tutorial__no_op(lily_state *s)
{
lily_return_unit(s);
}
----- test
print(tutorial.no_op) # <built-in function no_op>
Returning a more interesting value.
----- manifest
define return_10: Integer
----- source
void lily_tutorial__return_10(lily_state *s)
{
lily_return_integer(s, 10);
}
----- test
print(tutorial.return_10()) # 10
Add two Integer values to produce another.
----- manifest
define add(left: Integer, right: Integer): Integer
----- source
void lily_tutorial__add(lily_state *s)
{
int64_t arg_left = lily_arg_integer(s, 0);
int64_t arg_right = lily_arg_integer(s, 1);
int64_t result = arg_left + arg_right;
lily_return_integer(s, result);
}
----- test
print(tutorial.add(10, 45)) # 55
Variable arguments are always packaged in a List.
----- manifest
define combine(values: Integer...): Integer
----- source
void lily_tutorial__combine(lily_state *s)
{
lily_container_val *arg_values = lily_arg_container(s, 0);
int64_t total = 0;
uint32_t count = lily_con_size(arg_values);
uint32_t i;
for (i = 0;i < count;i++) {
lily_value *boxed_value = lily_con_get(arg_values, i);
int64_t value = lily_as_integer(boxed_value);
total += value;
}
lily_return_integer(s, total);
}
----- test
print(tutorial.combine(1, 20, 300)) # 321
Optional arguments are best handled with lily_optional_* function. Optional
functions return the argument at the given index, or the default provided. With
optional arguments, the source function is responsible for returning the values
mentioned in the manifest file.
The interpreter carries a msgbuf for simple string operations. lily_mb_get
returns it as well as clearing out the contents. The lily_mb_sprintf returns a
pointer to the msgbuf's underlying buffer, so that lily_return_string can copy
it. The pointer should not be stored for later, because subsequent msgbuf add
operations may invalidate it.
----- manifest
define strings(a: String, b: *String="two", c: *String="three"): String
----- source
void lily_tutorial__strings(lily_state *s)
{
const char *arg_a = lily_arg_string_raw(s, 0);
const char *arg_b = lily_optional_string_raw(s, 1, "two");
const char *arg_c = lily_optional_string_raw(s, 2, "three");
lily_msgbuf *msgbuf = lily_msgbuf_get(s);
const char *result = lily_mb_sprintf(msgbuf,
"%s,%s,%s", arg_a, arg_b, arg_c);
lily_return_string(s, result);
}
----- test
print(tutorial.strings("one")) # one, two, three
print(tutorial.strings("four", "five")) # four, five, three
print(tutorial.strings("four", "five", "six")) # four, five, six
Keyword arguments are reorganized and passed positionally.
----- manifest
define keyed_add(:left a: Integer, :right b: Integer): Integer
----- source
void lily_tutorial__keyed_add(lily_state *s)
{
int64_t arg_left = lily_arg_integer(s, 0);
int64_t arg_right = lily_arg_integer(s, 1);
int64_t result = arg_left + arg_right;
lily_return_integer(s, result);
}
----- test
print(tutorial.keyed_add(:left 2, :right 1)) # 3
Keyed optional functions can be handled with optional functions as well.
----- manifest
define keyopt_abc(:a a: *Integer=1,
:b b: *Integer=10,
:c c: *Integer=100): Integer
----- source
void lily_tutorial__keyopt_abc(lily_state *s)
{
int64_t arg_a = lily_optional_integer(s, 0, 1);
int64_t arg_b = lily_optional_integer(s, 1, 10);
int64_t arg_c = lily_optional_integer(s, 2, 100);
int64_t result = arg_a + arg_b + arg_c;
lily_return_integer(s, result);
}
----- test
print(tutorial.keyopt_abc(:b 50)) # 151
Definitions in manifest mode are allowed to make use of the $1 type. When used
as a requirement, $1 allows any incoming type. As a result, $1 returns the
types provided to it. In an overwhelming majority of cases, $1 is unnecessary.
This type is how List.zip is able to handle any number of values. It's also
how FascinatedBox/lily-dis is able to export a single dis definition that
takes any Function type. Care must be taken when using this type.
This example takes two arguments so that it cannot create an empty Tuple.
----- manifest
define make_tuple[A](first: A, values: $1...): Tuple[A, $1]
----- source
void lily_tutorial__make_tuple(lily_state *s)
{
lily_value *first_arg = lily_arg_value(s, 0);
lily_container_val *value_list = lily_arg_container(s, 1);
uint32_t size = lily_con_size(value_list);
lily_container_val *result = lily_push_tuple(s, size + 1);
uint32_t i;
lily_con_set(result, 0, first_arg);
for (i = 0;i < size;i++) {
lily_value *source_value = lily_con_get(value_list, i);
lily_con_set(result, i + 1, source_value);
}
lily_return_top(s);
}
----- test
print(tutorial.make_tuple(1)) # <[1]>
print(tutorial.make_tuple(1, "asdf")) # <[1, "asdf"]>
print(tutorial.make_tuple(1, "asdf", [234])) # <[1, "asdf", [234]]>
A foreign class is a Lily class that wraps over a C typedef. Foreign classes cannot be inherited from and cannot contain native values. Foreign classes are tracked through reference counting only. They are not considered capable of a cycle, and cannot be gc tagged.
Methods in a class have the same abilities as they do in code mode, except that the definitions are empty like above.
----- manifest
foreign class Container(value: Integer)
{
public define return_value: Integer
}
----- source
typedef struct
{
LILY_FOREIGN_HEADER
int64_t value;
} lily_tutorial_Container;
static void destroy_Container(lily_tutorial_Container *con)
{
/* Delete anything allocated inside of the container.
The container itself is deleted automatically. */
}
void lily_tutorial_Container_new(lily_state *s)
{
/* Initializing this value also pushes it to the top of the stack. */
lily_tutorial_Container *result = INIT_Container(s);
int64_t arg_value = lily_arg_integer(s, 0);
result->value = arg_value;
lily_return_top(s);
}
void lily_tutorial_Container_return_value(lily_state *s)
{
lily_tutorial_Container *arg_container = ARG_Container(s, 0);
int64_t value = arg_container->value;
lily_return_integer(s, value);
}
----- test
print(tutorial.Container(5).return_value()) # 5
A foreign static class is a class that does not have a constructor. This is used
in situations like the Conn class of FascinatedBox/lily-postgres, which has
an open function that returns a Result to pattern match on.
----- manifest
foreign static class StatCon
{
public static define create(value: Integer): StatCon
public define return_value: Integer
}
----- source
typedef struct
{
LILY_FOREIGN_HEADER
int64_t value;
} lily_tutorial_StatCon;
static void destroy_StatCon(lily_tutorial_StatCon *con)
{
/* Delete anything allocated inside of the StatCon.
The StatCon itself is deleted automatically. */
}
void lily_tutorial_StatCon_create(lily_state *s)
{
lily_tutorial_StatCon *result = INIT_StatCon(s);
int64_t arg_value = lily_arg_integer(s, 0);
result->value = arg_value;
lily_return_top(s);
}
void lily_tutorial_StatCon_return_value(lily_state *s)
{
lily_tutorial_StatCon *arg_StatCon = ARG_StatCon(s, 0);
int64_t value = arg_StatCon->value;
lily_return_integer(s, value);
}
----- test
print(tutorial.StatCon.create(5).return_value())
A native class is the same kind of class defined in Lily code. Native classes are allowed to have properties of any scope, and methods that are protected or public. They can also be inherited by classes defined in native code. Native classes in foreign modules should only contain data types that cannot contain cycles.
----- manifest
class StringBox(text: String)
{
public var @text: String
public define return_text: String
}
----- source
void lily_tutorial_StringBox_new(lily_state *s)
{
/* 1 is how many properties the StringBox has. */
lily_container_val *con = lily_push_super(s, ID_StringBox(s), 1);
/* Self is not at 0, even if inheriting. */
lily_value *text_arg = lily_arg_value(s, 0);
lily_con_set(con, 0, text_arg);
lily_return_top(s);
}
void lily_tutorial_StringBox_return_text(lily_state *s)
{
lily_container_val *arg_box = lily_arg_container(s, 0);
lily_value *text_value = GET_StringBox__text(arg_box);
lily_return_value(s, text_value);
}
----- test
print(tutorial.StringBox("asdf").return_text()) # asdf
print(tutorial.StringBox("qwerty").text) # qwerty
Variants of enums declared in manifest mode should not be capable of containing cycles. Outside of that, they carry the same abilities as enums declared in code mode. There are no extra qualifiers for enums.
The scoped keyword is available here, but is not included in the example
because the same bindings are generated.
----- manifest
class StringBox(text: String)
{
public var @text: String
public define return_text: String
}
----- source
void lily_tutorial_StringBox_new(lily_state *s)
{
/* 1 is how many properties StringBox has. */
lily_container_val *con = lily_push_super(s, ID_StringBox(s), 1);
const char *arg_text = lily_arg_string_raw(s, 0);
lily_push_string(s, arg_text);
lily_con_set_from_stack(s, con, 0);
lily_return_top(s);
}
void lily_tutorial_StringBox_return_text(lily_state *s)
{
lily_container_val *arg_box = lily_arg_container(s, 0);
lily_value *text_value = GET_StringBox__text(arg_box);
lily_return_value(s, text_value);
}
----- test
print(tutorial.StringBox("asdf").return_text()) # asdf
print(tutorial.StringBox("qwerty").text) # qwerty
Variants of enums declared in manifest mode should not be capable of containing cycles. Outside of that, they carry the same abilities as enums declared in code mode. There are no extra qualifiers for enums.
The scoped keyword is available here, but is not included in the example
because the same bindings are generated.
----- manifest
enum Example {
Empty,
Value(Integer, String)
define first_or(fallback: Integer): Integer
define second_or(fallback: String): String
}
define make_empty: Example
define make_value(first: Integer, second: String): Example
----- source
void lily_tutorial_Example_first_or(lily_state *s)
{
// Do not do this.
// Empty variants are not containers, and this will crash.
// lily_container_val *con = lily_arg_container(s, 0);
int64_t result;
/* Note: Empty variants are not containers, */
if (lily_arg_isa(s, 0, ID_Empty(s)))
result = lily_arg_integer(s, 1);
else {
lily_container_val *con = lily_arg_container(s, 0);
lily_value *value = lily_con_get(con, 0);
result = lily_as_integer(value);
}
lily_return_integer(s, result);
}
void lily_tutorial_Example_second_or(lily_state *s)
{
const char *result;
/* Note: Empty variants are not containers, */
if (lily_arg_isa(s, 0, ID_Empty(s)))
result = lily_arg_string_raw(s, 1);
else {
lily_container_val *con = lily_arg_container(s, 0);
lily_value *value = lily_con_get(con, 1);
result = lily_as_string_raw(value);
}
lily_return_string(s, result);
}
void lily_tutorial__make_empty(lily_state *s)
{
PUSH_Empty(s);
lily_return_top(s);
}
void lily_tutorial__make_value(lily_state *s)
{
/* This pushes the variant to the top of the stack. */
lily_container_val *con = INIT_Value(s);
lily_value *first_arg = lily_arg_value(s, 0);
lily_value *second_arg = lily_arg_value(s, 1);
/* Filled variants are containers with the first element at 0. */
lily_con_set(con, 0, first_arg);
lily_con_set(con, 1, second_arg);
lily_return_top(s);
}
----- test
var enum_values = [
tutorial.Empty,
tutorial.Value(10, "asdf"),
tutorial.make_empty(),
tutorial.make_value(100, "qwerty"),
]
print(enum_values[0]) # Empty
print(enum_values[1]) # Value(10, "asdf")
print(enum_values[2]) # Empty
print(enum_values[3]) # Value(100, "qwerty")
print(enum_values[0].first_or(5)) # 5
print(enum_values[1].first_or(5)) # 10
print(enum_values[2].first_or(5)) # 5
print(enum_values[3].first_or(5)) # 100
print(enum_values[0].second_or("fallback")) # fallback
print(enum_values[1].second_or("fallback")) # asdf
print(enum_values[2].second_or("fallback")) # fallback
print(enum_values[3].second_or("fallback")) # qwerty
Variables declared in manifest mode require a type, but do not allow an initial value. Instead, initialization is done by a foreign function. Variables declared by a foreign function should have a type that is not capable of having a cycle.
Variable initialization functions are executing during parsing. They are allowed to use configuration information and push values onto the stack. However, they must not call into the interpreter, raise an error, or otherwise modify the interpreter.
A key difference between variable initialization functions and other functions
is how they yield a value to the interpreter. The functions above all finish by
pushing an extra value onto the stack, then calling a lily_return function. In
contrast, variable initialization functions must finish by having one extra
value on the top of the stack, and must not call a lily_return function.
The top of stack will become the value of the variable.
The following are examples of creating various kinds of values.
----- manifest
var falsey: Boolean
var truthy: Boolean
----- source
void lily_tutorial_var_falsey(lily_state *s)
{
lily_push_boolean(s, 0);
}
void lily_tutorial_var_truthy(lily_state *s)
{
lily_push_boolean(s, 1);
}
----- test
print(tutorial.falsey) # false
print(tutorial.truthy) # true
----- manifest
var byte: Byte
----- source
void lily_tutorial_var_byte(lily_state *s)
{
lily_push_byte(s, 32);
}
----- test
print(tutorial.byte.to_i()) # 32
----- manifest
var bytestring: ByteString
----- source
void lily_tutorial_var_bytestring(lily_state *s)
{
const char *to_push = "asdf";
lily_push_bytestring(s, to_push, strlen(to_push));
}
----- test
print(tutorial.bytestring) # asdf
----- manifest
var double: Double
----- source
void lily_tutorial_var_double(lily_state *s)
{
lily_push_double(s, 1.5);
}
----- test
print(tutorial.double) # 1.5
Variables of type Function should not be created.
----- manifest
var integer_hash: Hash[Integer, String]
var string_hash: Hash[String, Double]
----- source
void lily_tutorial_var_integer_hash(lily_state *s)
{
/* Small hash, so reserve 4 slots. Underfilling is ok. */
lily_hash_val *result = lily_push_hash(s, 4);
lily_push_integer(s, 5);
lily_push_string(s, "asdf");
/* This takes two elements from the stack.
Value is on the top, key underneath. */
lily_hash_set_from_stack(s, result);
}
void lily_tutorial_var_string_hash(lily_state *s)
{
lily_hash_val *result = lily_push_hash(s, 4);
lily_push_string(s, "a");
lily_push_double(s, 1.5);
lily_hash_set_from_stack(s, result);
}
----- test
print(tutorial.integer_hash) # [5 => "asdf"]
print(tutorial.string_hash) # ["a" => 1.0]
----- manifest
var int: Integer
----- source
void lily_tutorial_var_int(lily_state *s)
{
lily_push_integer(s, 75);
}
----- test
print(tutorial.int) # 75
----- manifest
var empty_list: List[Double]
var string_list: List[String]
----- source
void lily_tutorial_var_empty_list(lily_state *s)
{
/* Create a List of 0 elements. */
lily_push_list(s, 0);
}
void lily_tutorial_var_string_list(lily_state *s)
{
/* 3 elements indexed from 0. All elements must be initialized.
Uninitialized values usually cause a crash when used. */
lily_container_val *result = lily_push_list(s, 3);
lily_push_string(s, "one");
lily_con_set_from_stack(s, result, 0);
lily_push_string(s, "two");
lily_con_set_from_stack(s, result, 1);
lily_push_string(s, "three");
lily_con_set_from_stack(s, result, 2);
}
----- test
print(tutorial.empty_list) # []
print(tutorial.string_list) # ["one", "two", "three"]
This example does not include an empty Tuple, because creating empty Tuple
values is not allowed.
----- manifest
var tuple: Tuple[String, Integer]
----- source
void lily_tutorial_var_tuple(lily_state *s)
{
/* 2 elements indexed from 0. All elements must be initialized.
Uninitialized values usually cause a crash when used. */
lily_container_val *result = lily_push_tuple(s, 2);
lily_push_string(s, "asdf");
lily_con_set_from_stack(s, result, 0);
lily_push_integer(s, 123);
lily_con_set_from_stack(s, result, 1);
}
----- test
print(tutorial.tuple) # <["asdf", 123]>
Creating a user-defined class.
----- manifest
class VarClass(value: Integer)
{
public var @value: Integer
}
var varclass: VarClass
----- source
void lily_tutorial_VarClass_new(lily_state *s)
{
/* 1 is how many properties VarClass has. */
lily_container_val *con = lily_push_super(s, ID_VarClass(s), 1);
/* Self is not at 0, even if inheriting. */
lily_value *value_arg = lily_arg_value(s, 0);
lily_con_set(con, 0, value_arg);
lily_return_top(s);
}
void lily_tutorial_var_varclass(lily_state *s)
{
/* 1 is how many properties VarClass has. */
lily_container_val *result = lily_push_instance(s, ID_VarClass(s), 1);
lily_push_integer(s, 123);
lily_con_set_from_stack(s, result, 0);
}
----- test
print(tutorial.varclass.value) # 123