Introduction

This guide assumes that you are reasonably familiar with C and Lily. This guide explains why this tool is necessary, how manifest files work, and provides some examples of extending Lily in C.

Bindgen is a tool for writing Lily extensions in C. The Lily interpreter does not provide a mechanism to directly use foreign libraries. Instead, a foreign library (dll, dylib, so) must export information tables for the interpreter.

Bindgen takes care of generating those tables. To use bindgen, pass it the path to a manifest file. For a given manifest file (ex: manifest/tutorial.lily), bindgen will generate a bindings file (ex: src/lily_tutorial_bindings.h).

Download

You will need to have Lily installed, because coretools (bindgen and docgen) use a C library (spawni) that links to Lily's library. The easiest way to get them is to download garden (the package manager) and have it install coretools. Garden is available here:

https://gitlab.com/FascinatedBox/lily-garden

Garden does not have any dependencies. Execute run_garden.lily with install FascinatedBox/lily-coretools to get and prepare coretools.

Bindgen and docgen are located in the src directory of coretools.

Why

Lily uses a loading mechanism called dynaload. Simply put, it only loads the symbols of a foreign module once they are referenced. This is done to save both time and memory. Dynaload is used with all foreign modules, including predefined modules where it makes a substantial difference.

The dynaload mechanism requires two tables to be present within a module that is loaded. One table holds stringified definitions of symbols, and the other holds a load function for any given symbol. Bindgen generates these tables so you don't have to think about them.

Since bindgen generates the tables for the interpreter's prelude, it needs to support the full range of interpreter symbol types. To make that work, the interpreter has a hidden manifest mode. Manifest mode allows defining symbols, but not initializing them.

Bindgen then uses introspection to determine what the subinterpreter picked up. From that information, it generates relevant bindings. This same strategy is used by docgen, and is what allows docgen to work on both native and foreign code.

Setup

The examples in this guide assume the following structure.

tutorial/
    CMakeLists.txt
    test.lily

    cmake/
        Findlily.cmake

    manifest/
        tutorial.lily

    src/
        lily_tutorial.c

--- CMakeLists.txt ---

cmake_minimum_required(VERSION 3.0.0)
project("lily-tutorial")
set(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake)
find_package(lily)
lily_add_library(tutorial src/lily_tutorial.c)

--- Findlily.cmake ---

(This file is in the interpreter's repository.)

--- tutorial.lily ---

(This line must be at the top of this file.)
import manifest

--- lily_tutorial.c ---

#include "lily.h"
#include "lily_tutorial_bindings.h"

(This line must be below all declarations.)
LILY_DECLARE_TUTORIAL_CALL_TABLE

--- test.lily ---

import "src/tutorial"

Run bindgen.lily against the tutorial.lily file to generate the missing lily_tutorial_bindings.h file. Afterward, build the project.

The project is working if lily test.lily does not produce any output.

Expectations

The language is designed to provide several assurances to native code. During native code execution, the interpreter is able to enforce those assurances. Extensions, on the other hand, are expected to operate within those assurances. The interpreter's extension api is designed to help mitigate issues, but it cannot get rid of them entirely. This section details how the api is designed, and common expectations.

Unless explicitly stated otherwise, functions in the interpreter's C api must not be sent NULL.

If a function takes an index, it starts at 0 and must not be negative.

New values are created by pushing a value onto the stack. Reference counting is handled automatically. The interpreter will not start a gc sweep because of a value being pushed onto the stack.

Variable initialization functions are executed at parse time, and must finish by having one extra value on the stack. They can use the interpreter's config and push values, but must not otherwise modify the interpreter. All other functions must call lily_return_* at the end. The most common ones are lily_return_top and lily_return_unit.

String values must always be valid utf-8 and zero terminated. Extensions can use lily_is_valid_utf8 to verify before creating a String. On the other hand, ByteString allows invalid utf-8 and may not be zero terminated.

List, Tuple, classes, and non-empty variants are all treated as containers and can use lily_con_* functions. All containers, aside from List, cannot be resized once they have been created.

When creating a container, an element count is required. All elements specified must be filled. An easy way to verify is to check if print crashes, since print is not designed to handle uninitialized values. Since print does not include the properties of classes, make sure to print the properties too.

Empty variants such as None are not containers. Use lily_arg_isa with an autogenerated ID_* macro to check an argument before using lily_arg_container.

The api is not all-inclusive. If you need functionality that is not present, feel free to open an issue or ask.

Definitions

The simplest definition is one that does not take arguments and returns Unit.

----- manifest

define no_op

----- source

void lily_tutorial__no_op(lily_state *s)
{
    lily_return_unit(s);
}

----- test

print(tutorial.no_op) # <built-in function no_op>

Returning a more interesting value.

----- manifest

define return_10: Integer

----- source

void lily_tutorial__return_10(lily_state *s)
{
    lily_return_integer(s, 10);
}

----- test

print(tutorial.return_10()) # 10

Add two Integer values to produce another.

----- manifest

define add(left: Integer, right: Integer): Integer

----- source

void lily_tutorial__add(lily_state *s)
{
    int64_t arg_left = lily_arg_integer(s, 0);
    int64_t arg_right = lily_arg_integer(s, 1);
    int64_t result = arg_left + arg_right;

    lily_return_integer(s, result);
}

----- test

print(tutorial.add(10, 45)) # 55

Variable arguments are always packaged in a List.

----- manifest

define combine(values: Integer...): Integer

----- source

void lily_tutorial__combine(lily_state *s)
{
    lily_container_val *arg_values = lily_arg_container(s, 0);
    int64_t total = 0;
    uint32_t count = lily_con_size(arg_values);
    uint32_t i;

    for (i = 0;i < count;i++) {
        lily_value *boxed_value = lily_con_get(arg_values, i);
        int64_t value = lily_as_integer(boxed_value);

        total += value;
    }

    lily_return_integer(s, total);
}

----- test

print(tutorial.combine(1, 20, 300)) # 321

Optional arguments are best handled with lily_optional_* function. Optional functions return the argument at the given index, or the default provided. With optional arguments, the source function is responsible for returning the values mentioned in the manifest file.

The interpreter carries a msgbuf for simple string operations. lily_mb_get returns it as well as clearing out the contents. The lily_mb_sprintf returns a pointer to the msgbuf's underlying buffer, so that lily_return_string can copy it. The pointer should not be stored for later, because subsequent msgbuf add operations may invalidate it.

----- manifest

define strings(a: String, b: *String="two", c: *String="three"): String

----- source

void lily_tutorial__strings(lily_state *s)
{
    const char *arg_a = lily_arg_string_raw(s, 0);
    const char *arg_b = lily_optional_string_raw(s, 1, "two");
    const char *arg_c = lily_optional_string_raw(s, 2, "three");
    lily_msgbuf *msgbuf = lily_msgbuf_get(s);
    const char *result = lily_mb_sprintf(msgbuf,
            "%s,%s,%s", arg_a, arg_b, arg_c);

    lily_return_string(s, result);
}

----- test

print(tutorial.strings("one"))                 # one, two, three
print(tutorial.strings("four", "five"))        # four, five, three
print(tutorial.strings("four", "five", "six")) # four, five, six

Keyword arguments are reorganized and passed positionally.

----- manifest

define keyed_add(:left a: Integer, :right b: Integer): Integer

----- source

void lily_tutorial__keyed_add(lily_state *s)
{
    int64_t arg_left = lily_arg_integer(s, 0);
    int64_t arg_right = lily_arg_integer(s, 1);
    int64_t result = arg_left + arg_right;

    lily_return_integer(s, result);
}

----- test

print(tutorial.keyed_add(:left 2, :right 1)) # 3

Keyed optional functions can be handled with optional functions as well.

----- manifest

define keyopt_abc(:a a: *Integer=1,
                  :b b: *Integer=10,
                  :c c: *Integer=100): Integer

----- source

void lily_tutorial__keyopt_abc(lily_state *s)
{
    int64_t arg_a = lily_optional_integer(s, 0, 1);
    int64_t arg_b = lily_optional_integer(s, 1, 10);
    int64_t arg_c = lily_optional_integer(s, 2, 100);
    int64_t result = arg_a + arg_b + arg_c;

    lily_return_integer(s, result);
}

----- test

print(tutorial.keyopt_abc(:b 50)) # 151

Definitions in manifest mode are allowed to make use of the $1 type. When used as a requirement, $1 allows any incoming type. As a result, $1 returns the types provided to it. In an overwhelming majority of cases, $1 is unnecessary.

This type is how List.zip is able to handle any number of values. It's also how FascinatedBox/lily-dis is able to export a single dis definition that takes any Function type. Care must be taken when using this type.

This example takes two arguments so that it cannot create an empty Tuple.

----- manifest

define make_tuple[A](first: A, values: $1...): Tuple[A, $1]

----- source

void lily_tutorial__make_tuple(lily_state *s)
{
    lily_value *first_arg = lily_arg_value(s, 0);
    lily_container_val *value_list = lily_arg_container(s, 1);
    uint32_t size = lily_con_size(value_list);
    lily_container_val *result = lily_push_tuple(s, size + 1);
    uint32_t i;

    lily_con_set(result, 0, first_arg);

    for (i = 0;i < size;i++) {
        lily_value *source_value = lily_con_get(value_list, i);

        lily_con_set(result, i + 1, source_value);
    }

    lily_return_top(s);
}

----- test

print(tutorial.make_tuple(1))                # <[1]>
print(tutorial.make_tuple(1, "asdf"))        # <[1, "asdf"]>
print(tutorial.make_tuple(1, "asdf", [234])) # <[1, "asdf", [234]]>

Foreign classes

A foreign class is a Lily class that wraps over a C typedef. Foreign classes cannot be inherited from and cannot contain native values. Foreign classes are tracked through reference counting only. They are not considered capable of a cycle, and cannot be gc tagged.

Methods in a class have the same abilities as they do in code mode, except that the definitions are empty like above.

----- manifest

foreign class Container(value: Integer)
{
    public define return_value: Integer
}

----- source

typedef struct
{
    LILY_FOREIGN_HEADER
    int64_t value;
} lily_tutorial_Container;

static void destroy_Container(lily_tutorial_Container *con)
{
    /* Delete anything allocated inside of the container.
       The container itself is deleted automatically. */
}

void lily_tutorial_Container_new(lily_state *s)
{
    /* Initializing this value also pushes it to the top of the stack. */
    lily_tutorial_Container *result = INIT_Container(s);
    int64_t arg_value = lily_arg_integer(s, 0);

    result->value = arg_value;
    lily_return_top(s);
}

void lily_tutorial_Container_return_value(lily_state *s)
{
    lily_tutorial_Container *arg_container = ARG_Container(s, 0);
    int64_t value = arg_container->value;

    lily_return_integer(s, value);
}

----- test

print(tutorial.Container(5).return_value()) # 5

A foreign static class is a class that does not have a constructor. This is used in situations like the Conn class of FascinatedBox/lily-postgres, which has an open function that returns a Result to pattern match on.

----- manifest

foreign static class StatCon
{
    public static define create(value: Integer): StatCon
    public define return_value: Integer
}

----- source

typedef struct
{
    LILY_FOREIGN_HEADER
    int64_t value;
} lily_tutorial_StatCon;

static void destroy_StatCon(lily_tutorial_StatCon *con)
{
    /* Delete anything allocated inside of the StatCon.
       The StatCon itself is deleted automatically. */
}

void lily_tutorial_StatCon_create(lily_state *s)
{
    lily_tutorial_StatCon *result = INIT_StatCon(s);
    int64_t arg_value = lily_arg_integer(s, 0);

    result->value = arg_value;
    lily_return_top(s);
}

void lily_tutorial_StatCon_return_value(lily_state *s)
{
    lily_tutorial_StatCon *arg_StatCon = ARG_StatCon(s, 0);
    int64_t value = arg_StatCon->value;

    lily_return_integer(s, value);
}

----- test

print(tutorial.StatCon.create(5).return_value())

Native classes

A native class is the same kind of class defined in Lily code. Native classes are allowed to have properties of any scope, and methods that are protected or public. They can also be inherited by classes defined in native code. Native classes in foreign modules should only contain data types that cannot contain cycles.

----- manifest

class StringBox(text: String)
{
    public var @text: String

    public define return_text: String
}

----- source

void lily_tutorial_StringBox_new(lily_state *s)
{
    /* 1 is how many properties the StringBox has. */
    lily_container_val *con = lily_push_super(s, ID_StringBox(s), 1);

    /* Self is not at 0, even if inheriting. */
    lily_value *text_arg = lily_arg_value(s, 0);

    lily_con_set(con, 0, text_arg);
    lily_return_top(s);
}

void lily_tutorial_StringBox_return_text(lily_state *s)
{
    lily_container_val *arg_box = lily_arg_container(s, 0);
    lily_value *text_value = GET_StringBox__text(arg_box);

    lily_return_value(s, text_value);
}

----- test

print(tutorial.StringBox("asdf").return_text()) # asdf
print(tutorial.StringBox("qwerty").text)        # qwerty

Enums

Variants of enums declared in manifest mode should not be capable of containing cycles. Outside of that, they carry the same abilities as enums declared in code mode. There are no extra qualifiers for enums.

The scoped keyword is available here, but is not included in the example because the same bindings are generated.

----- manifest

class StringBox(text: String)
{
    public var @text: String

    public define return_text: String
}

----- source

void lily_tutorial_StringBox_new(lily_state *s)
{
    /* 1 is how many properties StringBox has. */
    lily_container_val *con = lily_push_super(s, ID_StringBox(s), 1);
    const char *arg_text = lily_arg_string_raw(s, 0);

    lily_push_string(s, arg_text);
    lily_con_set_from_stack(s, con, 0);
    lily_return_top(s);
}

void lily_tutorial_StringBox_return_text(lily_state *s)
{
    lily_container_val *arg_box = lily_arg_container(s, 0);
    lily_value *text_value = GET_StringBox__text(arg_box);

    lily_return_value(s, text_value);
}

----- test

print(tutorial.StringBox("asdf").return_text()) # asdf
print(tutorial.StringBox("qwerty").text)        # qwerty

Enums

Variants of enums declared in manifest mode should not be capable of containing cycles. Outside of that, they carry the same abilities as enums declared in code mode. There are no extra qualifiers for enums.

The scoped keyword is available here, but is not included in the example because the same bindings are generated.

----- manifest

enum Example {
    Empty,
    Value(Integer, String)

    define first_or(fallback: Integer): Integer
    define second_or(fallback: String): String
}

define make_empty: Example
define make_value(first: Integer, second: String): Example

----- source

void lily_tutorial_Example_first_or(lily_state *s)
{
    // Do not do this.
    // Empty variants are not containers, and this will crash.
    // lily_container_val *con = lily_arg_container(s, 0);
    int64_t result;

    /* Note: Empty variants are not containers,  */
    if (lily_arg_isa(s, 0, ID_Empty(s)))
        result = lily_arg_integer(s, 1);
    else {
        lily_container_val *con = lily_arg_container(s, 0);
        lily_value *value = lily_con_get(con, 0);

        result = lily_as_integer(value);
    }

    lily_return_integer(s, result);
}

void lily_tutorial_Example_second_or(lily_state *s)
{
    const char *result;

    /* Note: Empty variants are not containers,  */
    if (lily_arg_isa(s, 0, ID_Empty(s)))
        result = lily_arg_string_raw(s, 1);
    else {
        lily_container_val *con = lily_arg_container(s, 0);
        lily_value *value = lily_con_get(con, 1);

        result = lily_as_string_raw(value);
    }

    lily_return_string(s, result);
}

void lily_tutorial__make_empty(lily_state *s)
{
    PUSH_Empty(s);
    lily_return_top(s);
}

void lily_tutorial__make_value(lily_state *s)
{
    /* This pushes the variant to the top of the stack. */
    lily_container_val *con = INIT_Value(s);
    lily_value *first_arg = lily_arg_value(s, 0);
    lily_value *second_arg = lily_arg_value(s, 1);

    /* Filled variants are containers with the first element at 0. */
    lily_con_set(con, 0, first_arg);
    lily_con_set(con, 1, second_arg);
    lily_return_top(s);
}

----- test

var enum_values = [
    tutorial.Empty,
    tutorial.Value(10, "asdf"),
    tutorial.make_empty(),
    tutorial.make_value(100, "qwerty"),
]

print(enum_values[0]) # Empty
print(enum_values[1]) # Value(10, "asdf")
print(enum_values[2]) # Empty
print(enum_values[3]) # Value(100, "qwerty")

print(enum_values[0].first_or(5)) # 5
print(enum_values[1].first_or(5)) # 10
print(enum_values[2].first_or(5)) # 5
print(enum_values[3].first_or(5)) # 100

print(enum_values[0].second_or("fallback")) # fallback
print(enum_values[1].second_or("fallback")) # asdf
print(enum_values[2].second_or("fallback")) # fallback
print(enum_values[3].second_or("fallback")) # qwerty

Variables

Variables declared in manifest mode require a type, but do not allow an initial value. Instead, initialization is done by a foreign function. Variables declared by a foreign function should have a type that is not capable of having a cycle.

Variable initialization functions are executing during parsing. They are allowed to use configuration information and push values onto the stack. However, they must not call into the interpreter, raise an error, or otherwise modify the interpreter.

A key difference between variable initialization functions and other functions is how they yield a value to the interpreter. The functions above all finish by pushing an extra value onto the stack, then calling a lily_return function. In contrast, variable initialization functions must finish by having one extra value on the top of the stack, and must not call a lily_return function. The top of stack will become the value of the variable.

The following are examples of creating various kinds of values.

Boolean

----- manifest

var falsey: Boolean
var truthy: Boolean

----- source

void lily_tutorial_var_falsey(lily_state *s)
{
    lily_push_boolean(s, 0);
}

void lily_tutorial_var_truthy(lily_state *s)
{
    lily_push_boolean(s, 1);
}

----- test

print(tutorial.falsey) # false
print(tutorial.truthy) # true

Byte

----- manifest

var byte: Byte

----- source

void lily_tutorial_var_byte(lily_state *s)
{
    lily_push_byte(s, 32);
}

----- test

print(tutorial.byte.to_i()) # 32

ByteString

----- manifest

var bytestring: ByteString

----- source

void lily_tutorial_var_bytestring(lily_state *s)
{
    const char *to_push = "asdf";

    lily_push_bytestring(s, to_push, strlen(to_push));
}

----- test

print(tutorial.bytestring) # asdf

Double

----- manifest

var double: Double

----- source

void lily_tutorial_var_double(lily_state *s)
{
    lily_push_double(s, 1.5);
}

----- test

print(tutorial.double) # 1.5

Function

Variables of type Function should not be created.

Hash

----- manifest

var integer_hash: Hash[Integer, String]
var string_hash: Hash[String, Double]

----- source

void lily_tutorial_var_integer_hash(lily_state *s)
{
    /* Small hash, so reserve 4 slots. Underfilling is ok. */
    lily_hash_val *result = lily_push_hash(s, 4);

    lily_push_integer(s, 5);
    lily_push_string(s, "asdf");

    /* This takes two elements from the stack.
       Value is on the top, key underneath. */
    lily_hash_set_from_stack(s, result);
}

void lily_tutorial_var_string_hash(lily_state *s)
{
    lily_hash_val *result = lily_push_hash(s, 4);

    lily_push_string(s, "a");
    lily_push_double(s, 1.5);
    lily_hash_set_from_stack(s, result);
}

----- test

print(tutorial.integer_hash) # [5 => "asdf"]
print(tutorial.string_hash)  # ["a" => 1.0]

Integer

----- manifest

var int: Integer

----- source

void lily_tutorial_var_int(lily_state *s)
{
    lily_push_integer(s, 75);
}

----- test

print(tutorial.int) # 75

List

----- manifest

var empty_list: List[Double]
var string_list: List[String]

----- source

void lily_tutorial_var_empty_list(lily_state *s)
{
    /* Create a List of 0 elements. */
    lily_push_list(s, 0);
}

void lily_tutorial_var_string_list(lily_state *s)
{
    /* 3 elements indexed from 0. All elements must be initialized.
       Uninitialized values usually cause a crash when used. */
    lily_container_val *result = lily_push_list(s, 3);

    lily_push_string(s, "one");
    lily_con_set_from_stack(s, result, 0);
    lily_push_string(s, "two");
    lily_con_set_from_stack(s, result, 1);
    lily_push_string(s, "three");
    lily_con_set_from_stack(s, result, 2);
}

----- test

print(tutorial.empty_list)  # []
print(tutorial.string_list) # ["one", "two", "three"]

Tuple

This example does not include an empty Tuple, because creating empty Tuple values is not allowed.

----- manifest

var tuple: Tuple[String, Integer]

----- source

void lily_tutorial_var_tuple(lily_state *s)
{
    /* 2 elements indexed from 0. All elements must be initialized.
       Uninitialized values usually cause a crash when used. */
    lily_container_val *result = lily_push_tuple(s, 2);

    lily_push_string(s, "asdf");
    lily_con_set_from_stack(s, result, 0);
    lily_push_integer(s, 123);
    lily_con_set_from_stack(s, result, 1);
}

----- test

print(tutorial.tuple) # <["asdf", 123]>

User class

Creating a user-defined class.

----- manifest

class VarClass(value: Integer)
{
    public var @value: Integer
}

var varclass: VarClass

----- source

void lily_tutorial_VarClass_new(lily_state *s)
{
    /* 1 is how many properties VarClass has. */
    lily_container_val *con = lily_push_super(s, ID_VarClass(s), 1);

    /* Self is not at 0, even if inheriting. */
    lily_value *value_arg = lily_arg_value(s, 0);

    lily_con_set(con, 0, value_arg);
    lily_return_top(s);
}

void lily_tutorial_var_varclass(lily_state *s)
{
    /* 1 is how many properties VarClass has. */
    lily_container_val *result = lily_push_instance(s, ID_VarClass(s), 1);

    lily_push_integer(s, 123);
    lily_con_set_from_stack(s, result, 0);
}

----- test

print(tutorial.varclass.value) # 123