add modules section

Signed-off-by: Diogo Behrens <[email protected]>

Author: db7

doc/design.md

@@ -150,9 +150,9 @@ The Pubsub system introduces several key advantages:
     metadata to the subscribers. Actual type defined by `chain`.
 
 2. **Publishing**: Publishers (such as intercept modules or the Self module)
-   publish events by calling `PS_PUBLISH(chain_id, type_id, void*, metadata_t*)`. The
-   publisher specifies the chain, event type, event payload, and a potentially
-   `NULL` metadata object.
+   publish events by calling `PS_PUBLISH(chain_id, type_id, void*,
+   metadata_t*)`. The publisher specifies the chain, event type, event payload,
+   and a potentially `NULL` metadata object.
 
 3. **Callback Handling**: The callback function is where the action happens for
    the subscriber. Upon receiving an event, the callback can inspect and process
@@ -482,3 +482,74 @@ By using these interpose modules, Dice can gather detailed execution data,
 track thread behavior, monitor memory usage, and enable advanced testing and
 debugging features.
 
+
+# 6. Loading Modules
+
+Dice is designed to be loaded as a shared library with `LD_PRELOAD`.
+The core library in Dice has only the Pubsub and the Mempool modules inside.
+Addtional modules can be loaded in two ways.
+
+
+## 6.1. Shared Library Modules
+
+Consider a multithreaded application `foo`. To run `foo` with Dice the user
+simply starts it with the following command:
+
+```
+env LD_PRELOAD=/path/to/libdice.so foo <arg1>
+```
+
+Additional modules can be loaded with the `LD_PRELOAD` variable as well:
+
+```
+env LD_PRELOAD=/path/to/libdice.so:/path/to/mod-pthread_create.so:... foo <arg1>
+```
+
+Subscription callbacks are internally kept as lists of function pointers. The
+corresponding indirection cost has to be considered when deploying Dice in this
+way.
+
+### Subscription Order
+
+One has to be careful in which order the constructors of the modules are
+executed.  On NetBSD the constructors are executed left-to-right. Given a list
+of libraries `LD_PRELOAD=libdice.so:mod1.so:mod2.so`, the subscribers in
+`mod1.so` will be registered earlier than subscribers in `mod2.so`; therefore,
+callbacks in `mod1.so` will be called first.  On Linux, the constructors are
+executed right-to-left. So the opposite subscription order will result.
+
+In general, the user has to figure out the order the constructors are called by
+the linker and order the shared libraries accordingly.
+
+## 6.2. Monolithic Shared Library
+
+Modules in Dice can also linked together with the core components in a single
+shared library. In this configuration, each module is a compilation unit, i.e.,
+a `.o` file.  When compiling each of these files, the user should specify a
+priority for the subscription order by passing `-DDICE_XTOR_PRIO=VAL` to the
+compiler. `VAL` should be a number between 200 and 1000. The lower the value,
+the higher the priority when subscribing chains.
+
+Assume the user creates a library `libmydice.so` containing `pubsub.o`,
+`mempool.o`, `mod-pthread_create.o` and a user-defined module `mymod.o`. To load
+this bundle, the user simply uses `LD_PRELOAD`:
+
+```
+env LD_PRELOAD=/path/to/libmydice.so foo <arg1>
+```
+
+If the user compiles all modules with LTO option, the resulting library can be
+much faster than relying on the approach of section 6.1. Nevertheless, without
+further steps, the resulting library above will use function pointers when
+executing the callbacks of the subscribers.
+
+### Fast-Chain Module
+
+Dice provides a auto-generated module called Fast-Chain module or `fastch.o`,
+which directly call the subscriber callbacks without relying on function
+pointers. To do that, the Fast-Chain module employs large switch cases on the
+`chain_id`, `type_id` and priority values.  However, the ranges of these three
+dimensions has to be limited to keep the size of the generated code manageable.
+
+For instructions on how to use Fast-Chain, see the `CMakeLists.txt` inside
+`src/fastch`.