Tiny Dream - Embedded, Stable Diffusion Inference in C

Version 1.9.72 (Release Notes)

Tiny Dream is a header only, dependency free*, partially uncensored, Stable Diffusion implementation** written in C with primary focus on CPU efficiency, and smaller memory footprint. Tiny Dream runs reasonably fast on the average consumer hardware, require only 1.7 ~ 5.5 GB of RAM to execute, does not enforce Nvidia GPUs presence, and is designed to be embedded on larger codebases (host programs) with an easy to use C API. The possibilities are literally endless, or at least extend to the boundaries of Stable Diffusion's latent manifold.

Illustration courtesy of Binxu Wang , Ph.D candidate, Harvard University.

Stable Diffusion , a powerful, open-source text-to-image generation model launched publicly by Stability.ai in August, 2022, and is designed to produce images matching input text prompt. Stable Diffusion rely on the latent diffusion model architecture, a variant of the diffusion model which maps to the latent space using a fixed Markov chain . The goal of diffusion models is to learn the latent structure of a dataset by modeling the way in which data points diffuse through the latent space . They are designed to learn the underlying structure of a dataset by mapping it to a lower-dimensional latent space. This latent space represents the data in which the relationships between different data points are more easily understood and analyzed.

• Features
• Getting Started
• Download
• Github Repository
• Project Roadmap
• Licensing
• C API Reference
• Related Projects
• Issues Tracker

Features

• Smallest, Run-Time Memory Footprint for Running Stable Diffusion on Inference
  • Tiny Dream maximizes memory efficiency thanks to ONNX quantization , allowing for seamless image generation with minimal memory footprint.
  • As of this release, only 5.5GB of RAM are needed for Stable Diffusion Inference with high-resolution, 2048x2048 pixels output.
  • Further memory gain (1.7GB ~ 4GB) can be achieved with standard, medium quality 512x512 pixels output.
• OpenCV Dependency Free
  • You do not need to link to OpenCV or any other bloated image processing library.
  • Only stb_image_write.h from the excellent stb single-header, public domain C library is required for saving images to disk.
• Straightforward to Integrate on Existing Codebases
  • Just drop tinydream.hpp and stb_image_write.h on your source tree with the Pre-trained assets.
  • The library exposes a single C class named tinyDream with just 8 public methods.
  • No output to stdout/stderr. You just supply your log consumer callback, and redirect the generated log messages to your favorite stream (eg Terminal, Disk File or Network Socket).
• Reasonably fast on Intel/AMD CPUs
  • With TBB threading, Intel MKL*, and AVX vectorization.

  • Intel i9-9990XE	Intel i7-13700	Ryzen 9-5900X
    2.98s ~ 5.07s Clock: 5.0GHz ~ 4.0GHz	10.09s ~ 8.16s Clock: 2.8GHz ~ 3.9GHz	5.38s ~ 4.02s Clock: 3.7GHz ~ 4.8GHz

    Slowest to fastest inference time (single seed) expressed in seconds with clock speed (base & turbo-boost) under various background loads for a standard 512x512, non-upscaled PNG image. Please note that PixLab maintain a private fork of NCNN that is extremely optimized for the x86 architecture so inference speed may vary if reproduced on your machine.
• Support Real-ESRGAN , A Super Resolution Network Upscaler
  • Generate extremely high-resolution, 2048x2048 pixels output thanks to this neural network. This extra step is a CPU intensive operation, and takes few seconds to complete.
• Full Support for Negative Words
  • Supply an extra set of words that allows you to list what you don't want to see generated such as gore or NSFW  content besides your Positive Prompts.
• Full Support for Words Priority
  • Instruct the model to pay attention, and give higher priority to word (keywords) surrounded by parenthesis ().
  • square brackets [] on the other side are used to reduce word importance.
• Support for Output Metadata
  • Link meta information to your output images such as copyright notice, comments, or any other meta data you would like to see linked to your image.
• Support for Stable Diffusion Extra Parameters
  • Adjust Seed resizing: A way to generate the same image but at slightly different resolution.
  • Control how much the image generation process follows the text prompt using guidance scale.
  • Adjust sampling steps during Stable Diffusion Inference.

Getting Started

Integrating Tiny Dream on your existing code base is straightforward. Here is what to do without having to do a lot of tedious reading and configuration:

1. Download The Tiny-Dream Source Code

2. Embedding Tiny-Dream

#include "tinydream.hpp"
int main(int argc, char *argv[]) 
{
    tinyDream td; // stack allocated tinyDream object

    // Display the library current inference engine, version number, and copyright notice
    std::cout << tinyDream::about() << std::endl;
    
    // At least a positive prompt must be supplied via command line
    if (argc < 2) {
        std::cout << "Example of Prompts:" << std::endl;
        // Example of built-in Positive/Negative Prompts
        auto prompts = tinyDream::promptExample();
        std::cout << "\tPositive Prompt: " << prompts.first << std::endl;
        std::cout << "\tNegative Prompt: " << prompts.second << std::endl;
        return -1;
    }

    // Register a log handler callback responsible of 
    // consuming log messages generated during inference.
    td.setLogCallback(logCallback, nullptr);
    
    // Optionally, set the assets path if the pre-trained models
    // are not extracted on the same directory as your executable
    td.setAssetsPath("/path/to/tinydream/assets"); // Remove or comment this if your assets are located on the same directory as your executable
    
    // Optionally, set a prefix of your choice to each freshly generated image name
    td.setImageOutputPrefix("tinydream-");
    
    // Optionally, set the directory where you want
    // the generated images to be stored
    td.setImageOutputPath("/home/photos/");

    /*
   * Finally, run Stable Diffusion in inference
   * 
   * The supplied log consumer callback registered previously should shortly receive
   * all generated log messages (including errors if any) during inference.
   * 
   * Refer to the official documentation at: https://pixlab.io/tiny-dream#tiny-dream-method
   * for the expected parameters the tinyDream::dream() method takes.
   */
    for (int seed = 1; seed < seedMax; seed  ) {
        std::string outputImagePath;

        td.dream(
            positivePrompt, 
            negativePrompt, 
            outputImagePath, 
            true, /* Set to false if you want 512x512 pixels output instead of 2048x2048 output */
            seed,
            step
        );

        std::cout << "Output Image location: " << outputImagePath << std::endl; // uncomment this if too intrusive
    }
    return 0;
}

git clone https://github.com/symisc/tiny-dream.git
cd tiny-dream
g   -o tinydream boileplate.cpp -funsafe-math-optimizations -Ofast -flto=auto  -funroll-all-loops -pipe -march=native -std=c  17 -Wall -Wextra `pkg-config --cflags --libs ncnn` -lstdc   -pthread -Wl -flto -fopt-info-vec-optimized
./tinydream "pyramid, desert, palm trees, river, (landscape), (high quality)"

4. Get the Pre-Trained Models & Assets

5. C Usage Example

6. Continue with The C API Reference Guide

Downloads

Roadmap

As we continue to develop and improve Tiny Dream, we have an exciting roadmap of future addons and enhancements planned. Here's a glimpse into what you can expect in the future:

• Highest Priority Move the Tensor library from NCNN to a non bloated one such as SOD or GGML with focus on CPU performance.
• Highest Priority Output SVG, and easy to alter formats (potentially PSD) rather than static PNGs.
• Hot Provide a Cross-Platform GUI to Tiny Dream implemented in Dear imGUI.
• Hot Provide a Web-Assembly port to the library once the future Tensor library (SOD or GGML) ported to WASM.
• Low Provide an Android, proof of concept, show-case APK.

You may refer to the PixLab Blog for the upcoming updates & announcements, and to the Tiny Dream repository for the roadmap progress & implementation.

Licensing

• Tiny Dream is a dual-licensed, open source product. The complete source code of the library, and related utility are freely available on Github.
• Tiny Dream is released under the GNU Affero General Public License (AGPLv3) .
• The AGPLv3 license permits you to use Tiny Dream at no charge under the condition that if you use the library in a host application, the complete source code for your application must be available and freely redistributable under reasonable conditions.
• If you wish to derive a commercial advantage by not releasing your application under the AGPLv3 or any other compatible open source license, you must purchase a non-exclusive commercial Tiny Dream license.
• By purchasing a commercial license, you do not longer have to release your application's source code. Please reach out to [email protected] to place an order and/or for additional information about the licensing situation.

C API Reference

This section define the C language interface to Tiny Dream. For a tutorial introduction, please refer to the Getting Started section above. As of this release, the library expose a single class with just eight methods, making it easy to use & integrate in your existing C projects. The exposed methods are documented below.

tinyDream Public Methods:

Syntax

tinyDream::tinyDream(const std::string& assetsPath = "./assets/");

Description

Constructor for the tinyDream class.

• Instantiate a new tinyDream object ready for Stable Diffusion Inference. The constructor takes an optional argument assetsPath which specifies the path to the pre-trained models required by the tinyDream::dream() method in order to accept prompts and generate images that matches the input criteria.
• Once the object instantiated, you can start registering log callback, assets path location, image storage directory, name prefixes, etc. and finally calls tinyDream::dream() for inference.
• You can download the pre-trained assets from the Download section above. Once downloaded, extract the assets ZIP archive in a directory of your choice (usually the directory where your executable is located), and optionally set the path via the constructor or tinyDream::setAssetsPath().

Parameters

Return Value

Syntax

void tinyDream::setLogCallback(std::function<void(const char* /* zLogMsg */, int /*msg length*/, void* /* pCookie */) xLogHandler, void* pUserData);

Description

Consume log messages via an external log handler callback. The main task of the supplied callback is to consume log messages generated during Stable Diffusion inference. Inference may take some time to execute depending on the available resources so it make sense to log everything to the terminal or text file for example.

Parameters

Return Value

Syntax

void tinyDream::setAssetsPath(const std::string& assetsPath);

Description

Set the pre-trained models path. The Tiny Dream assets comprise all pre-trained models (over 2GB as of this release) required by the tinyDream::dream() method for stable diffusion inference (image generation).

You can download the pre-trained assets from the Download section above. Once downloaded, extract the assets ZIP archive in a directory of your choice (usually the directory where your executable is located), and set the full path via this method or from the Tiny Dream constructor. You do not need to call this method if the assets are located on the same directory where your executable reside.

Parameters

Return Value

Syntax

void tinyDream::setImageOutputPath(const std::string& outputPath = "./");

Description

Set a directory of your choice where you want your output images to be stored. The default path is set to the current directory where the executable reside.

Parameters

Return Value

Syntax

void tinyDream::setImageOutputPrefix(const std::string& outputImgPrefix = "tinydream-");

Description

Set a prefix of your choice to the yet to be generated image names. Output image names must be unique during each inference step. The assigned prefix alongside a random string plus the current step, and seed generation will be used to generate this unique name that will reside on the directory of your choice.

Parameters

Return Value

Syntax

bool tinyDream::dream(const std::string& positivePrompt, const std::string& negativePrompt,
              std::string& outputImgPath, bool upScale = true, int step = 30, int seed = 42);

Description

Stable diffusion inference - Generate high definition output images that matches the input criteria.

Prompts (Positive or Negative) in this implementation are keywords separated by commas where each word describe the thing you'd like to see (or not) generated.

Prior to calling this method, Pre-trained Models & Assets must be accessible to your executable. You can download the Pre-trained models from the Download section above. Once downloaded, extract the assets ZIP archive in a directory of your choice (usually the directory where your executable is located), and optionally set the full path (if located outside the current executable directory) via tinyDream::setAssetsPath() or from the Tiny Dream constructor.

Depending on the step, and seed parameters, it make sense to call this method more than once (ie dozen times) to achieve the desired result. On which case, we recommend that you install a log consumer callback via tinyDream::setLogCallback() to capture log messages generated during inference, and get a detailed overview of what's going on under the hood.

Stable Diffusion is based on the Latent Diffusion architecture, introduced in the High-Resolution Image Synthesis with Latent Diffusion Models paper. There are three main building blocks in Stable Diffusion:

• Text Encoder: The text-encoder is responsible for converting the input prompt into an embedding space (latent vector).
• Diffusion Model & Image Encoder: Diffusion models are trained to denoise random Gaussian noise (64x64 latent image patch) step by step, to get to a sample of interest.
• Image Decoder: Finally, the decoder is responsible for converting the latent patches into 512x512 pixels output.

Parameters

Return Value

Boolean true is returned on successful inference, false is returned otherwise. On which case, log messages captured by your log consumer callback should give you an insight of what went wrong during the whole image generation process.

Syntax

 static std::pair<std::string /*Positive Prompt */, std::string /* Negative Prompt*/> tinyDream::promptExample();

Description

Return a hard-coded, prompt example template to be passed to the tinyDream::dream() method. This static method is of no particular interest except to familiarize the developer (in lack of imagination) with the library prompt inputs.

Parameters

Return Value

This static method never fail, and always return a standard template std::pair object holding the positive prompt in the first field of the std::pair, while the negative prompt is stored in the second field of the pair object.

Syntax

static const char * tinyDream::about();

Description

Return the current tensor engine, copyright notice, library identification and version number.

Parameters

Return Value

This static method never fail, and always return a pointer to a null terminated string holding the copyright notice, and the current inference engine.

Related Projects

You may find useful the following open source*, production-ready products developed & maintained by PixLab | Symisc Systems: