# MBASE

MBASE inference library is a high-level non-blocking LLM inference library written on top of the [llama.cpp](https://github.com/ggml-org/llama.cpp) library to provide the necessary tools and APIs to allow developers to integrate popular LLMs into their applications with minimal performance loss and development time. The inference SDK will allow developers the utilize the LLMs and create their own solutions, tooling mechanisms etc. When the phrase "local LLM inference" is thrown around, it usually means that hosting an Openai API compatible HTTP server and using the completions API locally. The MBASE inference library is expected to change this notion by providing you the LLM inference capability through its low-level objects and procedures so that you can integrate and embedd LLM into your high-performance applications such as games, server applications and many more. Also you still have the option of hosting Openai server using the [mbase_openai_server](doc_link) program or you can code a similar one yourself! There also is a benchmarking tool developed for testing the performance of the LLM but more specifically, the impact of the inference operation on your main program loop. For further details refer to [benchmark](doc_link) documentation. ## Github Repository https://github.com/Emreerdog/mbase/tree/main ## Features - Non-blocking TextToText LLM inference SDK. - Non-blocking Embedder model inference SDK. - GGUF file meta-data manipulation SDK. - Openai server program supporting both TextToText and Embedder endpoints with system prompt caching support which implies significant performance boost. - Hosting multiple models in a single Openai server program. - Using [llama.cpp](https://github.com/ggml-org/llama.cpp) as an inference backend so that models that are supported by the [llama.cpp](https://github.com/ggml-org/llama.cpp) library are supported by default. - Benchmark application for measuring the impact of LLM inference on your application. - Plus anything [llama.cpp](https://github.com/ggml-org/llama.cpp) supports. ## Supported Models Since the MBASE SDK uses [llama.cpp](https://github.com/ggml-org/llama.cpp) as a backend inference engine, the models that are supported by [llama.cpp](https://github.com/ggml-org/llama.cpp) are supported by default which includes major models such as **Phi**, **Deepseek**, **Llama** and **Qwen**. You can see the full list [here](https://github.com/ggml-org/llama.cpp?tab=readme-ov-file#description). ## Implementation Matrix | Type | SDK Support | Openai API Support | Engine | | ---------- | ----------- | ------------------ | -------------------------------------------------- | | TextToText | ✅ | ✅ | [llama.cpp](https://github.com/ggml-org/llama.cpp) | | Embedder | ✅ | ✅ | [llama.cpp](https://github.com/ggml-org/llama.cpp) | ## Supported Platform - Mac OS - Linux - Windows ## Documentation Detailed MBASE SDK documentation can be found [here](https://docs.mbasesoftware.com/inference/about). ## SDK Usage Examples - [Single-Prompt Example](https://docs.mbasesoftware.com/inference/quickstart/single_prompt_ex/about): Simple prompt and answer example. At the end of the example, the user will supply a prompt in the terminal and LLM will give the response. - [Dialogue-Example](https://docs.mbasesoftware.com/inference/quickstart/dialogue_ex/about): More complex dialogue based prompt and answer example. At the end of the example, the user will be able to have a dialogue with LLM using terminal. - [Embedding Example](https://docs.mbasesoftware.com/inference/quickstart/embeddings_ex/about): Vector embedding generator which is generally used by RAG programs and more. At the end of the example, the user will supply an input and vector embeddings will be generated by using embedder LLM model. ## What do I mean by Non-blocking **Since there are multiple questions arised from people asking what non-blocking means in this context, this title is addressing those questions** Let's don't think about the LLMs for a second and think about IO management in the program. IO and network operations are expensive operations. Their performance are limited by the read/write speed or the network operations highly influenced by your network environment speec etc. In IO scenario, when you want to write multiple Gigs of data into a disk, you should write a mechanism in your program so that the write won't block your main application logic. You may do this by writing data to a file by specifying a threshold let's say writing 1KB every iteration. Or you may do your IO operations in seperate thread and write a synchronization mechanism based-off of your needs. Or, you can use async io libraries that can do this for you. In my opinion, LLM inference deserve its own non-blocking terminology because the operations such as language model initialization(llama_model_load_from_file), destruction(llama_model_free), context creation(llama_init_from_model), encoder/decoder(llama_encode/llama_decode) methods are extremelly expensive which makes them really difficult to integrate into your main application logic. Even with high-end GPU, the amount your program halts on llama_model_load_from_file, llama_init_from_model, llama_encode/llama_decode prevents people from integrating LLMs into their applications. This SDK apply those operations in a non-blocking manner or in other words, the model initialization, destruction, context creation, encoder/decoder methods doesn't block your main thread and synchronization is handled by the MBASE SDK. Using this, you will be able to load/unload multiple models, create contexts, and apply encode/decoder operations all at the same time while not blocking your main application thread because MBASE will handle all those operations in parallel and will provide you the synchronized callbacks so that you won't need to consider issues arise from parallel programming. ## Download and Setting up Download page: [quickstart/setup/download](https://docs.mbasesoftware.com/inference/quickstart/setup/download) SDK setup and compiling from source: [quickstart/setup/setting_up](https://docs.mbasesoftware.com/inference/quickstart/setup/setting_up) ## Useful Programs ### Openai Server Detailed documentation: [programs/openai-server/about](https://docs.mbasesoftware.com/inference/programs/openai-server/about) An Openai API compatible HTTP/HTTPS server for serving LLMs. This program provides chat completion API for TextToText models and embeddings API For embedder models. **Usage:** ```bash mbase_openai_server *[option [value]] mbase_openai_server --hostname "127.0.0.1" -jsdesc description.json mbase_openai_server --hostname "127.0.0.1" --port 8080 -jsdesc description.json mbase_openai_server --hostname "127.0.0.1" --port 8080 --ssl-pub public_key_file --ssl-key private_key_file -jsdesc description.json ``` ### Benchmark T2T Detailed documentation: [programs/benchmark-t2t/about](https://docs.mbasesoftware.com/inference/programs/benchmark-t2t/about) It is a program written to measure the performance of the given T2T LLM and its impact on your main application logic. **Usage:** ```bash mbase_benchmark_t2t model_path *[option [value]] mbase_benchmark_t2t model.gguf -uc 1 -fps 500 -jout . mbase_benchmark_t2t model.gguf -uc 1 -fps 500 -jout . -mdout . ``` ### Embedding Detailed documentation: [programs/embedding/about](https://docs.mbasesoftware.com/inference/programs/embedding/about) An example program for generating the embeddings of the given prompt or prompts. **Usage:** ```bash mbase_embedding_simple model_path *[option [value]] mbase_embedding_simple model.gguf -gl 80 -p 'What is life?' mbase_embedding_simple model.gguf -gl 80 -pf prompt1.txt -pf prompt2.txt ``` ### Retrieval Detailed documentation: [programs/retrieval/about](https://docs.mbasesoftware.com/inference/programs/retrieval/about) An example program for generating the embeddings of the given prompt or prompts. **Usage:** ```bash mbase_retrieval model_path *[option [value]] mbase_retrieval model.gguf -q 'What is MBASE' -pf file1.txt -pf file2.txt -gl 80 ``` ### Simple Conversation Detailed documentation: [programs/simple-conversation/about](https://docs.mbasesoftware.com/inference/programs/simple-conversation/about) It is a simple executable program where you are having a dialogue with the LLM you provide. It is useful for examining the answer of the LLM since the system prompt and sampler values can be altered. **Usage:** ```bash mbase_simple_conversation model_path *[option [value]] mbase_simple_conversation model.gguf mbase_simple_conversation model.gguf -gl 80 mbase_simple_conversation model.gguf -gl 80 -sys 'You are a helpful assistant.' ``` ### Typo Fixer Detailed documentation: [programs/typo-fixer/about](https://docs.mbasesoftware.com/inference/programs/typo-fixer/about) This is an applied example use case of the MBASE library. The program is reading a user-supplied text file and fixing the typos. ```bash mbase_typo_fixer model_path *[option [value]] mbase_typo_fixer model.gguf -gl 80 -s typo.txt -o fixed.txt ``` ## Finding the SDK Detailed documentation: [Finding the SDK](https://docs.mbasesoftware.com/inference/info-reference/sdk-structure#finding-the-sdk) If you have installed the MBASE SDK, you can find the library using CMake find_package function with components specification. In order to find the library using cmake, write the following: ```cmake find_package(mbase.libs REQUIRED COMPONENTS inference) ``` This will find the inference SDK. In order to set the include directories and link the libraries, write the following: ```cmake target_compile_features( PUBLIC cxx_std_17) target_link_libraries( PRIVATE mbase-inference) target_include_directories( PUBLIC mbase-inference) ``` ## GGUF, Displaying General Metadata Detailed documentation: [About GGUF Files](https://docs.mbasesoftware.com/inference/info-reference/gguf-files) ```cpp #include #include int main() { mbase::GgufMetaConfigurator metaConfigurator(L""); if(!metaConfigurator.is_open()) { std::cout << "Unable to open gguf file." << std::endl; return 1; } mbase::string modelArchitecture; mbase::string modelName; mbase::string modelAuthor; mbase::string modelVersion; mbase::string modelOrganization; mbase::string modelSizeLabel; mbase::string modelLicense; mbase::string modelLicenseName; mbase::string modelLicenseLink; mbase::string modelUuid; uint32_t modelFileType; metaConfigurator.get_key("general.architecture", modelArchitecture); metaConfigurator.get_key("general.name", modelName); metaConfigurator.get_key("general.author", modelAuthor); metaConfigurator.get_key("general.version", modelVersion); metaConfigurator.get_key("general.organization", modelOrganization); metaConfigurator.get_key("general.size_label", modelSizeLabel); metaConfigurator.get_key("general.license", modelLicense); metaConfigurator.get_key("general.license.name", modelLicenseName); metaConfigurator.get_key("general.license.link", modelLicenseLink); metaConfigurator.get_key("general.uuid", modelUuid); metaConfigurator.get_key("general.file_type", modelFileType); std::cout << "Architecture: " << modelArchitecture << std::endl; std::cout << "Name: " << modelName << std::endl; std::cout << "Author: " << modelAuthor << std::endl; std::cout << "Version: " << modelVersion << std::endl; std::cout << "Organization: " << modelOrganization << std::endl; std::cout << "Size label: " << modelSizeLabel << std::endl; std::cout << "License: " << modelLicense << std::endl; std::cout << "License name: " << modelLicenseName << std::endl; std::cout << "License link: " << modelLicenseLink << std::endl; std::cout << "Model UUID: " << modelUuid << std::endl; std::cout << "File type: " << modelFileType << std::endl; return 0; } ``` ## Project State and Goals The MBASE SDK is in its early stages for now and it is expected to err on some scenarios because not all cases are tested and the product is not yet fully-polished. The project has been developed only by me and I was planning to open-source in the near future. However, the complication and the workload forced me to open-source the project much earlier before being perfectly tested and polished. In other words, I am once again asking for your contribution to this project. The company is being established to accelerate the development of the MBASE SDK and for creating both proprietary and open-source products built using the MBASE SDK. The goal of the MBASE SDK is expected to provide non-blocking LLM inference to both beginner and advanced users of the C++ library and very useful tools and programs to non-programmer users.