v0.1.6 (#54)

* Minor fix (#46)

* leftover minor

* minor

* fix dockerfile for x86_64 to include hopper

* fix benchmark script

* minor

* Models/gemma 3 (#47)

* wip: revamp model registration

* fix gemma3 for causal LM

* gemma3

* update dockerfile

* Bump transformers from 4.46.3 to 4.48.0 in /meta (#49)

Bumps [transformers](https://github.com/huggingface/transformers) from 4.46.3 to 4.48.0.
- [Release notes](https://github.com/huggingface/transformers/releases)
- [Commits](huggingface/transformers@v4.46.3...v4.48.0)

---
updated-dependencies:
- dependency-name: transformers
  dependency-version: 4.48.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xiaozhe Yao <[email protected]>

* Support #44

* redo docs (#52)

* WIP: Offloading and other utilities (#51)

* wip: offload support

* debug

* fix sampling

* fix aarch64 dockerfile

* update aarch script

* remove sgl-kernel from dependencies

* update dockerfile

* minor reorg

* remove templated message printout

* fix dockerfile in aarch build

* relax triton req

* move `triton` dep to requirements-cuda

* minor fix

* fix build issues

* test dependencies

* move torch-memory-saver

* minor

* fix build issues & add initial metrics ui

* ready to build

* wip: multistage build

* Models/qwen3 (#53)

* init: qwen3

* qwen3

* minor

* minor refactor: health check on server starts

* minor update

* update buildfile

* minor

* minor bug fix

* logger

* update dev

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Author: xzyaoi

.dockerignore

@@ -2,3 +2,23 @@ docs/
 scripts
 *.egg-info
 .local
+.github/
+.vscode/
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+*.db
+*.sqlite
+*.log
+*.egg
+*.egg-info
+*.whl
+*.tar.gz
+*.zip
+*.tar
+*.gz
+*.bz2
+.buildcache
+.pytest_cache
+.git

.gitignore

@@ -8,3 +8,4 @@ pyrightconfig.json
 .data
 *.ipynb
 .coverage*
+.buildcache/

README.md

@@ -1,3 +1,8 @@
 # Scratchpad
 
-This is an experimental LLM serving system, forked and built on top of [SGLang SRT](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt), and is used to support [SwissAI Model Serving](https://fmapi.swissai.cscs.ch/).
+This is an experimental LLM serving system, forked and built on top of [SGLang SRT](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt), and is used to support [SwissAI Model Serving](https://serving.swissai.cscs.ch/).
+
+## Adoption
+
+- [SwissAI](https://serving.swissai.cscs.ch/) - as the primary serving engine for LLMs.
+- [HexGen-Flow](https://github.com/Relaxed-System-Lab/Hexgen-Flow) - as a LLM execution simulator for Text-to-SQL application.

docker/Dockerfile.aarch64-cuda

@@ -1 +1,21 @@
-build/
+# build output
+dist/
+# generated types
+.astro/
+
+# dependencies
+node_modules/
+
+# logs
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+
+
+# environment variables
+.env
+.env.production
+
+# macOS-specific files
+.DS_Store

docker/build_image.sh

@@ -0,0 +1,54 @@
+# Starlight Starter Kit: Basics
+
+[![Built with Starlight](https://astro.badg.es/v2/built-with-starlight/tiny.svg)](https://starlight.astro.build)
+
+```
+npm create astro@latest -- --template starlight
+```
+
+[![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/github/withastro/starlight/tree/main/examples/basics)
+[![Open with CodeSandbox](https://assets.codesandbox.io/github/button-edit-lime.svg)](https://codesandbox.io/p/sandbox/github/withastro/starlight/tree/main/examples/basics)
+[![Deploy to Netlify](https://www.netlify.com/img/deploy/button.svg)](https://app.netlify.com/start/deploy?repository=https://github.com/withastro/starlight&create_from_path=examples/basics)
+[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fwithastro%2Fstarlight%2Ftree%2Fmain%2Fexamples%2Fbasics&project-name=my-starlight-docs&repository-name=my-starlight-docs)
+
+> 🧑‍🚀 **Seasoned astronaut?** Delete this file. Have fun!
+
+## 🚀 Project Structure
+
+Inside of your Astro + Starlight project, you'll see the following folders and files:
+
+```
+.
+├── public/
+├── src/
+│   ├── assets/
+│   ├── content/
+│   │   ├── docs/
+│   └── content.config.ts
+├── astro.config.mjs
+├── package.json
+└── tsconfig.json
+```
+
+Starlight looks for `.md` or `.mdx` files in the `src/content/docs/` directory. Each file is exposed as a route based on its file name.
+
+Images can be added to `src/assets/` and embedded in Markdown with a relative link.
+
+Static assets, like favicons, can be placed in the `public/` directory.
+
+## 🧞 Commands
+
+All commands are run from the root of the project, from a terminal:
+
+| Command                   | Action                                           |
+| :------------------------ | :----------------------------------------------- |
+| `npm install`             | Installs dependencies                            |
+| `npm run dev`             | Starts local dev server at `localhost:4321`      |
+| `npm run build`           | Build your production site to `./dist/`          |
+| `npm run preview`         | Preview your build locally, before deploying     |
+| `npm run astro ...`       | Run CLI commands like `astro add`, `astro check` |
+| `npm run astro -- --help` | Get help using the Astro CLI                     |
+
+## 👀 Want to learn more?
+
+Check out [Starlight’s docs](https://starlight.astro.build/), read [the Astro documentation](https://docs.astro.build), or jump into the [Astro Discord server](https://astro.build/chat).

docs/.gitignore

@@ -0,0 +1,28 @@
+// @ts-check
+import { defineConfig } from 'astro/config';
+import starlight from '@astrojs/starlight';
+import starlightThemeFlexoki from 'starlight-theme-flexoki'
+
+// https://astro.build/config
+export default defineConfig({
+	integrations: [
+		starlight({
+			title: 'Scratchpad',
+			plugins: [starlightThemeFlexoki()],
+			social: [{ icon: 'github', label: 'GitHub', href: 'https://github.com/eth-easl/Scratchpad' }],
+			sidebar: [
+				{
+					label: 'Guides',
+					items: [
+						// Each item here is one entry in the navigation menu.
+						{ label: 'Example Guide', slug: 'guides/example' },
+					],
+				},
+				{
+					label: 'Reference',
+					autogenerate: { directory: 'reference' },
+				},
+			],
+		}),
+	],
+});