ai-deployment-english
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| ai-deployment-english [2026/09/16 13:42] – misa | ai-deployment-english [2026/09/16 13:57] (current) – [6. End-to-end test] misa | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== | + | ====== Local AI Stack - Part 1: Base Setup (LM Studio + Open WebUI + ComfyUI) ====== |
| - | '' | + | '' |
| - | All components run locally | + | |
| - | ===== 0. Prerequisites ===== | + | '' |
| - | || Requirement | + | || Component |
| - | |-----| | + | || LM Studio |
| - | || GPU || NVIDIA, 16-19 GB VRAM (Qwen3 27B) || | + | || Open WebUI || chat interface, user accounts, knowledge base (RAG) || 3000 || Podman container || |
| - | || Driver | + | || ComfyUI |
| - | || Podman | + | || Qwen3 27B (GGUF) || the LLM weights, about 16 GB || - || stored in ~/ |
| - | || User || ``rootai``, home ``/ | + | |
| - | || Ports || 1234 (LM Studio), 3000 (Open WebUI), 8188 (ComfyUI) | + | |
| - | ===== 1. Install | + | '' |
| - | 1. '' | + | ===== 1. Prerequisites ===== |
| - | 2. Make it executable: | + | || Requirement || Details || Check command || |
| - | | + | || GPU || NVIDIA card with 16 to 19 GB VRAM (needed for Qwen3 27B) || nvidia-smi || |
| - | + | || NVIDIA driver || current driver, working || nvidia-smi shows card, driver and CUDA version || | |
| - | 3. '' | + | || RAM || 32 GB comfortable, |
| - | | + | || Disk space || about 25 GB free for model + container + documents || df -h /home || |
| - | | + | || Podman || installed, used for Open WebUI || podman |
| - | 4. '' | + | || User account || rootai, home directory |
| - | | + | || Ports || 1234, 3000, 8188 must be free || sudo lsof -i :1234 -i :3000 -i :8188 || |
| - | 5. '' | + | |
| - | ``sudo ln -sf / | + | |
| - | Note: after every AppImage update with a new filename, recreate | + | If nvidia-smi is not found, install |
| - | '' | + | ===== 2. Install LM Studio (AppImage) ===== |
| - | ``lms ls'' | + | |
| - | ===== 2. Start LM Studio Server | + | '' |
| + | |||
| + | '' | ||
| + | |||
| + | Download the latest LM Studio AppImage and store it in / | ||
| + | |||
| + | '' | ||
| + | |||
| + | chmod +x / | ||
| + | |||
| + | '' | ||
| + | |||
| + | Fedora-based systems do not enable FUSE for normal users by default, so the AppImage may fail with a FUSE or mount error. If that happens: | ||
| + | |||
| + | cd / | ||
| + | ./ | ||
| + | ./ | ||
| + | |||
| + | This unpacks the AppImage into a folder and starts it via AppRun. It behaves exactly like the normal AppImage. | ||
| + | |||
| + | '' | ||
| + | |||
| + | In LM Studio: Models tab, search for Qwen3 27B, download the GGUF variant (about 16 GB). Models are stored in ~/.lmstudio - this directory survives every AppImage update. | ||
| + | |||
| + | '' | ||
| + | |||
| + | LM Studio provides the lms command (used by start-ai.sh). Create a stable symlink so scripts do not depend on the exact AppImage filename: | ||
| + | |||
| + | sudo ln -sf / | ||
| + | |||
| + | '' | ||
| + | |||
| + | '' | ||
| + | |||
| + | lms ls | ||
| + | |||
| + | Expected: the command runs and lists the model. | ||
| + | |||
| + | ===== 3. Start the LM Studio Server | ||
| + | |||
| + | '' | ||
| + | |||
| + | In LM Studio: Developer tab, set port 1234, click Start Server. (The lms load command below also starts the server automatically.) | ||
| + | |||
| + | '' | ||
| - | # start the server (LM Studio -> Developer -> Start Server, port 1234) | ||
| lms load qwen/ | lms load qwen/ | ||
| - | # API test | + | Wait until the load finishes (about 16 GB into VRAM, takes a few seconds). |
| + | |||
| + | '' | ||
| curl -s http:// | curl -s http:// | ||
| - | '' | + | Expected: JSON output that contains |
| - | ===== 3. Open WebUI (Podman container) ===== | + | ===== 4. Open WebUI (Podman container) ===== |
| + | |||
| + | '' | ||
| - | # pull the image | ||
| podman pull ghcr.io/ | podman pull ghcr.io/ | ||
| - | # start the container | + | '' |
| podman run -d --name open-webui -p 3000:8080 -v open-webui-data:/ | podman run -d --name open-webui -p 3000:8080 -v open-webui-data:/ | ||
| - | '' | + | '' |
| - | 1. '' | + | * --name open-webui |
| - | * Base URL: ``http:// | + | * -p 3000:8080 - host port 3000 mapped to container port 8080 |
| - | * API key: any value (LM Studio does not require one) | + | * -v open-webui-data:/ |
| - | * Test: model ``qwen3.8-27b'' | + | * --restart unless-stopped |
| - | 2. '' | + | |
| - | 3. '' | + | '' |
| - | 4. '' | + | |
| - | '' | ||
| podman ps | grep open-webui | podman ps | grep open-webui | ||
| - | curl -s http:// | + | curl -sI http:// |
| - | ===== 4. ComfyUI (Z-Image) ===== | + | Expected: an HTTP 200 or 302 line from curl. |
| + | |||
| + | '' | ||
| + | |||
| + | Open http://< | ||
| + | |||
| + | # Settings, Connections, | ||
| + | * Base URL: http:// | ||
| + | * API key: any non-empty value (LM Studio ignores it) | ||
| + | * Test: qwen3.8-27b must appear in the model list | ||
| + | * Why host.docker.internal: | ||
| + | |||
| + | # Settings, Models: set nomic-embed-text-v1.5 as the embedding model (needed for RAG, see Part 2) | ||
| + | |||
| + | |||
| + | ===== 5. ComfyUI (Z-Image) ===== | ||
| + | |||
| + | '' | ||
| - | '' | ||
| nohup ~/ | nohup ~/ | ||
| - | '' | + | '' |
| curl -s http:// | curl -s http:// | ||
| - | '' | + | Expected: JSON with system statistics. For problems: |
| - | * URL: ``http:// | + | |
| - | * Model: '' | + | |
| - | Log on problems: | + | |
| - | ===== 5. Auto-start on boot ===== | + | tail -f ~/ |
| - | Script: ``/ | + | '' |
| + | |||
| + | Settings, Connections, | ||
| + | |||
| + | * URL: http:// | ||
| + | * Model: Z-Image | ||
| + | |||
| + | ===== 6. Auto-start on boot ===== | ||
| + | |||
| + | '' | ||
| #!/bin/bash | #!/bin/bash | ||
| - | # wait for system services and the LM Studio server | + | # wait for system services and the LM Studio server |
| sleep 15 | sleep 15 | ||
| - | | + | |
| + | | ||
| lms load qwen/ | lms load qwen/ | ||
| - | | + | |
| + | | ||
| sleep 10 | sleep 10 | ||
| - | | + | |
| + | | ||
| podman restart open-webui | podman restart open-webui | ||
| - | '' | + | '' |
| - | ===== 6. Final smoke test ===== | + | chmod +x / |
| + | |||
| + | '' | ||
| + | |||
| + | '' | ||
| + | |||
| + | ===== 7. Final smoke test ===== | ||
| + | |||
| + | Run all three checks: | ||
| # 1. LM Studio API | # 1. LM Studio API | ||
| Line 106: | Line 180: | ||
| curl -s http:// | curl -s http:// | ||
| - | 1. Browser -> ``http://< | + | Then in the browser: |
| - | 2. Ask a free question to ``qwen3.8-27b'' | + | |
| - | ===== 7. Troubleshooting ===== | + | # Open http://< |
| + | # Start a chat with qwen3.8-27b, ask a simple question without a knowledge base | ||
| + | # Expected: a normal answer, no citations (citations come with RAG, see Part 2) | ||
| - | || Symptom || Cause || Fix || | + | ===== 8. Troubleshooting ===== |
| - | |-----| | + | |
| - | || AppImage does not start || FUSE missing (Fedora/ | + | |
| - | || lms: command not found || broken symlink, old filename || ``sudo ln -sf / | + | |
| - | || WebUI cannot find the model || wrong base URL || use ``host.docker.internal: | + | |
| - | || Container will not restart || wrong container name || check real name with ``podman ps -a'' | + | |
| - | || Port already in use || stale process || ``sudo lsof -i : | + | |
| - | ====== Deployment: Local AI Stack (Dental-Billing Assistant) - Part 2: RAG Knowledge Base (EN) ====== | + | || Symptom || Likely cause || Fix || |
| + | || AppImage will not start || FUSE not enabled for the user (Fedora default) || use --appimage-extract and AppRun (step 2.3) || | ||
| + | || lms: command not found || symlink points to an old AppImage filename || redo step 2.5 with the new path || | ||
| + | || WebUI cannot find the model || wrong base URL from the container || use http:// | ||
| + | || podman restart fails || wrong container name || check the real name with podman ps -a || | ||
| + | || Port already in use || an old process still holds the port || sudo lsof -i :1234 (same for 3000 and 8188), kill the process || | ||
| + | || Slow first answer after boot || model still loading || wait for the full load, watch the LM Studio status bar || | ||
| - | '' | ||
| - | ===== 1. Set the embedding model (prerequisite) ===== | ||
| - | RAG does not work without an embedding model. | + | ====== Local AI Stack - Part 2: RAG Knowledge Base (Zahnarztabrechnung) ====== |
| - | 1. Open WebUI -> Settings -> Models | + | '' |
| - | 2. '' | + | |
| - | 3. '' | + | |
| - | '' | + | '' |
| + | |||
| + | ===== 1. Set the embedding model (must be first) ===== | ||
| + | |||
| + | RAG works in two steps: | ||
| + | |||
| + | # The document text is split into chunks and converted into vectors (embeddings) by the embedding model. | ||
| + | # At chat time, the question is embedded the same way and the most similar chunks are retrieved and shown as citations. | ||
| + | |||
| + | Without an embedding model, step one cannot happen and the knowledge base stays empty. | ||
| + | |||
| + | # Open WebUI, Settings, Models | ||
| + | # Embedding model: set nomic-embed-text-v1.5 (built-in, no Ollama or extra LM Studio model needed) | ||
| + | # Verify: the model list shows the embedding model as active | ||
| + | |||
| + | '' | ||
| ===== 2. Prepare the PDFs ===== | ===== 2. Prepare the PDFs ===== | ||
| || # || Document || Source || Note || | || # || Document || Source || Note || | ||
| - | |-----| | + | || 1 || SGB V (full text) || gesetze-im-internet.de || 577 pages, |
| - | || 1 || SGB V (full text) || gesetze-im-internet.de || 577 pages, | + | |
| || 2 || KZBV FZ-Kompendium 2025 || KZBV || statutory co-payments (Festzuschüsse) || | || 2 || KZBV FZ-Kompendium 2025 || KZBV || statutory co-payments (Festzuschüsse) || | ||
| || 3 || GOÄ/GOZ || pkv.de || private fee schedules || | || 3 || GOÄ/GOZ || pkv.de || private fee schedules || | ||
| - | || 4 || BEL-Preisliste 2025 || kvzbw.de || - || | + | || 4 || BEL-Preisliste 2025 || kvzbw.de || billing positions for dental labs || |
| - | || 5 || BEB-Preisliste 2024 || itzgmbh.de || image-based (scanned) - OCR required, see section | + | || 5 || BEB-Preisliste 2024 || itzgmbh.de || image-based (scanned) - OCR required, see step 3 || |
| - | '' | + | '' |
| - | * text can be selected/ | + | |
| - | * images only -> OCR problem (BEB) | + | |
| - | ===== 3. Enable OCR for image-based PDFs (BEB) ===== | + | # Open it in a browser or PDF viewer |
| + | # Try to select and copy some text | ||
| + | # If text copies - OK | ||
| + | # If the page shows only an image - OCR problem | ||
| - | '' | + | '' |
| + | |||
| + | The full SGB V is 577 pages and pulls in pharmacy and hospital content that can show up as citations for dental questions. If you see such noise in the answer sources, replace the document with a trimmed version containing only Part 3, sections 55 to 66 (the dental-related chapters), and re-index (delete the old document first). | ||
| + | |||
| + | ===== 3. Enable OCR for the image-based BEB file ===== | ||
| + | |||
| + | '' | ||
| '' | '' | ||
| - | 1. Open WebUI -> Admin -> Settings | + | |
| - | 2. turn '' | + | # Open WebUI, Admin, Settings, Documents |
| - | 3. '' | + | # Turn ON the option PDF Extract Images (OCR) |
| + | # Delete | ||
| + | # Upload the BEB PDF again | ||
| + | # Wait until the status shows processed | ||
| + | |||
| + | OCR is slower than normal extraction - a larger PDF can take several minutes. That is expected. | ||
| ===== 4. Create the knowledge base ===== | ===== 4. Create the knowledge base ===== | ||
| - | 1. Open WebUI -> Knowledge | + | # Open WebUI, Knowledge, |
| - | 2. '' | + | # Name: Zahnarztabrechnung |
| - | 3. wait until every document shows status | + | # Add documents: upload all five PDFs (BEB after the OCR step) |
| - | 4. '' | + | # Wait until every document shows the status processed, no errors |
| + | # Set permissions: user artz, access | ||
| - | '' | + | '' |
| - | * new chat session, attach the '' | + | |
| - | * test question: "What statutory co-payment | + | # New chat session, attach the knowledge base Zahnarztabrechnung |
| - | * expected: answer with source citations from the FZ-Kompendium | + | # Ask: What statutory co-payment applies to a crown? |
| + | # Expected: an answer with visible | ||
| ===== 5. Create the Zahn-Abrechnungs-Assistent model ===== | ===== 5. Create the Zahn-Abrechnungs-Assistent model ===== | ||
| - | 1. Open WebUI -> Settings | + | # Open WebUI, Settings, Models, create |
| - | 2. '' | + | # Name: Zahn-Abrechnungs-Assistent |
| - | 3. '' | + | # Base model: qwen3.8-27b (from LM Studio) |
| - | 4. '' | + | # Attach the knowledge base Zahnarztabrechnung |
| + | # System prompt | ||
| - | | + | |
| - | (GOÄ/GOZ, FZ-Kompendium, | + | (GOA/GOZ, KZBV FZ-Kompendium, |
| - | of the documents provided. Give concrete position numbers, prices, the | + | |
| - | edition (e.g. BEB 2024, BEL 2025) and the source. If a topic is not | + | |
| - | covered in the documents, say so clearly. This is not legal advice. | + | |
| - | 5. '' | + | Antworte direkt auf Basis der bereitgestellten Dokumente. |
| + | Nenne immer: | ||
| + | - die konkrete Positionsnummer | ||
| + | - den Preis | ||
| + | - die Edition der Preisliste (z. B. BEB 2024, BEL 2025) | ||
| + | - das Quelldokument | ||
| + | |||
| + | Stelle keine Rueckfragen. Wenn Informationen fehlen, sage das klar, | ||
| + | statt zu raten. | ||
| + | Dieses Tool erteilt keine Rechtsberatung. | ||
| + | |||
| + | # Disable | ||
| ===== 6. End-to-end test ===== | ===== 6. End-to-end test ===== | ||
| - | 1. Browser | + | # Browser: http://< |
| - | 2. Select model '' | + | # Select |
| - | 3. Sample test questions | + | # Ask the following |
| - | * " | + | |
| - | * "What does a full-ceramic crown cost per BEB?" -> expect | + | || Test question || Expected answer elements || |
| - | * "What is the statutory co-payment for inlays?" -> expect | + | || What GOZ position applies for a composite filling on an incisor? |
| - | 4. '' | + | || What does a full ceramic crown cost per BEB? || BEB position |
| - | * source citations | + | || What is the statutory co-payment for an inlay? || FZ-Kompendium 2025, amount, source || |
| - | * concrete | + | || What BEL position covers a partial denture? || BEL position, price, edition 2025 || |
| - | * edition | + | |
| - | * for missing | + | '' |
| + | |||
| + | * Source citation | ||
| + | * Concrete | ||
| + | * Edition | ||
| + | * For missing | ||
| ===== 7. Troubleshooting ===== | ===== 7. Troubleshooting ===== | ||
| || Symptom || Cause || Fix || | || Symptom || Cause || Fix || | ||
| - | |-----| | + | || Answer without sources, |
| - | || Answer without sources, made up || RAG not connected || check model prompt, | + | || BEB gives no sensible citations || PDF is image-based, no text layer || enable OCR (step 3), delete and re-upload |
| - | || BEB gives no sensible citations || PDF is image-based || enable OCR (section | + | || SGB V citations show pharmacy |
| - | || SGB V citations show pharmacy/hospital noise || 577-page full text too broad || trim SGB V to §§ 55-66, replace the document, re-index | + | || Model asks questions |
| - | || Slow answers on large PDFs || index size || normal, cache warms up; optionally check chunk size in Settings | + | || artz cannot see the knowledge base || no permission set || Knowledge, select the base, set Read access for artz || |
| - | || Model asks instead of answering || Ask User tool active || disable it in the model settings || | + | || Slow first answer on a large document || indexing and embedding in progress || wait until the document shows processed, then retry || |
| + | || After a re-index, old citations still appear || stale index entry || delete the document from the knowledge base completely before re-uploading | ||
ai-deployment-english.1789566170.txt.gz · Last modified: by misa
