Release v6.0.2 (#294)

- Add the "echo" parameter to OpenAI completions (@danielholanda)
- New dedicated report tool for LLM CSVs, as well as ASCII tables (@amd-pworfolk)
- Properly raise and transmit server model load failures (@jeremyfowers)
- Add documentation for Lemonade_Server_Installer.exe (@jeremyfowers)
- Add telemetry to server: performance, input tokens, output tokens, and prompt tracing (@danielholanda)
- Ensure that Ryzen AI Hybrid support is not installed on incompatible devices (@danielholanda) 

Co-authored-by: amd-pworfolk <[email protected]>
Co-authored-by: Daniel Holanda <[email protected]>

Author: 3 people

docs/lemonade/getting_started.md

@@ -58,6 +58,10 @@ To install `lemonade` from source code:
 1. Follow the same instructions as in the [PyPI installation](#from-pypi), except replace the `turnkeyml` with a `.`.
     - For example: `pip install -e .[llm-oga-igpu]`
 
+## From Lemonade_Server_Installer.exe
+
+The `lemonade` server is available as a standalone tool with a one-click Windows installer `.exe`. Check out the [Lemonade_Server_Installer.exe guide](lemonade_server_exe.md) for installation instructions and the [server spec](https://github.com/onnx/turnkeyml/blob/main/docs/lemonade/server_spec.md) to learn more about the functionality.
+
 # CLI Commands
 
 The `lemonade` CLI uses a unique command syntax that enables convenient interoperability between models, frameworks, devices, accuracy tests, and deployment options.  
@@ -135,6 +139,14 @@ That command will run a few warmup iterations, then a few generation iterations
 
 The prompt size, number of output tokens, and number iterations are all parameters. Learn more by running `lemonade oga-bench -h` or `lemonade huggingface-bench -h`.
 
+## LLM Report
+
+To see a report that contains all the benchmarking results and all the accuracy results, use the `report` tool with the `--perf` flag:
+
+`lemonade report --perf`
+
+The results can be filtered by model name, device type and data type.  See how by running `lemonade report -h`.
+
 ## Memory Usage
 
 The peak memory used by the `lemonade` build is captured in the build output. To capture more granular

docs/lemonade/lemonade_server_exe.md

@@ -0,0 +1,117 @@
+# Lemonade Server Installer
+
+The `lemonade` server is available as a standalone tool with a one-click Windows installer `.exe`. Check out the [server spec](https://github.com/onnx/turnkeyml/blob/main/docs/lemonade/server_spec.md) to learn more about the functionality.
+
+## GUI Installation and Usage
+
+> *Note:* you may need to give your browser or OS permission to download or install the .exe.
+
+1. Navigate to the [latest release](https://github.com/onnx/turnkeyml/releases/latest).
+1. Scroll to the bottom and click `Lemonade_Server_Installer.exe` to download.
+1. Double-click the `Lemonade_Server_Installer.exe` and follow the instructions.
+
+Now that you have the server installed, you can double click the desktop shortcut to run the server process. From there, you can connect it to applications that are compatible with the OpenAI completions API.
+
+## Silent Installation and Command Line Usage
+
+Silent installation and command line usage are useful if you want to fully integrate `lemonade` server into your own application. This guide provides fully automated steps for downloading, installing, and running `lemonade` server so that your users don't have to install `lemonade` separately.
+
+Definitions:
+- "Silent installation" refers to an automatic command for installing `lemonade` server without running any GUI or prompting the user for any questions. It does assume that the end-user fully accepts the license terms, so be sure that your own application makes this clear to the user.
+- Command line usage allows the server process to be launched programmatically, so that your application can manage starting and stopping the server process on your user's behalf.
+
+### Download
+
+Follow these instructions to download a copy of `Lemonade_Server_Installer.exe`.
+
+#### cURL Download
+
+In a `bash` terminal, such as `git bash`:
+
+Download the latest version:
+
+```bash
+curl -L -o ".\Lemonade_Server_Installer.exe" https://github.com/onnx/turnkeyml/releases/latest/download/Lemonade_Server_Installer.exe
+```
+
+Download a specific version:
+
+```bash
+curl -L -o ".\Lemonade_Server_Installer.exe" https://github.com/onnx/turnkeyml/releases/download/v6.0.0/Lemonade_Server_Installer.exe
+```
+
+#### PowerShell Download
+
+In a powershell terminal:
+
+Download the latest version:
+
+```powershell
+Invoke-WebRequest -Uri "https://github.com/onnx/turnkeyml/releases/latest/download/Lemonade_Server_Installer.exe" -OutFile "Lemonade_Server_Installer.exe"
+```
+
+Download a specific version:
+
+```powershell
+Invoke-WebRequest -Uri "https://github.com/onnx/turnkeyml/releases/download/v6.0.0/Lemonade_Server_Installer.exe" -OutFile "Lemonade_Server_Installer.exe"
+```
+
+### Silent Installation
+
+Silent installation runs `Lemonade_Server_Installer.exe` without a GUI and automatically accepts all prompts.
+
+In a `cmd.exe` terminal:
+
+Install *with* Ryzen AI hybrid support: 
+
+```bash
+Lemonade_Server_Installer.exe /S /Extras=hybrid
+```
+
+Install *without* Ryzen AI hybrid support:
+
+```bash
+Lemonade_Server_Installer.exe /S
+```
+
+The install directory can also be changed from the default by using `/D` as the last argument. 
+
+For example: 
+
+```bash
+Lemonade_Server_Installer.exe /S /Extras=hybrid /D=C:\a\new\path`
+```
+
+### Command Line Invocation
+
+Command line invocation starts the `lemonade` server process so that your application can connect to it via REST API endpoints. 
+
+#### Foreground Process
+
+These steps will open lemonade server in a terminal window that is visible to users. The user can exit the server by closing the window.
+
+In a `cmd.exe` terminal:
+
+```bash
+conda run --no-capture-output -p INSTALL_DIR\lemonade_server\lemon_env lemonade serve
+```
+
+Where `INSTALL_DIR` is the installation path of `lemonade_server`. 
+
+For example, if you used the default installation directory and your username is USERNAME: 
+
+```bash
+C:\Windows\System32\cmd.exe /C conda run --no-capture-output -p C:\Users\USERNAME\AppData\Local\lemonade_server\lemon_env lemonade serve
+```
+
+#### Background Process
+
+This command will open lemonade server without opening a window. Your application needs to manage terminating the process and any child processes it creates.
+
+In a powershell terminal:
+
+```powershell
+$serverProcess = Start-Process -FilePath "C:\Windows\System32\cmd.exe" -ArgumentList "/C conda run --no-capture-output -p INSTALL_DIR\lemonade_server\lemon_env lemonade serve" -RedirectStandardOutput lemonade_out.txt -RedirectStandardError lemonade_err.txt -PassThru -NoNewWindow
+```
+
+Where `INSTALL_DIR` is the installation path of `lemonade_server`.

docs/lemonade/server_spec.md

@@ -35,7 +35,13 @@ The additional endpoints under development are:
 
 > **NOTE:** This server is intended for use on local systems only. Do not expose the server port to the open internet.
 
-First, install lemonade with your desired backend (e.g., `pip install lemonade[llm]`). Then, run the following command to start the server:
+### Windows Installer
+
+See the [Lemonade_Server_Installer.exe instructions](lemonade_server_exe.md) to get started. 
+
+### Python Environment
+
+If you have `lemonade` [installed in a Python environment](getting_started.md#from-pypi), simply activate it and run the following command to start the server:
 
 ```bash
 lemonade serve
@@ -124,6 +130,7 @@ Text Completions API. You provide a prompt and receive a completion. This API wi
 | `model` | Yes | The model to use for the completion.  | <sub>![Status](https://img.shields.io/badge/available-green)</sub> |
 | `stream` | No | If true, tokens will be sent as they are generated. If false, the response will be sent as a single message once complete. Defaults to false. | <sub>![Status](https://img.shields.io/badge/available-green)</sub> |
 | `stop` | No | Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence. Can be a string or an array of strings. | <sub>![Status](https://img.shields.io/badge/available-green)</sub> |
+| `echo` | No | Echo back the prompt in addition to the completion. Available on non-streaming mode. | <sub>![Status](https://img.shields.io/badge/available-green)</sub> |
 | `logprobs` | No | Include log probabilities of the output tokens. If true, returns the log probability of each output token. Defaults to false. | <sub>![Status](https://img.shields.io/badge/WIP-yellow)</sub> |
 
 
@@ -343,3 +350,20 @@ curl http://localhost:8000/api/v0/stats
   "decode_token_times": [0.01, 0.02, 0.03, 0.04, 0.05]
 }
 ```
+
+# Debugging
+
+To help debug the Lemonade server, you can use the `--log-level` parameter to control the verbosity of logging information. The server supports multiple logging levels that provide increasing amounts of detail about server operations.
+
+```
+lemonade serve --log-level [level]
+```
+
+Where `[level]` can be one of:
+
+- **critical**: Only critical errors that prevent server operation.
+- **error**: Error conditions that might allow continued operation.
+- **warning**: Warning conditions that should be addressed.
+- **info**: (Default) General informational messages about server operation.
+- **debug**: Detailed diagnostic information for troubleshooting, including metrics such as input/output token counts, Time To First Token (TTFT), and Tokens Per Second (TPS).
+- **trace**: Very detailed tracing information, including everything from debug level plus all input prompts.

installer/Installer.nsi

@@ -20,6 +20,17 @@ Var LEMONADE_CONDA_ENV
 Var HYBRID_SELECTED
 Var HYBRID_CLI_OPTION
 
+; Variables for CPU detection
+Var cpuName
+Var isCpuSupported
+Var ryzenAiPos
+Var seriesStartPos
+Var currentChar
+
+; Used for string manipulation
+!include "StrFunc.nsh"
+${StrLoc}
+
 ; Define a section for the installation
 Section "Install Main Components" SEC01
 SectionIn RO ; Read only, always installed
@@ -296,7 +307,7 @@ LangString MUI_BUTTONTEXT_FINISH "${LANG_ENGLISH}" "Finish"
 LangString MUI_TEXT_LICENSE_TITLE ${LANG_ENGLISH} "AMD License Agreement"
 LangString MUI_TEXT_LICENSE_SUBTITLE ${LANG_ENGLISH} "Please review the license terms before installing AMD Ryzen AI Hybrid Execution Mode."
 LangString DESC_SEC01 ${LANG_ENGLISH} "The minimum set of dependencies for a lemonade server that runs LLMs on CPU."
-LangString DESC_HybridSec ${LANG_ENGLISH} "Add support for running LLMs on Ryzen AI hybrid execution mode, which uses both the NPU and iGPU for improved performance on Ryzen AI 300-series processors."
+LangString DESC_HybridSec ${LANG_ENGLISH} "Add support for running LLMs on Ryzen AI hybrid execution mode, which uses both the NPU and iGPU for improved performance. Only available on Ryzen AI 300-series processors."
 
 ; Insert the description macros
 !insertmacro MUI_FUNCTION_DESCRIPTION_BEGIN
@@ -317,6 +328,56 @@ Function .onInit
     StrCpy $InstDir "$LOCALAPPDATA\lemonade_server"
   ${EndIf}
 
+  ; Check CPU name to determine if Hybrid section should be enabled
+  DetailPrint "Checking CPU model..."
+  
+  ; Use registry query to get CPU name
+  nsExec::ExecToStack 'reg query "HKEY_LOCAL_MACHINE\HARDWARE\DESCRIPTION\System\CentralProcessor\0" /v ProcessorNameString'
+  Pop $0 ; Return value
+  Pop $cpuName ; Output (CPU name)
+  DetailPrint "Detected CPU: $cpuName"
+  
+  ; Check if CPU name contains "Ryzen AI" and a 3-digit number starting with 3
+  StrCpy $isCpuSupported "false" ; Initialize CPU allowed flag to false
+  
+  ${StrLoc} $ryzenAiPos $cpuName "Ryzen AI" ">"
+  ${If} $ryzenAiPos != ""
+    ; Found "Ryzen AI", now look for 3xx series
+    ${StrLoc} $seriesStartPos $cpuName " 3" ">"
+    ${If} $seriesStartPos != ""
+      ; Check if the character after "3" is a digit (first digit of model number)
+      StrCpy $currentChar $cpuName 1 $seriesStartPos+2
+      ${If} $currentChar >= "0"
+        ${AndIf} $currentChar <= "9"
+        ; Check if the character after that is also a digit (second digit of model number)
+        StrCpy $currentChar $cpuName 1 $seriesStartPos+3
+        ${If} $currentChar >= "0"
+          ${AndIf} $currentChar <= "9"
+          ; Check if the character after the third digit is a space or end of string
+          StrCpy $currentChar $cpuName 1 $seriesStartPos+4
+          ${If} $currentChar == " "
+            ${OrIf} $currentChar == ""
+            ; Found a complete 3-digit number starting with 3
+            StrCpy $isCpuSupported "true"
+            DetailPrint "Detected Ryzen AI 3xx series processor"
+          ${EndIf}
+        ${EndIf}
+      ${EndIf}
+    ${EndIf}
+  ${EndIf}
+  
+  DetailPrint "CPU is compatible with Ryzen AI hybrid software: $isCpuSupported"
+  
+  ; Check if CPU is in the allowed models list
+  ${If} $isCpuSupported != "true"
+    ; Disable Hybrid section if CPU is not in allowed list
+    SectionGetFlags ${HybridSec} $0
+    IntOp $0 $0 & ${SECTION_OFF}    ; Turn off selection
+    IntOp $0 $0 | ${SF_RO}          ; Make it read-only (can't be selected)
+    SectionSetFlags ${HybridSec} $0
+    StrCpy $HYBRID_SELECTED "false"
+  ${EndIf}
+
   ; Disable hybrid mode by default in silent mode
   ; Use /Extras="hybrid" option to enable it
   ${If} ${Silent}
@@ -327,6 +388,10 @@ Function .onInit
     ${IfNot} $HYBRID_CLI_OPTION == "hybrid"
       SectionSetFlags ${HybridSec} 0
       StrCpy $HYBRID_SELECTED "false"
+    ${ElseIf} $isCpuSupported != "true"
+      ; Don't allow hybrid mode if CPU is not in allowed list, even if specified in command line
+      SectionSetFlags ${HybridSec} 0
+      StrCpy $HYBRID_SELECTED "false"
     ${EndIf}
   ${EndIf}
 

setup.py

@@ -21,6 +21,7 @@
         "lemonade.tools",
         "lemonade.tools.ort_genai",
         "lemonade.tools.quark",
+        "lemonade.tools.report",
         "turnkeyml_models",
         "turnkeyml_models.graph_convolutions",
         "turnkeyml_models.selftest",

src/lemonade/cli.py

@@ -4,7 +4,6 @@
 import turnkeyml.cli.cli as cli
 from turnkeyml.sequence import Sequence
 from turnkeyml.tools.management_tools import Cache, Version, SystemInfo
-from turnkeyml.tools.report import Report
 from turnkeyml.state import State
 
 from lemonade.tools.huggingface_load import (
@@ -24,6 +23,7 @@
 from lemonade.tools.prompt import LLMPrompt
 from lemonade.tools.quark.quark_load import QuarkLoad
 from lemonade.tools.quark.quark_quantize import QuarkQuantize
+from lemonade.tools.report.llm_report import LemonadeReport
 from lemonade.tools.serve import Server
 
 
@@ -43,9 +43,9 @@ def main():
         OgaBench,
         QuarkQuantize,
         QuarkLoad,
+        LemonadeReport,
         Server,
         # Inherited from TurnkeyML
-        Report,
         Cache,
         Version,
         SystemInfo,