From 2acc6a315c552b37ed64a2184321ae622d856457 Mon Sep 17 00:00:00 2001 From: Dan Gazineu Date: Sun, 8 Jun 2025 13:43:10 -0400 Subject: [PATCH 1/4] docs: overhaul root and internal READMEs Provide up-to-date documentation that reflects the current state of the codebase. The new README in the repository root offers a clearer overview, installation steps, environment setup, core concepts (modules, metadata, annotations), and a complete command reference. Internal READMEs for llm/, workspace/, and the newly added cmd/template/README.md were expanded to explain their subpackages, public APIs, and how they interact. This makes it easier for new contributors and users to navigate the project and understand how the pieces fit together. --- README.md | 211 ++++++++++++++++++++++++++--------------- cmd/template/README.md | 26 +++++ llm/README.md | 41 ++++++-- workspace/README.md | 40 +++++--- 4 files changed, 218 insertions(+), 100 deletions(-) create mode 100644 cmd/template/README.md diff --git a/README.md b/README.md index d9bb35e..92b5532 100644 --- a/README.md +++ b/README.md @@ -1,79 +1,136 @@ # vyb -`vyb` is a CLI that helps you iteratively develop applications, faster. -With 'vyb', you can refine your application spec, generate or update code, or provide your own command extensions to -perform tasks that are relevant to your application workspace. - -## Vision -The vision for `vyb` is that it will provide a workflow for developers to collaborate with AI, through well-defined -commands that encapsulate intent and avoid repetitive activities like drafting a prompt or choosing which files to -include in every interaction with the LLM. - -Each `vyb` command has a filter for which workspace files should be included in the request to the LLM, and which files -are allowed to be modified by the LLM's response. - -### Workflow -The envisioned workflow for `vyb` is as follows: -- `vyb refine` will help the developer draft requirements for their application, marking which features are already -implemented and which still need to be implemented. This could be connected with GitHub API, to link the issue # to the -spec element. In this case, the first step in implementing an issue would be to add it as a TODO in the SPEC; -- `vyb code` implements either a `TODO(vyb)` in the code, or a TODO SPEC element. As of now, this command relies -heavily on `TODO(vyb)` for prioritization, and doesn't really know how to pick a SPEC element to implement. It would be -good to find a better way to tell the CLI which specific task needs to be executed each time the command is called; -- `vyb document` updates README.md files to reflect the application code. Ideally, this activity would be done -atomically when `vyb code` runs, but the prompt isn't there yet. But even after that is resolved, this command will -continue to be useful because the developer may manually change code and allow it to drift over time; -- `vyb inferspec` ensures any manual modifications made to the code are reflected back in the SPEC.md files; - -### Git Integration -Every command execution gets a list of files to be modified, along with a commit message. For now, the commit message is -only printed in the output for the user to see, but the goal is to store it in the `.vyb/` folder, alongside the list of -modified files. After reviewing the proposed modifications, the user will then run `vyb accept`, at which point `vyb` -would stage and commit the changes, using the commit message provided by the LLM. - -### LLM Quotas -`vyb` only includes files that match the command filtering criteria under the directory in which it is executed, plus -any of its sub-directories. This is to limit the number of tokens sent to the LLM, but it could also hurt the LLM's -ability to resolve a problem if it doesn't have all the context it needs. -Still to be developed, is a summarization logic that will produce embeddings at each relevant application folder, -allowing for better contextualization even when executing `vyb` within a module and not passing it the entire codebase. - -This is a high priority feature to be implemented soon, since even vyb's codebase is already reaching o1's token limits -for certain prompts. - -## Features - -- Iterative specification refinement and code generation -- Automated project file detection and filtering -- Integration with OpenAI for advanced language-based tasks -- Modular command structure using Cobra - -## Getting Started - -1. Install Go 1.24 or later. -2. Clone this repository. -3. Build the CLI using: - go build -o vyb -4. Obtain an OpenAI API key and store it in the OPENAI_API_KEY env var. - -## Basic Usage - -Below are some key commands: - -• Initialize project: - vyb init - -• Remove project metadata: - vyb remove [--force-root] - -• Summarize code and docs: - vyb summarize - -• Refine specifications: - vyb refine [SPEC.md] - -• Implement code: - vyb code [SPEC.md] - -Check out the subdirectory README files for deeper insights into the -application's architecture. +`vyb` is an AI-assisted command-line companion that helps developers +iterate on code, documentation and specifications **without having to +craft prompts by hand**. It wraps common workflows (code generation, +refactoring, documentation, spec maintenance, …) into deterministic CLI +commands that: + +1. **Select the right context** – only the files relevant for the task + are sent to the LLM. +2. **Apply guarded changes** – proposed modifications are validated + against allow/deny patterns before touching the working tree. +3. **Generate commit messages** – every LLM reply already contains a + Conventional Commit message so you can just `git commit -F` it. (WIP) + +--- + +## Installation + +Requirements: + +* Go ≥ 1.24 +* A valid OpenAI API key (export `OPENAI_API_KEY`) + +```bash +# install the latest directly from github +$ go install github.com/vybdev/vyb + +# make the binary discoverable +# if #GOPATH is not set in your environment, it is usually under $HOME/go +$ export PATH=$GOPATH/bin:$PATH +``` + +> TIP Run `vyb --help` at any time to see all registered commands. + +--- + +## Quick-start + +```bash +# initialise metadata at the repository root +$ vyb init + +# ask the LLM to implement a TODO in the current module +$ vyb code my/pkg/handler.go + +# refresh documentation after a big refactor +$ vyb document -a # -a ⇒ include *all* modules +``` + +All commands only **stage changes** locally; no commit is created. After +reviewing the alterations you can commit them with the message suggested +by `vyb`. + +--- + +## Built-in commands + +| Command | Purpose | +|----------------|------------------------------------------------------------| +| `init` | Create `.vyb/metadata.yaml` in the project root | +| `update` | Re-scan workspace, merge & (re)generate annotations | +| `remove` | Delete `.vyb` completely | +| `code` | Implement `TODO(vyb)`s or the file passed as argument | +| `document` | Generate / refresh `README.md` files | +| `refine` | Polish `SPEC.md` content | +| `inferspec` | Make spec match the *current* codebase | + +Flags accepted by **all** AI-driven commands: + +* `-a, --all` – include every file in the project, not only the current + module. + +--- + +## Core concepts + +### Project metadata (`.vyb/metadata.yaml`) + +A hierarchical representation of the workspace: + +* **Module** – a folder treated as a logical unit. Stores token counts, + MD5 digests and an *Annotation* (see below). +* **FileRef** – path, checksum and token count for each file. + +The metadata is fully derived from the file system; you should never +edit it manually. + +### Annotations + +`vyb` records three complementary summaries for every module: + +* **Internal context** – what lives *inside* the module (private view). +* **Public context** – what the module (and its children) expose. +* **External context** – how the module fits in the *overall* application. + +These texts are generated with the help of the LLM and later injected +into prompts to reduce the number of files that need to be submitted in each request. + +--- + +## Architecture overview + +``` +cmd/ entry-points and Cobra command wiring + template/ YAML + Mustache definitions used by AI commands +llm/ OpenAI API wrapper + strongly typed JSON payloads +workspace/ file selection, .gitignore handling, metadata evolution +``` + +Flow of an AI command (`vyb code` for instance): + +1. "template" loads the prompt YAML, computes inclusion/exclusion sets. +2. "selector" walks the workspace to gather the right files. +3. The user & system messages are built, then sent to `llm/openai`. +4. The JSON reply is validated and applied to the working tree. + +--- + +## Extending `vyb` + +Put additional `.vyb` YAML templates under: + +* `$VYB_HOME/cmd/` – globally available commands. +* `.vyb/cmd/` inside your project – repo-local commands *(planned)*. + +See `cmd/template/embedded/code.vyb` for the field reference. + +--- + +## Development & Testing + +* Unit tests: `go test ./...` +* Lint / CI: see `.github/workflows/go.yml` + +Feel free to open issues or PRs – all contributions are welcome! diff --git a/cmd/template/README.md b/cmd/template/README.md new file mode 100644 index 0000000..617d9a8 --- /dev/null +++ b/cmd/template/README.md @@ -0,0 +1,26 @@ +# cmd/template Directory + +This folder contains the *prompt templates* that power every AI-driven +`vyb` command. Each template is a `.vyb` YAML file with the following +fields: + +| Field | Description | +|---------------------------------|-------------------------------------------| +| `name` | The CLI sub-command to register | +| `prompt` | User-facing task description (Markdown) | +| `targetSpecificPrompt` *(opt)* | Extra instructions when a file is passed | +| `argInclusionPatterns` | Glob patterns accepted as CLI arguments | +| `argExclusionPatterns` | … patterns that cannot be passed | +| `requestInclusionPatterns` | Files to embed in the LLM payload | +| `requestExclusionPatterns` | Files to never embed | +| `modificationInclusionPatterns` | Files the LLM is allowed to touch | +| `modificationExclusionPatterns` | Guard-rails against accidental edits | + +At runtime the loader merges three sources (by precedence): + +1. Embedded templates bundled at compile time (`embedded/*.vyb`). +2. User-wide templates under `$VYB_HOME/cmd`. +3. (planned) Project-local templates under `.vyb/cmd`. + +Templates use Mustache placeholders to inject dynamic data (e.g. the +command-specific prompt gets embedded into a global *system* prompt). diff --git a/llm/README.md b/llm/README.md index b205f30..873b074 100644 --- a/llm/README.md +++ b/llm/README.md @@ -1,15 +1,36 @@ -# llm Directory +# llm Package -This folder houses logic for interacting with the OpenAI API as well as -for building and parsing the requests and responses used by the `vyb` CLI. +`llm` wraps all interaction with OpenAI and exposes strongly-typed data +structures so the rest of the codebase never has to deal with raw JSON. -## openai Package +## Sub-packages -Implements the HTTP calls to the OpenAI API, using a structured request -and response format. The `CallOpenAI` function returns proposed file -changes that the CLI then applies. +### `llm/openai` -## payload Package +* Builds requests (`model`, messages, `response_format`). +* Retries on `rate_limit_exceeded`. +* Dumps every request/response pair to a temporary JSON file for easy +debugging. +* Public helpers: + * `GetWorkspaceChangeProposals` – returns a list of file edits + commit + message. + * `GetModuleContext` – summarises a module into *internal* & *public* + contexts. + * `GetModuleExternalContexts` – produces *external* contexts in bulk. -Defines the data structures for building the user message payload and -for interpreting the AI's proposed modifications to the workspace. +### `llm/payload` + +Pure data & helper utilities: + +* `BuildUserMessage` – turns a list of files into a Markdown payload. +* `BuildModuleContextUserMessage` – embeds annotations into the payload + according to precise inclusion rules. +* Go structs mirroring every JSON schema (WorkspaceChangeProposal, + ModuleSelfContainedContext, …). + +## JSON Schema enforcement + +The JSON responses expected from the LLM are described under +`llm/openai/internal/schema/schemas/*.json`. Each request sets the +`json_schema` field so GPT returns **validatable, deterministic** output +that can be unmarshalled straight into Go types. diff --git a/workspace/README.md b/workspace/README.md index 79c6574..ae4f155 100644 --- a/workspace/README.md +++ b/workspace/README.md @@ -1,17 +1,31 @@ -# workspace Directory +# workspace Package -The code in this directory manages project-specific details, such as -finding and validating the project's root, matching and selecting -files for inclusion, and handling various metadata operations. +Everything related to the **local file system** lives here: matching, +selection, metadata generation and evolution. -## Subdirectories +## Sub-packages -- matcher: Implements .gitignore-style file matching logic. -- project: Manages project metadata (the .vyb directory) and checks - for correct project root location. -- selector: Orchestrates how files are scanned within the workspace, - applying inclusion/exclusion patterns as well as .gitignore rules. +| Package | Responsibility | +|------------|-----------------------------------------------------| +| `matcher` | Lightweight implementation of `.gitignore` wildcards| +| `selector` | Walks the project applying inclusion/exclusion rules | +| `project` | Creates/updates `.vyb/metadata.yaml` & annotations | +| `context` | Runtime-only struct capturing paths for a command | -This abstraction allows the `vyb` CLI to work in a straightforward way -across diverse project structures by consistently determining which -files should be processed or ignored. +### File selection flow + +1. A `context.ExecutionContext` pins *project root*, *working dir* and + (optionally) a *target file*. +2. `selector.Select` starts at `TargetDir` and walks down, pruning: + * directories excluded by user patterns or inherited `.gitignore`s; + * files outside inclusion patterns. +3. Relative paths of the remaining files are returned for payload + construction. + +### Metadata lifecycle + +* **Create (`vyb init`)** – scans the workspace and writes a brand-new + `metadata.yaml` with empty annotations. +* **Update (`vyb update`)** – regenerates structural data, merges + untouched annotations and refreshes/creates missing ones via the LLM. +* **Remove (`vyb remove`)** – deletes the whole `.vyb` folder. From faae7a9a8e485b549c490d522323a13119611db3 Mon Sep 17 00:00:00 2001 From: Dan Gazineu Date: Sun, 8 Jun 2025 13:51:52 -0400 Subject: [PATCH 2/4] changed module name to github.com/vybdev/vyb --- cmd/init.go | 2 +- cmd/remove.go | 2 +- cmd/root.go | 2 +- cmd/template/template.go | 12 +- cmd/template/user_msg_builder.go | 210 +++++++++++++------------- cmd/template/user_msg_builder_test.go | 4 +- cmd/update.go | 2 +- go.mod | 2 +- llm/openai/openai.go | 4 +- main.go | 2 +- workspace/project/annotation.go | 4 +- workspace/project/metadata.go | 4 +- workspace/selector/selector.go | 4 +- workspace/selector/selector_test.go | 2 +- 14 files changed, 128 insertions(+), 128 deletions(-) diff --git a/cmd/init.go b/cmd/init.go index 7c0d1ff..68fb91c 100644 --- a/cmd/init.go +++ b/cmd/init.go @@ -4,8 +4,8 @@ import ( "fmt" "os" - "github.com/dangazineu/vyb/workspace/project" "github.com/spf13/cobra" + "github.com/vybdev/vyb/workspace/project" ) var initCmd = &cobra.Command{ diff --git a/cmd/remove.go b/cmd/remove.go index 2067857..5c4fad0 100644 --- a/cmd/remove.go +++ b/cmd/remove.go @@ -2,8 +2,8 @@ package cmd import ( "fmt" - "github.com/dangazineu/vyb/workspace/project" "github.com/spf13/cobra" + "github.com/vybdev/vyb/workspace/project" "os" ) diff --git a/cmd/root.go b/cmd/root.go index 1c397c9..9d65687 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -2,8 +2,8 @@ package cmd import ( "fmt" - "github.com/dangazineu/vyb/cmd/template" "github.com/spf13/cobra" + "github.com/vybdev/vyb/cmd/template" "os" ) diff --git a/cmd/template/template.go b/cmd/template/template.go index 8eedbef..2f78f08 100644 --- a/cmd/template/template.go +++ b/cmd/template/template.go @@ -7,13 +7,13 @@ import ( "strings" "github.com/cbroglie/mustache" - "github.com/dangazineu/vyb/llm/openai" - "github.com/dangazineu/vyb/llm/payload" - "github.com/dangazineu/vyb/workspace/context" - "github.com/dangazineu/vyb/workspace/matcher" - "github.com/dangazineu/vyb/workspace/project" - "github.com/dangazineu/vyb/workspace/selector" "github.com/spf13/cobra" + "github.com/vybdev/vyb/llm/openai" + "github.com/vybdev/vyb/llm/payload" + "github.com/vybdev/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/matcher" + "github.com/vybdev/vyb/workspace/project" + "github.com/vybdev/vyb/workspace/selector" ) var systemExclusionPatterns = []string{ diff --git a/cmd/template/user_msg_builder.go b/cmd/template/user_msg_builder.go index 6ab2967..ab5b292 100644 --- a/cmd/template/user_msg_builder.go +++ b/cmd/template/user_msg_builder.go @@ -1,14 +1,14 @@ package template import ( - "fmt" - "io/fs" - "path/filepath" - "strings" - - "github.com/dangazineu/vyb/llm/payload" - "github.com/dangazineu/vyb/workspace/context" - "github.com/dangazineu/vyb/workspace/project" + "fmt" + "io/fs" + "path/filepath" + "strings" + + "github.com/vybdev/vyb/llm/payload" + "github.com/vybdev/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/project" ) // buildExtendedUserMessage composes the user-message payload that will be @@ -17,101 +17,101 @@ import ( // nil or when any contextual information is missing the function falls // back gracefully, emitting only what is available. func buildExtendedUserMessage(rootFS fs.FS, meta *project.Metadata, ec *context.ExecutionContext, filePaths []string) (string, error) { - // If metadata is missing we revert to the original behaviour – emit - // just the files. - if meta == nil || meta.Modules == nil { - return payload.BuildUserMessage(rootFS, filePaths) - } - - // Helper to clean/normalise relative paths. - rel := func(abs string) string { - if abs == "" { - return "" - } - r, _ := filepath.Rel(ec.ProjectRoot, abs) - return filepath.ToSlash(r) - } - - workingRel := rel(ec.WorkingDir) - targetRel := rel(ec.TargetDir) - - workingMod := project.FindModule(meta.Modules, workingRel) - targetMod := project.FindModule(meta.Modules, targetRel) - - if workingMod == nil || targetMod == nil { - return "", fmt.Errorf("failed to locate working and target modules") - } - - var sb strings.Builder - - // ------------------------------------------------------------ - // 1. External context of working module. - // ------------------------------------------------------------ - if ann := workingMod.Annotation; ann != nil && ann.ExternalContext != "" { - sb.WriteString(fmt.Sprintf("# Module: `%s`\n", workingMod.Name)) - sb.WriteString("## External Context\n") - sb.WriteString(ann.ExternalContext + "\n") - } - - // ------------------------------------------------------------ - // 2. Internal context of modules between working and target. - // ------------------------------------------------------------ - for m := targetMod.Parent; m != nil && m != workingMod; m = m.Parent { - if ann := m.Annotation; ann != nil && ann.InternalContext != "" { - sb.WriteString(fmt.Sprintf("# Module: `%s`\n", m.Name)) - sb.WriteString("## Internal Context\n") - sb.WriteString(ann.InternalContext + "\n") - } - } - - // ------------------------------------------------------------ - // 3. Public context of sibling modules along the path from the - // parent of the target module up to (and including) the working - // module. This replaces the previous logic that only considered - // direct children of the working module. - // ------------------------------------------------------------ - - isAncestor := func(a, b string) bool { - return a == b || (a != "." && strings.HasPrefix(b, a+"/")) - } - - for ancestor := targetMod.Parent; ancestor != nil; ancestor = ancestor.Parent { - for _, child := range ancestor.Modules { - // Skip the target itself and all its ancestor path. - if isAncestor(child.Name, targetMod.Name) { - continue - } - if ann := child.Annotation; ann != nil && ann.PublicContext != "" { - sb.WriteString(fmt.Sprintf("# Module: `%s`\n", child.Name)) - sb.WriteString("## Public Context\n") - sb.WriteString(ann.PublicContext + "\n") - } - } - if ancestor == workingMod { - break - } - } - - // ------------------------------------------------------------ - // 4. Public context of immediate sub-modules of target module. - // ------------------------------------------------------------ - for _, child := range targetMod.Modules { - if ann := child.Annotation; ann != nil && ann.PublicContext != "" { - sb.WriteString(fmt.Sprintf("# Module: `%s`\n", child.Name)) - sb.WriteString("## Public Context\n") - sb.WriteString(ann.PublicContext + "\n") - } - } - - // ------------------------------------------------------------ - // 5. Append file contents (only files from target module were - // selected by selector.Select). - // ------------------------------------------------------------ - filesMsg, err := payload.BuildUserMessage(rootFS, filePaths) - if err != nil { - return "", err - } - sb.WriteString(filesMsg) - - return sb.String(), nil + // If metadata is missing we revert to the original behaviour – emit + // just the files. + if meta == nil || meta.Modules == nil { + return payload.BuildUserMessage(rootFS, filePaths) + } + + // Helper to clean/normalise relative paths. + rel := func(abs string) string { + if abs == "" { + return "" + } + r, _ := filepath.Rel(ec.ProjectRoot, abs) + return filepath.ToSlash(r) + } + + workingRel := rel(ec.WorkingDir) + targetRel := rel(ec.TargetDir) + + workingMod := project.FindModule(meta.Modules, workingRel) + targetMod := project.FindModule(meta.Modules, targetRel) + + if workingMod == nil || targetMod == nil { + return "", fmt.Errorf("failed to locate working and target modules") + } + + var sb strings.Builder + + // ------------------------------------------------------------ + // 1. External context of working module. + // ------------------------------------------------------------ + if ann := workingMod.Annotation; ann != nil && ann.ExternalContext != "" { + sb.WriteString(fmt.Sprintf("# Module: `%s`\n", workingMod.Name)) + sb.WriteString("## External Context\n") + sb.WriteString(ann.ExternalContext + "\n") + } + + // ------------------------------------------------------------ + // 2. Internal context of modules between working and target. + // ------------------------------------------------------------ + for m := targetMod.Parent; m != nil && m != workingMod; m = m.Parent { + if ann := m.Annotation; ann != nil && ann.InternalContext != "" { + sb.WriteString(fmt.Sprintf("# Module: `%s`\n", m.Name)) + sb.WriteString("## Internal Context\n") + sb.WriteString(ann.InternalContext + "\n") + } + } + + // ------------------------------------------------------------ + // 3. Public context of sibling modules along the path from the + // parent of the target module up to (and including) the working + // module. This replaces the previous logic that only considered + // direct children of the working module. + // ------------------------------------------------------------ + + isAncestor := func(a, b string) bool { + return a == b || (a != "." && strings.HasPrefix(b, a+"/")) + } + + for ancestor := targetMod.Parent; ancestor != nil; ancestor = ancestor.Parent { + for _, child := range ancestor.Modules { + // Skip the target itself and all its ancestor path. + if isAncestor(child.Name, targetMod.Name) { + continue + } + if ann := child.Annotation; ann != nil && ann.PublicContext != "" { + sb.WriteString(fmt.Sprintf("# Module: `%s`\n", child.Name)) + sb.WriteString("## Public Context\n") + sb.WriteString(ann.PublicContext + "\n") + } + } + if ancestor == workingMod { + break + } + } + + // ------------------------------------------------------------ + // 4. Public context of immediate sub-modules of target module. + // ------------------------------------------------------------ + for _, child := range targetMod.Modules { + if ann := child.Annotation; ann != nil && ann.PublicContext != "" { + sb.WriteString(fmt.Sprintf("# Module: `%s`\n", child.Name)) + sb.WriteString("## Public Context\n") + sb.WriteString(ann.PublicContext + "\n") + } + } + + // ------------------------------------------------------------ + // 5. Append file contents (only files from target module were + // selected by selector.Select). + // ------------------------------------------------------------ + filesMsg, err := payload.BuildUserMessage(rootFS, filePaths) + if err != nil { + return "", err + } + sb.WriteString(filesMsg) + + return sb.String(), nil } diff --git a/cmd/template/user_msg_builder_test.go b/cmd/template/user_msg_builder_test.go index 5177578..0817a44 100644 --- a/cmd/template/user_msg_builder_test.go +++ b/cmd/template/user_msg_builder_test.go @@ -6,8 +6,8 @@ import ( "testing" "testing/fstest" - "github.com/dangazineu/vyb/workspace/context" - "github.com/dangazineu/vyb/workspace/project" + "github.com/vybdev/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/project" ) func Test_buildExtendedUserMessage(t *testing.T) { diff --git a/cmd/update.go b/cmd/update.go index 9f45066..1ba2496 100644 --- a/cmd/update.go +++ b/cmd/update.go @@ -4,8 +4,8 @@ import ( "fmt" "os" - "github.com/dangazineu/vyb/workspace/project" "github.com/spf13/cobra" + "github.com/vybdev/vyb/workspace/project" ) var updateCmd = &cobra.Command{ diff --git a/go.mod b/go.mod index 889caf5..a1c487a 100644 --- a/go.mod +++ b/go.mod @@ -1,4 +1,4 @@ -module github.com/dangazineu/vyb +module github.com/vybdev/vyb go 1.24 diff --git a/llm/openai/openai.go b/llm/openai/openai.go index 61515ca..0adb270 100644 --- a/llm/openai/openai.go +++ b/llm/openai/openai.go @@ -5,8 +5,8 @@ import ( "encoding/json" "errors" "fmt" - "github.com/dangazineu/vyb/llm/openai/internal/schema" - "github.com/dangazineu/vyb/llm/payload" + "github.com/vybdev/vyb/llm/openai/internal/schema" + "github.com/vybdev/vyb/llm/payload" "io" "net/http" "os" diff --git a/main.go b/main.go index cf50994..729359b 100644 --- a/main.go +++ b/main.go @@ -1,6 +1,6 @@ package main -import "github.com/dangazineu/vyb/cmd" +import "github.com/vybdev/vyb/cmd" func main() { cmd.Execute() diff --git a/workspace/project/annotation.go b/workspace/project/annotation.go index eb73fb3..d9bae9a 100644 --- a/workspace/project/annotation.go +++ b/workspace/project/annotation.go @@ -2,8 +2,8 @@ package project import ( "fmt" - "github.com/dangazineu/vyb/llm/openai" - "github.com/dangazineu/vyb/llm/payload" + "github.com/vybdev/vyb/llm/openai" + "github.com/vybdev/vyb/llm/payload" "io/fs" "strings" ) diff --git a/workspace/project/metadata.go b/workspace/project/metadata.go index 90ccdb3..10ccebf 100644 --- a/workspace/project/metadata.go +++ b/workspace/project/metadata.go @@ -4,7 +4,7 @@ import ( "crypto/md5" "encoding/hex" "fmt" - "github.com/dangazineu/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/context" "io/fs" "os" "path/filepath" @@ -14,7 +14,7 @@ import ( "gopkg.in/yaml.v3" - "github.com/dangazineu/vyb/workspace/selector" + "github.com/vybdev/vyb/workspace/selector" ) // Metadata represents the project-specific metadata file. Only one Metadata diff --git a/workspace/selector/selector.go b/workspace/selector/selector.go index aade5a2..98a8240 100644 --- a/workspace/selector/selector.go +++ b/workspace/selector/selector.go @@ -8,8 +8,8 @@ import ( "path/filepath" "strings" - "github.com/dangazineu/vyb/workspace/context" - "github.com/dangazineu/vyb/workspace/matcher" + "github.com/vybdev/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/matcher" ) // Select walks the workspace starting from ec.TargetDir (relative to the diff --git a/workspace/selector/selector_test.go b/workspace/selector/selector_test.go index 241bebf..0c2f839 100644 --- a/workspace/selector/selector_test.go +++ b/workspace/selector/selector_test.go @@ -2,9 +2,9 @@ package selector import ( "fmt" - "github.com/dangazineu/vyb/workspace/context" "github.com/google/go-cmp/cmp" "github.com/google/go-cmp/cmp/cmpopts" + "github.com/vybdev/vyb/workspace/context" "path/filepath" "testing" "testing/fstest" From 4708e8648759350fc255b1b139fcb4c28f2a438c Mon Sep 17 00:00:00 2001 From: Dan Gazineu Date: Sun, 8 Jun 2025 13:56:37 -0400 Subject: [PATCH 3/4] Add detailed READMEs for workspace internals MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce dedicated documentation files for the most complex sub-packages under the workspace tree: • workspace/project – explains the metadata model (Module, FileRef), annotation workflow and lifecycle of `.vyb/metadata.yaml`. • workspace/matcher – describes the advanced .gitignore-style pattern engine and how inclusion/exclusion logic is applied. • workspace/selector – documents the file-selection algorithm that powers every AI command, including traversal rules and interaction with matcher. These additions complement the existing high-level docs and make it much easier for contributors to navigate the codebase. --- workspace/matcher/README.md | 31 +++++++++++++++++++++ workspace/project/README.md | 54 ++++++++++++++++++++++++++++++++++++ workspace/selector/README.md | 32 +++++++++++++++++++++ 3 files changed, 117 insertions(+) create mode 100644 workspace/matcher/README.md create mode 100644 workspace/project/README.md create mode 100644 workspace/selector/README.md diff --git a/workspace/matcher/README.md b/workspace/matcher/README.md new file mode 100644 index 0000000..4fc210f --- /dev/null +++ b/workspace/matcher/README.md @@ -0,0 +1,31 @@ +# matcher sub-package + +Tiny but powerful implementation of **.gitignore-style pattern +matching** used across `vyb` for: + +* argument validation (`argInclusionPatterns`, `argExclusionPatterns`), +* request trimming (what gets embedded in the LLM prompt), +* write-guard rails (what the LLM is allowed to change). + +## Highlights + +* Supports `*`, `?`, ranges `[a-z]` and the double-star `**` semantics + documented in the official Git spec. +* Implements negated rules (`!important.txt`) and directory-only + patterns (`build/`). +* Works directly with `fs.FS` so it can be unit-tested against an + in-memory filesystem. + +### Public helpers + +| Function | Description | +|--------------------|-------------| +| `IsIncluded` | True when a path **is not** excluded **and** matches at +| | least one inclusion rule. | +| `IsExcluded` | Convenience wrapper that checks only the exclusion set. | + +The actual globbing logic lives in `matchesPattern`, which performs a +recursive token comparison to honour `**` behaviour without relying on +`filepath.Match` (that API lacks double-star support). + +Extensive test coverage can be found in `matchers_test.go`. \ No newline at end of file diff --git a/workspace/project/README.md b/workspace/project/README.md new file mode 100644 index 0000000..e00bfb2 --- /dev/null +++ b/workspace/project/README.md @@ -0,0 +1,54 @@ +# project sub-package + +This folder contains everything related to **workspace metadata**. +The main artifact managed here is `.vyb/metadata.yaml`, a structured +inventory of the project that helps `vyb` decide *what* to send to the +LLM and *when* to regenerate summaries. + +## Core concepts + +| Concept | Purpose | +|--------- | ---------------------------------------------------------------- | +| Module | Logical grouping that mirrors a directory (e.g. `api/user`). | +| FileRef | Lightweight descriptor for a single file (path, MD5, token cnt). | +| Metadata | Root object that keeps the full Module tree. | +| Annotation | Three complementary summaries (internal, public, external). | + +### Token accounting & hashing + +Each Module aggregates token counts from its children and computes an +MD5 digest of their hashes. When two Module objects share the same MD5 +we can safely reuse previous annotations. + +### Annotation workflow (high level) + +1. `vyb init` – creates metadata **and** calls the LLM to fill missing + annotations bottom-up (leaf modules first). +2. `vyb update` – rebuilds a fresh snapshot from disk, *patches* it into + the stored tree preserving still-valid annotations and asks the LLM + to fill only the gaps. +3. `vyb remove` – deletes the whole `.vyb` folder. + +### Files of interest + +| File | Responsibility | +|--------------------------------|------------------------------------------------| +| metadata.go | CRUD helpers + `Update` logic | +| filesystem.go | Walks `fs.FS`, builds Module/FileRef objects | +| annotation.go | Parallel LLM calls that populate annotations | +| root.go | Utility to locate project root from any path | + +### Example `metadata.yaml` (truncated) + +```yaml +modules: + name: . + modules: + - name: api + token_count: 3712 + annotation: + public-context: >- + Package `api` exposes HTTP handlers … +``` + +> NOTE: The file is **fully generated** – do not edit by hand. \ No newline at end of file diff --git a/workspace/selector/README.md b/workspace/selector/README.md new file mode 100644 index 0000000..db51c25 --- /dev/null +++ b/workspace/selector/README.md @@ -0,0 +1,32 @@ +# selector sub-package + +Responsible for discovering which files should be **sent to the LLM** +after all inclusion/exclusion rules are applied. + +## Walk strategy + +1. Determine `relStart` (directory that contains the *target* file or, + if no target is given, the current working directory). +2. Walk the `fs.FS` from root. + * Skip directories not relevant to the target (cheap pruning). + * Merge inherited exclusion patterns with any `.gitignore` found on + the way. +3. Every non-excluded file that matches inclusion patterns and lives + *under* the target subtree is returned. + +## Key invariants + +* **Isolation** – never leaks files outside `TargetDir` into the prompt. +* **Consistency** – all paths are returned relative to workspace root + using forward slashes. +* **Composable** – pure functions working on `fs.FS`, facilitating unit + tests with `fstest.MapFS`. + +### Interaction with other packages + +* Delegates pattern checks to `workspace/matcher`. +* Relies on `workspace/context.ExecutionContext` for validated path + boundaries. + +See `selector_test.go` for practical scenarios, including protection +against context leakage. From 1847f369759cca4046f652dbcf0ec170f13495f9 Mon Sep 17 00:00:00 2001 From: Dan Gazineu Date: Sun, 8 Jun 2025 14:00:04 -0400 Subject: [PATCH 4/4] removed outdated spec file --- SPEC.md | 98 --------------------------------------------------------- 1 file changed, 98 deletions(-) delete mode 100644 SPEC.md diff --git a/SPEC.md b/SPEC.md deleted file mode 100644 index 299f5db..0000000 --- a/SPEC.md +++ /dev/null @@ -1,98 +0,0 @@ -# vyb CLI Specification - -This document describes the design and functionality of the -"vyb" command-line application, based on the existing codebase. If new -requirements emerge or unimplemented features are discovered, they are -prefixed with `[TODO]:`. - -## Programming Language - -This application is written in Go (version 1.24 or higher). - -## Frameworks and Libraries - -- [Cobra](https://github.com/spf13/cobra) for CLI creation. -- [Mustache](https://github.com/cbroglie/mustache) for template - processing. -- [YAML.v3](https://pkg.go.dev/gopkg.in/yaml.v3) for reading and - writing metadata. -- [OpenAI API](https://openai.com/) integration implemented over REST. -- [TODO]: Add support for other LLM backend vendors. - - -## Application Type - -- Command-Line Interface (CLI) - -## Architecture and Design - -- The application is organized into multiple Go packages: - - `cmd`: Entry point for commands and subcommands using Cobra. - - `llm`: Handles interactions with the OpenAI API, including - request/response payloads. - - `workspace`: Manages file/directory patterns, metadata creation, - and project structure. -- The root command (`vyb`) registers subcommands (e.g., `init`, - `remove`, etc.) and dynamically loads additional commands (such as - `refine`, `code`, `inferspec`) using embedded `.vyb` config files. -- User-defined commands are loaded from `.vyb` files stored under - `$VYB_HOME/cmd/`. -- [TODO]: Also load user-defined commands from the `.vyb/` folder in - the application root. -- Project metadata is stored under a `.vyb` directory in the project's - root, keeping track of configuration details in `metadata.yaml`. -- The code uses a structured output schema to parse JSON responses - from OpenAI. - -## Dependencies and Integrations - -- Integrates with the OpenAI API to process instructions and handle - generated proposals. -- Requires a valid `OPENAI_API_KEY` environment variable for - authentication. -- Uses the standard Go tooling for builds and tests. - -## User Interface - -- Text-based CLI interface with subcommands. Examples include: - - `vyb init` for creating `.vyb/metadata.yaml`. - - `vyb remove` for deleting all metadata. - - `vyb refine` or `vyb inferspec` for updating specification files. - - `vyb code` and `vyb document` for code generation and doc updates. - -## Performance and Scalability Requirements - -- The CLI is designed for local usage. No special scalability measures - are currently in place. -- [TODO]: Explore concurrency or caching mechanisms for large - workspaces. - -## Security Considerations - -- The application reads `OPENAI_API_KEY` from the environment. -- No extra encryption or advanced security measures currently. -- [TODO]: Evaluate secure storage or encryption for API keys. - -## Deployment Environment - -- Project can be built and run locally with `go build -o vyb`. -- The CLI runs on any environment supporting Go 1.24+. - -## Operational Aspects - -- Commands log errors to stdout and exit with non-zero status on - failure. -- [TODO]: Integrate deeper logging or monitoring if needed. - -## Testing and Quality Assurance - -- Uses Go testing framework (`go test`) in each directory. -- Contains unit tests for file matching, metadata handling, and - OpenAI payloads. -- [TODO]: Evaluate coverage levels and add integration tests. - -## Documentation and Maintenance - -- The repository includes `README.md` files at various levels. -- This `SPEC.md` documents the established functionality. -- [TODO]: Update or extend as new features are introduced.