diff --git a/README.md b/README.md index d9bb35e..92b5532 100644 --- a/README.md +++ b/README.md @@ -1,79 +1,136 @@ # vyb -`vyb` is a CLI that helps you iteratively develop applications, faster. -With 'vyb', you can refine your application spec, generate or update code, or provide your own command extensions to -perform tasks that are relevant to your application workspace. - -## Vision -The vision for `vyb` is that it will provide a workflow for developers to collaborate with AI, through well-defined -commands that encapsulate intent and avoid repetitive activities like drafting a prompt or choosing which files to -include in every interaction with the LLM. - -Each `vyb` command has a filter for which workspace files should be included in the request to the LLM, and which files -are allowed to be modified by the LLM's response. - -### Workflow -The envisioned workflow for `vyb` is as follows: -- `vyb refine` will help the developer draft requirements for their application, marking which features are already -implemented and which still need to be implemented. This could be connected with GitHub API, to link the issue # to the -spec element. In this case, the first step in implementing an issue would be to add it as a TODO in the SPEC; -- `vyb code` implements either a `TODO(vyb)` in the code, or a TODO SPEC element. As of now, this command relies -heavily on `TODO(vyb)` for prioritization, and doesn't really know how to pick a SPEC element to implement. It would be -good to find a better way to tell the CLI which specific task needs to be executed each time the command is called; -- `vyb document` updates README.md files to reflect the application code. Ideally, this activity would be done -atomically when `vyb code` runs, but the prompt isn't there yet. But even after that is resolved, this command will -continue to be useful because the developer may manually change code and allow it to drift over time; -- `vyb inferspec` ensures any manual modifications made to the code are reflected back in the SPEC.md files; - -### Git Integration -Every command execution gets a list of files to be modified, along with a commit message. For now, the commit message is -only printed in the output for the user to see, but the goal is to store it in the `.vyb/` folder, alongside the list of -modified files. After reviewing the proposed modifications, the user will then run `vyb accept`, at which point `vyb` -would stage and commit the changes, using the commit message provided by the LLM. - -### LLM Quotas -`vyb` only includes files that match the command filtering criteria under the directory in which it is executed, plus -any of its sub-directories. This is to limit the number of tokens sent to the LLM, but it could also hurt the LLM's -ability to resolve a problem if it doesn't have all the context it needs. -Still to be developed, is a summarization logic that will produce embeddings at each relevant application folder, -allowing for better contextualization even when executing `vyb` within a module and not passing it the entire codebase. - -This is a high priority feature to be implemented soon, since even vyb's codebase is already reaching o1's token limits -for certain prompts. - -## Features - -- Iterative specification refinement and code generation -- Automated project file detection and filtering -- Integration with OpenAI for advanced language-based tasks -- Modular command structure using Cobra - -## Getting Started - -1. Install Go 1.24 or later. -2. Clone this repository. -3. Build the CLI using: - go build -o vyb -4. Obtain an OpenAI API key and store it in the OPENAI_API_KEY env var. - -## Basic Usage - -Below are some key commands: - -• Initialize project: - vyb init - -• Remove project metadata: - vyb remove [--force-root] - -• Summarize code and docs: - vyb summarize - -• Refine specifications: - vyb refine [SPEC.md] - -• Implement code: - vyb code [SPEC.md] - -Check out the subdirectory README files for deeper insights into the -application's architecture. +`vyb` is an AI-assisted command-line companion that helps developers +iterate on code, documentation and specifications **without having to +craft prompts by hand**. It wraps common workflows (code generation, +refactoring, documentation, spec maintenance, …) into deterministic CLI +commands that: + +1. **Select the right context** – only the files relevant for the task + are sent to the LLM. +2. **Apply guarded changes** – proposed modifications are validated + against allow/deny patterns before touching the working tree. +3. **Generate commit messages** – every LLM reply already contains a + Conventional Commit message so you can just `git commit -F` it. (WIP) + +--- + +## Installation + +Requirements: + +* Go ≥ 1.24 +* A valid OpenAI API key (export `OPENAI_API_KEY`) + +```bash +# install the latest directly from github +$ go install github.com/vybdev/vyb + +# make the binary discoverable +# if #GOPATH is not set in your environment, it is usually under $HOME/go +$ export PATH=$GOPATH/bin:$PATH +``` + +> TIP Run `vyb --help` at any time to see all registered commands. + +--- + +## Quick-start + +```bash +# initialise metadata at the repository root +$ vyb init + +# ask the LLM to implement a TODO in the current module +$ vyb code my/pkg/handler.go + +# refresh documentation after a big refactor +$ vyb document -a # -a ⇒ include *all* modules +``` + +All commands only **stage changes** locally; no commit is created. After +reviewing the alterations you can commit them with the message suggested +by `vyb`. + +--- + +## Built-in commands + +| Command | Purpose | +|----------------|------------------------------------------------------------| +| `init` | Create `.vyb/metadata.yaml` in the project root | +| `update` | Re-scan workspace, merge & (re)generate annotations | +| `remove` | Delete `.vyb` completely | +| `code` | Implement `TODO(vyb)`s or the file passed as argument | +| `document` | Generate / refresh `README.md` files | +| `refine` | Polish `SPEC.md` content | +| `inferspec` | Make spec match the *current* codebase | + +Flags accepted by **all** AI-driven commands: + +* `-a, --all` – include every file in the project, not only the current + module. + +--- + +## Core concepts + +### Project metadata (`.vyb/metadata.yaml`) + +A hierarchical representation of the workspace: + +* **Module** – a folder treated as a logical unit. Stores token counts, + MD5 digests and an *Annotation* (see below). +* **FileRef** – path, checksum and token count for each file. + +The metadata is fully derived from the file system; you should never +edit it manually. + +### Annotations + +`vyb` records three complementary summaries for every module: + +* **Internal context** – what lives *inside* the module (private view). +* **Public context** – what the module (and its children) expose. +* **External context** – how the module fits in the *overall* application. + +These texts are generated with the help of the LLM and later injected +into prompts to reduce the number of files that need to be submitted in each request. + +--- + +## Architecture overview + +``` +cmd/ entry-points and Cobra command wiring + template/ YAML + Mustache definitions used by AI commands +llm/ OpenAI API wrapper + strongly typed JSON payloads +workspace/ file selection, .gitignore handling, metadata evolution +``` + +Flow of an AI command (`vyb code` for instance): + +1. "template" loads the prompt YAML, computes inclusion/exclusion sets. +2. "selector" walks the workspace to gather the right files. +3. The user & system messages are built, then sent to `llm/openai`. +4. The JSON reply is validated and applied to the working tree. + +--- + +## Extending `vyb` + +Put additional `.vyb` YAML templates under: + +* `$VYB_HOME/cmd/` – globally available commands. +* `.vyb/cmd/` inside your project – repo-local commands *(planned)*. + +See `cmd/template/embedded/code.vyb` for the field reference. + +--- + +## Development & Testing + +* Unit tests: `go test ./...` +* Lint / CI: see `.github/workflows/go.yml` + +Feel free to open issues or PRs – all contributions are welcome! diff --git a/SPEC.md b/SPEC.md deleted file mode 100644 index 299f5db..0000000 --- a/SPEC.md +++ /dev/null @@ -1,98 +0,0 @@ -# vyb CLI Specification - -This document describes the design and functionality of the -"vyb" command-line application, based on the existing codebase. If new -requirements emerge or unimplemented features are discovered, they are -prefixed with `[TODO]:`. - -## Programming Language - -This application is written in Go (version 1.24 or higher). - -## Frameworks and Libraries - -- [Cobra](https://github.com/spf13/cobra) for CLI creation. -- [Mustache](https://github.com/cbroglie/mustache) for template - processing. -- [YAML.v3](https://pkg.go.dev/gopkg.in/yaml.v3) for reading and - writing metadata. -- [OpenAI API](https://openai.com/) integration implemented over REST. -- [TODO]: Add support for other LLM backend vendors. - - -## Application Type - -- Command-Line Interface (CLI) - -## Architecture and Design - -- The application is organized into multiple Go packages: - - `cmd`: Entry point for commands and subcommands using Cobra. - - `llm`: Handles interactions with the OpenAI API, including - request/response payloads. - - `workspace`: Manages file/directory patterns, metadata creation, - and project structure. -- The root command (`vyb`) registers subcommands (e.g., `init`, - `remove`, etc.) and dynamically loads additional commands (such as - `refine`, `code`, `inferspec`) using embedded `.vyb` config files. -- User-defined commands are loaded from `.vyb` files stored under - `$VYB_HOME/cmd/`. -- [TODO]: Also load user-defined commands from the `.vyb/` folder in - the application root. -- Project metadata is stored under a `.vyb` directory in the project's - root, keeping track of configuration details in `metadata.yaml`. -- The code uses a structured output schema to parse JSON responses - from OpenAI. - -## Dependencies and Integrations - -- Integrates with the OpenAI API to process instructions and handle - generated proposals. -- Requires a valid `OPENAI_API_KEY` environment variable for - authentication. -- Uses the standard Go tooling for builds and tests. - -## User Interface - -- Text-based CLI interface with subcommands. Examples include: - - `vyb init` for creating `.vyb/metadata.yaml`. - - `vyb remove` for deleting all metadata. - - `vyb refine` or `vyb inferspec` for updating specification files. - - `vyb code` and `vyb document` for code generation and doc updates. - -## Performance and Scalability Requirements - -- The CLI is designed for local usage. No special scalability measures - are currently in place. -- [TODO]: Explore concurrency or caching mechanisms for large - workspaces. - -## Security Considerations - -- The application reads `OPENAI_API_KEY` from the environment. -- No extra encryption or advanced security measures currently. -- [TODO]: Evaluate secure storage or encryption for API keys. - -## Deployment Environment - -- Project can be built and run locally with `go build -o vyb`. -- The CLI runs on any environment supporting Go 1.24+. - -## Operational Aspects - -- Commands log errors to stdout and exit with non-zero status on - failure. -- [TODO]: Integrate deeper logging or monitoring if needed. - -## Testing and Quality Assurance - -- Uses Go testing framework (`go test`) in each directory. -- Contains unit tests for file matching, metadata handling, and - OpenAI payloads. -- [TODO]: Evaluate coverage levels and add integration tests. - -## Documentation and Maintenance - -- The repository includes `README.md` files at various levels. -- This `SPEC.md` documents the established functionality. -- [TODO]: Update or extend as new features are introduced. diff --git a/cmd/init.go b/cmd/init.go index 7c0d1ff..68fb91c 100644 --- a/cmd/init.go +++ b/cmd/init.go @@ -4,8 +4,8 @@ import ( "fmt" "os" - "github.com/dangazineu/vyb/workspace/project" "github.com/spf13/cobra" + "github.com/vybdev/vyb/workspace/project" ) var initCmd = &cobra.Command{ diff --git a/cmd/remove.go b/cmd/remove.go index 2067857..5c4fad0 100644 --- a/cmd/remove.go +++ b/cmd/remove.go @@ -2,8 +2,8 @@ package cmd import ( "fmt" - "github.com/dangazineu/vyb/workspace/project" "github.com/spf13/cobra" + "github.com/vybdev/vyb/workspace/project" "os" ) diff --git a/cmd/root.go b/cmd/root.go index 1c397c9..9d65687 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -2,8 +2,8 @@ package cmd import ( "fmt" - "github.com/dangazineu/vyb/cmd/template" "github.com/spf13/cobra" + "github.com/vybdev/vyb/cmd/template" "os" ) diff --git a/cmd/template/README.md b/cmd/template/README.md new file mode 100644 index 0000000..617d9a8 --- /dev/null +++ b/cmd/template/README.md @@ -0,0 +1,26 @@ +# cmd/template Directory + +This folder contains the *prompt templates* that power every AI-driven +`vyb` command. Each template is a `.vyb` YAML file with the following +fields: + +| Field | Description | +|---------------------------------|-------------------------------------------| +| `name` | The CLI sub-command to register | +| `prompt` | User-facing task description (Markdown) | +| `targetSpecificPrompt` *(opt)* | Extra instructions when a file is passed | +| `argInclusionPatterns` | Glob patterns accepted as CLI arguments | +| `argExclusionPatterns` | … patterns that cannot be passed | +| `requestInclusionPatterns` | Files to embed in the LLM payload | +| `requestExclusionPatterns` | Files to never embed | +| `modificationInclusionPatterns` | Files the LLM is allowed to touch | +| `modificationExclusionPatterns` | Guard-rails against accidental edits | + +At runtime the loader merges three sources (by precedence): + +1. Embedded templates bundled at compile time (`embedded/*.vyb`). +2. User-wide templates under `$VYB_HOME/cmd`. +3. (planned) Project-local templates under `.vyb/cmd`. + +Templates use Mustache placeholders to inject dynamic data (e.g. the +command-specific prompt gets embedded into a global *system* prompt). diff --git a/cmd/template/template.go b/cmd/template/template.go index 8eedbef..2f78f08 100644 --- a/cmd/template/template.go +++ b/cmd/template/template.go @@ -7,13 +7,13 @@ import ( "strings" "github.com/cbroglie/mustache" - "github.com/dangazineu/vyb/llm/openai" - "github.com/dangazineu/vyb/llm/payload" - "github.com/dangazineu/vyb/workspace/context" - "github.com/dangazineu/vyb/workspace/matcher" - "github.com/dangazineu/vyb/workspace/project" - "github.com/dangazineu/vyb/workspace/selector" "github.com/spf13/cobra" + "github.com/vybdev/vyb/llm/openai" + "github.com/vybdev/vyb/llm/payload" + "github.com/vybdev/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/matcher" + "github.com/vybdev/vyb/workspace/project" + "github.com/vybdev/vyb/workspace/selector" ) var systemExclusionPatterns = []string{ diff --git a/cmd/template/user_msg_builder.go b/cmd/template/user_msg_builder.go index 6ab2967..ab5b292 100644 --- a/cmd/template/user_msg_builder.go +++ b/cmd/template/user_msg_builder.go @@ -1,14 +1,14 @@ package template import ( - "fmt" - "io/fs" - "path/filepath" - "strings" - - "github.com/dangazineu/vyb/llm/payload" - "github.com/dangazineu/vyb/workspace/context" - "github.com/dangazineu/vyb/workspace/project" + "fmt" + "io/fs" + "path/filepath" + "strings" + + "github.com/vybdev/vyb/llm/payload" + "github.com/vybdev/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/project" ) // buildExtendedUserMessage composes the user-message payload that will be @@ -17,101 +17,101 @@ import ( // nil or when any contextual information is missing the function falls // back gracefully, emitting only what is available. func buildExtendedUserMessage(rootFS fs.FS, meta *project.Metadata, ec *context.ExecutionContext, filePaths []string) (string, error) { - // If metadata is missing we revert to the original behaviour – emit - // just the files. - if meta == nil || meta.Modules == nil { - return payload.BuildUserMessage(rootFS, filePaths) - } - - // Helper to clean/normalise relative paths. - rel := func(abs string) string { - if abs == "" { - return "" - } - r, _ := filepath.Rel(ec.ProjectRoot, abs) - return filepath.ToSlash(r) - } - - workingRel := rel(ec.WorkingDir) - targetRel := rel(ec.TargetDir) - - workingMod := project.FindModule(meta.Modules, workingRel) - targetMod := project.FindModule(meta.Modules, targetRel) - - if workingMod == nil || targetMod == nil { - return "", fmt.Errorf("failed to locate working and target modules") - } - - var sb strings.Builder - - // ------------------------------------------------------------ - // 1. External context of working module. - // ------------------------------------------------------------ - if ann := workingMod.Annotation; ann != nil && ann.ExternalContext != "" { - sb.WriteString(fmt.Sprintf("# Module: `%s`\n", workingMod.Name)) - sb.WriteString("## External Context\n") - sb.WriteString(ann.ExternalContext + "\n") - } - - // ------------------------------------------------------------ - // 2. Internal context of modules between working and target. - // ------------------------------------------------------------ - for m := targetMod.Parent; m != nil && m != workingMod; m = m.Parent { - if ann := m.Annotation; ann != nil && ann.InternalContext != "" { - sb.WriteString(fmt.Sprintf("# Module: `%s`\n", m.Name)) - sb.WriteString("## Internal Context\n") - sb.WriteString(ann.InternalContext + "\n") - } - } - - // ------------------------------------------------------------ - // 3. Public context of sibling modules along the path from the - // parent of the target module up to (and including) the working - // module. This replaces the previous logic that only considered - // direct children of the working module. - // ------------------------------------------------------------ - - isAncestor := func(a, b string) bool { - return a == b || (a != "." && strings.HasPrefix(b, a+"/")) - } - - for ancestor := targetMod.Parent; ancestor != nil; ancestor = ancestor.Parent { - for _, child := range ancestor.Modules { - // Skip the target itself and all its ancestor path. - if isAncestor(child.Name, targetMod.Name) { - continue - } - if ann := child.Annotation; ann != nil && ann.PublicContext != "" { - sb.WriteString(fmt.Sprintf("# Module: `%s`\n", child.Name)) - sb.WriteString("## Public Context\n") - sb.WriteString(ann.PublicContext + "\n") - } - } - if ancestor == workingMod { - break - } - } - - // ------------------------------------------------------------ - // 4. Public context of immediate sub-modules of target module. - // ------------------------------------------------------------ - for _, child := range targetMod.Modules { - if ann := child.Annotation; ann != nil && ann.PublicContext != "" { - sb.WriteString(fmt.Sprintf("# Module: `%s`\n", child.Name)) - sb.WriteString("## Public Context\n") - sb.WriteString(ann.PublicContext + "\n") - } - } - - // ------------------------------------------------------------ - // 5. Append file contents (only files from target module were - // selected by selector.Select). - // ------------------------------------------------------------ - filesMsg, err := payload.BuildUserMessage(rootFS, filePaths) - if err != nil { - return "", err - } - sb.WriteString(filesMsg) - - return sb.String(), nil + // If metadata is missing we revert to the original behaviour – emit + // just the files. + if meta == nil || meta.Modules == nil { + return payload.BuildUserMessage(rootFS, filePaths) + } + + // Helper to clean/normalise relative paths. + rel := func(abs string) string { + if abs == "" { + return "" + } + r, _ := filepath.Rel(ec.ProjectRoot, abs) + return filepath.ToSlash(r) + } + + workingRel := rel(ec.WorkingDir) + targetRel := rel(ec.TargetDir) + + workingMod := project.FindModule(meta.Modules, workingRel) + targetMod := project.FindModule(meta.Modules, targetRel) + + if workingMod == nil || targetMod == nil { + return "", fmt.Errorf("failed to locate working and target modules") + } + + var sb strings.Builder + + // ------------------------------------------------------------ + // 1. External context of working module. + // ------------------------------------------------------------ + if ann := workingMod.Annotation; ann != nil && ann.ExternalContext != "" { + sb.WriteString(fmt.Sprintf("# Module: `%s`\n", workingMod.Name)) + sb.WriteString("## External Context\n") + sb.WriteString(ann.ExternalContext + "\n") + } + + // ------------------------------------------------------------ + // 2. Internal context of modules between working and target. + // ------------------------------------------------------------ + for m := targetMod.Parent; m != nil && m != workingMod; m = m.Parent { + if ann := m.Annotation; ann != nil && ann.InternalContext != "" { + sb.WriteString(fmt.Sprintf("# Module: `%s`\n", m.Name)) + sb.WriteString("## Internal Context\n") + sb.WriteString(ann.InternalContext + "\n") + } + } + + // ------------------------------------------------------------ + // 3. Public context of sibling modules along the path from the + // parent of the target module up to (and including) the working + // module. This replaces the previous logic that only considered + // direct children of the working module. + // ------------------------------------------------------------ + + isAncestor := func(a, b string) bool { + return a == b || (a != "." && strings.HasPrefix(b, a+"/")) + } + + for ancestor := targetMod.Parent; ancestor != nil; ancestor = ancestor.Parent { + for _, child := range ancestor.Modules { + // Skip the target itself and all its ancestor path. + if isAncestor(child.Name, targetMod.Name) { + continue + } + if ann := child.Annotation; ann != nil && ann.PublicContext != "" { + sb.WriteString(fmt.Sprintf("# Module: `%s`\n", child.Name)) + sb.WriteString("## Public Context\n") + sb.WriteString(ann.PublicContext + "\n") + } + } + if ancestor == workingMod { + break + } + } + + // ------------------------------------------------------------ + // 4. Public context of immediate sub-modules of target module. + // ------------------------------------------------------------ + for _, child := range targetMod.Modules { + if ann := child.Annotation; ann != nil && ann.PublicContext != "" { + sb.WriteString(fmt.Sprintf("# Module: `%s`\n", child.Name)) + sb.WriteString("## Public Context\n") + sb.WriteString(ann.PublicContext + "\n") + } + } + + // ------------------------------------------------------------ + // 5. Append file contents (only files from target module were + // selected by selector.Select). + // ------------------------------------------------------------ + filesMsg, err := payload.BuildUserMessage(rootFS, filePaths) + if err != nil { + return "", err + } + sb.WriteString(filesMsg) + + return sb.String(), nil } diff --git a/cmd/template/user_msg_builder_test.go b/cmd/template/user_msg_builder_test.go index 5177578..0817a44 100644 --- a/cmd/template/user_msg_builder_test.go +++ b/cmd/template/user_msg_builder_test.go @@ -6,8 +6,8 @@ import ( "testing" "testing/fstest" - "github.com/dangazineu/vyb/workspace/context" - "github.com/dangazineu/vyb/workspace/project" + "github.com/vybdev/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/project" ) func Test_buildExtendedUserMessage(t *testing.T) { diff --git a/cmd/update.go b/cmd/update.go index 9f45066..1ba2496 100644 --- a/cmd/update.go +++ b/cmd/update.go @@ -4,8 +4,8 @@ import ( "fmt" "os" - "github.com/dangazineu/vyb/workspace/project" "github.com/spf13/cobra" + "github.com/vybdev/vyb/workspace/project" ) var updateCmd = &cobra.Command{ diff --git a/go.mod b/go.mod index 889caf5..a1c487a 100644 --- a/go.mod +++ b/go.mod @@ -1,4 +1,4 @@ -module github.com/dangazineu/vyb +module github.com/vybdev/vyb go 1.24 diff --git a/llm/README.md b/llm/README.md index b205f30..873b074 100644 --- a/llm/README.md +++ b/llm/README.md @@ -1,15 +1,36 @@ -# llm Directory +# llm Package -This folder houses logic for interacting with the OpenAI API as well as -for building and parsing the requests and responses used by the `vyb` CLI. +`llm` wraps all interaction with OpenAI and exposes strongly-typed data +structures so the rest of the codebase never has to deal with raw JSON. -## openai Package +## Sub-packages -Implements the HTTP calls to the OpenAI API, using a structured request -and response format. The `CallOpenAI` function returns proposed file -changes that the CLI then applies. +### `llm/openai` -## payload Package +* Builds requests (`model`, messages, `response_format`). +* Retries on `rate_limit_exceeded`. +* Dumps every request/response pair to a temporary JSON file for easy +debugging. +* Public helpers: + * `GetWorkspaceChangeProposals` – returns a list of file edits + commit + message. + * `GetModuleContext` – summarises a module into *internal* & *public* + contexts. + * `GetModuleExternalContexts` – produces *external* contexts in bulk. -Defines the data structures for building the user message payload and -for interpreting the AI's proposed modifications to the workspace. +### `llm/payload` + +Pure data & helper utilities: + +* `BuildUserMessage` – turns a list of files into a Markdown payload. +* `BuildModuleContextUserMessage` – embeds annotations into the payload + according to precise inclusion rules. +* Go structs mirroring every JSON schema (WorkspaceChangeProposal, + ModuleSelfContainedContext, …). + +## JSON Schema enforcement + +The JSON responses expected from the LLM are described under +`llm/openai/internal/schema/schemas/*.json`. Each request sets the +`json_schema` field so GPT returns **validatable, deterministic** output +that can be unmarshalled straight into Go types. diff --git a/llm/openai/openai.go b/llm/openai/openai.go index 61515ca..0adb270 100644 --- a/llm/openai/openai.go +++ b/llm/openai/openai.go @@ -5,8 +5,8 @@ import ( "encoding/json" "errors" "fmt" - "github.com/dangazineu/vyb/llm/openai/internal/schema" - "github.com/dangazineu/vyb/llm/payload" + "github.com/vybdev/vyb/llm/openai/internal/schema" + "github.com/vybdev/vyb/llm/payload" "io" "net/http" "os" diff --git a/main.go b/main.go index cf50994..729359b 100644 --- a/main.go +++ b/main.go @@ -1,6 +1,6 @@ package main -import "github.com/dangazineu/vyb/cmd" +import "github.com/vybdev/vyb/cmd" func main() { cmd.Execute() diff --git a/workspace/README.md b/workspace/README.md index 79c6574..ae4f155 100644 --- a/workspace/README.md +++ b/workspace/README.md @@ -1,17 +1,31 @@ -# workspace Directory +# workspace Package -The code in this directory manages project-specific details, such as -finding and validating the project's root, matching and selecting -files for inclusion, and handling various metadata operations. +Everything related to the **local file system** lives here: matching, +selection, metadata generation and evolution. -## Subdirectories +## Sub-packages -- matcher: Implements .gitignore-style file matching logic. -- project: Manages project metadata (the .vyb directory) and checks - for correct project root location. -- selector: Orchestrates how files are scanned within the workspace, - applying inclusion/exclusion patterns as well as .gitignore rules. +| Package | Responsibility | +|------------|-----------------------------------------------------| +| `matcher` | Lightweight implementation of `.gitignore` wildcards| +| `selector` | Walks the project applying inclusion/exclusion rules | +| `project` | Creates/updates `.vyb/metadata.yaml` & annotations | +| `context` | Runtime-only struct capturing paths for a command | -This abstraction allows the `vyb` CLI to work in a straightforward way -across diverse project structures by consistently determining which -files should be processed or ignored. +### File selection flow + +1. A `context.ExecutionContext` pins *project root*, *working dir* and + (optionally) a *target file*. +2. `selector.Select` starts at `TargetDir` and walks down, pruning: + * directories excluded by user patterns or inherited `.gitignore`s; + * files outside inclusion patterns. +3. Relative paths of the remaining files are returned for payload + construction. + +### Metadata lifecycle + +* **Create (`vyb init`)** – scans the workspace and writes a brand-new + `metadata.yaml` with empty annotations. +* **Update (`vyb update`)** – regenerates structural data, merges + untouched annotations and refreshes/creates missing ones via the LLM. +* **Remove (`vyb remove`)** – deletes the whole `.vyb` folder. diff --git a/workspace/matcher/README.md b/workspace/matcher/README.md new file mode 100644 index 0000000..4fc210f --- /dev/null +++ b/workspace/matcher/README.md @@ -0,0 +1,31 @@ +# matcher sub-package + +Tiny but powerful implementation of **.gitignore-style pattern +matching** used across `vyb` for: + +* argument validation (`argInclusionPatterns`, `argExclusionPatterns`), +* request trimming (what gets embedded in the LLM prompt), +* write-guard rails (what the LLM is allowed to change). + +## Highlights + +* Supports `*`, `?`, ranges `[a-z]` and the double-star `**` semantics + documented in the official Git spec. +* Implements negated rules (`!important.txt`) and directory-only + patterns (`build/`). +* Works directly with `fs.FS` so it can be unit-tested against an + in-memory filesystem. + +### Public helpers + +| Function | Description | +|--------------------|-------------| +| `IsIncluded` | True when a path **is not** excluded **and** matches at +| | least one inclusion rule. | +| `IsExcluded` | Convenience wrapper that checks only the exclusion set. | + +The actual globbing logic lives in `matchesPattern`, which performs a +recursive token comparison to honour `**` behaviour without relying on +`filepath.Match` (that API lacks double-star support). + +Extensive test coverage can be found in `matchers_test.go`. \ No newline at end of file diff --git a/workspace/project/README.md b/workspace/project/README.md new file mode 100644 index 0000000..e00bfb2 --- /dev/null +++ b/workspace/project/README.md @@ -0,0 +1,54 @@ +# project sub-package + +This folder contains everything related to **workspace metadata**. +The main artifact managed here is `.vyb/metadata.yaml`, a structured +inventory of the project that helps `vyb` decide *what* to send to the +LLM and *when* to regenerate summaries. + +## Core concepts + +| Concept | Purpose | +|--------- | ---------------------------------------------------------------- | +| Module | Logical grouping that mirrors a directory (e.g. `api/user`). | +| FileRef | Lightweight descriptor for a single file (path, MD5, token cnt). | +| Metadata | Root object that keeps the full Module tree. | +| Annotation | Three complementary summaries (internal, public, external). | + +### Token accounting & hashing + +Each Module aggregates token counts from its children and computes an +MD5 digest of their hashes. When two Module objects share the same MD5 +we can safely reuse previous annotations. + +### Annotation workflow (high level) + +1. `vyb init` – creates metadata **and** calls the LLM to fill missing + annotations bottom-up (leaf modules first). +2. `vyb update` – rebuilds a fresh snapshot from disk, *patches* it into + the stored tree preserving still-valid annotations and asks the LLM + to fill only the gaps. +3. `vyb remove` – deletes the whole `.vyb` folder. + +### Files of interest + +| File | Responsibility | +|--------------------------------|------------------------------------------------| +| metadata.go | CRUD helpers + `Update` logic | +| filesystem.go | Walks `fs.FS`, builds Module/FileRef objects | +| annotation.go | Parallel LLM calls that populate annotations | +| root.go | Utility to locate project root from any path | + +### Example `metadata.yaml` (truncated) + +```yaml +modules: + name: . + modules: + - name: api + token_count: 3712 + annotation: + public-context: >- + Package `api` exposes HTTP handlers … +``` + +> NOTE: The file is **fully generated** – do not edit by hand. \ No newline at end of file diff --git a/workspace/project/annotation.go b/workspace/project/annotation.go index eb73fb3..d9bae9a 100644 --- a/workspace/project/annotation.go +++ b/workspace/project/annotation.go @@ -2,8 +2,8 @@ package project import ( "fmt" - "github.com/dangazineu/vyb/llm/openai" - "github.com/dangazineu/vyb/llm/payload" + "github.com/vybdev/vyb/llm/openai" + "github.com/vybdev/vyb/llm/payload" "io/fs" "strings" ) diff --git a/workspace/project/metadata.go b/workspace/project/metadata.go index 90ccdb3..10ccebf 100644 --- a/workspace/project/metadata.go +++ b/workspace/project/metadata.go @@ -4,7 +4,7 @@ import ( "crypto/md5" "encoding/hex" "fmt" - "github.com/dangazineu/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/context" "io/fs" "os" "path/filepath" @@ -14,7 +14,7 @@ import ( "gopkg.in/yaml.v3" - "github.com/dangazineu/vyb/workspace/selector" + "github.com/vybdev/vyb/workspace/selector" ) // Metadata represents the project-specific metadata file. Only one Metadata diff --git a/workspace/selector/README.md b/workspace/selector/README.md new file mode 100644 index 0000000..db51c25 --- /dev/null +++ b/workspace/selector/README.md @@ -0,0 +1,32 @@ +# selector sub-package + +Responsible for discovering which files should be **sent to the LLM** +after all inclusion/exclusion rules are applied. + +## Walk strategy + +1. Determine `relStart` (directory that contains the *target* file or, + if no target is given, the current working directory). +2. Walk the `fs.FS` from root. + * Skip directories not relevant to the target (cheap pruning). + * Merge inherited exclusion patterns with any `.gitignore` found on + the way. +3. Every non-excluded file that matches inclusion patterns and lives + *under* the target subtree is returned. + +## Key invariants + +* **Isolation** – never leaks files outside `TargetDir` into the prompt. +* **Consistency** – all paths are returned relative to workspace root + using forward slashes. +* **Composable** – pure functions working on `fs.FS`, facilitating unit + tests with `fstest.MapFS`. + +### Interaction with other packages + +* Delegates pattern checks to `workspace/matcher`. +* Relies on `workspace/context.ExecutionContext` for validated path + boundaries. + +See `selector_test.go` for practical scenarios, including protection +against context leakage. diff --git a/workspace/selector/selector.go b/workspace/selector/selector.go index aade5a2..98a8240 100644 --- a/workspace/selector/selector.go +++ b/workspace/selector/selector.go @@ -8,8 +8,8 @@ import ( "path/filepath" "strings" - "github.com/dangazineu/vyb/workspace/context" - "github.com/dangazineu/vyb/workspace/matcher" + "github.com/vybdev/vyb/workspace/context" + "github.com/vybdev/vyb/workspace/matcher" ) // Select walks the workspace starting from ec.TargetDir (relative to the diff --git a/workspace/selector/selector_test.go b/workspace/selector/selector_test.go index 241bebf..0c2f839 100644 --- a/workspace/selector/selector_test.go +++ b/workspace/selector/selector_test.go @@ -2,9 +2,9 @@ package selector import ( "fmt" - "github.com/dangazineu/vyb/workspace/context" "github.com/google/go-cmp/cmp" "github.com/google/go-cmp/cmp/cmpopts" + "github.com/vybdev/vyb/workspace/context" "path/filepath" "testing" "testing/fstest"