Save the current state of training

[Serginho]

I think that's a good idea to make a function that let us to save the current state of training and able to continue training with new content in the future.
I know that we can use marshaling, but it would be nice to make a method 'save'.

[cardmagic]

Implementation Plan: JSON Import/Export for Classifier State

After reviewing the codebase, here's a detailed plan for implementing JSON-based persistence for both Classifier::Bayes and Classifier::LSI.

---

1. Bayes Classifier (lib/classifier/bayes.rb)

Internal state to serialize:

• @categories - Hash mapping category names to word frequency hashes
• @total_words - Integer total word count
• @category_counts - Hash mapping category names to document counts
• @category_word_count - Hash mapping category names to total words per category

JSON Structure:

{
  "version": 1,
  "type": "bayes",
  "categories": {
    "Spam": { "free": 5, "money": 3, ... },
    "Ham": { "meeting": 2, "project": 4, ... }
  },
  "total_words": 1250,
  "category_counts": { "Spam": 50, "Ham": 120 },
  "category_word_count": { "Spam": 500, "Ham": 750 }
}

New Methods:

def to_json(*args)
def self.from_json(json_string)
def save_to_file(path)
def self.load_from_file(path)

---

2. LSI Classifier (lib/classifier/lsi.rb)

Internal state to serialize:

• @items - Hash mapping content keys to ContentNode objects
• @word_list - WordList object (word → index mapping)
• @auto_rebuild - Boolean
• @version / @built_at_version - Version tracking

ContentNode data:

• @word_hash - Term frequencies (essential, used to rebuild vectors)
• @categories - Array of category labels
• Vectors (raw_vector, lsi_vector, etc.) - computed, can be rebuilt

Design Decision: Export only the source data (word_hash, categories), not computed vectors. On import, call build_index to reconstruct the vector space. This keeps JSON portable across GSL/non-GSL environments.

JSON Structure:

{
  "version": 1,
  "type": "lsi",
  "auto_rebuild": true,
  "items": {
    "This text involves dogs": {
      "word_hash": { "text": 1, "involv": 1, "dog": 1 },
      "categories": ["Animals"]
    },
    ...
  }
}

New Methods:

def to_json(*args)
def self.from_json(json_string)
def save_to_file(path)
def self.load_from_file(path)

---

3. Implementation Steps

1. Add require 'json' to both classifier files
2. Bayes: Implement to_json - straightforward hash serialization
3. Bayes: Implement from_json - parse JSON, reconstruct instance with proper symbol handling for category names
4. LSI: Implement to_json - iterate @items, extract word_hash and categories from each ContentNode
5. LSI: Implement from_json - recreate LSI instance, add items, trigger rebuild
6. File helpers - Convenience methods wrapping File.write/File.read
7. Tests - Round-trip tests verifying classification behavior is preserved after export/import

---

4. Considerations

• Version field - Allows future format changes with backward compatibility
• Symbol vs String keys - JSON uses strings; need careful handling for category names (use prepare_category_name)
• Large datasets - Consider streaming JSON for very large classifiers (future enhancement)
• GSL portability - By not serializing vectors, JSON files work regardless of GSL availability

---

Let me know if you'd like me to proceed with this implementation or if you have any feedback on the approach!