[Feature Request]: support for custom transforms via function calling

[PorridgeBear]

What needs to be done?

Firstly - great library. I've hit upon a couple of use cases that would benefit from the XPath/CSS extractors being able to call into custom code to "do something with the value" the XPath/CSS gets hold of today:

1. When scraping an AEM website's web components e.g. <foo-bar data-model="<stringified json>" /> I have needed to use the type=attribute, attribute=data-model to get ahold of the JSON string, but it would be amazing to be able to route this into a further function for further processing that returns a final value to the field based on custom logic

2. When scraping a blog and obtaining e.g.a date string like 23 June 2025, I would rather have an ISO date string. Here, it seems like the type=transform would be handy except it's a fairly limited set. So much like (1) if it were possible to send the result of the XPath/CSS to a function, then I could return the ISO string.

Some further observations - I noticed in the code that there is a type=computed field which then looks for expression or function and so I played with this. It looks like the function idea here is what I need, except it is not passed the current XPath/CSS selected value to do something with - it is only passed already processed fields.

What problem does this solve?

It opens up the schema-based extractors to produce highly custom output that is far more useful for onward processing.

Target users/beneficiaries

Anyone wanting to extract structured data from the scrape.

Current alternatives/workarounds

A custom processing step that needs to read all the schema JSON output files to perform the transforms/customisations rather than doing it with the existing library features.

Proposed approach

Code appears to be present/have been started but did not work for me as expected.

What is key is that the standard extraction can still work i.e. extracting an attribute, or extracting html or text, but then it should be possible to route this value into a custom function to get a final value.

# example 1
def colour_from_json(v: str, fields_so_far: dict):
  return json.loads(v)["the"]["colour"]
  
{
  "name": "colour",
  "selector": "div#foo",
  "type": "attribute",
  "attribute": "data-model",
  "function": colour_from_json
}

# example 2
{
  "name": "date",
  "selector": "p#date",
  "type": "text",
  "function": lambda v: datetime.strptime(v, "%d %b %Y").date().isoformat() if x else ""
}