Support transform value by regex

[takato1314]

Hi, i would like to transform the following source json:

{
  "friends": [
    {"name": "Chris", "url": "https://localhost/feature/logs?IncludeDetails=true&StartDate=2023-01-14&api_key=58fd29c2-af2d-4d60-80b1-e2f6fbba621d&format=json"}
  ]
}

So that i can map it to {apiKey: value} where the value is derived from the api_key query string. I tried to use something like this but to no avail.

.friends
  | map({
    name: .name, 
    apiKey: regex(.url, "[?&]api_key=([^&]+)")
  })

The output is:

[{"name": "Chris", "apiKey": true}]

which i understand it makes sense because the regex function only asserts whether the text provided matches the regex or not. However, what i need here is to extract the api_key and map it to a new value. Is this doable?

[josdejong]

That is a good idea! Right now the regex function only tests whether a string matches or not: it returns true or false. It will be very useful to be able to extract one (or multiple) matches from a string using a regex.

It is probably best to create a new function that allows doing that. We will have to think through what this function should return and what options it should have. A regex can have one or multiple match groups for example. Also, you can either want the first match or all matches.

Alternatively we could change the existing function regex to return match results. That would make it slower when used to just filter. It would be nice if we could still optimize for that case.

Any thoughts on this?

[iamevn]

How about new functions regexExtract and regexExtractAll that returns match results as a match object/array of match objects?
Could model match objects based on how javascript does things (but make it an actual object instead of an array with extra properties tacked on).

Might look something like this, where the match object has a .results property that contains an array with the first element being the full match followed by any matched groups. There would also be a .groups property that contains an object mapping named capture groups to their captures. (Javascript includes .index, .length, and .input properties as well that should probably be included but I'm leaving them out on these examples for the sake of brevity)

regexExtract("Hello World!", "[A-Z][a-z]*")
=> {results: ["Hello"]}

regexExtract("Hello World!", "ASDF")
=> null

regexExtract("Hello World!", "[A-Z]([a-z]*)")
=> {results: ["Hello", "ello"]}

regexExtract("Hello World!", "([A-Z])([a-z]*)")
=> {results: ["Hello", "H", "ello"]}

regexExtract("Hello World!", "(?<firstLetter>[A-Z])([a-z]*)")
=> {results: ["Hello", "H", "ello"], groups: {firstLetter: "H"}}

regexExtract("Hello World!", "(?<firstLetter>[A-Z])(?<rest>[a-z]*)")
=> {results: ["Hello", "H", "ello"], groups: {firstLetter: "H", rest: "ello"}}

regexExtract("Hello World!", "(?<same>[A-Z])(?<same>[a-z]*)")
=> SyntaxError: duplicate capture group name in regular expression

regexExtractAll("Hello World!", "[A-Z][a-z]*")
=> [{results: ["Hello"]}, {results: ["World"]}]

regexExtractAll("Hello World!", "ASDF")
=> []

regexExtractAll("Hello World!", "^[A-Z][a-z]*")
=> [{results: ["Hello"]}]

regexExtractAll("Hello World!", "[A-Z]([a-z]*)")
=> [{results: ["Hello", "ello"]}, {results: ["World", "orld"]}]

regexExtractAll("Hello World!", "([A-Z])([a-z]*)")
=> [{results: ["Hello", "H", "ello"]},
    {results: ["World", "W", "orld"]}]

regexExtractAll("Hello World!", "(?<firstLetter>[A-Z])([a-z]*)")
=> [{results: ["Hello", "H", "ello"], groups: {firstLetter: "H"}},
    {results: ["World", "W", "orld"], groups: {firstLetter: "W"}}]

regexExtractAll("Hello World!", "(?<firstLetter>[A-Z])(?<rest>[a-z]*)")
=> [{results: ["Hello", "H", "ello"], groups: {firstLetter: "H", rest: "ello"}},
    {results: ["World", "W", "orld"], groups: {firstLetter: "W", rest: "orld"}}]

(a js implementation of this would be a pretty thin wrapper around String.prototype.match()/String.prototype.matchAll())

[josdejong]

Thanks for working out this proposal Evan, I like it!

The names regexExtract and regexExtractAll make sense, though they are quite long. How about match and matchAll?

I've made a start implementing this idea in #26. Feedback is welcome.

[josdejong]

I've worked out a PR and ported this to the Python library too. I'm still thinking about the most suitable return type for the new functions. JavaScript returns an array where the first item is the actual match, optionally followed by capturing groups. In our case we want to return named capturing groups too, so that is three separate things. Inspired by the regex functions of Python and Kotlin, it may be good to explicitely return those. Right now I have the following syntax:

// function match
jsonquery(
  'Save the date: 2025-12-25!',
  'match(get(), "(?<year>[0-9]{4})-(?<month>[0-9]{2})-(?<date>[0-9]{2})")'
)
// {
//   "value": "2025-12-25", 
//   "groups": ["2025", "12", "25"],
//   "namedGroups": {"year": "2025", "month": "12", "date": "25"}
// }

// function matchAll
jsonquery(
  'We will be on holiday from 2026-08-14 till 2026-08-28',
  'matchAll(get(), "(?<year>[0-9]{4})-(?<month>[0-9]{2})-(?<date>[0-9]{2})")'
)
// [
//   {
//     "value": "2026-08-14",
//     "groups": ["2026", "08", "14"],
//     "namedGroups": {"year": "2026", "month": "08", "date": "14"}
//   },
//   {
//     "value": "2026-08-28",
//     "groups": ["2026", "08", "28"],
//     "namedGroups": {"year": "2026", "month": "08", "date": "28"}
//   }
// ]

See docs of the new functions:

• /src/content/reference.md#match
• /src/content/reference.md#matchall

Pull requests:

• JavaScript PR
• Python PR
• Website PR

Instead of { value, groups, namedGroups } we could also name the properties { value, groups, groupValues } where groups then contains the named capturing groups and groupValues the array with the unnamed capturing groups.

Any thoughts on this?

[josdejong]

@iamevn and @takato1314 do you have any feedback/thoughts?

[iamevn]

I really like splitting out the matched value from the capture groups like this and { value, groups, namedGroups } seems intuitive to me 👍

Good callout on the docs for (?<name...) vs (?P<name>...).

[josdejong]

Thanks for the feedback Evan!

[josdejong]

Implemented now via #26, published in v5.1.0