Enhancement: Implement server-side filters for records API

[PassionateBytes]

Summary

This PR adds three new optional query parameters to the GET /api/v2/records endpoint to enable efficient server-side filtering:

• running: Filter records by their running state (running vs. stopped)
• hidden: Filter records by their hidden state (deleted records)
• tag: Filter records by tags in their description (supports single and multiple tags)

Motivation

This enhancement is driven by two key needs:

1. External Tooling Integration

Third-party CLI tools (including projects like better-timetagger-cli and other custom integrations) need efficient ways to query specific subsets of records - ideally without having to load large data sets that require a lot of local filtering.

2. Performance at Scale

Server-side SQL-based filtering is more efficient than client-side filtering, especially when dealing with larger datasets spanning months or years of time tracking data. By filtering records on the server before transmission, we:

• Reduce network bandwidth usage
• Decrease response payload sizes
• Minimize client-side processing overhead
• Improve overall API responsiveness

Implementation Details

Query Parameter Specifications

running - Filter by Running State

• Truthy values (true, yes, on, 1, y): returns only running records (where t1 == t2)
• Falsy values (false, no, off, 0, n): returns only stopped records
• Empty or unset: returns all records (no filter applied)

hidden - Filter by Hidden State

• Truthy values: returns only hidden records (description starts with "HIDDEN")
• Falsy values: returns only non-hidden records
• Empty or unset: returns all records (no filter applied)

tag - Filter by Tags

• Accepts single tag: ?tag=work
• Accepts multiple comma-separated tags: ?tag=work,urgent
• Multiple tags use AND logic (records must contain ALL specified tags)
• Supports URL-encoded # prefix (%23) or plain tag names
• Empty or unset: returns all records (no filter applied)

Filter Behavior

All filters:

• Are case-insensitive for boolean values
• Can be safely combined with each other and with the existing timerange parameter
• Handle edge cases gracefully (empty values, whitespace, invalid values)
• Are safe against SQL injection attacks - User-provided filter values are properly escaped and passed to SQLite using safe parameter binding

Changes Made

1. API Implementation (timetagger/server/_apiserver.py)

• Implemented query parameter parsing and sanitizing in the get_records() handler:
  • Parse running parameter as None (omitted) or bool.
  • Parse hidden parameter as None (omitted) or bool.
  • Parse tag as comma-separated list of sanitized tag names
• Implemented filtering logic in the get_records() handler:
  • running filter: selects only records where t1 matches t2 or not
  • hidden filter: selects only records where _ob.ds starts with "HIDDEN" or not
  • tag filter: selects only records where all tags are present in _ob.ds, either within the description (followed by whitespace) or at the end of the description (at end of string)
• Added FALSY_VALUES constant for consistent boolean parsing across the API codebase
• Propagated the use of FALSY_VALUES constant to other API handlers (get_webtoken, get_apitoken) for code consistency and reduced duplication

2. Documentation (docs/docs/webapi.md)

• Updated GET /records endpoint signature to include new query parameters
• Added detailed "Optional query parameters" section with:
  • Behavior descriptions for each parameter
  • Usage examples

3. Test Suite (tests/test_server_apiserver.py)

• Added four comprehensive test functions:
  • test_records_get_running_filter(): Tests running/stopped record filtering
  • test_records_get_hidden_filter(): Tests hidden/visible record filtering
  • test_records_get_tag_filter(): Tests single and multiple tag filtering
  • test_records_get_combined_filters(): Tests using multiple filter parameters together
• Test coverage includes:
  • Individual filter behavior with various truthy/falsy values
  • Case-insensitive value handling
  • Empty and invalid parameter values
  • Various combinations of filters
  • Integration with timerange parameter
  • Edge cases (whitespace handling, non-existent tags, empty filter values, etc.)
  • SQL injection safety
• Minor code quality improvements: removed f-string prefixes where not needed

All tests pass successfully ✅

# entire test suite:
invoke tests

# or just new tests:
pytest tests/test_server_apiserver.py::test_records_get_running_filter -v
pytest tests/test_server_apiserver.py::test_records_get_hidden_filter -v
pytest tests/test_server_apiserver.py::test_records_get_tag_filter -v
pytest tests/test_server_apiserver.py::test_records_get_combined_filters -v

Backward Compatibility

These changes are fully backward compatible. All new query parameters are optional, and the API behaves identically to the previous version when they are not specified.

Example Usage

Filter for running records

GET /api/v2/records?timerange=0-9999999999&running=true

Filter for non-hidden records with specific tags

GET /api/v2/records?timerange=0-9999999999&hidden=false&tag=work
GET /api/v2/records?timerange=0-9999999999&hidden=false&tag=work,urgent

Combine all filters

GET /api/v2/records?timerange=1609459200-1640995200&running=false&hidden=false&tag=project1

[almarklein]

Looks like this duplicates the exact same query-part.

[PassionateBytes]

Hey Almar, appreciate the review!

This is deliberate and actually not a duplicate. But the difference is subtle - the difference lies in the safe_params, which render slightly differently.

Here's an example: searching for tag "foo" will render this query part:

json_extract(_ob, '$.ds') LIKE '℅#foo %' ESCAPE '\' OR json_extract(_ob, '$.ds') LIKE '%#foo' ESCAPE '\'

That first bit matches tags at the beginning or somewhere in the middle of the description. The exact match here is:
LIKE '℅#foo %'

• ℅ arbitrary-length wildcard
• # literal hashtag
• foo literal tag nane
• whitespace
• % arbitrary-length wildcard

The second bit matches tags at the end of the description, i.e. followed explicitly by the end of the string. It is matching exactly:
LIKE '℅#foo'

• ℅ arbitrary-length wildcard
• # literal hashtag
• foo literal tag nane
• end of string

---

The reason why I am not using a simpler query here is because I want to prevent partial matches. For example, this simplified query:

json_extract(_ob, '$.ds') LIKE '℅#foo%' ESCAPE '\'

...would also match #fooBar, which is not the intended behavior.

This would be more straightforward, if sqlite supported regular expressions. - then we could simply regex-search for .*#foo(\s.*|$) - But without regex, the solution above is the only way to make this work effectively.

[almarklein]

Thanks for the explanation. I indeed overlooked that subtle difference :)