Feature/osh integration #91

ikrispin · 2025-10-19T16:40:51Z

Added integration with Open Scan Hub (OSH)
Automated Polling System: Implemented OshSchedulerService that automatically polls OSH for new scans on configured packages with configurable intervals and batch processing. NOTE: currently the packages list is empty, meaning all tasks will get scanned - we should decide on a list.
API Client Layer: Created OshClient and OshClientService with health checks, JSON/HTML response parsing, and error handling for OSH API interactions.
Job Creation Pipeline: Built OshJobCreationService that converts OSH scan results into SAST-AI workflow jobs, including NVR parsing and metadata extraction
Report Download Service: Added OshJsonDownloadService to fetch SAST scan results from OSH log files and convert them into processable formats

Retry Algorithm Implementation:

Two-Phase Polling System: Enhanced OshSchedulerService with incremental scan processing (new scans) and retry processing (failed scans) in a single polling cycle
Retry Configuration: Added OshRetryConfiguration with master toggle, configurable max attempts, exponential backoff timing, retention policies, and batch sizing controls
Retry Queue Management: Implemented OshRetryService for the retry lifecycle - recording failures with classification, fetching retry-eligible scans with backoff enforcement, tracking attempt counts and automated cleanup
Failure Classification: Built failure categorization system (network errors, JSON parsing, data validation, database issues, API failures) for targeted retry strategies (if needed)
Database Layer: Created OshUncollectedScanRepository for core retry operations with database-level locking (FOR UPDATE SKIP LOCKED) for concurrent scheduler safety, and OshRetryStatisticsRepository for monitoring and statistical queries
Admin Monitoring: Added OshAdminResource with comprehensive admin endpoints for retry queue status, statistics, manual cleanup operations, and troubleshooting tools
Concurrent Safety: Implemented database-level row locking (FOR UPDATE SKIP LOCKED) and optimistic locking in case we need to handle multiple scheduler instances safely, with additional protection against concurrent manual database access
Retention Management: Automated cleanup of expired retry records and scans exceeding maximum attempts to prevent unbounded growth

Data Model Extensions:

Extended Job entity with oshScanId field and unique constraints to prevent duplicate processing
Added OshSchedulerCursor entity for tracking polling state and resuming from where it left off
Added OshUncollectedScan entity for retry queue management with attempt tracking, failure classification, and JSON scan data storage
Enhanced JobCreationDto to support OSH scan data input with transient JSON content handling

Key Capabilities:

Monitors specific Red Hat packages for new security scans
Automatically creates SAST-AI analysis jobs from OSH findings
Maintains polling state across application restarts
Resilient Processing: Failed scans are automatically retried with backoff strategies
Failure Recovery: System can recover from temporary network issues, API failures, and database problems without losing scan data
Configurable batch processing with rate limiting and timeout controls
Production Monitoring: Comprehensive admin interface for monitoring retry queue health, performance metrics, and manual intervention capabilities

…e management

…hecks and configuration validation implement sequential scan ID discovery for batch processing

… and add OSH_SCAN to InputSourceType enum

…N content

…ry processing), configurable backoff, retry limits, retention policies, failure classification and automatic cleanup

…s for retry queue inspection and statistics and manual cleanup operations and polling triggers for troubleshooting

…sitory split, and named constants

…ting, and thread interruption

…, Thread.sleep

…all failure types are retryable)

Signed-off-by: Jude Niroshan <[email protected]>

src/main/java/com/redhat/sast/api/service/OshSchedulerService.java

src/main/java/com/redhat/sast/api/config/OshConfiguration.java

src/main/java/com/redhat/sast/api/service/OshSchedulerService.java

src/main/java/com/redhat/sast/api/config/OshRetryConfiguration.java

Signed-off-by: Jude Niroshan <[email protected]>

…ature/osh-integration

Signed-off-by: Jude Niroshan <[email protected]>

…th large diff upgraded model reviewer to gemini-2.5-flash

github-actions · 2025-10-29T13:35:52Z